SO-ARM100/101 Record Dataset
Add Cameras
The camera used in this tutorial is IMX335 5MP USB Camera (B)
• Using Jetson Orin Nano as an example, if you have two cameras and two robot arms, at least four USB ports are required. If USB ports are insufficient, you can connect the USB cables of the two robot arms to a USB Hub, and connect the USB cables of the two cameras directly to the Jetson Orin Nano.
• Note: It is best not to connect cameras to a USB Hub, as insufficient bandwidth allocation may result in only one of the two cameras functioning.
To find the camera indices of the cameras connected to your system, run the following script:
lerobot-find-cameras opencv
The terminal will print relevant camera information. For example:
• The camera IDs recognized by the system in the image above are /dev/video0 and /dev/video2.
• You can find images captured by each camera in the ~/lerobot/outputs/captured_images directory.
Afterwards, run the following code to display the camera feed during teleoperation.
lerobot-teleoperate \
--robot.type=so101_follower \
--robot.port=/dev/ttyACM1 \
--robot.id=my_awesome_follower_arm \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30, fourcc: "MJPG"}, side: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30, fourcc: "MJPG"}}" \
--teleop.type=so101_leader \
--teleop.port=/dev/ttyACM0 \
--teleop.id=my_awesome_leader_arm \
--display_data=true
• The port numbers for the Follower and Leader robot arms need to be filled in according to the actual ports.
• Please note the format of index_or_path. It is determined by the last digit of the camera ID output by the lerobot-find-cameras opencv command. For example: if the camera device ID is /dev/video0, then the device index in the system is index_or_path: 0.
• The fourcc: "MJPG" format image is compressed. You can try higher resolutions, or you can try the YUYV format image, but this may cause a reduction in image resolution and FPS, leading to lag in robot arm operation. Currently, the MJPG format can support three cameras at 1920*1080 resolution while maintaining 30FPS, but it is still not recommended to connect two cameras through the same USB Hub to the computer.
If you have only one camera, you can run the following code:
lerobot-teleoperate \
--robot.type=so101_follower \
--robot.port=/dev/ttyACM1 \
--robot.id=my_awesome_follower_arm \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30, fourcc: "MJPG"}}" \
--teleop.type=so101_leader \
--teleop.port=/dev/ttyACM0 \
--teleop.id=my_awesome_leader_arm \
--display_data=true
If you have more cameras, you can add them by modifying the --robot.cameras parameter.
Dataset Creation and Collection
If you want the dataset to be saved locally, you can directly run:
lerobot-record \
--robot.type=so101_follower \
--robot.port=/dev/ttyACM1 \
--robot.id=my_awesome_follower_arm \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30, fourcc: "MJPG"}, side: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30, fourcc: "MJPG"}}" \
--teleop.type=so101_leader \
--teleop.port=/dev/ttyACM0 \
--teleop.id=my_awesome_leader_arm \
--display_data=true \
--dataset.repo_id=waveshare/test \
--dataset.num_episodes=3 \
--dataset.single_task="Grab the black cube" \
--dataset.push_to_hub=false \
--dataset.episode_time_s=30 \
--dataset.reset_time_s=30
You can customize the repo_id. With the push_to_hub=false, the final dataset will be saved under ~/.cache/huggingface/lerobot in your home directory, creating the aforementioned waveshare/test folder.
Recording Functionality
The record functionality provides a set of tools for capturing and managing data during robot operation.
1. Data Storage
• Data is stored in the LeRobotDataset format and saved to disk during the recording process.
• By default, the dataset is pushed to your Hugging Face page after recording is complete.
• To disable upload, use: --dataset.push_to_hub=False.
2. Checkpoints and Resumption
• Checkpoints are automatically created during recording.
• If recording is interrupted, it can be resumed by running the same command again and adding --resume=true.
⚠️ Important: When resuming, set --dataset.num_episodes to the number of additional episodes to record (not the total target number of episodes in the dataset).
• To start recording from the beginning, manually delete the dataset directory.
3. Recording Parameters
Set the data recording process via command-line parameters:
| Parameter | Description | Default Value |
| --dataset.episode_time_s | Duration of each dataset episode (seconds) | 60 |
| --dataset.reset_time_s | Environment reset time after each episode (seconds) | 60 |
| --dataset.num_episodes | Total number of episodes to record | 50 |
4. Keyboard Controls During Recording
Use keyboard shortcuts to control the data recording flow:
| Key | Action |
| → (Right Arrow) | Prematurely terminate the current episode / reset; proceed to the next. |
| ← (Left Arrow) | Cancel the current episode; re-record. |
| ESC | Immediately stop the session, encode videos, and upload the dataset. |
If your keyboard presses have no effect, you may need to downgrade your pynput version, for example, install version 1.6.8.
pip install pynput==1.6.8
Visualize the Dataset
If you used --dataset.push_to_hub=false and did not upload the data, you can also visualize it locally with the following command:
lerobot-dataset-viz \
--repo-id waveshare/test \
--episode-index 0
Replay a Dataset
Try replaying the first dataset on the robot arm:
lerobot-replay \ --robot.type so101_follower \ --robot.port /dev/ttyACM1 \ --robot.id my_awesome_follower_arm \ --dataset.repo_id waveshare/test \ --dataset.episode 0
At this point, the robot should perform the same actions you recorded during teleoperation.