Template: Jetbot AI kit tutorial

From Waveshare Wiki
Jump to: navigation, search

Basic motion

  • Access jetbot by going to http://<jetbot_ip_address>:8888, navigate to ~/Notebooks/basic_motion/
  • Open basic_motion.ipynb file and following the notebook.

【Note】You can click icon ▶ to run codes, or select Run -> Run Select Cells. Make sure the JetBot has enough space to run.
JetBot AI Kit03.jpg

  • When running robot.left(speed=0.3), the car will turn left in a circle (need to ensure that the car has enough space to move). If the car does not turn left, it may be that the wiring is wrong, and you need to check it again.
  • After running this program, two sliders on the left and right will be output. Drag the slider to change the speed of the left and right clicks.

JetBot AI Kit04.jpg

  • After running this code, you can control the front, back, left and right movement of the car through the web buttons. [Note] You can open the output window as a new page by right-clicking the code ->Create New View for Output.

JetBot AI Kit06.jpg

  • This code uses "heartbeat" to maintain the normal operation of the car. After dragging the slider to reduce the heartbeat frequency, the car will stop rotating.

JetBot AI Kit05.jpg


  • Access JetBot by going to https://<jetbot_ip_address>:8888, navigate to ~/Notebooks/teleoperation/.
  • Open teleoperation.ipynb file and following notebook.
  • Connect USB adapter to PC.
  • Go to https://html5gamepad.com, check the INDEX of Gamepad.
Before you run the example, let's learn how the gamepad work.
The gamepad included supports two working modes. One is PC/PS3/Andorid mode and another is Xbox 360 mode.
The gamepad is set to PC/PS3/Andorid mode by default, in this mode, the gamepad has two sub-modes. You can press the HOME button to switch it. In Mode 1, the front panel lights on only one LED, the right joystick is mapped to buttons 0,1,2 and 3, and you can get only 0 or -1/1 value from the joysticks. In Mode 2, the front panel lights on two LEDs, the right joystick is mapped to axes[2] and axes[5]. In this mode, you can get No intermediate values from joysticks.
JetBot AI Kit Manual 7.jpg
To switch between PC/PS3/Andorid mode and the Xbox 360 mode, you can long-press the HOME button for about 7s. In Xbox mode, the left joystick is mapped to axes[0] and axes[1], right joystick is mapped to axes[2] and axes[3]. This mode is exactly what the NVIDIA examples use. We recommend you to set your gamepad to this mode when you use it. Otherwise, you need to modify the codes.
JetBot AI Kit Manual 43.JPG
  • Modify the index. Run and test Gamepad.
JetBot AI Kit Manual 8.jpg
  • Modify axes values if required, here we use axes[0] and axes[1]
JetBot AI Kit Manual 9.jpg
  • At this point, the window will display the picture captured by the current camera.
JetBot AI Kit Manual 10.jpg
  • "Heartbeat" check, the car will stop automatically when the car is disconnected from the network.
JetBot AI Kit Manual 11.jpg
  • Control the car to take pictures by pressing the button. You can choose different buttons by changing the button. Here, buttons[0] is selected for the convenience of control.
Gamepad Control 07.jpg
  • After running the program, you can control Jetbot through the remote control handle. Asex[0] controls the left wheel, Asex[1] controls the right wheel, button[0] controls the camera, the left image shows the real-time camera image, the right is the captured image, and the image is saved in the snapshots file.


In this example, we'll collect an image classification dataset that will be used to help keep JetBot safe! We'll teach JetBot to detect two scenarios free and blocked. We'll use this AI classifier to prevent JetBot from entering dangerous territory.

Step 1. Collect data on JetBot
  • Access JetBot by going to https://<jetbot_ip_address>:8888, navigate to ~/Notebooks/collision_avoidance/.
  • Open data_collection.ipynb file and following notebook.
  • After running the program, the interface as shown in the figure appears, put the car in a different position, and click "add free" if there is no obstacle in front of the car. If there is an obstacle in front of the car, please click "add blocked". The captured pictures will be saved in the dataset folder, and as many pictures of various situations as possible will be taken. You can try different orientations, brightness, object or collision types (walls, ledges, etc.), and can try untextured floors/objects (patterned, smooth, glass, etc.).
JetBot AI Kit Manual 13.jpg
  • The more scene data the car collects, the better the obstacle avoidance effect will be. Therefore, it is very important to obtain as much different data as possible for the obstacle avoidance effect. Generally, at least 100 pictures are required for each situation.
  • Finally, run the program to package the pictures. After packaging, a dataset.zip compressed file will appear in the current directory.
JetBot AI Kit Manual 14.jpg
Step 2. Train neural network
  • Access JetBot by going to https://<jetbot_ip_address>:8888, navigate to ~/Notebooks/collision_avoidance/
  • Open and follow the tain_model.ipynb notebook.
  • If you already have the dataset.zip file you just compressed, you do not need to run this statement to decompress it, otherwise you will be prompted to overwrite the existing file.
JetBot AI Kit Manual 15.jpg
  • When the program runs here, the alexnet model will be downloaded, and the download time is a bit long. After downloading the program, an alexnet-owt-4df8aa71.pth file will appear in the /home/hetbot/.torch/models directory.
JetBot AI Kit Manual 16.jpg
  • Finally, run the program to train the neural network, and the running time is relatively long. After the training is completed, a best_mode.pth file will appear in the current directory.
JetBot AI Kit Manual 17.jpg
Step 3. Automatic Obstacle Avoiding
  • Access JetBot by going to https://<jetbot_ip_address>:8888, navigate to ~/Notebooks/collision_avoidance/
  • Open and follow the live_demo.ipynb notebook.
  • After running the program, the camera live image and a slider are displayed. Intermodulation represents the probability of encountering an obstacle, 0.00 means that there is no obstacle ahead, and 1.00 means that the obstacle ahead needs to be turned to avoid.
JetBot AI Kit Manual 18.jpg
  • Here, adjust the speed a little to avoid hitting the obstacles too fast. If obstacle avoidance cannot be achieved in some places, it is recommended to collect more data.
JetBot AI Kit Manual 19.jpg
  • [Note] Some statements may take a long time to run. There is a program running promptly in the upper right corner of JupyterLab. When the small dot is black, it means the program is running, and white means it is idle.
JetBot AI Kit Manual 20.jpg


The current image of the latest jetpack4.5 does not support the target tracking demo.
In this chapter we demonstrate how to use jetbot to track objects, we use a pre-trained coco dataset neural network that can detect 90 different objects. Including people (index 0), cups (index 47), apples (index 54) and other objects. For specific objects that can be detected, you can refer to data list

  • Enter http://<jetbot_ip_address>:8888 in the browser to connect to the car, open Notebook/object_following/ on the left side and the live_demo.ipynb file.
  • Before running the program, you need to download the pre-trained ssd_mobilenet_v2_coco.engine model, decompress it, and copy it to the current folder directory.
  • It should be noted that the program in this chapter needs to use the module established in the previous chapter for autonomous obstacle avoidance, and the car needs to be performed in the same environment.
  • Before running this code, you need to put the detection object in front of the camera. The detected coco objects are output after running the program. Empty data [[ ]] is output if no object is detected. The output information can be seen through the table lookup that an apple has been detected, and the apple's ID is 53. If multiple objects are detected at the same time, multiple messages are output.
JetBot AI Kit Manual 023.jpg

After running the program, the output is as shown in the figure, a blue box is drawn around the detected object, and the target object (jetbot follows the target) will display a green frame.

JetBot AI Kit Manual 23.jpg
  • You can adjust the speed and turn gain appropriately to avoid the jetbot running too fast.
  • When jetbot detects the target, it will turn to the target, if blocked by an obstacle, jetbot will turn left.

Line tracking

This chapter we will use data collect, link tracking and auto-detecting to realize robot auto line-tracking function.

Step 1. Collect data by JetBot
  • Access JetBot by going to https://<jetbot_ip_address>:8888, navigate to ~/Notebooks/road_following/
  • Open data-collection.ipynb file.
JetBot AI Kit Manual 24.jpg
  • Running the codes and a video is played, you can follow it.
JetBot AI Kit Manual 25.jpg
  • On the image captured by camera, there are a green point and a blue line. The point and line is the expected road which Robot run.
  • The content below is similar to [#3. Teleoperation], modify the index and axes values.

【Note】The axes keys here must be analogized by Wie, that is, decimals can be output. If you use the gamepad we configured, you need to press the HOME button to switch the mode. Make the indicator light in the state of two lights.

JetBot AI Kit Manual 26.jpg
  • Modify button value for capturing. (you can also keep default setting).
JetBot AI Kit Manual 27.jpg
  • Collecting data. Set JetBot to different places of the lines, and use Gamepad to move the green point to the black line. The blue line is the way Jetbot expected to run. You can press the capture button to capture a picture. You should collect pictures as soon as possible, count shows the amount of the pictures captured.

【Note】If Gamepad is inconvenient for you, you can set the position of green point by dragging steering and throttle sliders.

  • Save pictures
JetBot AI Kit Manual 28.jpg
Step 2. Training model
  • Access Jetbo by going to https://<jetbot_ip_address>:8888, navigate to ~/Notebooks/road_following/
  • Open train_model.ipynb file.
  • If you use the data collected above, you needn't to unzip files next.
  • If you use external data, you need to modify the name road_following.zip to the corresponding file name.
JetBot AI Kit Manual 29.jpg
JetBot AI Kit Manual 30.jpg
  • Download Model
JetBot AI Kit Manual 31.jpg
  • Train model, it will generate best_steerin_mdel_xy.pth file.
Step 3. Road following
  • Access Jetbo by going to https://<jetbot_ip_address>:8888, navigate to ~/Notebooks/road_following/.
  • Open live_demo.ipynb file.
  • Load model and open camera for living video.
  • You can drag the sliders to modify parameters.
JetBot AI Kit Manual 32.jpg
  • x, y are forecast values. Speed is VSL of jetbot, steering is steering speed of jetbot.
JetBot AI Kit Manual 33.jpg
  • Move Jetbot by change the speed gain.
【Note】 You cannot set the speed gain too high, otherwise, JetBot may run fast and go off the rail. You can also set the steering smaller to make the motion of jetbot much more smooth.
JetBot AI Kit Manual 34.jpg