21 Auto Driving of Patrol Lines Based on OpenCV
Line Following Autonomous Driving with OpenCV
In this chapter, we will use basic OpenCV functionality to detect yellow (default color) lines in the image, and then control the chassis steering by detecting the position of that yellow line (the chassis will not move in this example; this example only demonstrates the OpenCV algorithm on screen). For safety reasons, we will not integrate motion control into the example, because this function is heavily affected by external factors. Users must fully understand the code before adding their own motion control functions.
If you want to use this example to control the robot's movement, please refer to the previous Python Chassis Motion Control chapter to add the relevant motion control functions (our open‑source example is located in robot_ctrl.py).
Preparation
Since the product runs the main program automatically at startup by default, which occupies the camera resource, you cannot use this tutorial under that condition. You need to terminate the main program or disable its auto-start, then restart the robot.
Note that the robot's main program uses multi‑threading and is configured to run at boot via crontab, so a conventional sudo killall python usually does not work. Therefore we describe here how to disable the auto-start of the main program.
If you have already disabled the auto-start of the robot's main program, you do not need to execute the Terminate the Main Program section below.
Terminate the Main Program
1. Click the "+" icon next to the current page tab to open a new Launcher tab.
2. Click "Terminal" under "Other" to open a terminal window.
3. In the terminal window, type bash and press Enter.
4. You can now control the robot using the Bash shell.
5. Enter the command: crontab -e
6. If asked which editor to use, type 1 and press Enter to select nano.
7. After opening the crontab configuration file, you should see the following two lines:
@reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1 @reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1
8. Add a # at the very beginning of the line that starts with ……app.py >> …… to comment it out.
# @reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1 @reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1
9. In the terminal page, press Ctrl+X to exit. It will ask Save modified buffer? Type Y and press Enter to save the changes.
10. Reboot the device. Note that this process will temporarily close the current Jupyter Lab session. If you did not comment out the line ……start_jupyter.sh >> …… in the previous step, you will still be able to use Jupyter Lab normally after the robot restarts (JupyterLab and the robot main program app.py run independently). You may need to refresh the page.
11. One important point: because the lower computer continuously communicates with the upper computer via the serial port, a voltage glitch on the serial line during the upper computer reboot may prevent it from booting correctly. For example, on a Raspberry Pi as the upper computer, after a reboot the Pi may shut down but not restart – the red LED stays on while the green LED does not light. In that case, you can turn off the robot power switch and then turn it on again; the robot will then restart normally.
12. Enter the reboot command: sudo reboot
13. Wait for the device to restart (during reboot the green LED on the Raspberry Pi will blink; when the blinking slows down or stops, it indicates that startup has succeeded), refresh the page, and continue with the remaining parts of this tutorial.
Example
The following code block can be executed directly:
1. Select the code block below.
2. Press Shift+Enter to run the code block.
3. Watch the real-time video window.
4. Press STOP to close the real-time video and release the camera resource.
If you cannot see the camera's real-time video when running
- Click "Kernel" → "Shut down all kernels"
- Close this chapter's tab and reopen it
- Press STOP to release the camera resource, then re‑run the code block
- Reboot the device
Notes
If you are using a USB camera, uncomment the line frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).
Operation
After running the code block below, place yellow tape in front of the camera and observe whether the contour of the yellow tape appears in the black image, and whether the two detection lines can detect the yellow tape.
import cv2 # import OpenCV for image processing
import imutils, math # helper libraries for image processing and math
from picamera2 import Picamera2 # library to access Raspberry Pi Camera
import numpy as np
from IPython.display import display, Image # display images in Jupyter Notebook
import ipywidgets as widgets # create interactive widgets such as buttons
import threading # create new threads for asynchronous execution
# Stop button
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# findline autodrive
# Upper detection line, 0.6 represents the position; larger value moves it down
sampling_line_1 = 0.6
# Lower detection line, value must be greater than sampling_line_1 and less than 1
sampling_line_2 = 0.9
# Influence of detection line slope on turning
slope_impact = 1.5
# Influence of the line position detected at the lower detection line on turning
base_impact = 0.005
# Influence of current speed on turning
speed_impact = 0.5
# Line following speed
line_track_speed = 0.3
# Influence of slope on line following speed
slope_on_speed = 0.1
# Target line color in HSV color space
line_lower = np.array([25, 150, 70])
line_upper = np.array([42, 255, 255])
def view(button):
# If you are using a CSI camera, uncomment the picam2 code and comment out the camera code
# Because newer versions of OpenCV (4.9.0.80) no longer support CSI cameras, you need to use picamera2 to capture camera frames
# picam2 = Picamera2() # create a Picamera2 instance
# Configure camera parameters: set video format and size
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
# picam2.start() # start the camera
camera = cv2.VideoCapture(-1) # create a camera instance
# set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle = display(None, display_id=True)
while True:
# img = picam2.capture_array()
_, img = camera.read() # capture a frame from the camera
# frame = cv2.flip(frame, 1) # if your camera reverses your image
height, width = img.shape[:2]
center_x, center_y = width // 2, height // 2
# Image preprocessing: color space conversion, Gaussian blur, color range filtering, etc.
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
line_mask = cv2.inRange(hsv, line_lower, line_upper) # filter out the target line by color range
line_mask = cv2.erode(line_mask, None, iterations=1) # erode to remove noise
line_mask = cv2.dilate(line_mask, None, iterations=1) # dilate to enhance the target line
# Detect the target line at the positions of the upper and lower sampling lines, and compute steering and speed control signals
sampling_h1 = int(height * sampling_line_1)
sampling_h2 = int(height * sampling_line_2)
get_sampling_1 = line_mask[sampling_h1]
get_sampling_2 = line_mask[sampling_h2]
# Compute the width of the target line at the upper and lower sampling lines
sampling_width_1 = np.sum(get_sampling_1 == 255)
sampling_width_2 = np.sum(get_sampling_2 == 255)
if sampling_width_1:
sam_1 = True
else:
sam_1 = False
if sampling_width_2:
sam_2 = True
else:
sam_2 = False
# Get the edge indices of the target line at the upper and lower sampling lines
line_index_1 = np.where(get_sampling_1 == 255)
line_index_2 = np.where(get_sampling_2 == 255)
# If the target line is detected at the upper sampling line, compute its center
if sam_1:
sampling_1_left = line_index_1[0][0] # leftmost index of the target line at the upper sampling line
sampling_1_right = line_index_1[0][sampling_width_1 - 1] # rightmost index
sampling_1_center = int((sampling_1_left + sampling_1_right) / 2) # center index
# If the target line is detected at the lower sampling line, compute its center
if sam_2:
sampling_2_left = line_index_2[0][0]
sampling_2_right = line_index_2[0][sampling_width_2 - 1]
sampling_2_center = int((sampling_2_left + sampling_2_right) / 2)
# Initialize steering and speed control signals
line_slope = 0
input_speed = 0
input_turning = 0
# If the target line is detected at both sampling lines, compute the line slope and then the speed/turning signals
if sam_1 and sam_2:
line_slope = (sampling_1_center - sampling_2_center) / abs(sampling_h1 - sampling_h2) # compute line slope
impact_by_slope = slope_on_speed * abs(line_slope) # compute speed impact from slope
input_speed = line_track_speed - impact_by_slope # compute speed control signal
input_turning = -(line_slope * slope_impact + (sampling_2_center - center_x) * base_impact) #+ (speed_impact * input_speed) # compute turning control signal
elif not sam_1 and sam_2: # target line detected only at the lower sampling line
input_speed = 0 # set speed to 0
input_turning = (sampling_2_center - center_x) * base_impact # compute turning control signal
elif sam_1 and not sam_2: # target line detected only at the upper sampling line
input_speed = (line_track_speed / 3) # reduce speed
input_turning = 0 # no turning
else: # target line not detected at either sampling line
input_speed = - (line_track_speed / 3) # reverse
input_turning = 0 # no turning
# base.base_json_ctrl({"T":13,"X":input_speed,"Z":input_turning})
cv2.putText(line_mask, f'X: {input_speed:.2f}, Z: {input_turning:.2f}', (center_x+50, center_y+0), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1)
# Visualization: draw lines at the sampling positions, mark sampling results, and display steering/speed signals
cv2.line(line_mask, (0, sampling_h1), (img.shape[1], sampling_h1), (255, 0, 0), 2)
cv2.line(line_mask, (0, sampling_h2), (img.shape[1], sampling_h2), (255, 0, 0), 2)
if sam_1:
# Draw green marker lines at both ends of the target line at the upper sampling line
cv2.line(line_mask, (sampling_1_left, sampling_h1+20), (sampling_1_left, sampling_h1-20), (0, 255, 0), 2)
cv2.line(line_mask, (sampling_1_right, sampling_h1+20), (sampling_1_right, sampling_h1-20), (0, 255, 0), 2)
if sam_2:
# Draw green marker lines at both ends of the target line at the lower sampling line
cv2.line(line_mask, (sampling_2_left, sampling_h2+20), (sampling_2_left, sampling_h2-20), (0, 255, 0), 2)
cv2.line(line_mask, (sampling_2_right, sampling_h2+20), (sampling_2_right, sampling_h2-20), (0, 255, 0), 2)
if sam_1 and sam_2:
# If the target line is detected at both sampling lines, draw a red line connecting the two centers
cv2.line(line_mask, (sampling_1_center, sampling_h1), (sampling_2_center, sampling_h2), (255, 0, 0), 2)
_, frame = cv2.imencode('.jpeg', line_mask)
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value == True:
# picam2.close()
cv2.release() # if yes, release the camera
display_handle.update(None)
# Display the "Stop" button and start the display function thread
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()