13 Display Real-Time Screen in Jupyter Lab
Displaying Real-Time Video in Jupyter Lab
In the previous chapter, we used Flask to display real-time camera footage. That method requires opening a new browser tab or using another device to access it. In this chapter, we will present a solution for viewing real-time video directly in Jupyter Lab.
Preparation
Since the product runs the main program automatically at startup by default, which occupies the camera resource, you cannot use this tutorial under that condition. You need to terminate the main program or disable its auto-start, then restart the robot.
Note that the robot's main program uses multi‑threading and is configured to run at boot via crontab, so a conventional sudo killall python usually does not work. Therefore we describe here how to disable the auto-start of the main program.
If you have already disabled the auto-start of the robot's main program, you do not need to execute the Terminate the Main Program section below.
Terminate the Main Program
1. Click the "+" icon next to the current page tab to open a new Launcher tab.
2. Click "Terminal" under "Other" to open a terminal window.
3. In the terminal window, type bash and press Enter.
4. You can now control the robot using the Bash shell.
5. Enter the command: crontab -e
6. If asked which editor to use, type 1 and press Enter to select nano.
7. After opening the crontab configuration file, you should see the following two lines:
@reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1 @reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1
8. Add a # at the very beginning of the line that starts with ……app.py >> …… to comment it out.
# @reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1 @reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1
9. In the terminal page, press Ctrl+X to exit. It will ask Save modified buffer? Type Y and press Enter to save the changes.
10. Reboot the device. Note that this process will temporarily close the current Jupyter Lab session. If you did not comment out the line ……start_jupyter.sh >> …… in the previous step, you will still be able to use Jupyter Lab normally after the robot restarts (JupyterLab and the robot main program 'app.py' run independently). You may need to refresh the page.
11. One important point: because the lower computer continuously communicates with the upper computer via the serial port, a voltage glitch on the serial line during the upper computer reboot may prevent it from booting correctly. For example, on a Raspberry Pi as the upper computer, after a reboot the Pi may shut down but not restart – the red LED stays on while the green LED does not light. In that case, you can turn off the robot power switch and then turn it on again; the robot will then restart normally.
12. Enter the reboot command: sudo reboot
13. Wait for the device to restart (during reboot the green LED on the Raspberry Pi will blink; when the blinking slows down or stops, it indicates that startup has succeeded), refresh the page, and continue with the remaining parts of this tutorial.
Example
The following code block can be executed directly:
1. Select the code block below.
2. Press Shift+Enter to run the code block.
3. Watch the real-time video window.
4. Press STOP to close the real-time video and release the camera resource.
If you cannot see the camera's real-time video when running
- Click "Kernel" → "Shut down all kernels"
- Close this chapter's tab and reopen it
- Press STOP to release the camera resource, then re‑run the code block
- Reboot the device
import matplotlib.pyplot as plt # import matplotlib for plotting
import cv2 # import OpenCV for image processing
from picamera2 import Picamera2 # import Picamera2 to access Raspberry Pi Camera
import numpy as np # import NumPy for mathematical operations
from IPython.display import display, Image # import IPython display functions
import ipywidgets as widgets # import ipywidgets for interactive controls
import threading # import threading for multi‑threading
# Create a toggle button as a stop button
stopButton = widgets.ToggleButton(
value=False, # initial state: not selected
description='Stop', # text displayed on the button
disabled=False, # button initially enabled
button_style='danger', # red button style
tooltip='Description', # tooltip on hover
icon='square' # icon displayed on the button
)
# Define a function to display the video stream
def view(button):
# If you are using a CSI camera, uncomment the picam2 code and comment out the camera code
# Because newer versions of OpenCV (4.9.0.80) no longer support CSI cameras, you need to use picamera2 to capture camera frames
# picam2 = Picamera2() # create a Picamera2 instance
# Configure camera parameters: set video format and size
# picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
# picam2.start() # start the camera
camera = cv2.VideoCapture(-1) # create a camera instance
# set resolution
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
display_handle = display(None, display_id=True) # create a display handle for updating content
while True:
# frame = picam2.capture_array() # capture a frame from the camera
_, frame = camera.read() # capture a frame from the camera
# Optional processing (flipping, color conversion, etc.) can be done here
_, frame = cv2.imencode('.jpeg', frame) # encode the frame to JPEG format
display_handle.update(Image(data=frame.tobytes())) # update the displayed image
if stopButton.value == True: # check if the stop button has been pressed
# picam2.close() # if yes, close the camera
cv2.release() # if yes, release the camera
display_handle.update(None) # clear the displayed content
# Display the stop button
display(stopButton)
# Create and start a thread with target function view, passing the stop button as argument
thread = threading.Thread(target=view, args=(stopButton,))
thread.start() # start the thread