Hand Gesture Recognition with MediaPipe

This chapter introduces how to use MediaPipe + OpenCV to implement hand gesture recognition.

What is MediaPipe?

MediaPipe is an open‑source framework developed by Google for building machine‑learning‑powered multimedia processing applications. It provides a set of tools and libraries for processing video, audio, and image data, and applying machine learning models to perform various tasks such as pose estimation, gesture recognition, face detection, and more. MediaPipe aims to offer efficient, flexible, and easy‑to‑use solutions, allowing developers to quickly build various multimedia processing applications.

Preparation

Since the product runs the main program automatically at startup by default, which occupies the camera resource, you cannot use this tutorial under that condition. You need to terminate the main program or disable its auto-start, then restart the robot.

Note that the robot's main program uses multi‑threading and is configured to run at boot via crontab, so a conventional sudo killall python usually does not work. Therefore we describe here how to disable the auto-start of the main program.

If you have already disabled the auto-start of the robot's main program, you do not need to execute the Terminate the Main Program section below.

Terminate the Main Program

1. Click the "+" icon next to the current page tab to open a new Launcher tab.

2. Click "Terminal" under "Other" to open a terminal window.

3. In the terminal window, type bash and press Enter.

4. You can now control the robot using the Bash shell.

5. Enter the command: crontab -e

6. If asked which editor to use, type 1 and press Enter to select nano.

7. After opening the crontab configuration file, you should see the following two lines:

@reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1
@reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1

8. Add a # at the very beginning of the line that starts with ……app.py >> …… to comment it out.

# @reboot ~/ugv_pt_rpi/ugv-env/bin/python ~/ugv_pt_rpi/app.py >> ~/ugv.log 2>&1
@reboot /bin/bash ~/ugv_pt_rpi/start_jupyter.sh >> ~/jupyter_log.log 2>&1

9. In the terminal page, press Ctrl+X to exit. It will ask Save modified buffer? Type Y and press Enter to save the changes.

10. Reboot the device. Note that this process will temporarily close the current Jupyter Lab session. If you did not comment out the line ……start_jupyter.sh >> …… in the previous step, you will still be able to use Jupyter Lab normally after the robot restarts (JupyterLab and the robot main program app.py run independently). You may need to refresh the page.

11. One important point: because the lower computer continuously communicates with the upper computer via the serial port, a voltage glitch on the serial line during the upper computer reboot may prevent it from booting correctly. For example, on a Raspberry Pi as the upper computer, after a reboot the Pi may shut down but not restart – the red LED stays on while the green LED does not light. In that case, you can turn off the robot power switch and then turn it on again; the robot will then restart normally.

12. Enter the reboot command: sudo reboot

13. Wait for the device to restart (during reboot the green LED on the Raspberry Pi will blink; when the blinking slows down or stops, it indicates that startup has succeeded), refresh the page, and continue with the remaining parts of this tutorial.

Example

The following code block can be executed directly:

1. Select the code block below.

2. Press Shift+Enter to run the code block.

3. Watch the real-time video window.

4. Press STOP to close the real-time video and release the camera resource.

If you cannot see the camera's real-time video when running

Click "Kernel" → "Shut down all kernels"
Close this chapters tab and reopen it
Press STOP to release the camera resource, then re‑run the code block
Reboot the device

Notes

If you are using a USB camera, uncomment the line frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).

Features of this chapter

When the code block is running normally, you can place your hand in front of the camera. The real‑time video will annotate the joints of the hand, which will change as your hand moves. The positions of each joint are also output, facilitating further development for gesture control.

MediaPipe's hand gesture recognition uses different names for different joints. You can obtain the position of a specific joint by calling its corresponding index.

MediaPipe Hand

WRIST
THUMB_CMC
THUMB_MCP
THUMB_IP
THUMB_TIP
INDEX_FINGER_MCP
INDEX_FINGER_PIP
INDEX_FINGER_DIP
INDEX_FINGER_TIP
MIDDLE_FINGER_MCP
MIDDLE_FINGER_PIP
MIDDLE_FINGER_DIP
MIDDLE_FINGER_TIP
RING_FINGER_MCP
RING_FINGER_PIP
RING_FINGER_DIP
RING_FINGER_TIP
PINKY_MCP
PINKY_PIP
PINKY_DIP
PINKY_TIP

import cv2
import imutils, math
from picamera2 import Picamera2  # library to access Raspberry Pi Camera
from IPython.display import display, Image  # display images in Jupyter Notebook
import ipywidgets as widgets  # create interactive widgets such as buttons
import threading  # create new threads for asynchronous execution
import mediapipe as mp  # import MediaPipe library for hand landmark detection


# Create a "Stop" button that users can click to stop the video stream
# ================
stopButton = widgets.ToggleButton(
    value=False,
    description='Stop',
    disabled=False,
    button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Description',
    icon='square' # (FontAwesome names without the `fa-` prefix)
)

# Initialize MediaPipe drawing utilities and hand landmark detection model
mpDraw = mp.solutions.drawing_utils

mpHands = mp.solutions.hands
hands = mpHands.Hands(max_num_hands=1) # initialize hand landmark detection model, detect at most one hand

# Define the display function, which processes video frames and performs hand landmark detection
def view(button):
    # If you are using a CSI camera, uncomment the picam2 code and comment out the camera code
    # Because newer versions of OpenCV (4.9.0.80) no longer support CSI cameras, you need to use picamera2 to capture camera frames
    
    # picam2 = Picamera2()  # create a Picamera2 instance
    # Configure camera parameters: set video format and size
    # picam2.configure(picam2.create_video_configuration(main={"format": 'XRGB8888', "size": (640, 480)}))
    # picam2.start()  # start the camera

    camera = cv2.VideoCapture(-1) # create a camera instance
    # set resolution
    camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    
    display_handle = display(None, display_id=True)  # create a display handle for updating the displayed image
    
    while True:
        # frame = picam2.capture_array()
        _, frame = camera.read() # capture a frame from the camera
        # frame = cv2.flip(frame, 1) # if your camera reverses your image

        img = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
        results = hands.process(img)

        # If hand landmarks are detected
        if results.multi_hand_landmarks:
            for handLms in results.multi_hand_landmarks: # iterate over each detected hand
                # Draw hand landmarks
                for id, lm in enumerate(handLms.landmark):
                    h, w, c = img.shape
                    cx, cy = int(lm.x * w), int(lm.y * h)  # compute the position of the landmark in the image
                    cv2.circle(img, (cx, cy), 5, (255, 0, 0), -1)  # draw a circle at the landmark position

                
                frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
                mpDraw.draw_landmarks(frame, handLms, mpHands.HAND_CONNECTIONS) # draw hand skeleton connections
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) 

                target_pos = handLms.landmark[mpHands.HandLandmark.INDEX_FINGER_TIP]

        _, frame = cv2.imencode('.jpeg', frame)
        display_handle.update(Image(data=frame.tobytes()))
        if stopButton.value == True:
            # picam2.close() # if yes, close the camera
            cv2.release() # if yes, release the camera
            display_handle.update(None)

# Display the "Stop" button and start the display function thread
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()

Navigation menu

22 Gesture Recognition Based on MediaPipe

Contents