Hand tracking MIDI Controller with Ableton (Max Msp + Python)

I wanted to control a few parameters in Ableton Live’s Drift synth using hand gestures on a webcam. A lot of the setups you see online use TouchDesigner for this, but I wanted to do it with Max Msp and a python script.
I ended up writing a Python script that handles the tracking via MediaPipe, then streams that data straight into Ableton using OSC and a basic Max for Live patch.
Here is exactly how the routing works and how to set it up.

How the data flows:

1. Webcam: Grabs the live video feed.
2. Python: Processes the frames, calculates the hand position (–>scales the numbers between 0.0 and 1.0., this is directly compatible with MIDI (no need to scale it to 0-127)
3. OSC: Sends those values over your local network port.
4. Max for Live: Receives the network data inside Ableton and maps it to your synth knobs.

1. The Python Script

You’ll need to install this package via your terminal first (copy this in terminal):

pip install opencv-python mediapipe python-osc

This script mirrors the camera view, calculates the X/Y position of your wrist and measures the distance between your fingers to detect a fist clench.

FULL PYTHON CODE:

import cv2
import mediapipe as mp
import math
from pythonosc.udp_client import SimpleUDPClient

# Target Max for live local ports
client = SimpleUDPClient("127.0.0.1", 8000)

mp_hands = mp.solutions.hands
mp_draw = mp.solutions.drawing_utils

cap = cv2.VideoCapture(0)

def calculate_distance(p1, p2):
    """Finds the distance between two landmarks."""
    # HIER: De **2 toegevoegd voor de X en Y as
    return math.sqrt((p1.x - p2.x)**2 + (p1.y - p2.y)**2 + (p1.z - p2.z)**2)

with mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.7
) as hands:
    
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break

        frame = cv2.flip(frame, 1) # Mirror view
        h, w, _ = frame.shape

        rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        results = hands.process(rgb)

        if results.multi_hand_landmarks:
            hand = results.multi_hand_landmarks[0]

            # eextract landmarks
            wrist = hand.landmark[0]
            index_tip = hand.landmark[8]
            middle_tip = hand.landmark[12]
            ring_tip = hand.landmark[16]
            pinky_tip = hand.landmark[20]

            # 1. Octave (Fist clench)
            d1 = calculate_distance(index_tip, wrist)
            d2 = calculate_distance(middle_tip, wrist)
            d3 = calculate_distance(ring_tip, wrist)
            d4 = calculate_distance(pinky_tip, wrist)
            avg_finger_distance = (d1 + d2 + d3 + d4) / 4

            fist_value = (avg_finger_distance - 0.15) / (0.35 - 0.15)
            fist_value = max(0.0, min(1.0, fist_value))

            # 2. Filter Cutoff (X-Axis)
            x_value = wrist.x
            x_value = max(0.0, min(1.0, x_value))

            # 3. Wave Shape (Y-Axis / Height)
            height_value = 1.0 - wrist.y 
            height_value = max(0.0, min(1.0, height_value))

            # send OSCdata
            client.send_message("/hand/fist", fist_value)
            client.send_message("/hand/x", x_value)          
            client.send_message("/hand/height", height_value) 

            # Draw lines (skeletonsss)
            mp_draw.draw_landmarks(frame, hand, mp_hands.HAND_CONNECTIONS)

            # coordinates
            wrist_x = int(wrist.x * w)
            wrist_y = int(wrist.y * h)

            # Text (depends on ableton parameters:) )
            cv2.putText(frame, f"Octave (Fist): {fist_value:.2f}", (wrist_x + 15, wrist_y - 40), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
            cv2.putText(frame, f"Cutoff (X): {x_value:.2f}", (wrist_x + 15, wrist_y - 20), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
            cv2.putText(frame, f"Wave Shape (Y): {height_value:.2f}", (wrist_x + 15, wrist_y), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

        cv2.imshow("Hand Tracking Jam", frame)

        if cv2.waitKey(1) & 0xFF == 27: # Press ESC to close
            break

cap.release()
cv2.destroyAllWindows()

Put this full code in a text document on your computer (make sure to convert this to a .py document)
To run it, just copy these one by one in terminal (press enter after each line):

cd ~/mediapipe-max

source venv/bin/activate

python “name_of_your_script”.py (make sure to replace “name_of_your_script” with the name of your script 😉 )

2. Max MSP & Ableton Setup

To get the data into Ableton, you need a Max for Live device that receives UDP/OSC strings (like the standard Ableton Connection Kit or any basic custom patch)

1. Go into ableton
2. Create a Max For Live Midi Instrument
3. Click “edit in Max”
4. Make this patch:

This patch sets the incoming UDP port to 8000 and simply uses a route /hand/fist /hand/x /hand/height object to split the data streams.

If you now save and close this patch, you should be able to map paramaters within ableton!

While I kept it to 3 parameters to stay focused during the live jam, MediaPipe actually tracks 21 individual 3D points on your hand at the same time.
Because you have the coordinates for every single finger joint inside the Python script, you can easily expand this. You could write a few lines of logic to track individual finger pinches (like thumb to index for an effects send), calculate the rotation/tilt of your hand for panning, or trigger clip launches when you hold up a specific number of fingers.

Shoutout to these videos on Youtube for inspiring me:

“[Max/MSP] gesture controlled vocoder” by Sunkyung Ryu (insta @i____eyre)
“Handmate MIDI Demo – MediaPipe Hand-Tracking and Gesture Recognition for Web MIDI” by Monica Lim (insta @monicalimmusic)
“How To Tutorial – Gesture Recognition in Ableton – SubjectSound” by Subject Sound (insta @subjectsound)

Thanks for reading!