← Engineering
Completed2025

RTHT-3D

A webcam-based interface that turns hand movements into real-time controls for Blender.

Runtime data flowTwo Python processes connected over a small local UDP protocol
RTHT-3D runtime architectureA webcam frame is processed by OpenCV and MediaPipe, classified into gestures, encoded as CSV and sent over UDP to a Blender listener that dispatches scene mutations on Blender's main thread.PYTHON PROCESS · HAND TRACKINGTRANSPORTBLENDER PROCESSWebcam + OpenCVVideoCapture(0)flip · BGR → RGBMediaPipe Handsmodel complexity 0detection 0.5tracking 0.4maximum 2 hands21 landmarks / handGesture classifierpoint · pinch · Vpalm · fistpinch distance < 0.1coordinates x,y∈ [0,1]Message encoderUTF-8 CSVgesture,x,yoptional second handno timestamp /sequence IDEffective input controlscamera FPS: hardware-dependentwaitKey(5 ms): UI polling onlymessages sent when hand 1 ≠ noneUDPlocalhostport 5006fire-and-forgetListener threadrecvfrom 1024 Bsocket timeout1000 msdaemon threadMain-thread bridgebpy.app.timerssafe scene mutationPayload parser3 fields / handfloat x,yone or two handsGesture dispatchersingle-hand actionstwo-hand combinationstransition-awareselectionMotion filtersposition history: 3movement factor: 0.1rotation + scale:0.02jitter threshold:0.005Blender sceneselect · move ·transformcreate · duplicate ·deletepaint · RGBseparationviewport feedbackRGB framenormalized landmarksgesture + x,ydatagramUTF-8 bytesscheduledatatyped valuesdeltasscene operation

Gesture contract

Point transition
Select object
Pinch
Move selected object
Two pinches
Rotate and scale
Two palms
Create plane · 1000 ms cooldown
Two V signs
Duplicate selected object
Two fists
Delete selected object
Fist + point
Toggle painting mode
Fist + palm
Clear paint trail
Pinch + palm
Toggle RGB separation

Problem

Project brief

Control a 3D scene from a standard webcam without a physical controller or dedicated tracking hardware.

Contribution

Role and scope

Designed and implemented the vision pipeline, gesture model, UDP protocol and Blender integration.

Python · MediaPipe · Blender · UDP

Implementation

Key technical decisions

  1. Used MediaPipe hand landmarks to derive semantic gesture states instead of coupling raw coordinates to scene actions.
  2. Separated webcam tracking and Blender into two Python processes connected over local UDP.
  3. Mapped one-hand and two-hand gestures to distinct operations so complex actions remain intentional.

Constraints

Engineering constraints

  • Translate noisy landmark data into gestures that remain understandable in motion.
  • Keep the vision process separate from Blender without making interaction feel delayed.
  • Support selection, movement, scaling and scene actions through a small gesture vocabulary.

Outcome

Result

The finished prototype supports one- and two-hand scene controls; its public demonstration reached 343K views.

View sourceWatch demo