Motion Capture Hand

A glove that drives a rigged 3D hand in Blender in real time, built from an IMU, five flex sensors, and an Arduino.

▶ Watch the demo · Read the write-up

Built October to December 2021 as a university computer-animation project. The goal was a motion capture rig for a whole hand, fingers included, cheap enough to build on a desk out of hobby parts. Rotation of the hand comes from an inertial measurement unit; each finger gets its own flex sensor.

It is archived and I am not developing it further, but it still works exactly as filmed.

How it works

flowchart LR
  A["BNO055 IMU<br/>hand orientation"] --> C
  B["5x flex sensor<br/>finger bend"] --> C
  C["Arduino<br/>read, filter, format"] -->|"USB serial, 115200 baud"| D
  D["Blender<br/>imuDataManipulator.py"] --> E["Hand armature<br/>quaternion retarget"]

Loading

The Arduino reads both sensor types, does the filtering and unit conversion, and prints a formatted frame to the serial monitor. Blender reads that same port from Python, converts each frame into a rotation, and writes it onto the matching bone of the armature. Calling bpy.ops.wm.redraw_timer after each frame is what makes the viewport update live instead of only when the script finishes.

Getting rotation out of the IMU

This was the part that took the longest, and the route matters more than the destination.

I started by estimating roll and pitch from raw accelerometer data, taking the angle in the Euclidean plane of the x and z axes for roll and the y and z axes for pitch, and visualising it on a rectangle in Processing. Accelerometer-only orientation turned out to be very sensitive to vibration.

So I added a complementary filter, blending a small percentage of the newly computed angle with a large percentage of the previous filtered value, then brought the gyroscope in to help. Yaw stayed shakier than I wanted.

Then, reading the datasheet properly, I found the BNO055 fuses its own sensors and exposes a quaternion directly. That solved orientation outright and sidestepped the drift and gimbal problems I had been fighting. The Euler path is still in the firmware behind a RotationType enum, so you can switch back and compare the two.

Getting bend out of the flex sensors

A flex sensor is a variable resistor, so the reading is whatever the ADC says and nothing more. Even sensors of the same model disagreed with each other, so each finger is calibrated by hand: read the value flat, read it at 90 degrees, then map that range onto 0 to 90 degrees. The index finger, for example, sat near 480 open and near 250 fully bent. Those degrees are what get retargeted onto the finger bone.

Hardware

Part	Role
Adafruit BNO055	Absolute orientation, fused on-chip, read over I2C
Adafruit Unified Sensor library	Driver layer for the BNO055
5x flex sensor	One per finger, read as analog resistance
Arduino	Samples both sensor types and writes frames to serial
Latex glove	What everything is mounted to

Repository layout

Source/Arduino/Global/
  Global.ino                globals, sensor objects, RotationType enum
  Execute.ino               setup() and loop(), quaternion and Euler paths
  Helper.ino                filtering, calibration, flex mapping, serial output
Source/Python/
  imuDataManipulator.py     reads serial, retargets rotations onto the armature
  operator_hand.py          wraps it as a Blender operator, "Activate Hand"
Blender/SkeletonHand.blend  the rigged hand
Report.pdf                  full write-up, including what did not work

Running it

You need Blender with pyserial available to its bundled Python, plus the Adafruit BNO055 and Unified Sensor libraries in the Arduino IDE.

1. Wire the sensors and connect the Arduino.
2. Open Source/Arduino/Global/Global.ino and upload. The sketch spans three .ino files in one folder, so the IDE compiles them together.
3. Open Blender/SkeletonHand.blend.
4. In the Scripting workspace, run imuDataManipulator.py and then operator_hand.py with Alt+P.
5. Back in Layout, press F3, search for Activate Hand, and hit Enter.
6. Give it about three seconds. The bones need a moment to settle into the correct orientation before the motion looks right.

Two things you will have to change first. operator_hand.py appends a hardcoded absolute path to sys.path so Blender can find imuDataManipulator, and it opens com3 at 115200 baud. Point the path at wherever you cloned this, and set the port to whatever your Arduino enumerates as. The armature is expected to be named HandsRig, with a hand bone and finger bones named pinky, ring, middle, index and thumb.

What never got finished

Kept here because the write-up is honest about it and the limitations are the interesting part.

• The flex sensors never made it onto the glove. They work, but they stayed on the breadboard. Soldering a resistor to the leads snapped the sensor tips, and with a presentation coming I chose a working demo over more broken parts.
• One bone per finger. Each flex sensor drives a single joint, so fingers curl rather than articulating across all three joints.
• No collision handling. It was in the proposal and did not happen.
• A second IMU on the elbow was attempted, to extend capture up the arm, after the first one worked.

References

• Madgwick, S. (2010). An efficient orientation filter for inertial and inertial/magnetic sensor arrays
• Adafruit. BNO055 Absolute Orientation Sensor
• Blender. Python API Documentation
• Wikipedia. Conversion between quaternions and Euler angles
• An, L. et al. (2019). A Novel Method for Estimating Pitch and Yaw of Rotating Projectiles Based on Dynamic Constraints
• Campbell, S. Basics of the I2C Communication Protocol
• Paul McWhorter's Arduino series

License

MIT. See LICENSE.