01 · Research · Robotics & computer vision
Building a fiddler crab, then tracking the real ones
Two parts of one field study in Thailand: a robotic claw that replicated a male’s mating wave, and a computer-vision pipeline that timestamps every real crab’s wave from drone footage.
I worked on both halves of a Northwestern ESAM × NICO field study of fiddler crabs in Thailand. First I led the hardware and embedded software for a robotic claw that performed a male’s mating wave on camera. Then, with the MMCS Lab, I built the software that reads the drone footage from the field: it pins the ground in place to within a fraction of a pixel, finds every crab’s burrow, and timestamps each claw wave, so we can test whether neighbouring crabs wave in sync.
Part one
The robot

Preserving the character of the wave
Turning a mating display into a machine meant preserving the character of the wave, not just the geometry. I broke observed movement into engineering requirements, then tuned the timing and amplitude until the gesture read correctly in the field.
A compact field-ready mechanism
The claw used a 2-DOF servo-actuated mechanism, custom 3D-printed parts, and an Arduino Nano controller. Python tools handled real-time motion tuning and data capture, so the wave could be adjusted between takes rather than between builds.
A custom tool for shaping the claw’s path
To get the wave right I built a custom tool for authoring the claw’s motion path directly. It let me tweak keyframes, timing, and easing for each servo and preview the resulting trajectory, then push the refined path to the controller — so refining the gesture was a matter of adjusting curves rather than reflashing hand-edited values.
Designed for work outside the lab
The system was built for an international science and wildlife media production in Thailand, where reliability, portability, and fast iteration mattered as much as mechanical fidelity.
Part two
The tracking
Making a hovering drone sit still
Wind pushes a hovering drone more than a hundred pixels around a 4K frame. The camera looks straight down at flat ground, so every frame maps onto the first by a homography. I register each frame directly to frame 0 using about 2,000 tracked points, reject moving crabs as outliers, and re-render once in native 10-bit. Maximum drift drops from 133 px to 0.45 px, and a full 7-minute 4K60 flight is analysed in about 4 minutes on a laptop.
Watch the burrow, not the crab
Waving males stay at their burrows. So instead of chasing crabs frame to frame, the pipeline finds where each crab lives from a map of crab-scale motion, then watches a fixed window there in every frame. Identities can’t swap. The same signals are recorded on patches of empty mud, so codec flicker and leftover jitter get measured instead of passing for synchrony.
From 5% of frames to 99.9%
On the same 13-second clip, the existing MATLAB tracker found 28 crabs, located them in 5% of frames, and produced no wave events in 315 s. Mine found 55 sites with data in 99.9% of frames and 156 timestamped waves in 63 s on the same machine, with zero false events on the empty-mud controls. On a full 6:50 flight it found 200 crab sites and 6,638 waves. Next: testing synchrony against a time-shifted null.
A labeling tool built for speed
Motion also picks up wind-blown stalks and shells. I built a local labeling page that loops each candidate site in real time next to a strip of its whole clip. You press 1 for crab, 2 for not, 3 for unsure, and the unsure sites come back for a second look. The labels train a classifier on 16 motion features plus DINOv2 image embeddings. On held-out clips it cut false sites by more than a third while keeping 94% of real crabs.