A fingerspelling classifier, built entirely by describing it.
An ASL learning game that reads your handshape through the webcam and grades it in real time

No flashcards. No typing. No IDE.

Sign the letter, hold it steady, and 21 hand landmarks decide if you got it right.

How it came together

Built entirely in Figma Make, entirely through prompts — no manual code written. I described the interaction loop I wanted — show a letter, sign it back, confirm it — and iterated by describing what was wrong until the detection held up.

No ML model. No training data. No labeled dataset to collect.

Each letter is a hand-authored geometric rule — which fingers are curled, which are extended, where the thumb sits relative to the palm — scored against MediaPipe's live landmark output every frame. Fingerspelling reduced to classifiable geometry.

A few of the iterations

• Detection fired on any matching frame — too sensitive to flicker. Prompted in a hold-to-confirm window so a sign has to hold steady before it registers.

• A, O, and C kept collapsing into each other. Refined the finger-curl thresholds by prompt until each letter had a distinct signature.

• A bare video feed gave no confidence the system was actually tracking you. Added a live landmark skeleton, an "I see: L" readout, and a tracking-status badge — the interface proving its own accuracy back to the player.

• A wrong sign used to end the round. Rebuilt the loop so a miss just resets the streak — the game keeps going, no dead ends.

Small prompts, each one making the detection feel less like a guess and more like an instrument.

Big challenges

Turning raw hand geometry into a sign-language classifier

MediaPipe has no concept of ASL — it returns coordinates, nothing else. Every letter had to be specified as a rule: finger extension states, joint angles, thumb position. Separating visually adjacent letters like A, O, and C meant tuning thresholds against a live feed by prompt, iteration after iteration, not reading off a spec.

Extending a static-pose system into motion

Letters are a single frame. Gestures like BIRD and HELLO are a hand moving relative to the face over time. That meant running a second face landmarker alongside the hand one, anchoring each gesture to a facial reference point — mouth, chin, forehead — and prompting a small state machine into existence to track motion instead of a static pose.

Making it playable without a camera at all

Not every player can, or wants to, sign at a webcam. Every gesture-based level needed a full non-camera equivalent — multiple-choice, keyboard-operable, identical scoring — so accessibility wasn't a fallback bolted on after, it was built into the same prompt as the camera mode.

Try it

→ Works best on desktop Chrome with good lighting

→ Hold up your hand and sign the letter on screen — hold it steady and watch it check off