← work

Verlet Robotics · annotation tooling & case study

Turning estimated hand motion into reviewable annotations

GitHub profile

A review and correction workflow for egocentric hand motion. The tools turn HaWoR estimates into annotations that can be inspected, corrected, and exported with a record of how each frame was produced.

What is MANO annotation?

MANO is a compact, deformable 3D hand model introduced by Javier Romero, Dimitrios Tzionas, and Michael J. Black. Its parameters describe hand shape and articulation, allowing a posed hand surface to be reconstructed from a small representation.

In this workflow, annotation means fitting and reviewing that representation over time: which hand is present, how its fingers are posed, where the wrist sits, and how the hand relates to the camera. Camera motion provides the bridge from camera-relative estimates to placement in the world. A useful annotation therefore carries both geometry and temporal context.

The problem: plausible hands can still be wrong

Egocentric video makes hand tracking difficult: fingers disappear behind objects, both hands overlap, and the camera moves with the wearer. An estimate can look plausible in one frame while switching hand identities, jumping in scale, or moving inconsistently across a sequence.

The correction layer organizes these failures into reviewable time spans. It combines motion signals with independent image evidence when available, then presents suggestions that can be accepted, rejected, or adjusted. This gives a reviewer a concrete place to intervene instead of treating every predicted frame as equally trustworthy.

Five correction tools, in a deliberate order

The review engine works over a shared annotation store. Corrections run in a fixed order because an early mistake can distort the evidence used by later steps.

  • Hand identity: exchange left/right assignments across an affected span while keeping the associated pose and evidence together.
  • Scale: adjust the camera trajectory’s scale consistently, preserving the relationship between camera-frame hands and world placement.
  • Pose: refine articulation around trusted anchors, using independent image evidence when available.
  • Contact: address implausible inter-hand penetration through geometric correction or the available refitting path.
  • Missing spans: interpolate between trusted boundaries, retaining an explicit record that those frames were filled in.

Review, correct, re-score, export

The workflow starts with hand estimates and the camera trajectory. It computes quality signals, groups suspect frames into segments, and generates review suggestions. In the interactive workflow, the reviewer chooses which suggestions to accept and can establish protected human keyframes.

After a correction, the affected spans are scored again. Locked human anchors survive subsequent passes. The export contains corrected hand parameters, derived world placement, timestamps, and per-frame provenance: a downstream consumer can distinguish an original estimate, an edited frame, and an interpolation.

The same review engine is available through a command-line workflow, including explicitly configured automatic passes. The design keeps the correction logic separate from the browser interface so the underlying operations can be tested independently.

Why EgoScale makes this worth studying

EgoScale, by Ruijie Zheng and collaborators, studies how large-scale egocentric human demonstrations can support dexterous robot learning. It trains on 20,854 hours of human video, combining relative wrist motion with finger actions retargeted to a robot hand. Human pretraining is followed by aligned human–robot mid-training and task-specific robot training.

The paper represents human hands through 21 keypoint transforms and retargets articulation into a 22-degree-of-freedom hand space. It reports a 54% improvement in average success rate over its no-pretraining baseline. These are the authors’ results, not measurements of this correction toolset.

My connection to this work is the annotation problem it motivates: recovering useful motion supervision from video. Reviewing identity, articulation, scale, and temporal consistency is a practical way to make that supervision inspectable. The toolset is a separate annotation project; EgoScale is the related research case study.

What is tested, and what still needs measurement

The toolset includes synthetic fixtures with injected hand swaps, scale errors, articulation corruption, penetration, and missing detections. Those fixtures make it possible to test the correction passes, protected anchors, and export behavior against known failures.

Synthetic tests do not establish accuracy on unconstrained recordings. Thresholds need calibration on real clips, and pose/contact fitting depends on the available evidence and compute path. When independent keypoints or masks are absent, some operations fall back to interpolation or geometric proxies. The next useful evaluation is a reviewed real-video benchmark that measures correction quality and reviewer time.