The labels
How we label it.
We use the formats robot-learning teams already know: Ego4D hand-object interactions and EPIC-KITCHENS verb-noun actions. Every clip gets narration and keysteps. Every interaction gets boxes on its key frames. And the worker's own spoken reasons are linked to the steps they explain.
left hand
right hand
tool
object of change
- 4 · rationale“If you put a screw close to the corner you need to pre-drill, otherwise it will split the wood.”
- 4 · rationale“I used a 2.5 inch screw, not 3 inch. At this angle a 3 inch would poke through.”
- 4 · plan“We make it 24 inch and cut four pieces. 96 inch total, four out of it. 30 inch will waste too much material.”
- 4 · technique“Always use the same first piece to do the marks.” “Keep your jobsite neat and clean.”
layer 4 · his spoken reasons: transcribed, corrected by hand, tagged goal / plan / rationale / technique
{
"clip": "ThumbPW_0005.MP4", "duration_s": 300.0, "fps": 50,
"keysteps": [
{"start_s": 0, "end_s": 52, "verb": "drive", "noun": "screw", "essential": true},
{"start_s": 105, "end_s": 185, "verb": "cut", "noun": "board", "essential": true}
],
"hoi_interactions": [
{"verb": "drive", "noun": "screw", "tool": "impact_driver", "state_change": "attach",
"critical_frames": { "PRE": {...}, "CONTACT": {...}, "PNR": {...}, "POST": {...} }}
],
"spoken_intent": [
{"t_start": 206.5, "tag": "rationale", "links_keystep": 8,
"text": "Pre-drill first near a corner, otherwise it will split the wood."}
]
} // ships as LeRobot v2.0: episodes, per-frame IMU, language, splits
Why nobody else has this: off-the-shelf hand trackers find zero hands in this footage. We tried MediaPipe. Work gloves and the steep helmet angle break them. So we verify a seed set by hand, train our own gloved-hand detector on it, and have the tradesperson review every label. Hard to copy when you can't even get a first detection.
