FIG.05 — AUTO-LABEL PIPELINE

Physical AI · Robotics data foundry

You're on Auto-Label

8 models · one auto-label pipeline

From raw rgb.mp4 to a fully-provenanced manifest in ≤48h. Hand kpts, body kpts, object masks, depth, verb-noun, and a full provenance trail — visualized on real captures, not mockups.

Models

Latency

≤ 48h

Schema

v3.0

Provenance

per-field

FIG.04 — FOUNDRY PIPELINE

One pipeline · five phases · every stage diffable

From factory floor to LeRobot v2 in ≤48h. Every phase leaves a machine-readable trace so any downstream claim can be verified.

FIG.08 — AUTO-LABEL MODEL STACK

Eight production models · one pipeline

Every capture flows through eight pinned, versioned, VRAM-profiled models before a human sees it. Each is swappable behind a role interface — the pipeline stays the same regardless of the underlying weights.

Segmenter

Video object + hand masks

FRONTIER

Frame-accurate masks for every object and both hands.

INRGB video · 720p

OUTMasks · per-frame · track-linked

VRAM10 GB / 24

LAT0.3 s/frame·RUNgpu · isolated

Video-native · masks travel with track_id.

3D Hand + Camera

MANO 21-kpt + SLAM

RESEARCH

Reconstructs 3D hand pose plus the camera trajectory.

INRGB video

OUT21-kpt MANO hand · R,t camera pose

VRAM13 GB / 24

LAT1.2× clip length·RUNgpu · isolated

DROID-style SLAM · MANO body pinned.

Monocular Depth

Per-pixel depth + intrinsics

FOUNDATION

Predicts dense depth and camera intrinsics from a single frame.

INRGB frame

OUTDepth map · 3×3 intrinsics K

VRAM5 GB / 24

LAT0.4 s/frame·RUNgpu · isolated

Feeds 6-DoF pose tracker.

Mesh Reconstruction

Object → .glb mesh

FRONTIER

Reconstructs a per-object 3D mesh from monocular input.

INRGB frame · object mask

OUT.glb mesh · watertight

VRAM14 GB / 24

LAT2–4 s / object·RUNgpu · isolated

Multi-checkpoint encoder/decoder chain.

6-DoF Pose Tracker

Per-frame per-object pose

RESEARCH

Tracks object 6-DoF pose using depth + mask + prior mesh.

INDepth · mask · mesh

OUT4×4 pose matrix · per object · per frame

VRAM6 GB / 24

LAT0.4 s / frame / obj·RUNgpu · isolated

Swappable behind a common interface.

Frontier VLA

Verb + noun · 8-frame windows

FRONTIERDETERM

Classifies action verb and object noun on every 8-frame window.

IN8 RGB frames · task prompt

OUTVerb · noun · confidence

VRAM22 GB / 24

LAT2.5 s / window·RUNgpu-node · remote

Deterministic decoding · 86 % verb-noun accuracy on internal eval.

Lightweight Hand

21-kpt fallback

LIGHTWEIGHT

Fast bbox → 21 keypoints when the frontier model drops the hand.

INRGB bbox

OUT21 hand keypoints · confidence

VRAM0 GB / 24

LAT0.02 s / frame·RUNcpu

Patches gaps · runs on CPU.

Lightweight Body

33-kpt body pose

LIGHTWEIGHT

Full-body keypoints for humanoid-transfer downstream tasks.

INRGB frame

OUT33 body keypoints

VRAM0 GB / 24

LAT0.05 s / frame·RUNcpu

Runs on CPU · humanoid retarget input.

Peak sequential VRAM: ~22 GB · verified on NVIDIA data-center GPUs.

FIG.05A/B — DESCRIPTION + METADATA

Every clip has a verb-noun · every field has a version

Beyond kpts and masks, every capture ships with structured semantics — action segments plus a provenance trail that names the exact model and git SHA that produced each field.

FIG.05A — DESCRIPTIONQwen3-VL

Description · verb-noun action segments

A vision-language model watches every clip and emits verb-noun action segments with confidence scores. Each segment maps to a canonical noun ID from a 200-entry industrial ontology.

{
  "start_t": 0.0,
  "end_t": 6.54,
  "verb": "pick",
  "noun": "cup",
  "noun_id": 36,
  "confidence": 0.9,
  "source": "qwen_vl"
}

FIG.05B — METADATAmanifest · versioned schema

Metadata · provenance trail

Every capture ships with a manifest that records not just the labels but the exact model + version + git SHA that produced each field. Any claim we make is diffable.

output · manifest · models { hands, object_seg, object_mesh, object_pose, depth, action }

{
  "schema_version": "3.0_tbrain_ego",
  "clip_id": "pick_up_the_cup__t01",
  "provenance": { "git_sha": "1b0cce1", "captured_at": "…" },
  "models": {
    "hands":      "hand-tracker · v0.7",
    "object_seg": "segmenter · v3.1",
    "object_pose": "6dof-pose · v0.9",
    "depth":      "mono-depth · v1.2",
    "action":     "vlm · v3.8b"
  },
  "action_segments": [ … ],
  "frames": [ … ]
}

FIG.05C — KEYPOINTS · HANDMANO 21-kpt + per-hand SLAM

Hand keypoints

Per-frame 21-keypoint MANO mesh + SLAM camera trajectory for each hand independently. Interpolated frames flagged; low-coverage caps escalated to Label Studio.

output · hands.left/right · kpts_2d · kpts_3d_world · kpts_3d_cam · source

iron_01MANO 21-kpt

sew_01MANO 21-kpt

arrange_01MANO 21-kpt

FIG.05D — KEYPOINTS · BODYExocentric mocap primary · Sapiens 308-kpt secondary

Body pose

High-fidelity body pose comes from partner-signed exocentric mocap sessions. Sapiens 308-kpt runs on every capture and lands in the manifest, but after the pipeline hardening pass (burn v1b0cce1) the dense body layer is OFF by default in the annotated.mp4 — the bystander skeleton no longer leaks. Kpts remain in the manifest for downstream research + retrained gates surface partial-body detections.

output · body_dense (308 × 2 · conf) · exo mocap skeleton (partner) · min_kpts gate · dense default off

Exocentric mocap · body pose primarypartner · signed

Sapiens 308-kpt · ego capturesecondary · gated

Sapiens body on egocentric capture, face + dense gated

DIAGNOSTIC · sapiens_body/kpts.npz · topology gateSUPPRESSED · 2 egocentric caps

cap 1 · textile ego

cap

iron_product · op mobile · 20260626T01

kpts.npz shape

(450, 308, 2) · float32

Y_nose (kpt 0)

477.6 · frame 225

Y_hip mean (11,12)

440.1 · frame 225

Δ (nose - hip)

+37.5 · positive = TOPOLOGY_INVALID

body kpt count

0 / 17 above threshold

gate condition

nose_Y < hip_Y AND body_count ≥ 6

action

suppress face + dense from viz · retain raw kpts in the manifest

cap 2 · tabletop ego

cap

pick_up_the_cup · op unknown · 20260617T01

kpts.npz shape

(273, 308, 2) · float32

Y_nose (kpt 0)

477.6 · frame 136

Y_hip mean (11,12)

447.3 · frame 136

Δ (nose - hip)

+30.3 · TOPOLOGY_INVALID (head-mount top-down)

body kpt count

12 / 17 above threshold

gate condition

nose_Y < hip_Y AND body_count ≥ 6

action

suppress face + dense · body-17 still rendered where coherent

gate rationale · On egocentric captures the wearer's head-mounted camera looks down; Sapiens predicts "nose" below "hips" — impossible in a coherent skeleton. The gate flags TOPOLOGY_INVALID and suppresses face + dense kpts from the visualization. The raw kpts stay in the manifest so downstream research on partial-body detection retains full access. High-fidelity body pose comes from partner-signed exocentric mocap instead.

pick_up_the_cup · body 0/17

iron_product · body 0/17

sew_hem · body 0/17

arrange_fabric · body 0/17

honest note · The watermark surface exposes silent Sapiens failures. The dense body layer is off in the visualization by default (bystander skeleton hidden). Landing viz suppresses ego frames where topology is invalid (nose Y > hip Y). Raw kpts still ride in the manifest with the full provenance trail.

FIG.05E — OBJECT MASKSSAM3 · tracked per episode

Object masks

Text-prompted video segmenter finds and tracks every relevant object across the full episode. Emits per-frame masks + tracklet IDs consumed by 6-DoF pose.

output · objects[].track_id · mask · bbox · pose_6dof

· PASS · 4 tracked objects

pick_up_the_cup · cupOK

pick_up_the_cup · right handOK

sew_hem · fabricOK

arrange_fabric · fabricOK

· HONEST FAILURE · we ship this flag, not silence

iron_product · target: iron · SAM tracked: pantsWRONG_OBJECT

summary.json flag

{
  "check": "sam_target_match",
  "expected": "iron",
  "tracked":  "pants",
  "confidence": 0.62,
  "reason": "prompt disambiguation
   fabric ≈ pants when hand
   overlaps the iron",
  "action": "escalate → LS
   correction task"
}

The segmenter locked onto the shorts instead of the iron. The QC flag surfaces this — no silent overwrite. Downstream retrain uses this diff to sharpen prompt disambiguation.

FIG.05F — DEPTH · POINTMAPMoGe · monocular pointmap

Depth

Metric monocular depth + pointmap for the object camera view. Feeds object 6-DoF pose (world-scale ‖t‖ sanity-checked against 0.1–5m industrial range).

output · depth (H × W) · pointmap (H × W × 3) · intrinsics

pick_up_the_cup · 20260617T01
MoGe pointmap · ‖t‖_mean 1.45m

sew_hem · 20260626T01
MoGe pointmap · z 0.91–3.22m

arrange_fabric · 20260626T01
MoGe pointmap · z 1.20–3.16m

iron_product · 20260626T01
MoGe pointmap · z 1.27–2.71m

output tensor

pointmap.shape:
  (H, W, 3) · metric · f32
intrinsics.txt:
  fx, fy, cx, cy · per-cap

world_scale check:
  ||t||_mean = 1.45 m
  bound       0.1 .. 5 m
  · PASS

feeds → object 6-DoF pose
feeds → SLAM alignment

1.45m

‖t‖ mean

PASS

scale check

K err vs hand-cam

FIG.05G — RERUN VIEWERrerun v0.24

Rerun · every episode is a scrubbable scene

Every finished capture ships with a .rrd file. RGB, depth, hand skeleton, object pose, camera trajectory — all frame-scrubbable in the same viewer our engineers use to debug.

output · .rrd · public web viewer link

Rerun · pick_up_the_cup.rrd · v0.25.1273 frames · 15/15 PASS

44MB · streams RGB · depth · MANO 21-kpt · body 308-kpt ·
object masks + pose · SLAM trajectory · action_segments

FIG.05H — PROVENANCE PALETTE · BURN v1b0cce1

Read every annotation by color · read every failure by watermark

After the pipeline hardening pass, the annotated.mp4 encodes provenance and failure modes in the visual itself. Buyer never has to grep the manifest to know which model produced which pixel.

· per-source palette

HaWoR MANO

src=hawor · Primary hand tracker · SLAM per hand · 21-kpt

interp

src=interp · Frame-level interpolation across a HaWoR gap

SAM3 (mask only)

src=sam3 · Object segmenter mask · hand-tracking suppressed

Sapiens wrist

src=sapiens_wrist · Wrist fallback where HaWoR fails · gated

MoGe depth

src=moge · Metric depth + pointmap · K_hawor priority

human correction

src=human · Label Studio diff · manifest override

· watermarks · failure surface

NO_DET

Hand missing this frame · flagged

CLOSE_HAND

Skin-close-to-hand · HITL priority

MESH_OOB

Mesh projected outside frame · dropped

MESH_DISPUTE

Mesh vs. hand-track disagree · gate

MASK_DRIFT

Mask/bbox ratio > 1.5× · skipped from ship

BODY_HIDDEN

Dense body off default · off-frame body suppressed

burn contract · Every colored dot / mesh in labels/annotated.mp4 maps 1:1 to a source in the manifest. Every watermark maps to a flag. LeRobot exports strip debug frames; HITL queue reads CLOSE_HAND priority. All 20 caps rebuilt + reburned at git 1b0cce1 · 152/152 tests PASS.

FIG.12 — LeROBOT v2 EXPORT

Shipped in the format frontier labs already train on

No proprietary schema. No conversion contract. Every batch exports directly to LeRobot v2 parquet + video, drops into your pipeline the day you sign, and mirrors to RLDS on request.

Episodes

chunk-000

Frames

annotated

Tasks

verb-noun

Videos

RGB + depth

+8 auto-labeled·textile · iron + sew + sort + package·6 solid · 2 intermittent640×480 · h264 · yuv420p · 15 fps

meta/info.json · chunk-000LeRobot v2.0

{
  "dataset_name": "tbrain_ego_v2",
  "robot_type":   "egocentric_human",
  "total_episodes": 8,
  "total_frames":   5016,
  "total_tasks":    6,
  "total_videos":   16,
  "fps": 15.0,
  "features": {
    "observation.images.rgb":   { "dtype": "video", "shape": [3, 480, 640] },
    "observation.images.depth": { "dtype": "video", "shape": [3, 480, 640], "is_depth_map": true },
    "action_type": { "dtype": "string" }
  }
}

Streams

· observation.images.rgb
· observation.images.depth
· action_type

Ship in RLDS, LeRobot, or your schema

Every capture ships with the full manifest — no proprietary format, no conversion contract.

See QC playbook Ask for a sample

Physical AI overview

Landing · foundry story · 20 sections

QC playbook

15 hard rules · 3 human review layers