FIG.06 — QC PLAYBOOK

Physical AI · Robotics data foundry

You're on QC

Zero-trust QC · hard rules + AI filter + 3-layer human review

Every capture crosses 15 machine-checkable hard rules, an AI filter, and up to three human review layers before it ships. Every fix keeps the provenance trail intact, and every episode leaves a Rerun scene the buyer can open.

Hard rules

Human layers

Ship rate

92%

Provenance

per-field

FIG.04 — FOUNDRY PIPELINE

One pipeline · five phases · every stage diffable

From factory floor to LeRobot v2 in ≤48h. Every phase leaves a machine-readable trace so any downstream claim can be verified.

FIG.06A — LAYER 1 · 15 HARD RULES

15 machine-checkable rules · every capture · zero human cost

Every rule maps to a real failure mode we've caught in the field. If any fires, the capture routes into human review with the exact reason attached.

· calibration2 checks

Camera intrinsics + trajectory sanity — every capture's K matches what the SLAM pipeline expects, no per-clip drift.

01
Camera intrinsics agreement
Hand-camera and object-camera intrinsics must agree within tolerance; large drift means SLAM diverged on one hand.
gate · fx err < 15%→fx_hand=594.3 · fx_object=642.0 · err=8.0%
02
Camera trajectory sanity
SLAM rotation matrix deviation from identity — bounds catch SLAM divergence (which cascades into 3D kpts).
gate · ‖R‖ dev < bound→max_dev=2.83

· temporal2 checks

Frame timing + object continuity across the episode — flags timing drift, tracklet ID thrashing.

01
Frame count alignment
Auto-label frame count must match raw video frame count. Mismatch means a stage silently dropped frames.
gate · npz == video→npz=271 · video=271
02
Object tracklet continuity
Fraction of object-detected frames with a stable track_id. Below floor, segmenter is thrashing between labels.
gate · > 70% assigned→266/266 · 100%

· detection4 checks

Detector coverage across hands, body, filter pass, dense-body — flags silent model failures before they ship.

01
Hand detection rate
Below floor, hand tracker collapsed and interpolation would drift — capture escalated to Label Studio for human kpt.
gate · > 10% per hand→L=85% · R=68%
02
Filter pass rate
Post-filter labels count vs. raw detections. Too low means over-aggressive smoothing threw away real signal.
gate · > 40% (no over-filter)→labels=496 · det=414 · rate=120%
03
Body pose detection rate
Fraction of frames with a full body pose. Feeds retargeting; below floor the operator is off-camera too long.
gate · > 40%→191/271 · 70.5%
04
Body dense-kpt rate
Sapiens 308-kpt coverage. Includes mean confidence so a low-conf pass gets flagged before ship.
gate · > 60% · conf→271/271 · 100% · conf=0.57

· spatial3 checks

Keypoint outlier %, 3D dual-frame agreement, world-scale ‖t‖ in industrial workspace bounds.

01
Keypoint outlier percentage
Frames whose keypoints jump outside the physically plausible envelope get flagged and dropped from downstream.
gate · < 5%→1/271 = 0.4%
02
3D keypoints · dual frame
Every hand kpt must exist in both world and camera frames — enables both fixed-scene and moving-camera policies.
gate · world && cam present→world=true · cam=true
03
Object world-scale sanity
Object position in world coordinates must land within industrial workspace bounds; catches metric-depth failure.
gate · 0.1 ≤ ‖t‖ ≤ 5m→‖t‖_mean = 1.45m

· semantic3 checks

Ontology mapping, action-segment count, grasp density — the meaning of the frames, not the pixels.

01
Object class mapping rate
Fraction of tracked objects that map to the canonical ontology. Below floor, ontology needs an extension entry.
gate · > 70%→266/266 = 100%
02
Action segment count
At least one verb-noun segment per episode. Zero means VLM refused or the clip was truly empty.
gate · ≥ 1→5 segs · source=vlm
03
Grasp event density
Derived from hand + object overlap. Low density on a manipulation clip means one of hand-track or object-track failed.
gate · > 0.05 / frame→110 events / 271 frames · 40.6

· provenance1 checks

Manifest completeness — every field maps to model + version + git SHA. Non-negotiable.

01
Schema + provenance trail
Every field must record the model + version + git SHA that produced it. Non-negotiable — anchors the diffability contract.
gate · model + version + git_sha→provenance · git_sha=1b0cce1

FIG.06B — LAYER 2 · AI FILTER

AI filter · what the hard rules can't reason about

Where the hard-rules gate is thresholds, the AI filter is judgment: confidence-weighted smoothing, cross-modal consistency (does the mask agree with the depth?), and out-of-distribution detection.

AI FILTER · JUDGMENT

Confidence smoothing

Per-frame kpt confidence dips are smoothed if temporally isolated. Sustained dips escalate to Label Studio.

conf < 0.55 for 4+ frames → escalate

Example · wrist kpt dips 0.42 for 2 frames (occluded by cup), rebounds to 0.87 — smoothed. Would have flagged 40% of ~/. usable captures under a naive threshold.

Sapiens 308-kpt · temporal window: 8 frames

AI FILTER · JUDGMENT

Cross-modal consistency

The segmenter mask and the depth pointmap must agree on the object envelope within a tolerance — catches SAM-vs-object mismatch (iron_T02 flag).

mask ⊕ depth IoU < 0.68 → flag

Example · iron_T02: SAM3 mask included the towel behind the iron. Depth pointmap disagreed (Δ = 14cm on the mask boundary). Flagged, reviewer tightened the mask.

SAM3 mask ⊕ MoGe depth · IoU gate

AI FILTER · JUDGMENT

Out-of-distribution

Novel objects flagged with an ontology_missing tag. Not a reject — a signal to add an ontology entry before ship.

verb-noun ∉ ontology → tag ontology_missing

Example · capture used a fabric-clamp our ontology didn't cover. Not rejected — flagged, ontology PR opened, all 3 captures re-tagged before ship.

VLM verb-noun · 240-verb ontology

FIG.06C — LAYER 3 · LABEL STUDIO

Label Studio · humans on the last mile, not the first

Only PARTIAL/FAIL captures reach Label Studio, pre-populated with auto-label output. Annotators correct kpt drift, adjust masks, override verb-noun — never annotate from a blank slate. Every correction lands as a labeled diff back into the training loop.

FIG.06B · HUMAN-IN-THE-LOOP WORKFLOW

Human box → AI predict → Human finetune → Ship

Humans anchor the task and finetune the edges. The pipeline propagates predictions across every frame in between. Under 10% of frames touch a human directly.

Stage 01

Human framing box

Reviewer draws initial bounding box + verb-noun on 1 keyframe per capture.

in queue

Stage 02

AI predicts

Auto-label pipeline pre-fills 21-kpt hand + 308-kpt body + object mask across all 273 frames.

0 kpts/min

vs 12 blank

Stage 03

Human finetune

Annotator corrects drift, adjusts masks, overrides verb-noun. Every diff writes to the manifest.

0 min

avg per task

Stage 04

Ship

Reviewer signs off. Hard rules re-run. LeRobot v2 parquet + Rerun scene shipped.

first-pass · 6% escalation

FIG.06C · LABEL SPEED · WHY ≤48h HOLDS

Pre-populated annotators are 5.7× faster than from-blank

Every capture arrives in Label Studio with the pipeline output already drawn on the frame. Annotators correct — they don't create. That's how a 273-frame capture ships in ≤48h.

· annotator throughput · kpts / minute

Traditional · from blank0

Auto-label pre-populated0

Sample of 42 textile captures · Q2 2026 · reviewer sign-off log.

· ≤48h delivery · shipped capture breakdown

Capture8h

Operator wears the rig · offline record + sync

Auto-label6h

8 models in parallel · hard rules gate

HITL fix8h

Label Studio · <10% frames touched

QC + sign-off4h

Reviewer + escalation dashboard

Buffer22h

Reshoot / escalation slack · rarely used

project: physical_ai/textile_v2 task 1247 / 1892auto-label · pre-populated

Queue · 5 open

#1247
pick_up_the_cup · 20260617T01
manual sample
#1248
iron_product · 20260626T01
kpt_outlier · L_wrist
#1249
sew_hem · 20260626T02
hand_detect_rate · L=8%
#1250
arrange_fabric · 20260626T01
class_mapping
#1251
package_product · 20260626T02
action_seg

reviewer · nguyen.t

frame 136 / 273 · 15 fps

auto-label burn · MANO 21-kpt overlay

avg 2.3 min · this task

00:09.06 / 00:18.20

Correction form

Segment 1 · verb / noun

pickcupconf 0.90

Hand kpts · pre-populated

· wristdrift 2px
· middle tipdrift 3px
· index pipdrift 1px
· thumb tipdrift 2px

Object mask · SAM3

cup · track_id 3

1 accept2 rejectf flagdemo · not a live annotator

Live annotator · video review · bbox tracks + action verbs + metadatareal capture · 450 frames · MANO + SAM3 overlay

Video review · pre-populated

Reviewers land on a filled canvas, not an empty task

Auto-label output (MANO 21-kpt · Sapiens 308-kpt · SAM3 mask · MoGe depth · verb-noun) is pushed as pre-annotations. Reviewer corrects, never draws from scratch. Every correction lands as a labeled diff back into the training loop.

450

frames / task

21+

MANO kpts / hand

action verbs

BATCH · doasido / v2184 / 500 · 37%

168 pass

13 flag

3 reject

1accept2rejectFflagSPACEplay←/→step 1f

PRE-ANNOTATION MODEL STACK

bbox · labels · action_type · narration_edit · quality · skill_confirm · rgb_path · imu_path · audio_path

FIG.06D — LAYER 4 · REVIEWER SIGN-OFF

Reviewer sign-off + escalation dashboard

A second annotator reviews every correction. Accept, reject, flag. Systemic failures (segmenter locked wrong object, SLAM divergence across multiple caps) escalate to engineering and feed the auto-label training loop.

Reviewer dashboard · task queue · sign-off + escalate

WORKFLOW · 5 STEPS

01
Auto-label produces the manifest + rerun.rrd
Eight models run on the raw capture
02
15 hard rules run · reject or PASS/PARTIAL
Auto-reject removes obvious junk
03
PARTIAL/FAIL_LABEL routed to Label Studio
Only what needs a human touches a human
04
Reviewer signs off · updates the manifest
Every fix keeps the provenance trail
05
Re-run hard rules · PASS ships
Zero-trust · a passing rule set is ship-ready

FIG.06E — PROVENANCE TRAIL

Every field records the model that produced it

The manifest is not decorative — downstream diffs, model rollouts, and root-cause reports all read this trail. Any claim we make is diffable.

StageModelVersionGit SHARole

collectcapture-firmwarev0.7.2capsys@a1b2c3dHardware-clock sync · offline cache

handshand-trackerv0.7ego@1b0cce121-kpt MANO + SLAM per hand

bodysapiens-densev1.4ego@1b0cce1308-kpt whole-body regression

object_segvideo-segmenterv3.1ego@1b0cce1Text-prompted mask + tracklet

object_pose6dof-posev0.9ego@1b0cce1Object 6-DoF trajectory

depthmono-depthv1.2ego@1b0cce1Metric depth + pointmap

actionvlm-verbnounv3.8bego@1b0cce1Segment · verb-noun · confidence

exportlerobot-exporterv2.0ego@1b0cce1Parquet + video + RLDS bridge

FIG.06F — RERUN PROOF

Every ship-ready capture ships with its Rerun scene

No screenshots, no cherry-picked metrics — just the raw multi-track scene. Open it in the same viewer our engineers use. Any regression, any claim, any anomaly is scrubbable by the buyer, not just by us.

Rerun · pick_up_the_cup.rrd · v0.25.1273 frames · 15/15 PASS