Apple's HALO: two layers of a humanoid stack run open-weights models

On November 4, Apple's research team posted “Perceptive Humanoid Motion from Simulation”, unveiling HALO, a humanoid autonomy system; the next morning (November 5, 01:47Z) the companion repo went up (our November 8 API check: single main branch, 3 commits, 186 stars at observation). The paper states the scale and result: six unmapped real homes, 360 single-task trials, 91% average success; the same single tasks hit 91% in simulation across 6,000 trials.

The stack has three layers on their own clocks. The orchestrator ticks about every 2 seconds (~0.5 Hz): the open-weights Qwen3.6-35B-A3B, 4-bit (MLX, ~20 GB), on a Mac mini (M4 Pro, 64 GB) inside the robot's rear enclosure; it reads the language instruction plus the egocentric image and outputs a subtask sequence — crouch, sit, push, follow — from a fixed subtask vocabulary (skills.yaml). The paper notes the orchestrator can optionally route to frontier cloud models via APIs. The generator ticks ~3 Hz: a frozen Qwen3.5-2B encoder with a DiT decoder (flow matching) turns RGB + subtask into a 5-second full-body plan (150 frames, 30 Hz). The tracker ticks 50 Hz: an RL + distillation policy on the G1's stock Jetson Orin NX takes depth + proprioception and outputs 29 joint targets.

Why layers? The paper's reasoning is direct: different problems need different frequencies. Deciding what to do need not be re-thought every frame; a 5-second plan re-planned at ~3 Hz suffices; balance and contact stability need a hard real-time loop at 50 Hz. Each layer consumes only what it needs. Cloud is swappable, the bottom layer is not: the orchestrator can upgrade to stronger cloud models, while the tracker must stay on the onboard chip.

• Read the paper page ↗
• See the demo video and real-task gallery ↗

The data pipeline answers an old question: where does humanoid-robot data come from? The abstract: teleoperation and wearable capture “require at least an hour of human time to produce an hour of data”. HALO synthesizes vision-language-motion data across 1,000 Infinigen-generated home scenes: 300 hours of episodes, all from simulation. Synthesis took 250 H100s for 48 hours; the generator trains on 64 B200s, the tracker on 4 H100s.

“The corpus scales with compute rather than human labor” is the paper's own claim, with “quality is maintained by automated verifiers” in the introduction. The accurate reading: corpus size is no longer bounded by human hours — machine hours replace human hours — but “adding a new physical task still requires a human to write its prompt and author the keyframes” sits in the paper's limitations: scale is still bounded by human design capacity.

Sim-to-real is zero-shot: no real home appears in the training corpus; deployment went straight to six real homes. That bounds what 91% means: it measures “homes never seen in training”, not “all homes”. Six is a small sample; the paper claims no generalization to all households.

• Read the paper's full HTML ↗
• See the interactive sim viewer ↗

In the early hours of November 5 the repo went up (November 8 API check: created 01:47Z, last push 04:58Z, single main branch, 3 commits, 186 stars at observation). Six directories cover the full pipeline: datagen (Infinigen scenes + Kimodo motion + Isaac/MuJoCo rendering), generator, tracker (Isaac Lab three phases: privileged experts → multi-teacher distillation → multi-critic fine-tuning), orchestrator (co-served via halo_serve), system (run_sim.py / run_real.py launchers, halo_comm messaging, halo_serve on-device scheduling), assets (G1 MJCF/URDF/USD).

The boundary of “open” needs spelling out. A TODO comment left unrendered in the README: “No tracker weights ship; the proprio and depth tracker students are trained with tracker/training” — tracker weights do not ship; reproducing requires training them yourself with the repo's training code. The orchestrator model is a stock community 4-bit quantization (mlx-community/Qwen3.6-35B-A3B-4bit, ~20 GB), needing ≥32 GB Apple silicon. The license is not open source in the usual sense: the LICENSE is Apple's own terms (use/modify/redistribute with conditions, GitHub tags Other/NOASSERTION, not an OSI license), and third-party assets keep their original licenses.

This “open code, train-your-own-weights” mode sits alongside release paths we have recorded: Cosmos 3 put post-trained checkpoints on a download page (open weights, run them yourself); K3.1 Pro published open weights just under its promised deadline; HALO is “full recipe open, train the weights yourself”. The three differ in what the public actually gets: Cosmos gives runnable checkpoints, HALO gives the complete recipe from data synthesis to deployment.

• See the repo ↗
• See the orchestrator model on Hugging Face ↗

What 91% covers: the average success rate of 360 single-task trials (sit on chairs, navigate to named objects, sidestep half-open doors, crouch under tables and crawl, push boxes) across six unmapped real homes; the same tasks hit 91% in simulation over 6,000 trials. Composites are tiered: the paper's multi-task eval used 50 sim environments + 1 real home, 10 trials per task per environment, success requires all subtasks. Three combos real/sim: Door→Go-To 100%/71% (real beats sim), Crouch→Sit 60%/73%, Crouch→Go-To→Push 50%/64% — two combos trail in the real world, one beats sim; the direction is not uniform, and single-task 91% must not be projected onto composites.

Boundary one: vision is the only sense. Limitations section: “once an object leaves the camera's field of view or is occluded, the policy has no signal that it is still there, and contact-rich tasks degrade accordingly. Tactile sensing would be a natural solution.” Boundary two: six homes is a small sample; no claim of generalization to all households. Boundary three: the orchestrator is itself simple; the authors call for a more agentic orchestration system, and the narrow inter-layer interface (fixed subtask vocabulary) is a listed limitation.

In this site's picture: our Qiyuan Q1/T1 piece recorded domestic humanoids entering homes; our Genie 4 piece recorded DeepMind's world model wired to a robotics experiment; our weights piece recorded open-weights models as readable and writable. HALO ties the three lines together: a procedurally generated world as data engine (the Genie 4 direction), an open-weights model as the brain (Qwen3.6-35B-A3B ~20 GB local), landing on a G1 in real homes. Apple trained no model of its own and ships no weights — a stock open-weights model in a three-layer stack, a procedural world making data, moving “capability from source to real homes” one notch closer to life; touch, occlusion and a more agentic orchestrator are the next steps the authors name.

• Compare: Qiyuan Q1/T1 (robots at home) →
• Compare: Genie 4 (world model + robotics) →
• Compare: weights you can read and write →