The August evaluation
π0.5 at 12k steps, with its vision–language backbone frozen, ran 79 times over 15 commands on the Jetson Thor rig, 22–24 August 2026. A person scored every rollout.
Each command was run 5 or 7 times. In 13 commands the vial starts in a slot the demonstrations used; in 2 it starts in slot 2 or 5, which no demonstration used. The scorer marked each rollout as a complete success or not, and noted a missed pickup, a missed drop and any collision.
Where the takes are lost
In the order a take happens. The first grasp is the largest loss; getting past it is not enough.
- Rollouts79
- Grasped the vial on the first try36 of 79 (46%)
- Then completed the move16 of 36 (44%)
Another 4 rollouts completed after missing the first grasp, so 20 of 79 (25%) in all. The vial was let go away from its destination in 15 rollouts (19%). A collision happened in 10, never on its own.
Every command
Click a command in either chart to watch its trials below.
Completed rollouts per command
The dot is the share of rollouts that completed the move; the line is its 95% Wilson interval. With 5 rollouts a command, the intervals are wide.
- Source slot used in training
- Source slot held out of training
Show the numbers
| Command | Source slot in training | Rollouts | Completed | Rate | 95% interval |
|---|---|---|---|---|---|
| L6 → R6 | yes | 5 | 4 | 80% | 38%–96% |
| R1 → L3 | yes | 5 | 3 | 60% | 23%–88% |
| R3 → L6 | yes | 5 | 3 | 60% | 23%–88% |
| L5 → R3 | no | 5 | 2 | 40% | 12%–77% |
| R2 → bin | no | 5 | 2 | 40% | 12%–77% |
| L3 → R1 | yes | 5 | 1 | 20% | 4%–62% |
| L4 → R1 | yes | 5 | 1 | 20% | 4%–62% |
| L4 → R3 | yes | 5 | 1 | 20% | 4%–62% |
| L3 → L6 | yes | 5 | 1 | 20% | 4%–62% |
| R3 → L3 | yes | 5 | 1 | 20% | 4%–62% |
| L1 → R1 | yes | 7 | 1 | 14% | 3%–51% |
| L1 → R3 | yes | 7 | 0 | 0% | 0%–35% |
| L3 → R3 | yes | 5 | 0 | 0% | 0%–43% |
| L4 → bin | yes | 5 | 0 | 0% | 0%–43% |
| R6 → L6 | yes | 5 | 0 | 0% | 0%–43% |
Pooled, rollouts with a held-out source slot completed more often (4 of 10) than those with a trained one (16 of 69), but the intervals overlap and the held-out rate rests on two commands. It is not evidence that the policy generalises; those two slots may simply be easier to grasp.
How far each command’s rollouts got
Each rollout counted once: complete, otherwise a missed pickup, otherwise a missed drop. The rest picked the wrong vial or used the wrong slot.
- Missed the pickup
- Wrong vial or slot
- Missed the drop
- Complete
Show the numbers
| Command | Missed the pickup | Wrong vial or slot | Missed the drop | Complete | Rollouts |
|---|---|---|---|---|---|
| L6 → R6 | 1 | 0 | 0 | 4 | 5 |
| R1 → L3 | 1 | 0 | 1 | 3 | 5 |
| R3 → L6 | 1 | 0 | 1 | 3 | 5 |
| L5 → R3 | 2 | 0 | 1 | 2 | 5 |
| R2 → bin | 1 | 0 | 2 | 2 | 5 |
| L3 → R1 | 3 | 0 | 1 | 1 | 5 |
| L4 → R1 | 3 | 0 | 1 | 1 | 5 |
| L4 → R3 | 3 | 1 | 0 | 1 | 5 |
| L3 → L6 | 3 | 1 | 0 | 1 | 5 |
| R3 → L3 | 0 | 3 | 1 | 1 | 5 |
| L1 → R1 | 5 | 1 | 0 | 1 | 7 |
| L1 → R3 | 3 | 0 | 4 | 0 | 7 |
| L3 → R3 | 4 | 0 | 1 | 0 | 5 |
| L4 → bin | 4 | 0 | 1 | 0 | 5 |
| R6 → L6 | 5 | 0 | 0 | 0 | 5 |
Ordered left to right by how far the take got, in one blue ramp that stands out more the further the take got.
Watch the trials
Side, top and wrist cameras, as the run was recorded for scoring: about 3.5 frames a second, so motion looks jumpy. Full-rate demonstrations are in the August datasets.
One rollout of each outcome
The side camera of four scored trials.
Trained on, and done well, slot by slot
The 150 demonstrations against the evaluation, on the rack layout: the bin, the left rack and the right rack, slots 1 to 6 from left to right as the side camera sees them.
- Demonstrations:
- under 15
- 15–19
- 20–24
- 25–29
- 30 or more
- Completed:
- under 20%
- 20–39%
- 40–59%
- 60–79%
- 80% or more
With 5 rollouts a command, success does not follow how many demonstrations a command had. R1 → L3 had no demonstration of that exact move and completed 3 of 5; L4 → bin had 3 and completed none. Slots 2 and 5 were never a source in training, and slots 2, 4 and 5 never a destination.
What these numbers cannot say
Five rollouts a command is enough to see the failure modes, not to rank commands: the intervals above are 35 to 65 percentage points wide. The scores come from one person watching each run. The v5 evaluation tests one checkpoint of one recipe on one rig; the October results show what the later runs did.