Experiments

Everything about how the models were tested comes before the results: how a rollout is run, how it is scored, what each model is, and what each word means.

How a rollout is run

A rollout is one autonomous attempt at one command. The robot is the follower arm of the SO-101 pair on the workstation, with the five cameras of the Data page, and nobody drives it.

The arm is moved to the pose every demonstration started from, so a crash in the previous session cannot shift “home”.
The command is the sentence the model trained on. From 21 October a short code (“R3 L1”) expands to the exact training wording; before that it was typed by hand.
A person starts the rollout. The model reads the camera images and the joint angles, aims for a control rate of 30 per second, and sends joint targets to the arm. A person stops it, or it ends when a camera drops off the bus.
The model replans every few dozen actions. In sync mode the arm waits for each replan, which limits π0.5 to about 24 to 26 actions a second; with RTC it does not. FLUX has no RTC and ran at 15.5 a second in the one session whose rate was logged.
Each session is stored as a LeRobot dataset with the cameras the model saw, one episode per rollout, and uploaded to Hugging Face.

These were test runs made during development, not a planned evaluation: 41 sessions, uneven in the moves tried, with no fixed number of rollouts per move.

How a rollout is scored

The scorer reads the arm’s joint angles. It finds each moment the gripper closed on something and each moment it opened, and decides which slot the arm was at by matching its pose to the same moment of the 100 demonstrations. It cannot see the vial. Each rollout is put in the furthest of four outcomes it reached:

Missed the vialThe gripper never closed on the vial in the asked source slot: it closed on another slot, on an empty hole, or not at all.
Grasped, not liftedThe gripper closed on the right vial but the arm never lifted it.
Lifted, let go elsewhere or not at allThe arm lifted the right vial, then let go at a different slot or never let go before the rollout ended.
Let go at the asked slotThe arm let go at the slot the command asked for (within half a slot). It does not prove the vial ended in the hole: the scorer reads the arm, not the vial.

174 rollouts are scored. Left out: 11 that ended within 3 seconds, 32 that asked for moves no demonstration contains (this includes the held-out slots), 2 whose command the scorer cannot read, and 11 from an earlier layout of the table.

Because the scorer reads the arm and not the vial, a person also watched 119 of the rollouts that let go at the asked slot: the vial was in the slot in 43, not in it in 73, and could not be told in 3. This is shown beside the scores and is never added to them.

The models

11 models. Most trained on the 100 demonstrations of the Data page, or on a copy of them; Sweet Corn trained on 25 earlier ones. The name is what the rest of the site uses.

Name	Model	What it is	Started from	Cameras it saw	Steps	Scored rollouts	Weights
Plain Bagel	π0.5	π0.5 trained from the base on the 220 demonstrations, where the destination cannot be read off the cameras	lerobot/pi05_base	All five	16,000	28	Hugging Face
Mango Pudding	π0.5	Mango Rice trained on for 16,000 steps on the 220 demonstrations	Mango Rice at 12k steps	All five	16,000	28	Hugging Face
Mango Lassi	π0.5	Mango Rice trained on for 4,000 steps on the 220 demonstrations	Mango Rice at 12k steps	All five	4,000	33	not published
Mango Sorbet	π0.5	Mango Lassi with the gripper and wrist ranges set back to Mango Rice's	Mango Lassi (the same weights)	All five	4,000	10	not published
Mango Chutney	π0.5	Mango Lassi with only the gripper range set back	Mango Lassi (the same weights)	All five	4,000	5	not published
Spicy Chicken	π0.5	π0.5 trained on the recordings as they were, all five cameras	lerobot/pi05_base	All five	12,000	21	Hugging Face
Mango Rice	π0.5	π0.5 trained from the base on a cleaned copy with one wording per move	lerobot/pi05_base	All five	12,000	9	Hugging Face
Pretzel One	π0.5	A 361-step fine-tune of Spicy Chicken to test the operator-free starts	Spicy Chicken at 12,000 steps	All five	361	3	not published
Pretzel Two	π0.5	A 500-step fine-tune of Spicy Chicken with the corrected trim	Spicy Chicken at 12,000 steps	All five	500	17	not published
Crispy Falafel	FLUX 3 Action	FLUX 3 Action, LoRA fine-tune, overhead and wrist cameras	black-forest-labs/flux-3-action-so101, with this dataset's own joint ranges	Overhead + On the gripper	30,000	20	Hugging Face
Sweet Corn	FLUX 3 Action	FLUX 3 Action on a single task, the first 25 demonstrations	black-forest-labs/flux-3-action-so101, with this dataset's own joint ranges	left (re-aimed high) and wrist, from the earlier camera layout	10,000	not scored	not published

Pretzel One and Pretzel Two were short fine-tunes of Spicy Chicken made to test one idea each, and their checkpoints are deleted. Sweet Corn is the first FLUX run, on one task; its rollouts are not scored.

One model in full

Pick a model. Its settings, its checkpoints, every move it was asked and what happened to it are on this page and nowhere else.

Plain Bagel

π0.5 trained from the base on the 220 demonstrations, where the destination cannot be read off the cameras.

Started from: lerobot/pi05_base
Trained on: vial-sort-slots-220, idle start and end cut, operator removed from the start frames, one sentence per move
Cameras it saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen
Steps: 16,000
Passes over the data: 2.27
Training time: 21.5 h
Peak GPU memory: 37.8 GB
Weights: on Hugging Face

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: 9 asked for moves no demonstration contains.

Beside the table, 16k · sync: 9 rollouts on moves no demonstration it trained on contains (L1 → L6, L2 → R6, L3 → R4, L5 → bin, R2 → bin, R3 → L5, R5 → L1). The right vial was grasped in 7, the arm let go at the asked slot in 5, and the vial was seen in that slot in 4.

Checkpoint	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
16k · sync	28	2	1	6	19

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

16k · sync: 28 rollouts, 23 moves

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
L3 → R3	4	2	0	0	1	1
L3 → R6	3	2	0	0	1	1
L6 → R1	3	2	0	1	0	1
L6 → R3	3	2	1	0	1	0
R1 → L3	3	2	1	0	0	1
L1 → R1	2	1	0	0	0	1
L1 → R3	4	1	0	0	1	0
L1 → R6	4	1	0	0	0	1
L3 → R1	3	1	0	0	0	1
L3 → bin	3	1	0	0	0	1
L6 → L1	0	1	0	0	0	1
L6 → R6	4	1	0	0	0	1
R1 → L1	3	1	0	0	0	1
R1 → L6	2	1	0	0	0	1
R1 → R6	0	1	0	0	0	1
R3 → L1	4	1	0	0	1	0
R3 → L3	2	1	0	0	0	1
R3 → L6	3	1	0	0	0	1
R3 → bin	4	1	0	0	0	1
R6 → L1	4	1	0	0	0	1
R6 → L3	3	1	0	0	0	1
R6 → L6	2	1	0	0	0	1
R6 → R1	0	1	0	0	1	0

Hyperparameters

Read from the train_config.json saved with the checkpoint.

What trains

Starting weights	lerobot/pi05_base	The checkpoint training began from.
Parameters trained	All of them	Vision encoder, language model and action expert all receive gradients: a full fine-tune, no LoRA.
Precision	bfloat16, gradient checkpointing	Number format of the weights and activations; checkpointing trades speed for memory.

Optimiser

Optimiser	AdamW	The update rule.
Peak learning rate	2.5e-05	The step size at its largest.
Betas	0.9, 0.95	How strongly AdamW remembers past gradients.
Weight decay	0.01	A pull of the weights toward zero.
Gradient clipping	1	Largest allowed size of one update's gradient.

Learning-rate schedule

Shape	Linear warm-up, then cosine decay	The learning rate rises, then falls smoothly.
Warm-up steps	533 (set to 1,000, scaled to fit the run)	Steps over which the rate rises to its peak.
Decay steps	16,000 (set to 30,000, scaled to fit the run)	Steps over which it falls to the final rate.
Final learning rate	2.5e-06	The rate at the end of the decay.

Batches and length

Batch size	16	Frames per optimiser step.
Steps	16,000	Optimiser steps. Checkpoints were saved every 1,000.
Passes over the data	2.27	Steps × batch size ÷ frames in the training data.
Validation split	none	Every demonstration is used for training, so the loss cannot show when the model starts to memorise.

What it sees and predicts

Cameras	top_down, left_top, left_bottom, right, wrist	The views fed to the model at every step.
Image size	224 × 224	Each camera image is resized to this many pixels.
Prompt length	200 tokens	Longest command the text tokenizer keeps.
Action chunk	50 actions	How many future joint targets one forward pass predicts (1.7 s at 30 Hz).
Denoising steps	10	Steps the action expert takes to turn noise into a chunk when running.
Normalisation	state and action by quantiles; images unchanged	Joint values are rescaled with the 1st and 99th percentile of this data set, so its own joint ranges are used.

Image augmentation

Augmentation	colour jitter (brightness ±20%, contrast ±20%, saturation ±50%, hue ±0.05), sharpness, and a small shift and rotation (≤5% and ±5°); up to 3 applied to each image	Random changes to each training image.

Reproducibility

Seed	1,000	Fixes the random choices.

What happened

How it was trained

7 November, on one A100 80 GB of a university GPU cluster, in two 12-hour jobs: the first stopped itself after the 8,000-step checkpoint and the second resumed from it. 16,000 steps at batch 16 (2.3 passes over the data), learning rate 2.5e-5, 4.8 s per step, 37.8 GiB of GPU memory. Loss fell from 0.099 to 0.009.

On the robot, 9 November

The same 31 moves as Mango Pudding, one after the other: 21 vials ended in the asked slot, against 11. It reached slot 6 every time it carried the vial there, and also slots 4 and 5, which no demonstration used as a destination. One move failed at the grasp: L1 → L6, the one move that is not in the 220 demonstrations. What is left is the drop: six times it reached the right slot and the vial ended in front of or behind the hole.

Tested without the robot

Every checkpoint from 8,000 steps on went through the offline gate. They score the same within a few points, so the last one was kept. Its predicted destination for slot 1, 3 and 6 is further apart than Mango Pudding's on the same frames.

Rollouts to watch

R3 → L6

Plain Bagel at 16k steps, R3 → L6, 9 November. Trained on the 220 demonstrations: carries the vial across the table to the far end of the left rack and lets go in L6.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

L3 → R4

Plain Bagel at 16k steps, L3 → R4. No demonstration ever put a vial in slot 4, and the vial ends in R4.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

Mango Pudding

Mango Rice trained on for 16,000 steps on the 220 demonstrations.

Started from: Mango Rice at 12k steps
Trained on: vial-sort-slots-220, idle start and end cut, operator removed from the start frames, one sentence per move
Cameras it saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen
Steps: 16,000
Passes over the data: 2.27
Training time: 21.5 h
Peak GPU memory: 37.8 GB
Weights: on Hugging Face

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: 9 asked for moves no demonstration contains.

Beside the table, 16k · sync: 9 rollouts on moves no demonstration it trained on contains (L1 → L6, L2 → R6, L3 → R4, L5 → bin, R2 → bin, R3 → L5, R5 → L1). The right vial was grasped in 6, the arm let go at the asked slot in 4, and the vial was seen in that slot in 3.

Checkpoint	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
16k · sync	28	8	4	8	8

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

16k · sync: 28 rollouts, 23 moves

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
L3 → R6	3	3	1	0	2	0
L6 → L1	0	2	1	1	0	0
R1 → L1	3	2	0	0	1	1
R6 → L1	4	2	1	1	0	0
L1 → R1	2	1	0	0	0	1
L1 → R3	4	1	0	0	1	0
L1 → R6	4	1	0	0	0	1
L3 → R1	3	1	0	0	1	0
L3 → R3	4	1	0	0	1	0
L3 → bin	3	1	0	0	0	1
L6 → R1	3	1	0	1	0	0
L6 → R3	3	1	1	0	0	0
L6 → R6	4	1	0	0	1	0
R1 → L3	3	1	1	0	0	0
R1 → L6	2	1	0	0	0	1
R1 → R6	0	1	1	0	0	0
R3 → L1	4	1	1	0	0	0
R3 → L3	2	1	0	0	1	0
R3 → L6	3	1	1	0	0	0
R3 → bin	4	1	0	0	0	1
R6 → L3	3	1	0	0	0	1
R6 → L6	2	1	0	1	0	0
R6 → R1	0	1	0	0	0	1

Hyperparameters

Read from the train_config.json saved with the checkpoint.

What trains

Starting weights	Mango Rice at 12k steps	The checkpoint training began from.
Parameters trained	All of them	Vision encoder, language model and action expert all receive gradients: a full fine-tune, no LoRA.
Precision	bfloat16, gradient checkpointing	Number format of the weights and activations; checkpointing trades speed for memory.

Optimiser

Optimiser	AdamW	The update rule.
Peak learning rate	1e-05	The step size at its largest.
Betas	0.9, 0.95	How strongly AdamW remembers past gradients.
Weight decay	0.01	A pull of the weights toward zero.
Gradient clipping	1	Largest allowed size of one update's gradient.

Learning-rate schedule

Shape	Linear warm-up, then cosine decay	The learning rate rises, then falls smoothly.
Warm-up steps	533 (set to 1,000, scaled to fit the run)	Steps over which the rate rises to its peak.
Decay steps	16,000 (set to 30,000, scaled to fit the run)	Steps over which it falls to the final rate.
Final learning rate	2.5e-06	The rate at the end of the decay.

Batches and length

Batch size	16	Frames per optimiser step.
Steps	16,000	Optimiser steps. Checkpoints were saved every 1,000.
Passes over the data	2.27	Steps × batch size ÷ frames in the training data.
Validation split	none	Every demonstration is used for training, so the loss cannot show when the model starts to memorise.

What it sees and predicts

Cameras	top_down, left_top, left_bottom, right, wrist	The views fed to the model at every step.
Image size	224 × 224	Each camera image is resized to this many pixels.
Prompt length	200 tokens	Longest command the text tokenizer keeps.
Action chunk	50 actions	How many future joint targets one forward pass predicts (1.7 s at 30 Hz).
Denoising steps	10	Steps the action expert takes to turn noise into a chunk when running.
Normalisation	state and action by quantiles; images unchanged	Joint values are rescaled with the 1st and 99th percentile of this data set, so its own joint ranges are used.

Image augmentation

Augmentation	colour jitter (brightness ±20%, contrast ±20%, saturation ±50%, hue ±0.05), sharpness, and a small shift and rotation (≤5% and ±5°); up to 3 applied to each image	Random changes to each training image.

Reproducibility

Seed	1,000	Fixes the random choices.

What happened

How it was trained

6 to 7 November, on one A100 80 GB of a university GPU cluster, in two 12-hour jobs. 16,000 steps at batch 16, learning rate 1e-5. The joint ranges used to scale the data were set back to Mango Rice's own, the fix found with Mango Lassi. Loss fell from 0.085 to 0.009.

On the robot, 8 and 9 November

31 moves: 11 vials ended in the asked slot. The destination now follows the command: with the vial always at R3, asking for slot 1, 3 and 6 sent it to slot 1, about 3 and 5. But the far slots were pulled toward the middle, and seven moves failed at the grasp, most of them from slot 6.

Rollouts to watch

R3 → L6

Mango Pudding at 16k steps, R3 → L6, 8 November: the same move as the first clip, from the model trained on from Mango Rice. It stops one slot short, in L5.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

Mango Lassi

Mango Rice trained on for 4,000 steps on the 220 demonstrations.

Started from: Mango Rice at 12k steps
Trained on: vial-sort-slots-220, idle start and end cut, operator removed from the start frames, one sentence per move
Cameras it saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen
Steps: 4,000
Passes over the data: 0.57
Training time: 4.1 h
Peak GPU memory: 37.9 GB

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: 3 ended within 3 seconds, 4 asked for moves no demonstration contains.

Beside the table, 4k · sync: 4 rollouts on moves no demonstration it trained on contains (L1 → L6, L3 → R4, R4 → L5). The right vial was grasped in 4, the arm let go at the asked slot in 0, and the vial was seen in that slot in 0.

Checkpoint	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
4k · sync	33	9	5	8	11

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

4k · sync: 33 rollouts, 23 moves

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
R3 → L1	4	3	0	1	2	0
R3 → L6	3	3	0	0	1	2
L1 → R1	2	2	1	0	0	1
L3 → R1	3	2	0	1	0	1
L3 → R6	3	2	2	0	0	0
L6 → R3	3	2	1	0	1	0
R1 → L6	2	2	0	1	0	1
R6 → L6	2	2	1	1	0	0
L1 → R3	4	1	0	0	0	1
L1 → R6	4	1	0	1	0	0
L3 → R3	4	1	0	0	1	0
L3 → bin	3	1	1	0	0	0
L6 → L1	0	1	1	0	0	0
L6 → bin	3	1	0	0	0	1
R1 → L1	3	1	0	0	1	0
R1 → L3	3	1	0	0	0	1
R1 → R6	0	1	0	0	0	1
R1 → bin	3	1	1	0	0	0
R3 → L3	2	1	0	0	1	0
R3 → bin	4	1	0	0	0	1
R6 → L1	4	1	0	0	0	1
R6 → L3	3	1	0	0	1	0
R6 → R1	0	1	1	0	0	0

Hyperparameters

Read from the train_config.json saved with the checkpoint.

What trains

Starting weights	Mango Rice at 12k steps	The checkpoint training began from.
Parameters trained	All of them	Vision encoder, language model and action expert all receive gradients: a full fine-tune, no LoRA.
Precision	bfloat16, gradient checkpointing	Number format of the weights and activations; checkpointing trades speed for memory.

Optimiser

Optimiser	AdamW	The update rule.
Peak learning rate	1e-05	The step size at its largest.
Betas	0.9, 0.95	How strongly AdamW remembers past gradients.
Weight decay	0.01	A pull of the weights toward zero.
Gradient clipping	1	Largest allowed size of one update's gradient.

Learning-rate schedule

Shape	Linear warm-up, then cosine decay	The learning rate rises, then falls smoothly.
Warm-up steps	133 (set to 1,000, scaled to fit the run)	Steps over which the rate rises to its peak.
Decay steps	4,000 (set to 30,000, scaled to fit the run)	Steps over which it falls to the final rate.
Final learning rate	2.5e-06	The rate at the end of the decay.

Batches and length

Batch size	16	Frames per optimiser step.
Steps	4,000	Optimiser steps. Checkpoints were saved every 500.
Passes over the data	0.57	Steps × batch size ÷ frames in the training data.
Validation split	none	Every demonstration is used for training, so the loss cannot show when the model starts to memorise.

What it sees and predicts

Cameras	top_down, left_top, left_bottom, right, wrist	The views fed to the model at every step.
Image size	224 × 224	Each camera image is resized to this many pixels.
Prompt length	200 tokens	Longest command the text tokenizer keeps.
Action chunk	50 actions	How many future joint targets one forward pass predicts (1.7 s at 30 Hz).
Denoising steps	10	Steps the action expert takes to turn noise into a chunk when running.
Normalisation	state and action by quantiles; images unchanged	Joint values are rescaled with the 1st and 99th percentile of this data set, so its own joint ranges are used.

Image augmentation

Augmentation	colour jitter (brightness ±20%, contrast ±20%, saturation ±50%, hue ±0.05), sharpness, and a small shift and rotation (≤5% and ±5°); up to 3 applied to each image	Random changes to each training image.

Reproducibility

Seed	1,000	Fixes the random choices.

What happened

Why it lost the grasp

On 4 November it failed at the grasp in 14 of 35 rollouts. Merging the datasets had widened the gripper's range in the scaling from 37.4° to 50.8°, so every closing command came out about a third too open. A model trained on from Mango Rice keeps Mango Rice's scale, so the change broke it. Mango Sorbet is the same weights with the old ranges written back.

Mango Sorbet

Mango Lassi with the gripper and wrist ranges set back to Mango Rice's.

Started from: Mango Lassi (the same weights)
Trained on: vial-sort-slots-220
Cameras it saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen
Steps: 4,000
Passes over the data: 0.57

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: 1 asked for moves no demonstration contains.

Beside the table, 4k · sync: 1 rollout on moves no demonstration it trained on contains (L1 → L6). The right vial was grasped in 1, the arm let go at the asked slot in 0, and the vial was seen in that slot in 0.

Checkpoint	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
4k · sync	10	1	1	6	2

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

4k · sync: 10 rollouts, 8 moves

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
L3 → R6	3	2	0	0	1	1
R6 → L6	2	2	0	0	2	0
L1 → R6	4	1	0	0	0	1
L3 → L1	0	1	0	1	0	0
L3 → bin	3	1	0	0	1	0
L6 → L1	0	1	0	0	1	0
R1 → L6	2	1	1	0	0	0
R6 → R1	0	1	0	0	1	0

Hyperparameters

Read from the train_config.json saved with the checkpoint.

What trains

Starting weights	Mango Lassi (the same weights)	The checkpoint training began from.
Parameters trained	All of them	Vision encoder, language model and action expert all receive gradients: a full fine-tune, no LoRA.
Precision	bfloat16, gradient checkpointing	Number format of the weights and activations; checkpointing trades speed for memory.

Optimiser

Optimiser	AdamW	The update rule.
Peak learning rate	1e-05	The step size at its largest.
Betas	0.9, 0.95	How strongly AdamW remembers past gradients.
Weight decay	0.01	A pull of the weights toward zero.
Gradient clipping	1	Largest allowed size of one update's gradient.

Learning-rate schedule

Shape	Linear warm-up, then cosine decay	The learning rate rises, then falls smoothly.
Warm-up steps	133 (set to 1,000, scaled to fit the run)	Steps over which the rate rises to its peak.
Decay steps	4,000 (set to 30,000, scaled to fit the run)	Steps over which it falls to the final rate.
Final learning rate	2.5e-06	The rate at the end of the decay.

Batches and length

Batch size	16	Frames per optimiser step.
Steps	4,000	Optimiser steps. Checkpoints were saved every 500.
Passes over the data	0.57	Steps × batch size ÷ frames in the training data.
Validation split	none	Every demonstration is used for training, so the loss cannot show when the model starts to memorise.

What it sees and predicts

Cameras	top_down, left_top, left_bottom, right, wrist	The views fed to the model at every step.
Image size	224 × 224	Each camera image is resized to this many pixels.
Prompt length	200 tokens	Longest command the text tokenizer keeps.
Action chunk	50 actions	How many future joint targets one forward pass predicts (1.7 s at 30 Hz).
Denoising steps	10	Steps the action expert takes to turn noise into a chunk when running.
Normalisation	state and action by quantiles; images unchanged	Joint values are rescaled with the 1st and 99th percentile of this data set, so its own joint ranges are used.

Image augmentation

Augmentation	colour jitter (brightness ±20%, contrast ±20%, saturation ±50%, hue ±0.05), sharpness, and a small shift and rotation (≤5% and ±5°); up to 3 applied to each image	Random changes to each training image.

Reproducibility

Seed	1,000	Fixes the random choices.

What happened

The fix, tested on the arm

4 November, 10 rollouts: it failed at the grasp once, against 14 of 35 for Mango Lassi. It carried the vial to the destination rack nine times but put it in the asked slot only once: it went to the middle of the rack, slot 4 or 5, whatever slot was asked for.

Mango Chutney

Mango Lassi with only the gripper range set back.

Started from: Mango Lassi (the same weights)
Trained on: vial-sort-slots-220
Cameras it saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen
Steps: 4,000
Passes over the data: 0.57

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: 2 asked for moves no demonstration contains.

Beside the table, 4k · sync: 2 rollouts on moves no demonstration it trained on contains (L1 → L6). The right vial was grasped in 1, the arm let go at the asked slot in 0, and the vial was seen in that slot in 0.

Checkpoint	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
4k · sync	5	4	0	0	1

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

4k · sync: 5 rollouts, 5 moves

Move	Demonstrations	Rollouts	Missed the vial	Let go at the asked slot
L1 → R6	4	1	1	0
L3 → L1	0	1	1	0
L3 → R6	3	1	1	0
R1 → L6	2	1	1	0
R6 → L6	2	1	0	1

Hyperparameters

Read from the train_config.json saved with the checkpoint.

What trains

Starting weights	Mango Lassi (the same weights)	The checkpoint training began from.
Parameters trained	All of them	Vision encoder, language model and action expert all receive gradients: a full fine-tune, no LoRA.
Precision	bfloat16, gradient checkpointing	Number format of the weights and activations; checkpointing trades speed for memory.

Optimiser

Optimiser	AdamW	The update rule.
Peak learning rate	1e-05	The step size at its largest.
Betas	0.9, 0.95	How strongly AdamW remembers past gradients.
Weight decay	0.01	A pull of the weights toward zero.
Gradient clipping	1	Largest allowed size of one update's gradient.

Learning-rate schedule

Shape	Linear warm-up, then cosine decay	The learning rate rises, then falls smoothly.
Warm-up steps	133 (set to 1,000, scaled to fit the run)	Steps over which the rate rises to its peak.
Decay steps	4,000 (set to 30,000, scaled to fit the run)	Steps over which it falls to the final rate.
Final learning rate	2.5e-06	The rate at the end of the decay.

Batches and length

Batch size	16	Frames per optimiser step.
Steps	4,000	Optimiser steps. Checkpoints were saved every 500.
Passes over the data	0.57	Steps × batch size ÷ frames in the training data.
Validation split	none	Every demonstration is used for training, so the loss cannot show when the model starts to memorise.

What it sees and predicts

Cameras	top_down, left_top, left_bottom, right, wrist	The views fed to the model at every step.
Image size	224 × 224	Each camera image is resized to this many pixels.
Prompt length	200 tokens	Longest command the text tokenizer keeps.
Action chunk	50 actions	How many future joint targets one forward pass predicts (1.7 s at 30 Hz).
Denoising steps	10	Steps the action expert takes to turn noise into a chunk when running.
Normalisation	state and action by quantiles; images unchanged	Joint values are rescaled with the 1st and 99th percentile of this data set, so its own joint ranges are used.

Image augmentation

Augmentation	colour jitter (brightness ±20%, contrast ±20%, saturation ±50%, hue ±0.05), sharpness, and a small shift and rotation (≤5% and ±5°); up to 3 applied to each image	Random changes to each training image.

Reproducibility

Seed	1,000	Fixes the random choices.

What happened

Only half the fix

4 November: it failed at the grasp in all four moves that Mango Sorbet had picked up. The wrist pitch range had changed too, by 8%, which tilts the gripper about 4° down at the grasp. Both joints have to be set back.

Spicy Chicken

π0.5 trained on the recordings as they were, all five cameras.

Started from: lerobot/pi05_base
Trained on: vial-sort-slots-100, as recorded
Cameras it saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen
Steps: 12,000
Passes over the data: 3.27
Training time: 11.8 h
Peak GPU memory: 37.9 GB
Weights: on Hugging Face

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: 2 ended within 3 seconds, 6 asked for moves no demonstration contains, 2 had a command the scorer cannot read.

Beside the table, 7k · sync: 5 rollouts on moves no demonstration it trained on contains (R5 → L3, R5 → bin). The right vial was grasped in 0, the arm let go at the asked slot in 0, and the vial was seen in that slot in 0.

Beside the table, 12k · sync: 1 rollout on moves no demonstration it trained on contains (R4 → L5). The right vial was grasped in 0, the arm let go at the asked slot in 0, and the vial was seen in that slot in 0.

Checkpoint	Rollouts	Missed the vial	Let go at the asked slot
7k · sync	4	2	2
9k · RTC	3	3	0
12k · RTC	5	5	0
12k · sync	9	8	1

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

7k · sync: 4 rollouts, 4 moves

Move	Demonstrations	Rollouts	Missed the vial	Let go at the asked slot
L3 → R3	4	1	1	0
L3 → bin	3	1	1	0
R1 → bin	3	1	0	1
R3 → L6	3	1	0	1

9k · RTC: 3 rollouts, 3 moves

Move	Demonstrations	Rollouts	Missed the vial
L1 → R1	2	1	1
L1 → bin	3	1	1
L4 → R1	2	1	1

12k · RTC: 5 rollouts, 4 moves

Move	Demonstrations	Rollouts	Missed the vial
R3 → L6	3	2	2
L3 → R1	3	1	1
L4 → R1	2	1	1
L4 → bin	3	1	1

12k · sync: 9 rollouts, 6 moves

Move	Demonstrations	Rollouts	Missed the vial	Let go at the asked slot
L1 → R6	4	4	4	0
L1 → R1	2	1	1	0
L1 → bin	3	1	1	0
L3 → bin	3	1	0	1
R4 → L6	3	1	1	0
R6 → bin	3	1	1	0

Hyperparameters

Read from the train_config.json saved with the checkpoint.

What trains

Starting weights	lerobot/pi05_base	The checkpoint training began from.
Parameters trained	All of them	Vision encoder, language model and action expert all receive gradients: a full fine-tune, no LoRA.
Precision	bfloat16, gradient checkpointing	Number format of the weights and activations; checkpointing trades speed for memory.

Optimiser

Optimiser	AdamW	The update rule.
Peak learning rate	2.5e-05	The step size at its largest.
Betas	0.9, 0.95	How strongly AdamW remembers past gradients.
Weight decay	0.01	A pull of the weights toward zero.
Gradient clipping	1	Largest allowed size of one update's gradient.

Learning-rate schedule

Shape	Linear warm-up, then cosine decay	The learning rate rises, then falls smoothly.
Warm-up steps	400 (set to 1,000, scaled to fit the run)	Steps over which the rate rises to its peak.
Decay steps	12,000 (set to 30,000, scaled to fit the run)	Steps over which it falls to the final rate.
Final learning rate	2.5e-06	The rate at the end of the decay.

Batches and length

Batch size	16	Frames per optimiser step.
Steps	12,000	Optimiser steps. Checkpoints were saved every 1,000.
Passes over the data	3.27	Steps × batch size ÷ frames in the training data.
Validation split	none	Every demonstration is used for training, so the loss cannot show when the model starts to memorise.

What it sees and predicts

Cameras	top_down, left_top, left_bottom, right, wrist	The views fed to the model at every step.
Image size	224 × 224	Each camera image is resized to this many pixels.
Prompt length	200 tokens	Longest command the text tokenizer keeps.
Action chunk	50 actions	How many future joint targets one forward pass predicts (1.7 s at 30 Hz).
Denoising steps	10	Steps the action expert takes to turn noise into a chunk when running.
Normalisation	state and action by quantiles; images unchanged	Joint values are rescaled with the 1st and 99th percentile of this data set, so its own joint ranges are used.

Image augmentation

Augmentation	colour jitter (brightness ±20%, contrast ±20%, saturation ±50%, hue ±0.05), sharpness, and a small shift and rotation (≤5% and ±5°); up to 3 applied to each image	Random changes to each training image.

Reproducibility

Seed	1,000	Fixes the random choices.

What happened

How it was trained

15 to 16 October, in three stretches: it was stopped at 7,000 steps to stay inside a time budget and again at 8,000 for a reboot, and resumed from the saved checkpoint each time. Loss fell from 0.125 to 0.009.

What it did on the robot

At 7,000 steps both valid rollouts on 16 October put the vial in the right place. At 12,000 steps it left home in only 2 of 11 rollouts on 19 October; at 9,000 in 0 of 3.

Why it froze

Found on 19 October with offline tests on the GPU: about half of all frames at the home pose are the idle start of a demonstration and half the idle end, from the identical pose, so the policy told them apart from the pictures alone. In every demonstration the operator leaned over the robot at the start, and the policy learned that as its cue to move. Swapping just the wrist or the left_bottom image of a frozen rollout frame for a demonstration start made it reach.

Rollouts to watch

R1 → bin

Spicy Chicken at 7k steps, R1 → bin. The vial ends in the bin.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

L3 → bin

Spicy Chicken at 12k steps, L3 → bin. The vial ends in the bin. This checkpoint left home in only a few of its other rollouts.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

Mango Rice

π0.5 trained from the base on a cleaned copy with one wording per move.

Started from: lerobot/pi05_base
Trained on: vial-sort-slots-100, idle start and end cut, operator removed from the start frames, one canonical sentence per move
Cameras it saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen
Steps: 12,000
Passes over the data: 4.14
Training time: 11.6 h
Peak GPU memory: 37.9 GB
Weights: on Hugging Face

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: 2 ended within 3 seconds, 1 asked for moves no demonstration contains.

Beside the table, 12k · sync: 1 rollout on moves no demonstration it trained on contains (L3 → L6). The right vial was grasped in 0, the arm let go at the asked slot in 0, and the vial was seen in that slot in 0.

Checkpoint	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
10k · RTC	1	0	1	0	0
10k · sync	2	0	0	1	1
12k · RTC	1	0	1	0	0
12k · sync	5	1	0	3	1

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

10k · RTC: 1 rollouts, 1 move

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
R3 → L6	3	1	0	1	0	0

10k · sync: 2 rollouts, 2 moves

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
L3 → R3	4	1	0	0	0	1
L6 → R1	3	1	0	0	1	0

12k · RTC: 1 rollouts, 1 move

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
R3 → L6	3	1	0	1	0	0

12k · sync: 5 rollouts, 5 moves

Move	Demonstrations	Rollouts	Missed the vial	Lifted, let go elsewhere or not at all	Let go at the asked slot
L1 → R6	4	1	1	0	0
L6 → R3	3	1	0	1	0
R3 → L1	4	1	0	1	0
R3 → L3	2	1	0	0	1
R3 → L6	3	1	0	1	0

Hyperparameters

Read from the train_config.json saved with the checkpoint.

What trains

Starting weights	lerobot/pi05_base	The checkpoint training began from.
Parameters trained	All of them	Vision encoder, language model and action expert all receive gradients: a full fine-tune, no LoRA.
Precision	bfloat16, gradient checkpointing	Number format of the weights and activations; checkpointing trades speed for memory.

Optimiser

Optimiser	AdamW	The update rule.
Peak learning rate	2.5e-05	The step size at its largest.
Betas	0.9, 0.95	How strongly AdamW remembers past gradients.
Weight decay	0.01	A pull of the weights toward zero.
Gradient clipping	1	Largest allowed size of one update's gradient.

Learning-rate schedule

Shape	Linear warm-up, then cosine decay	The learning rate rises, then falls smoothly.
Warm-up steps	400 (set to 1,000, scaled to fit the run)	Steps over which the rate rises to its peak.
Decay steps	12,000 (set to 30,000, scaled to fit the run)	Steps over which it falls to the final rate.
Final learning rate	2.5e-06	The rate at the end of the decay.

Batches and length

Batch size	16	Frames per optimiser step.
Steps	12,000	Optimiser steps. Checkpoints were saved every 1,000.
Passes over the data	4.14	Steps × batch size ÷ frames in the training data.
Validation split	none	Every demonstration is used for training, so the loss cannot show when the model starts to memorise.

What it sees and predicts

Cameras	top_down, left_top, left_bottom, right, wrist	The views fed to the model at every step.
Image size	224 × 224	Each camera image is resized to this many pixels.
Prompt length	200 tokens	Longest command the text tokenizer keeps.
Action chunk	50 actions	How many future joint targets one forward pass predicts (1.7 s at 30 Hz).
Denoising steps	10	Steps the action expert takes to turn noise into a chunk when running.
Normalisation	state and action by quantiles; images unchanged	Joint values are rescaled with the 1st and 99th percentile of this data set, so its own joint ranges are used.

Image augmentation

Augmentation	colour jitter (brightness ±20%, contrast ±20%, saturation ±50%, hue ±0.05), sharpness, and a small shift and rotation (≤5% and ±5°); up to 3 applied to each image	Random changes to each training image.

Reproducibility

Seed	1,000	Fixes the random choices.

What happened

What is different from Spicy Chicken

The same recipe from the same base weights, with three changes to the data: the idle start and end of each demonstration are cut (58,650 frames became 46,361), about 8% of the samples at the home pose have the images that showed the operator swapped for views of the arm with nobody near it, and the 100 wordings are replaced by 32 canonical sentences, one for each move. It trained 4.1 passes over the data against 3.3.

Tested every 1,000 steps without the robot

Each checkpoint went through the offline gate (see the Training page). It leaves home at all 12; from 2,000 steps on its reach follows the source slot named in the command; it never follows the destination slot.

On the robot, 21 October

Run as RTC first, which never lifted the vial with this checkpoint, then in sync mode. In sync mode it took the right vial every time the command was worded exactly as in training, and put it in the asked slot in two of six rollouts, both slot 3.

Rollouts to watch

R3 → L3

Mango Rice at 12k steps, R3 → L3. Takes the vial across the table and lets go in L3, the slot asked for. The same rollout as the one at the top of the home page.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

L3 → R3

Mango Rice at 10k steps, L3 → R3. Over the slot it opens, grips again within half a second, then lets go. The vial ends in R3.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

R3 → L6

Mango Rice at 12k steps, R3 → L6. Carries the vial to the right slot and never lets go.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

L1 → R6

Mango Rice at 12k steps, L1 → R6. Closes on an empty hole instead of the vial.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

Pretzel One

A 361-step fine-tune of Spicy Chicken to test the operator-free starts.

Started from: Spicy Chicken at 12,000 steps
Trained on: vial-sort-slots-100, first trim (42,089 frames), operator-free start frames
Cameras it saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen
Steps: 361

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: 1 ended within 3 seconds.

Checkpoint	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
250 · RTC	3	2	0	0	1

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

250 · RTC: 3 rollouts, 3 moves

Move	Demonstrations	Rollouts	Missed the vial	Let go at the asked slot
L4 → R1	2	1	0	1
R3 → L1	4	1	1	0
R3 → bin	4	1	1	0

Hyperparameters

Read from the project notes (the checkpoint and its config were deleted).

Recipe

Starting weights	Spicy Chicken at 12,000 steps	Fine-tuned from the finished model, not from the base.
Learning rate	1e-5	0.4 times the peak of the full runs (2.5e-5).
Batch size	16	Frames per optimiser step.
Steps	361 of 1,000 planned	Stopped early; the checkpoint at 250 was tested.
Checkpoint every	250 steps

What happened

What it was for

A test of the finding that Spicy Chicken froze at home because it had learned the operator leaning in as its cue to move. The first frames of each start were swapped for views with nobody near the arm.

What went wrong

It left home at once and grasped the right vial, but let go too early. The trim cut every demonstration at the first slight loosening of the gripper over the hole, before the vial was pushed in, so it learned to loosen over the rack and stop. The trim was corrected for Pretzel Two. The checkpoint was deleted on 22 October.

Rollouts to watch

L4 → R1

Pretzel One at 250 steps, L4 → R1. The arm lets go at R1, but the vial lands on the table beside the rack. Scored as let go at the asked slot; a person marked it not in the slot.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

Pretzel Two

A 500-step fine-tune of Spicy Chicken with the corrected trim.

Started from: Spicy Chicken at 12,000 steps
Trained on: vial-sort-slots-100, trimmed at the real let-go (46,361 frames), operator-free start frames
Cameras it saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen
Steps: 500

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: none.

Checkpoint	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
500 · RTC	17	7	4	3	3

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

500 · RTC: 17 rollouts, 8 moves

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
R3 → L6	3	5	0	2	1	2
R3 → L1	4	3	1	1	1	0
R6 → bin	3	3	2	1	0	0
R1 → bin	3	2	2	0	0	0
L3 → R3	4	1	1	0	0	0
L6 → R1	3	1	0	0	0	1
R3 → bin	4	1	0	0	1	0
R6 → L6	2	1	1	0	0	0

Hyperparameters

Read from the project notes (the checkpoint and its config were deleted).

Recipe

Starting weights	Spicy Chicken at 12,000 steps	Fine-tuned from the finished model, not from Pretzel One.
Learning rate	1e-5, cosine
Batch size	16	Frames per optimiser step.
Steps	500	Checkpoints at 250 and 500.
Training time	30 minutes	17:34 to 18:04 on 19 October; loss 0.023 to 0.014.

What happened

What changed from Pretzel One

The trim now cuts at the widest opening of the gripper after the grasp, which is the real let-go, and the fine-tune starts again from Spicy Chicken rather than from Pretzel One.

Offline, before the robot

It left home on all four frozen test frames (72 to 90 degrees against 3 for Spicy Chicken), kept the error at the moment of letting go at 1.5 degrees, and picked the right rack on 100% of training starts.

On the robot, 19 October

Its scored rollouts are in the table above. A separate table-top test of six prompts showed slot numbers barely read: a middle vial beat the asked number twice, and the destination digit pulled the reach toward the wrong source once. Destinations landed at slot 3 whatever the digit. The checkpoint was deleted on 22 October.

Rollouts to watch

R3 → L6

Pretzel Two at 500 steps, R3 → L6. Lifts the right vial and lets go over L2, four slots short of L6.

Cameras the model saw: Overhead: seenAcross the table: seenNear eye level: seenFrom the side: seenOn the gripper: seen

Crispy Falafel

FLUX 3 Action, LoRA fine-tune, overhead and wrist cameras.

Started from: black-forest-labs/flux-3-action-so101, with this dataset's own joint ranges
Trained on: vial-sort-slots-100, overhead and wrist cameras only
Cameras it saw: Overhead: seenAcross the table: not seenNear eye level: not seenFrom the side: not seenOn the gripper: seen
Steps: 30,000
Passes over the data: 2.05
Training time: 16.9 h
Peak GPU memory: 33.6 GB
Weights: on Hugging Face

On the robot

Scored rollouts only, one row per checkpoint and mode, never pooled. Left out for this model: 1 ended within 3 seconds.

Checkpoint	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
10k · sync	8	7	1	0	0
20k · sync	3	3	0	0	0
26k · sync	1	1	0	0	0
26k · sync · EMA	1	1	0	0	0
30k · sync	4	1	1	1	1
30k · sync · EMA	3	2	0	1	0

Move by move

Each move this model was asked, with the number of demonstrations of that exact move in the training data. One block per checkpoint; the one with the most rollouts is open.

10k · sync: 8 rollouts, 5 moves

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted
L1 → R3	4	4	3	1
L1 → bin	3	1	1	0
L3 → R1	3	1	1	0
R3 → L3	2	1	1	0
R3 → bin	4	1	1	0

20k · sync: 3 rollouts, 2 moves

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
L4 → R1	2	2	2	0	0	0
L3 → R1	3	1	1	0	0	0

26k · sync: 1 rollouts, 1 move

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
R3 → L6	3	1	1	0	0	0

26k · sync · EMA: 1 rollouts, 1 move

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
R3 → L6	3	1	1	0	0	0

30k · sync: 4 rollouts, 3 moves

Move	Demonstrations	Rollouts	Missed the vial	Grasped, not lifted	Lifted, let go elsewhere or not at all	Let go at the asked slot
L1 → R6	4	2	1	1	0	0
R1 → L3	3	1	0	0	1	0
R3 → L6	3	1	0	0	0	1

30k · sync · EMA: 3 rollouts, 3 moves

Move	Demonstrations	Rollouts	Missed the vial	Lifted, let go elsewhere or not at all
L1 → R6	4	1	1	0
R1 → L3	3	1	0	1
R6 → bin	3	1	1	0

Hyperparameters

Read from the train_config.json saved with the checkpoint.

What trains

Starting weights	black-forest-labs/flux-3-action-so101, with this dataset's own joint ranges	The checkpoint training began from; the joint ranges built into it were replaced by this data set's own.
Method	LoRA, rank 32, alpha 32	Small low-rank adapters are trained on the attention and feed-forward layers; the 7-billion-parameter base stays frozen. About 137 million parameters train.
Action layers	trained in full	The layers that read and write joint values are not adapted, they are trained directly.
Precision	bfloat16, gradient checkpointing	Number format of the weights and activations; checkpointing trades speed for memory.

Optimiser

Optimiser	AdamW	The update rule.
Learning rate	0.0001 (adapters), 0.0005 (action layers)	The step size; the action layers learn five times faster.
Schedule	constant	No warm-up and no decay, so every checkpoint is a finished model.
Betas	0.9, 0.999	How strongly AdamW remembers past gradients.
Weight decay	0	A pull of the weights toward zero.
Gradient clipping	1	Largest allowed size of one update's gradient.
EMA of the weights	decay 0.999	A slowly moving average of the weights, saved beside the raw ones. Rollouts were run with both.

Batches and length

Batch size	4 × 4 accumulation = 16	Frames per optimiser update: 4 per step, 4 steps added up before each update.
Steps	30,000	Steps of 4 frames, so 7,500 updates. Checkpoints were saved every 2,000.
Passes over the data	2.05	Steps × frames per step ÷ frames in the training data.
Validation split	none	Every demonstration is used for training.

What it sees and predicts

Cameras	top_down, wrist	The two views are placed side by side on one canvas.
Canvas	256 × 512 pixels	Each camera is 256 × 256; the picture is divided into 8 × 8 cells.
Past observations	8 frames, 2 history snapshots	How much recent past the model sees.
Action chunk	42 predicted, 32 executed	Joint targets per forward pass, and how many are run before it replans.
Denoising steps	4, guidance 3	Steps to turn noise into a chunk; guidance strengthens the pull toward the prompt. Each replan costs 8 passes of the 7B model.
Loss weights	video 1, action 0.5, gripper channel ×2	FLUX learns to predict future video as well as actions.
Prompt dropout	0.1	Share of training samples shown without their prompt.
Actions	delta joint targets	Each action is the change from the current joint position.
Normalisation	built into the base	The base was rebuilt with this data set's 1st and 99th percentiles.

Reproducibility

Seed	42	Fixes the random choices.

What happened

How it was trained

Run 1 from 14 to 15 October (0 to 10,000 steps, 5 h 42 min), run 2 from 18 to 19 October (10,000 to 30,000, 11 h 13 min, a checkpoint every 2,000). The learning rate is constant, so every checkpoint is a finished model. Loss fell from 0.249 to 0.087 and was flat from about 8,000 steps; 30,000 steps is 2.05 passes over the data.

Getting it to run

Three problems stacked on 14 October: the base repositories were re-published and asked for a new licence click, an attention library compiled for CUDA 13 silently fell back to a kernel that tried to allocate 882 GiB, and runs started from an interactive session were killed when the session ended. The first needed the cached revisions pinned, the second the CUDA 13 runtime put on the library path, the third a user-level systemd service.

On the robot

Always in sync mode: FLUX has no real-time chunking. It replans every 32 actions with a freeze of about 1.3 s, and ran at 15.5 Hz against the 30 Hz target. At 30,000 steps it grasped the asked vial in 4 of 4 grasps and put it in the asked slot in 1 of 4.

Why only two cameras

FLUX takes a scene view and a wrist view. The overhead camera was chosen on 14 October because it is the only high-angle view, has no people in it, and stays readable in the dark runs. The other cameras were not trained for FLUX.

Which camera FLUX looked through

Three FLUX runs, each with a different scene camera. The choice of camera is the main thing that changed between them; they also differ in data size, training length and normalisation, so this is a sequence, not a controlled comparison.

Run	Scene camera	Data	Result	Why
Flux five	Low side camera across the table, plus wrist	5 demonstrations, one task	Did not work. Deleted on 22 October.	Black Forest Labs trained on high-angle views and dropped side views and datasets with people in them; this was a low side view with a person walking behind the table. The shipped joint ranges did not fit the arm either, and 3,000 steps on 5 demonstrations was too little.
Sweet Corn	The same camera re-aimed high and angled down, plus wrist	25 demonstrations, one task	Ran at 15 Hz with freezes. No success count was kept.	A single task cannot show whether the prompt is followed. The wrist camera dropped off USB during one session.
Crispy Falafel	Overhead (top_down), plus wrist	100 demonstrations, 32 moves	Right vial in 4 of 4 grasps at 30,000 steps; the right slot in 1 of 4.	Source slots are read; destination slots are not (see Training). A rack of six slots covers about two of the 8 by 8 cells FLUX divides its picture into.

Rollouts to watch

R1 → L3

Crispy Falafel at 30k steps, R1 → L3. Takes the right vial across to the left rack, holds it over L4 and never lets go.

Cameras the model saw: Overhead: seenAcross the table: not seenNear eye level: not seenFrom the side: not seenOn the gripper: seen

R1 → L3

Crispy Falafel at 30k steps, EMA weights, R1 → L3. Takes the right vial to the left rack and lets go at L1, not L3.

Cameras the model saw: Overhead: seenAcross the table: not seenNear eye level: not seenFrom the side: not seenOn the gripper: seen

Sweet Corn

FLUX 3 Action on a single task, the first 25 demonstrations.

Started from: black-forest-labs/flux-3-action-so101, with this dataset's own joint ranges
Trained on: vial-flux25, one task ("take the vial from position 3 of the rack to the bin")
Cameras it saw: left (re-aimed high) and wrist, from the earlier camera layout
Steps: 10,000
Passes over the data: 4.25
Training time: 6.4 h
Peak GPU memory: 32.4 GB

On the robot

Its rollouts ran on an earlier layout of the table, which the scorer cannot read, so none are scored here (11 rollouts, 2 too short).

Hyperparameters

Read from the train_config.json saved with the checkpoint.

What trains

Starting weights	black-forest-labs/flux-3-action-so101, with this dataset's own joint ranges	The checkpoint training began from; the joint ranges built into it were replaced by this data set's own.
Method	LoRA, rank 32, alpha 32	Small low-rank adapters are trained on the attention and feed-forward layers; the 7-billion-parameter base stays frozen. About 137 million parameters train.
Action layers	trained in full	The layers that read and write joint values are not adapted, they are trained directly.
Precision	bfloat16, gradient checkpointing	Number format of the weights and activations; checkpointing trades speed for memory.

Optimiser

Optimiser	AdamW	The update rule.
Learning rate	0.0001 (adapters), 0.0005 (action layers)	The step size; the action layers learn five times faster.
Schedule	constant	No warm-up and no decay, so every checkpoint is a finished model.
Betas	0.9, 0.999	How strongly AdamW remembers past gradients.
Weight decay	0	A pull of the weights toward zero.
Gradient clipping	1	Largest allowed size of one update's gradient.
EMA of the weights	decay 0.999	A slowly moving average of the weights, saved beside the raw ones. Rollouts were run with both.

Batches and length

Batch size	4 × 4 accumulation = 16	Frames per optimiser update: 4 per step, 4 steps added up before each update.
Steps	10,000	Steps of 4 frames, so 2,500 updates. Checkpoints were saved every 1,000.
Passes over the data	4.25	Steps × frames per step ÷ frames in the training data.
Validation split	none	Every demonstration is used for training.

What it sees and predicts

Cameras	left (re-aimed high) and wrist, from the earlier camera layout	The two views are placed side by side on one canvas.
Canvas	256 × 512 pixels	Each camera is 256 × 256; the picture is divided into 8 × 8 cells.
Past observations	8 frames, 2 history snapshots	How much recent past the model sees.
Action chunk	42 predicted, 32 executed	Joint targets per forward pass, and how many are run before it replans.
Denoising steps	4, guidance 3	Steps to turn noise into a chunk; guidance strengthens the pull toward the prompt. Each replan costs 8 passes of the 7B model.
Loss weights	video 1, action 0.5, gripper channel ×2	FLUX learns to predict future video as well as actions.
Prompt dropout	0.1	Share of training samples shown without their prompt.
Actions	delta joint targets	Each action is the change from the current joint position.
Normalisation	built into the base	The base was rebuilt with this data set's 1st and 99th percentiles.

Reproducibility

Seed	42	Fixes the random choices.

What happened

What it was

The first FLUX run that used the arm's own joint statistics. 10,000 steps (2,500 updates), loss 0.345 to 0.089, 6 h 25 min on a GPU shared with other users' jobs. Its first launch died at step 141 for a reason never found.

On the robot, 7 October

Ran at 15.3 Hz, not 30: the 7-billion-parameter model takes 95% of each loop and froze for 1.3 s every 32 actions. Its rollouts were on the earlier scene layout, which the scoring script cannot read, so they are not scored on this site.

Words used on this site

Hover or focus a dotted word anywhere to see its definition. All of them are here.

Rollout: One autonomous attempt by the robot at one command, from the moment it is started until it stops or the session ends.
Session: One launch of the rollout program. A session holds several rollouts, all with the same checkpoint, the same mode and the same lighting.
Checkpoint: The model’s weights saved at one training step. “12k” is the checkpoint after 12,000 steps. Results are never pooled across checkpoints.
Step: One update of the model’s weights, made from one batch of frames.
Pass over the data: Every frame of the training data seen once. Steps × batch size ÷ number of frames.
Demonstration: One move recorded by a person driving the leader arm while the follower arm copied it, with all cameras and joint positions logged.
Command (prompt): The sentence typed to the robot, for example “Move the vial from position 3 of the right rack to slot 6 of the left rack.” It names a source and a destination by position, never by colour.
Wording: One way of phrasing a command. The 100 demonstrations use 100 different wordings of 32 moves.
Move: One source slot and one destination, written R3 → L6: take the vial from slot 3 of the right rack and put it in slot 6 of the left rack. A destination can also be the bin.
Slot: One of the six holes of a rack, numbered 1 to 6. Slot 1 is nearest the follower arm.
Left and right rack: Read from the follower arm’s point of view: the left rack is the rack to the left of the arm, which is the top rack in the overhead camera. The bin sits between the racks.
Source and destination: The source is the slot the vial starts in. The destination is the slot (or the bin) it must end in.
Held out: Never recorded on purpose: slots 2 and 5 are never a source, and slots 2, 4 and 5 never a destination. They test whether a model can reach slots no demonstration used.
Scorer: A script (assess_rollouts) that reads the arm’s joint angles during a rollout, finds each grasp and release, and works out which slot the arm was at by matching its pose to the same moment in the demonstrations. It is accurate to about half a slot and cannot see the vial.
Grasped: The gripper closed on the vial in the slot the command asked to take it from. It does not mean the vial was lifted.
Lifted (picked up): After a grasp, the arm raised the vial out of its slot.
Let go at the asked slot: The arm opened the gripper at the destination slot the command asked for, within half a slot. It does not prove the vial ended in the hole.
Eye check: A person watching the cameras to see where the vial actually ended. Done for only some rollouts, so it is reported beside the scores, never inside them.
n: The number of rollouts behind a number. Small n means one more good or bad rollout moves the result a long way.
Sync mode: The robot pauses while the model computes its next chunk of actions, then runs the chunk. Smooth enough, but it stalls at every replan.
RTC: Real-time chunking: the model computes the next chunk while the arm is still running the last one. π0.5 supports it; FLUX 3 Action does not, so FLUX is always sync.
EMA weights: A slowly moving average of the weights over training, saved next to the raw weights. A rollout is run with one or the other.
LoRA: Low-rank adaptation: the large base model stays frozen and small added matrices are trained instead.
Full fine-tune: Every weight of the model is trained, with no frozen part and no adapters.
Loss: The training error: how far the model’s predicted actions are from the recorded ones. It falls as the model fits the data and says nothing about how it behaves on the robot.
Offline gate: A test of a saved checkpoint without the robot. It is shown recorded frames and commands and asked what motion it would predict.
Operator-free start: Training frames from the start of a demonstration, where the person leaned over the robot, replaced by views of the arm with nobody near it.