Training
How each model’s training loss fell, and, under it, how every checkpoint that was tried on the robot did. Settings are on the Experiments page, and the rollouts are compared on the Results page.
There is no validation loss: every demonstration has its own wording, so the curve cannot show when a model starts to memorise. The robot is the only test, and the squares under each curve show how far each rollout got. They are scored from the arm, as explained under Experiments.
- Missed the vial
- Grasped, not lifted
- Lifted, let go elsewhere or not at all
- Let go at the asked slot
Plain Bagelπ0.5
π0.5 trained from the base on the 220 demonstrations, where the destination cannot be read off the cameras.
- Steps
- 16k
- Training time
- 21.5 h
- Per step
- 4.81 s
- Peak GPU memory
- 37.8 GB
On the robot: 28 scored rollouts, one square each
Several columns at one checkpoint are separate sessions: sync, RTC, or EMA weights. Hover a square for the session. The table for each checkpoint is on Plain Bagel’s Experiments entry.
Mango Puddingπ0.5
Mango Rice trained on for 16,000 steps on the 220 demonstrations.
- Steps
- 16k
- Training time
- 21.5 h
- Per step
- 4.81 s
- Peak GPU memory
- 37.8 GB
On the robot: 28 scored rollouts, one square each
Several columns at one checkpoint are separate sessions: sync, RTC, or EMA weights. Hover a square for the session. The table for each checkpoint is on Mango Pudding’s Experiments entry.
Mango Lassiπ0.5
Mango Rice trained on for 4,000 steps on the 220 demonstrations.
- Steps
- 4k
- Training time
- 4.1 h
- Per step
- 3.8 s
- Peak GPU memory
- 37.9 GB
On the robot: 33 scored rollouts, one square each
Several columns at one checkpoint are separate sessions: sync, RTC, or EMA weights. Hover a square for the session. The table for each checkpoint is on Mango Lassi’s Experiments entry.
Spicy Chickenπ0.5
π0.5 trained on the recordings as they were, all five cameras.
- Steps
- 12k
- Training time
- 11.8 h
- Per step
- 3.5 s
- Peak GPU memory
- 37.9 GB
On the robot: 21 scored rollouts, one square each
Several columns at one checkpoint are separate sessions: sync, RTC, or EMA weights. Hover a square for the session. The table for each checkpoint is on Spicy Chicken’s Experiments entry.
Mango Riceπ0.5
π0.5 trained from the base on a cleaned copy with one wording per move.
- Steps
- 12k
- Training time
- 11.6 h
- Per step
- 3.45 s
- Peak GPU memory
- 37.9 GB
On the robot: 9 scored rollouts, one square each
Several columns at one checkpoint are separate sessions: sync, RTC, or EMA weights. Hover a square for the session. The table for each checkpoint is on Mango Rice’s Experiments entry.
Crispy FalafelFLUX 3 Action
FLUX 3 Action, LoRA fine-tune, overhead and wrist cameras.
- Steps
- 30k
- Training time
- 16.9 h
- Per step
- 2.02 s
- Peak GPU memory
- 33.6 GB
On the robot: 20 scored rollouts, one square each
Several columns at one checkpoint are separate sessions: sync, RTC, or EMA weights. Hover a square for the session. The table for each checkpoint is on Crispy Falafel’s Experiments entry.
Sweet CornFLUX 3 Action
FLUX 3 Action on a single task, the first 25 demonstrations.
- Steps
- 10k
- Training time
- 6.4 h
- Per step
- 2.27 s
- Peak GPU memory
- 32.4 GB
Its rollouts ran on an earlier layout of the table that the scorer cannot read, so they are not scored here.
What Mango Rice could read, without the robot
An offline test of every saved checkpoint of Mango Rice, from 1k to 12k steps. It is shown frames from the demonstrations and from real rollouts and asked what motion it would predict. It leaves home at every checkpoint. From 5k steps on its reach follows the source slot named in the command. It does not follow the destination slot: it passed none of the 7 destination frames at any checkpoint.
Error against the recorded demonstrations
Degrees between the checkpoint's predicted motion while placing and the recorded one, on 15 demonstrations.
- Insertion error
- Gripper error
Show the numbers
| Checkpoint | Insertion error (°) | Gripper error (°) |
|---|---|---|
| 1k | 4.8 | 7.1 |
| 2k | 2.9 | 5 |
| 3k | 2.2 | 3.7 |
| 4k | 2 | 3.1 |
| 5k | 2.2 | 3 |
| 6k | 1.9 | 2.9 |
| 7k | 1.5 | 1.9 |
| 8k | 1.5 | 2.2 |
| 9k | 1.5 | 2 |
| 10k | 1.4 | 1.9 |
| 11k | 1.4 | 1.7 |
| 12k | 1.4 | 1.7 |
The errors fall steadily. The checks below do not move the same way.
| Check | 1k | 2k | 3k | 4k | 5k | 6k | 7k | 8k | 9k | 10k | 11k | 12k |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Leaves homeShown the first frame of nine real rollouts, with the arm at home, does it predict a move away? Counted when all three random seeds do. Frames from the start of a demonstration move 55 to 90°; a model that will sit still moves a few degrees. | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 | 9/9 |
| Reach follows the source slotVials stand in right-rack slots 1, 3 and 6 and the command names each in turn. The predicted shoulder angle 1.7 s ahead must order the slots as the demonstrations do (1 furthest, then 3, then 6) with at least 15° between the ends. | no | yes | no | no | yes | yes | yes | yes | yes | yes | yes | yes |
| Binds the two numbers to their roles“From position 3 … to slot 1” and “from position 1 … to slot 3” use the same two numbers. A model that understands which is the source reaches differently for them, by at least 8°; one that only collects the numbers does not. | no | no | no | no | no | no | no | no | no | no | no | no |
| Reads the destination slot7 frames taken one second after a grasp, the same vial each time, with commands that ask for destination slot 1, 3 and 6. A frame passes when the predicted reach is ordered as in the demonstrations, by at least 10°. The check passes when at least 4 of the 7 frames do. “0/7” is a real count: no frame passed, at any checkpoint. | 0/7 | 0/7 | 0/7 | 0/7 | 0/7 | 0/7 | 0/7 | 0/7 | 0/7 | 0/7 | 0/7 | 0/7 |
The robot shows the same split. Plain Bagel at 16k · sync grasped the right vial in 26 of 28 rollouts and let go at the asked slot in 19. See the results.
Why the destination is not followed
On 21 October a probe on training frames answered “underfit, overfit, or what?”: neither. It is a shortcut built into how the 100 demonstrations were recorded. Each run of five was recorded in one sitting with one destination, and the bin moved and the light changed between sittings, so the camera picture alone says which sitting a frame is from, and the sitting says the destination. The source changed in every demonstration within a sitting, so the picture could not give it away; that is why the source slot was learned and the destination was not.
| Test, on training frames | 12k steps | 2k steps |
|---|---|---|
| Replays its own demonstration’s destination | 100% | 88% |
| The command names another source vial: it goes there | 94% | 94% |
| The command names another destination: it follows it | 19% | 31% |
| No command at all: it follows the other destination | 19% | 27% |
| Same command, all five camera images from the other demonstration: it follows the cameras | 81% | 75% |
| Same command, joint positions from the other demonstration | 10% | 15% |
Same result at 2k and 12k steps, so this is not overtraining. Without a command the model behaves as if the destination words were not there. The probe was run on 21 October on 24 pairs of demonstrations with the same source and different destinations, for Mango Rice, and is written up in the project notes.
The two short fine-tunes
Pretzel One and Pretzel Two are quick fine-tunes of Spicy Chicken at 12k. Both train on demonstrations with the idle start and end cut, and replace the first frames with views of the arm with nobody near it; the second uses a corrected cut. They test the finding of the 19 October probes: the 12k checkpoint often waited at its start pose because it had learned the operator leaning in as its cue to move.
Every model of this work, and what comes next
Cells say “not logged” where a run happened and no record of the number exists, and a dash where a policy was never trained. The August policies are with the previous work.
| Policy | What was trained | Data | Steps | Training time | Peak GPU memory | On the robot | Source |
|---|---|---|---|---|---|---|---|
| October runs (workstation rig, five cameras) | |||||||
Plain Bagel π0.5 | π0.5 trained from the base on the 220 demonstrations, where the destination cannot be read off the cameras | vial-sort-slots-220, idle start and end cut, operator removed from the start frames, one sentence per move | 16k | 21.5 h | 37.8 GB | 28 scored rollouts. Each checkpoint has its own row on the Experiments page. | training log |
Mango Pudding π0.5 | Mango Rice trained on for 16,000 steps on the 220 demonstrations | vial-sort-slots-220, idle start and end cut, operator removed from the start frames, one sentence per move | 16k | 21.5 h | 37.8 GB | 28 scored rollouts. Each checkpoint has its own row on the Experiments page. | training log |
Mango Lassi π0.5 | Mango Rice trained on for 4,000 steps on the 220 demonstrations | vial-sort-slots-220, idle start and end cut, operator removed from the start frames, one sentence per move | 4k | 4.1 h | 37.9 GB | 33 scored rollouts. Each checkpoint has its own row on the Experiments page. | training log |
Spicy Chicken π0.5 | π0.5 trained on the recordings as they were, all five cameras | vial-sort-slots-100, as recorded | 12k | 11.8 h | 37.9 GB | 21 scored rollouts. Each checkpoint has its own row on the Experiments page. | training log |
Mango Rice π0.5 | π0.5 trained from the base on a cleaned copy with one wording per move | vial-sort-slots-100, idle start and end cut, operator removed from the start frames, one canonical sentence per move | 12k | 11.6 h | 37.9 GB | 9 scored rollouts. Each checkpoint has its own row on the Experiments page. | training log |
Crispy Falafel FLUX 3 Action | FLUX 3 Action, LoRA fine-tune, overhead and wrist cameras | vial-sort-slots-100, overhead and wrist cameras only | 30k | 16.9 h | 33.6 GB | 20 scored rollouts. Each checkpoint has its own row on the Experiments page. Ran at 15.46 Hz against a 30 Hz target. | training log |
Sweet Corn FLUX 3 Action | FLUX 3 Action on a single task, the first 25 demonstrations | vial-flux25, one task ("take the vial from position 3 of the rack to the bin") | 10k | 6.4 h | 32.4 GB | Rollouts not scored: an earlier scene layout. | training log |
| Planned next | |||||||
MolmoAct2 (Allen Institute for AI) | Not trained. Planned: fine-tune on the same 100 demonstrations and compare with π0.5 and FLUX | vial-sort-slots-100 (planned) | – | – | 16.5 to 48.3 GiB, as documented | Of interest for depth reasoning from the colour images alone, since the RealSense cannot see the translucent vials. Already in LeRobot as policy.type=molmoact2; its depth-reasoning (Think) variant is not in that port yet. Memory is the authors' figure for fine-tuning the action expert only (batch 8) up to the full model, on an H100 with LIBERO, not measured here. | LeRobot docs/source/molmoact2.mdx, allenai.org/blog/molmoact2, ee/DEPTH_PLAN.md |
| Considered, not trained | |||||||
LingBot-VA (world-action model) | Not trained | – | – | – | – | 5,230 ms per inference as published: far too slow for closed-loop control. | arXiv 2603.22078 |