Vial sortPrevious work

Language-conditioned robotic vial sorting with a vision–language–action policy

A low-cost SO-101 arm moves small, translucent plastic vials between two racks and a bin. Each command names the exact slot to take the vial from and where to put it. The latest work trains π0.5 on 220 demonstrations seen by five cameras, recorded so the destination can only come from the command.

Mango Rice (π0.5), 12k steps, 21 October 2026. Real time, one camera, cropped to the table and trimmed to the move.

binleft rackright rack123456123456

The policy reads the sentence and the camera images and drives the arm's six joints. Nothing else tells it which vial to take or where to put it.

What the rollouts show

The runs of October and November, scored from the arm's motion over 174 rollouts, a rollout being one autonomous attempt at one command. The newest π0.5 model, Plain Bagel, grasps the right vial. Putting it in the right slot is the open problem.

  • 26 of 28rollouts grasped the right vialPlain Bagel (π0.5), 16k · sync.
  • 19 of 28of those let go at the slot the command asked forRead from the arm, not the vial.
  • 3 of 4rollouts grasped the right vialCrispy Falafel (FLUX 3 Action), 30k · sync. 1 of 4 let go at the asked slot.
  • 0 of 12checkpoints read the destination slot in the offline testMango Rice, October. The source slot is read from 5k steps on.

These are small counts from test runs, not a person-scored evaluation, and they should not be set beside the August evaluation. Compare every model and checkpoint.

Rollouts from the October and November models

Every camera the model saw, at once. They loop; click one to pause it.

Plain Bagelπ0.5

R3 → L6

Plain Bagel at 16k steps, R3 → L6, 9 November. Trained on the 220 demonstrations: carries the vial across the table to the far end of the left rack and lets go in L6.

Cameras the model saw:

L3 → R4

Plain Bagel at 16k steps, L3 → R4. No demonstration ever put a vial in slot 4, and the vial ends in R4.

Cameras the model saw:

Mango Puddingπ0.5

R3 → L6

Mango Pudding at 16k steps, R3 → L6, 8 November: the same move as the first clip, from the model trained on from Mango Rice. It stops one slot short, in L5.

Cameras the model saw:

Mango Riceπ0.5

R3 → L3

Mango Rice at 12k steps, R3 → L3. Takes the vial across the table and lets go in L3, the slot asked for. The same rollout as the one at the top of the home page.

Cameras the model saw:

L3 → R3

Mango Rice at 10k steps, L3 → R3. Over the slot it opens, grips again within half a second, then lets go. The vial ends in R3.

Cameras the model saw:

R3 → L6

Mango Rice at 12k steps, R3 → L6. Carries the vial to the right slot and never lets go.

Cameras the model saw:

L1 → R6

Mango Rice at 12k steps, L1 → R6. Closes on an empty hole instead of the vial.

Cameras the model saw:

Spicy Chickenπ0.5

R1 → bin

Spicy Chicken at 7k steps, R1 → bin. The vial ends in the bin.

Cameras the model saw:

L3 → bin

Spicy Chicken at 12k steps, L3 → bin. The vial ends in the bin. This checkpoint left home in only a few of its other rollouts.

Cameras the model saw:

Crispy FalafelFLUX 3 Action

R1 → L3

Crispy Falafel at 30k steps, R1 → L3. Takes the right vial across to the left rack, holds it over L4 and never lets go.

Cameras the model saw:

R1 → L3

Crispy Falafel at 30k steps, EMA weights, R1 → L3. Takes the right vial to the left rack and lets go at L1, not L3.

Cameras the model saw:

What each model is and how it was tested · How each one trained

The setup

The arm is an SO-101, a 3D-printed six-joint design, built as a pair: a person moves the leader arm and the follower copies it. That is how the 100 demonstrations were recorded.

Five cameras record at 30 frames a second: one overhead, two across the table at different heights, one from the side and one on the wrist. The vials are translucent plastic in three colours, and the policies see only the colour images.

π0.5 is fine-tuned in full from lerobot/pi05_base: the vision encoder and the whole vision–language model train, for 12,000 steps at batch size 16. FLUX 3 Action is fine-tuned with LoRA and sees two of the cameras, overhead and wrist; its training loss covers both predicted video and actions. Every model has a name, and the Experiments page says which is which.

How the data was recorded and worded

The current rig, labelled: the right, top-down, left-top and left-bottom cameras on stands and an arm above the table, the wrist camera on the follower arm, two racks with coloured vials, the bin, the follower arm on the left and the leader arm in the foreground.
The current rig, with every part labelled: the five cameras, the two racks, the bin, the follower arm that does the work and the leader arm a person moves. The five camera views are on the Data page.

Data and code

Before this

The first complete system, in August, was π0.5 with a frozen backbone, trained on 150 demonstrations and scored by a person over 79 rollouts. Its evaluation, datasets and report are kept.

See the previous work