Language-conditioned robotic vial sorting with a vision–language–action policy

Vision–language models understand scenes and instructions remarkably well, yet turning that understanding into precise physical manipulation remains an open problem. Vision–language–action (VLA) models address it by adding an action generating component to a pretrained vision–language model, but their demonstrations typically involve coarse tasks, such as moving a large object into a bowl, that tolerate millimetres of error. We study a more demanding case: sorting small, translucent plastic vials of three colours between two racks of narrow slots and a discard bin with a low-cost SO-101 arm, where each command names an exact source and destination position. Success requires an accurate grasp of a thin tube, a precise placement into a small hole, and a policy that handles many positions through many phrasings of the prompt.

We fine-tune the π0.5 VLA policy, keeping its vision–language backbone frozen and training only its action expert, on 150 teleoperated demonstrations that we release publicly, and we add a deterministic release step for the bin. In 79 autonomous, human-scored rollouts across 15 commands, the policy completes the full pick-and-place in 25% of cases. It acts on the wrong vial or slot in only 8% of rollouts, which shows that the prompts are generally followed, while the initial grasp of the vial is missed in 54% and the vial is released off the destination target in 19%. Precise physical execution is therefore the main limit, and more grasp demonstrations per position together with closed-loop grasp correction are the most direct ways to raise it.