Data

Five cameras watched a person move a vial with a leader arm. There are two training sets: the first 100 demonstrations, and 220 that keep 70 of those and add 150 recorded so the destination can only come from the command. Both sets and every robot recording are public on Hugging Face in the LeRobot format.

Five cameras

One instant of one demonstration, seen by each camera. All five are recorded at 30 frames a second.

• Overhead top_downThe only high-angle view. It sees both racks, the bin and the arm at once, with no people in it.
• Across the table left_topHigh, across the table. The operator is in view in every frame, on purpose: someone usually stands there during a rollout too.
• Near eye level left_bottomNear table level, the colour stream of a RealSense camera. The vials look largest here and which holes hold a vial can be read.
• From the side rightA wide view of the workspace from the other side. A person is in view some of the time.
• On the gripper wristClose up, mounted on the gripper. The vial fills the frame during a grasp.

The 100 demonstrations

100 demonstrations, 58,650 frames, about 33 minutes. Each is one move: take the vial from one slot and put it in another slot or the bin. The set is sari-abdan/vial-sort-slots-100, and any episode can be watched in the browser.

Every move that was recorded

Rows are the slot the vial started in, columns where it was put, and each number is how many demonstrations there are of that move. Held-out slots are hatched: they were never recorded, so a model can be tested on them. Every cross-rack move between the other slots, and every slot to the bin, is there.

32 different moves; 75 go from one rack to the other and 25 to the bin. No move stays inside a rack. Slots 2 and 5 are never a source, and slots 2, 4 and 5 are never a destination (slot 4 is only ever a source).

The 220 demonstrations

220 demonstrations, 141,735 frames, about 79 minutes, the same five cameras. With the 100, every run of five had one destination and was recorded in one sitting, so the pictures gave the destination away and the models learned to read it there. In the 150 new demonstrations one layout is recorded three times in a row with three different destinations, so only the command can tell them apart. The set is sari-abdan/vial-sort-slots-220, and any episode can be watched in the browser.

Every move that was recorded

Laid out like the matrix above, now with moves inside one rack: 51 of the 220. The held-out slots are the same, and so is one gap: no demonstration goes from L1 to L6.

48 different moves; 41 go to the bin and 51 stay inside one rack. Every source slot has 24 to 31 demonstrations.

Where they come from

The training set leaves the depth stream out: the November models see the five colour cameras only. The across-the-table camera sits slightly differently in the first 70 than in the later 150, and it was knocked again on 4 November; see the note of 9 November.

The wording pools

The same six colours mark the parts of the example wordings below.

The recorded wordings of one move

The move R3 → L6 has 3 demonstrations. These are their commands, exactly as they are stored in the set:

• “Move the vial from position 3 of the right rack to slot 6 of the left rack.”
• “Take the vial in slot 3 of the right rack to the sixth slot of the left rack.”
• “Transfer the tube from the third slot of the rack on the right over to hole 6 of the left rack.”

Robot recordings

Autonomous sessions from October, one episode per rollout, uploaded with every camera the model saw. A rollout is one autonomous attempt at one command. The results score every session, including the ones not uploaded.

Filled chips are the cameras stored in that recording. The August datasets, versions 1 to 5, are under previous work.