Tick the models and checkpoints to compare. Every chart, map and video on this page follows the selection. A row is one model at one checkpoint in one mode: nothing is added across checkpoints.
How far each one got
Missed the vial
Grasped, not lifted
Lifted, let go elsewhere or not at all
Let go at the asked slot
Crispy Falafel · 10k · sync
71
n = 8
Crispy Falafel · 20k · sync
3
n = 3
Crispy Falafel · 26k · sync
1
n = 1
Crispy Falafel · 26k · sync · EMA
1
n = 1
Crispy Falafel · 30k · sync
1111
n = 4
Crispy Falafel · 30k · sync · EMA
21
n = 3
Mango Chutney · 4k · sync
41
n = 5
Beside it, not counted in n: 2 rollouts on moves it was never trained on (L1 → L6). The vial ended in the asked slot in 0 of them, by eye.
Mango Lassi · 4k · sync
95811
n = 33
Beside it, not counted in n: 4 rollouts on moves it was never trained on (L1 → L6, L3 → R4, R4 → L5). The vial ended in the asked slot in 0 of them, by eye.
Mango Pudding · 16k · sync
8488
n = 28
Beside it, not counted in n: 9 rollouts on moves it was never trained on (L1 → L6, L2 → R6, L3 → R4, L5 → bin, R2 → bin, R3 → L5, R5 → L1). The vial ended in the asked slot in 3 of them, by eye.
Mango Rice · 10k · RTC
1
n = 1
Mango Rice · 10k · sync
11
n = 2
Mango Rice · 12k · RTC
1
n = 1
Mango Rice · 12k · sync
131
n = 5
Beside it, not counted in n: 1 rollout on moves it was never trained on (L3 → L6). The vial ended in the asked slot in 0 of them, by eye.
Mango Sorbet · 4k · sync
1162
n = 10
Beside it, not counted in n: 1 rollout on moves it was never trained on (L1 → L6). The vial ended in the asked slot in 0 of them, by eye.
Plain Bagel · 16k · sync
21619
n = 28
Beside it, not counted in n: 9 rollouts on moves it was never trained on (L1 → L6, L2 → R6, L3 → R4, L5 → bin, R2 → bin, R3 → L5, R5 → L1). The vial ended in the asked slot in 4 of them, by eye.
Pretzel One · 250 · RTC
21
n = 3
Pretzel Two · 500 · RTC
7433
n = 17
Spicy Chicken · 7k · sync
22
n = 4
Beside it, not counted in n: 5 rollouts on moves it was never trained on (R5 → L3, R5 → bin). The vial ended in the asked slot in 0 of them, by eye.
Spicy Chicken · 9k · RTC
3
n = 3
Spicy Chicken · 12k · RTC
5
n = 5
Spicy Chicken · 12k · sync
81
n = 9
Beside it, not counted in n: 1 rollout on moves it was never trained on (R4 → L5). The vial ended in the asked slot in 0 of them, by eye.
Nothing ticked. Pick at least one row on the left.
Show the numbers as a table
Model and checkpoint
Rollouts
Missed the vial
Grasped, not lifted
Lifted, let go elsewhere or not at all
Let go at the asked slot
Untrained moves, beside n
of those, vial in the asked slot
Crispy Falafel · 10k · sync
8
7
1
0
0
0
–
Crispy Falafel · 20k · sync
3
3
0
0
0
0
–
Crispy Falafel · 26k · sync
1
1
0
0
0
0
–
Crispy Falafel · 26k · sync · EMA
1
1
0
0
0
0
–
Crispy Falafel · 30k · sync
4
1
1
1
1
0
–
Crispy Falafel · 30k · sync · EMA
3
2
0
1
0
0
–
Mango Chutney · 4k · sync
5
4
0
0
1
2
0
Mango Lassi · 4k · sync
33
9
5
8
11
4
0
Mango Pudding · 16k · sync
28
8
4
8
8
9
3
Mango Rice · 10k · RTC
1
0
1
0
0
0
–
Mango Rice · 10k · sync
2
0
0
1
1
0
–
Mango Rice · 12k · RTC
1
0
1
0
0
0
–
Mango Rice · 12k · sync
5
1
0
3
1
1
0
Mango Sorbet · 4k · sync
10
1
1
6
2
1
0
Plain Bagel · 16k · sync
28
2
1
6
19
9
4
Pretzel One · 250 · RTC
3
2
0
0
1
0
–
Pretzel Two · 500 · RTC
17
7
4
3
3
0
–
Spicy Chicken · 7k · sync
4
2
0
0
2
5
0
Spicy Chicken · 9k · RTC
3
3
0
0
0
0
–
Spicy Chicken · 12k · RTC
5
5
0
0
0
0
–
Spicy Chicken · 12k · sync
9
8
0
0
1
1
0
A rollout is counted once, in the furthest outcome it reached. Let go at the asked slot is read from the arm: a person who watched the rollouts that reached it found the vial in the slot 43 times, not in it 73 times, and could not tell 3 times.
Slot by slot
Where on the table each ticked row succeeded. The racks are drawn left rack, bin, right rack, slots 1 to 6 in each. A number is rollouts that got through out of rollouts that started or ended there; a darker slot is a better share. Four rows are shown at most.
Demonstrations:
10
11–12
13–14
15–19
20 or more
Demonstrations, by the slot the vial was taken fromDemonstrations, by where the vial was put
Share of rollouts that got through:
up to 25%
up to 50%
up to 75%
over 75%
none
Crispy Falafel · 10k · sync8 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Crispy Falafel · 20k · sync3 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Crispy Falafel · 26k · sync1 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Crispy Falafel · 26k · sync · EMA1 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Crispy Falafel · 30k · sync4 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Crispy Falafel · 30k · sync · EMA3 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Mango Chutney · 4k · sync5 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Mango Lassi · 4k · sync33 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Mango Pudding · 16k · sync28 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Mango Rice · 10k · RTC1 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Mango Rice · 10k · sync2 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Mango Rice · 12k · RTC1 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Mango Rice · 12k · sync5 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Mango Sorbet · 4k · sync10 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Plain Bagel · 16k · sync28 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Pretzel One · 250 · RTC3 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Pretzel Two · 500 · RTC17 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Spicy Chicken · 7k · sync4 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Spicy Chicken · 9k · RTC3 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Spicy Chicken · 12k · RTC5 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Spicy Chicken · 12k · sync9 rollouts
Grasped the right vial, by source slotLet go at the asked slot, by destination
Nothing ticked.
More than four rows are ticked: the first four are shown.
Rollouts to watch
Rollouts of the ticked rows, every camera the model saw at once. They loop; click one to pause it. Filter by how far they got:
R3 → L6
Plain Bagel at 16k steps, R3 → L6, 9 November. Trained on the 220 demonstrations: carries the vial across the table to the far end of the left rack and lets go in L6.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
L3 → R4
Plain Bagel at 16k steps, L3 → R4. No demonstration ever put a vial in slot 4, and the vial ends in R4.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
R3 → L6
Mango Pudding at 16k steps, R3 → L6, 8 November: the same move as the first clip, from the model trained on from Mango Rice. It stops one slot short, in L5.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
R3 → L3
Mango Rice at 12k steps, R3 → L3. Takes the vial across the table and lets go in L3, the slot asked for. The same rollout as the one at the top of the home page.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
L3 → R3
Mango Rice at 10k steps, L3 → R3. Over the slot it opens, grips again within half a second, then lets go. The vial ends in R3.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
R3 → L6
Mango Rice at 12k steps, R3 → L6. Carries the vial to the right slot and never lets go.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
L1 → R6
Mango Rice at 12k steps, L1 → R6. Closes on an empty hole instead of the vial.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
R1 → bin
Spicy Chicken at 7k steps, R1 → bin. The vial ends in the bin.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
L3 → bin
Spicy Chicken at 12k steps, L3 → bin. The vial ends in the bin. This checkpoint left home in only a few of its other rollouts.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
L4 → R1
Pretzel One at 250 steps, L4 → R1. The arm lets go at R1, but the vial lands on the table beside the rack. Scored as let go at the asked slot; a person marked it not in the slot.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
R3 → L6
Pretzel Two at 500 steps, R3 → L6. Lifts the right vial and lets go over L2, four slots short of L6.
Cameras the model saw: Overhead: seenOverheadAcross the table: seenAcross the tableNear eye level: seenNear eye levelFrom the side: seenFrom the sideOn the gripper: seenOn the gripper
R1 → L3
Crispy Falafel at 30k steps, R1 → L3. Takes the right vial across to the left rack, holds it over L4 and never lets go.
Cameras the model saw: Overhead: seenOverheadAcross the table: not seenAcross the tableNear eye level: not seenNear eye levelFrom the side: not seenFrom the sideOn the gripper: seenOn the gripper
R1 → L3
Crispy Falafel at 30k steps, EMA weights, R1 → L3. Takes the right vial to the left rack and lets go at L1, not L3.
Cameras the model saw: Overhead: seenOverheadAcross the table: not seenAcross the tableNear eye level: not seenNear eye levelFrom the side: not seenFrom the sideOn the gripper: seenOn the gripper
No rollout of the ticked rows matches. Only some rollouts were picked to be shown.
What these numbers cannot say
They are small. The biggest row has 33 rollouts and many have one to five, so one more good or bad rollout moves a row a long way, and the rows are not a controlled comparison: they differ in training data, length and mode. These were test runs during development, and the moves are uneven. The scorer reads the arm, not the vial. Left out: 11 rollouts under 3 seconds, 32 on moves no demonstration contains, 2 with a command the scorer cannot read and 11 from an earlier table layout. For a person-scored evaluation of a different, earlier system see the August evaluation; the two should not be compared.