Skip to content

Commit

Permalink
Patch off object (again) (#146)
Browse files Browse the repository at this point in the history
  • Loading branch information
@scottcanoe
scottcanoe authored Jan 23, 2025
1 parent 4cdbd2a commit caf2ef4
Show file tree
Hide file tree
Showing 7 changed files with 193 additions and 108 deletions.
14 changes: 7 additions & 7 deletions benchmarks/results/montymeetsworld.csv
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
randrot_noise_sim_on_scan_monty_world,80.00%,80.83%,412,0.86,52m,24s
world_image_on_scanned_model,72.92%,83.33%,442,2.15,18m,21s
dark_world_image_on_scanned_model,35.42%,83.33%,430,1.81,16m,20s
bright_world_image_on_scanned_model,43.75%,79.17%,428,1.87,18m,22s
hand_intrusion_world_image_on_scanned_model,39.58%,56.25%,344,2.00,8m,10s
multi_object_world_image_on_scanned_model,43.75%,52.08%,344,1.90,11m,14s
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
randrot_noise_sim_on_scan_monty_world,80.00%,80.83%,412,0.86,39m,18s
world_image_on_scanned_model,72.92%,83.33%,442,2.15,9m,11s
dark_world_image_on_scanned_model,35.42%,83.33%,430,1.81,7m,9s
bright_world_image_on_scanned_model,43.75%,79.17%,428,1.87,10m,11s
hand_intrusion_world_image_on_scanned_model,39.58%,56.25%,344,2.00,6m,7s
multi_object_world_image_on_scanned_model,43.75%,52.08%,344,1.90,7m,8s
26 changes: 13 additions & 13 deletions benchmarks/results/ycb_10objs.csv
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
base_config_10distinctobj_dist_agent,99.29%,3.57%,34,0.24,5m,19s
base_config_10distinctobj_surf_agent,100.00%,0.00%,28,0.18,4m,21s
randrot_noise_10distinctobj_dist_agent,99.00%,6.00%,48,0.46,5m,29s
randrot_noise_10distinctobj_dist_on_distm,100.00%,2.00%,36,0.25,4m,28s
randrot_noise_10distinctobj_surf_agent,100.00%,1.00%,29,0.36,5m,33s
randrot_10distinctobj_surf_agent,100.00%,0.00%,28,0.38,3m,17s
randrot_noise_10distinctobj_5lms_dist_agent,100.00%,6.00%,53,0.84,15m,82s
base_10simobj_surf_agent,93.57%,10.00%,78,0.16,10m,53s
randrot_noise_10simobj_dist_agent,82.00%,42.00%,189,0.59,16m,119s
randrot_noise_10simobj_surf_agent,91.00%,31.00%,162,0.42,23m,192s
randomrot_rawnoise_10distinctobj_surf_agent,72.00%,76.00%,16,1.56,15m,15s
base_10multi_distinctobj_dist_agent,73.57%,40.00%,27,0.73,1h5m,2s
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
base_config_10distinctobj_dist_agent,100.00%,0.00%,37,0.28,5m,19s
base_config_10distinctobj_surf_agent,100.00%,0.00%,28,0.23,4m,18s
randrot_noise_10distinctobj_dist_agent,100.00%,3.00%,55,0.39,5m,33s
randrot_noise_10distinctobj_dist_on_distm,99.00%,1.00%,38,0.21,4m,28s
randrot_noise_10distinctobj_surf_agent,100.00%,0.00%,29,0.39,5m,33s
randrot_10distinctobj_surf_agent,100.00%,1.00%,28,0.32,4m,21s
randrot_noise_10distinctobj_5lms_dist_agent,100.00%,0.00%,57,0.79,10m,84s
base_10simobj_surf_agent,94.29%,10.71%,81,0.24,10m,50s
randrot_noise_10simobj_dist_agent,84.00%,40.00%,237,0.61,18m,147s
randrot_noise_10simobj_surf_agent,93.00%,33.00%,176,0.52,24m,205s
randomrot_rawnoise_10distinctobj_surf_agent,68.00%,78.00%,16,1.78,17m,15s
base_10multi_distinctobj_dist_agent,79.29%,10.71%,31,0.35,43m,1s
12 changes: 6 additions & 6 deletions benchmarks/results/ycb_77objs.csv
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
base_77obj_dist_agent,93.07%,13.85%,86,0.32,56m,178s
base_77obj_surf_agent,98.70%,6.49%,56,0.16,35m,104s
randrot_noise_77obj_dist_agent,87.45%,30.74%,149,0.70,1h17m,264s
randrot_noise_77obj_surf_agent,96.54%,18.61%,103,0.58,1h3m,225s
randrot_noise_77obj_5lms_dist_agent,84.42%,9.09%,66,1.08,39m,843s
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
base_77obj_dist_agent,93.51%,12.99%,108,0.28,1h2m,212s
base_77obj_surf_agent,99.13%,6.06%,54,0.19,28m,82s
randrot_noise_77obj_dist_agent,89.61%,22.51%,152,0.65,1h25m,308s
randrot_noise_77obj_surf_agent,92.64%,25.11%,120,0.69,1h5m,227s
randrot_noise_77obj_5lms_dist_agent,88.31%,0.00%,70,1.02,32m,862s
6 changes: 3 additions & 3 deletions benchmarks/results/ycb_unsupervised.csv
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Experiment,%Correct - 1st Epoch,% Correct - >1st Epoch,Mean Objects per Graph,Mean Graphs per Object,Run Time,Episode Run Time (s)
surf_agent_unsupervised_10distinctobj,70.00%,83.33%,1.43,1.11,20m,12s
surf_agent_unsupervised_10distinctobj_noise,70.00%,67.78%,1.19,2.11,24m,15s
Experiment,%Correct - 1st Epoch,% Correct - >1st Epoch,Mean Objects per Graph,Mean Graphs per Object,Run Time,Episode Run Time (s)
surf_agent_unsupervised_10distinctobj,70.00%,83.33%,1.43,1.11,20m,12s
surf_agent_unsupervised_10distinctobj_noise,70.00%,67.78%,1.19,2.11,25m,15s
surf_agent_unsupervised_10simobj,40.00%,86.67%,2.60,1.30,28m,17s
48 changes: 24 additions & 24 deletions docs/overview/benchmark-experiments.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,18 +51,18 @@ The following results are obtained from experiments using the 10-object subsets

| Experiment | % Correct | % Used MLH | Num Matching Steps | Rotation Error (radians) | Run Time | Episode Run Time (s) |
|---------------------------------------------|-----------|------------|--------------------|--------------------------|----------|----------------------|
| base_config_10distinctobj_dist_agent | 99.29% | 3.57% | 34 | 0.24 | 5m | 19s |
| base_config_10distinctobj_surf_agent | 100.00% | 0.00% | 28 | 0.18 | 4m | 21s |
| randrot_noise_10distinctobj_dist_agent | 99.00% | 6.00% | 48 | 0.46 | 5m | 29s |
| randrot_noise_10distinctobj_dist_on_distm | 100.00% | 2.00% | 36 | 0.25 | 4m | 28s |
| randrot_noise_10distinctobj_surf_agent | 100.00% | 1.00% | 29 | 0.36 | 5m | 33s |
| randrot_10distinctobj_surf_agent | 100.00% | 0.00% | 28 | 0.38 | 3m | 17s |
| randrot_noise_10distinctobj_5lms_dist_agent | 100.00% | 6.00% | 53 | 0.84 | 15m | 82s |
| base_10simobj_surf_agent | 93.57% | 10.00% | 78 | 0.16 | 10m | 53s |
| randrot_noise_10simobj_dist_agent | 82.00% | 42.00% | 189 | 0.59 | 16m | 119s |
| randrot_noise_10simobj_surf_agent | 91.00% | 31.00% | 162 | 0.42 | 23m | 192s |
| randomrot_rawnoise_10distinctobj_surf_agent | 72.00% | 76.00% | 16 | 1.56 | 15m | 15s |
| base_10multi_distinctobj_dist_agent | 73.57% | 40.00% | 27 | 0.73 | 1h5m | 2s |
| base_config_10distinctobj_dist_agent | 100.00% | 0.00% | 37 | 0.28 | 5m | 19s |
| base_config_10distinctobj_surf_agent | 100.00% | 0.00% | 28 | 0.23 | 4m | 18s |
| randrot_noise_10distinctobj_dist_agent | 100.00% | 3.00% | 55 | 0.39 | 5m | 33s |
| randrot_noise_10distinctobj_dist_on_distm | 99.00% | 1.00% | 38 | 0.21 | 4m | 28s |
| randrot_noise_10distinctobj_surf_agent | 100.00% | 0.00% | 29 | 0.39 | 5m | 33s |
| randrot_10distinctobj_surf_agent | 100.00% | 1.00% | 28 | 0.32 | 4m | 21s |
| randrot_noise_10distinctobj_5lms_dist_agent | 100.00% | 0.00% | 57 | 0.79 | 10m | 84s |
| base_10simobj_surf_agent | 94.29% | 10.71% | 81 | 0.24 | 10m | 50s |
| randrot_noise_10simobj_dist_agent | 84.00% | 40.00% | 237 | 0.61 | 18m | 147s |
| randrot_noise_10simobj_surf_agent | 93.00% | 33.00% | 176 | 0.52 | 24m | 205s |
| randomrot_rawnoise_10distinctobj_surf_agent | 68.00% | 78.00% | 16 | 1.78 | 17m | 15s |
| base_10multi_distinctobj_dist_agent | 79.29% | 10.71% | 31 | 0.35 | 43m | 1s |

## Longer Experiments With all 77 YCB Objects

Expand All @@ -75,11 +75,11 @@ The following results are obtained from experiments on the entire YCB dataset (7

| Experiment | % Correct | % Used MLH | Num Matching Steps | Rotation Error (radians) | Run Time | Episode Run Time (s) |
|-------------------------------------|-----------|------------|--------------------|--------------------------|----------|----------------------|
| base_77obj_dist_agent | 93.07% | 13.85% | 86 | 0.32 | 56m | 178s |
| base_77obj_surf_agent | 98.70% | 6.49% | 56 | 0.16 | 35m | 104s |
| randrot_noise_77obj_dist_agent | 87.45% | 30.74% | 149 | 0.70 | 1h17m | 264s |
| randrot_noise_77obj_surf_agent | 96.54% | 18.61% | 103 | 0.58 | 1h3m | 225s |
| randrot_noise_77obj_5lms_dist_agent | 84.42% | 9.09% | 66 | 1.08 | 39m | 843s |
| base_77obj_dist_agent | 93.51% | 12.99% | 108 | 0.28 | 1h2m | 212s |
| base_77obj_surf_agent | 99.13% | 6.06% | 54 | 0.19 | 28m | 82s |
| randrot_noise_77obj_dist_agent | 89.61% | 22.51% | 152 | 0.65 | 1h25m | 308s |
| randrot_noise_77obj_surf_agent | 92.64% | 25.11% | 120 | 0.69 | 1h5m | 227s |
| randrot_noise_77obj_5lms_dist_agent | 88.31% | 0.00% | 70 | 1.02 | 32m | 862s |

### Explanation of Some of the Results

Expand Down Expand Up @@ -112,7 +112,7 @@ An object is classified as detected correctly if the detected object ID is in th
| Experiment | %Correct - 1st Epoch | % Correct - >1st Epoch | Mean Objects per Graph | Mean Graphs per Object | Run Time | Episode Run Time (s) |
|---------------------------------------------|----------------------|------------------------|------------------------|------------------------|----------|----------------------|
| surf_agent_unsupervised_10distinctobj | 70.00% | 83.33% | 1.43 | 1.11 | 20m | 12s |
| surf_agent_unsupervised_10distinctobj_noise | 70.00% | 67.78% | 1.19 | 2.11 | 24m | 15s |
| surf_agent_unsupervised_10distinctobj_noise | 70.00% | 67.78% | 1.19 | 2.11 | 25m | 15s |
| surf_agent_unsupervised_10simobj | 40.00% | 86.67% | 2.60 | 1.30 | 28m | 17s |

To obtain these results use `print_unsupervised_stats(train_stats, epoch_len=10)` (wandb logging is currently not written for unsupervised stats). Unsupervised, continual learning can, by definition, not be parallelized across epochs. Therefore these experiments were run without multiprocessing on the laptop (running on cloud CPUs works as well but since these are slower without parallelization these were run on the laptop).
Expand Down Expand Up @@ -159,12 +159,12 @@ See the [monty_lab project folder](https://github.com/thousandbrainsproject/mont

| Experiment | % Correct | % Used MLH | Num Matching Steps | [Rotation Error (radians)] | Run Time | Episode Run Time (s) |
|---------------------------------------------|-----------|------------|--------------------|----------------------------|----------|----------------------|
| randrot_noise_sim_on_scan_monty_world | 80.00% | 80.83% | 412 | 0.86 | 52m | 24s |
| world_image_on_scanned_model | 72.92% | 83.33% | 442 | 2.15 | 18m | 21s |
| dark_world_image_on_scanned_model | 35.42% | 83.33% | 430 | 1.81 | 16m | 20s |
| bright_world_image_on_scanned_model | 43.75% | 79.17% | 428 | 1.87 | 18m | 22s |
| hand_intrusion_world_image_on_scanned_model | 39.58% | 56.25% | 344 | 2.00 | 8m | 10s |
| multi_object_world_image_on_scanned_model | 43.75% | 52.08% | 344 | 1.90 | 11m | 14s |
| randrot_noise_sim_on_scan_monty_world | 80.00% | 80.83% | 412 | 0.86 | 39m | 18s |
| world_image_on_scanned_model | 72.92% | 83.33% | 442 | 2.15 | 9m | 11s |
| dark_world_image_on_scanned_model | 35.42% | 83.33% | 430 | 1.81 | 7m | 9s |
| bright_world_image_on_scanned_model | 43.75% | 79.17% | 428 | 1.87 | 10m | 11s |
| hand_intrusion_world_image_on_scanned_model | 39.58% | 56.25% | 344 | 2.00 | 6m | 7s |
| multi_object_world_image_on_scanned_model | 43.75% | 52.08% | 344 | 1.90 | 7m | 8s |

**Note that rotation errors are meaningless since no ground truth rotation is provided**

Expand Down
Loading

0 comments on commit caf2ef4

Please sign in to comment.