Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patch off object (again) #146

Merged
merged 20 commits into from
Jan 23, 2025
Merged

Patch off object (again) #146

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
03efabe
log error instead of assert
scottcanoe Jan 19, 2025
b789a32
Implement multiple orientation attempts
scottcanoe Jan 20, 2025
a3ff598
Fix multiobject
scottcanoe Jan 20, 2025
4efbd4b
docstring adjustment
scottcanoe Jan 20, 2025
094c03c
update tables
scottcanoe Jan 20, 2025
1890951
Merge branch 'main' into patch_off_object
scottcanoe Jan 20, 2025
cefcd34
reimplement compute_look_amounts
scottcanoe Jan 22, 2025
9ad7acf
Clean up
scottcanoe Jan 22, 2025
d7e7625
Clean up compute_look_amounts
scottcanoe Jan 22, 2025
8fc5323
Reduce number of orientation attempts for patch
scottcanoe Jan 22, 2025
9f86634
Update motor_policies.py
scottcanoe Jan 22, 2025
5bc5c83
Add is_on_object to motor system
scottcanoe Jan 22, 2025
4f8082b
typo
scottcanoe Jan 22, 2025
c70a47e
Finish moving is_on_target_object to motor system
scottcanoe Jan 22, 2025
018121a
Remove unnecessary check
scottcanoe Jan 22, 2025
c4dcc15
Correct variable
scottcanoe Jan 22, 2025
bc8ebf6
Update tables
scottcanoe Jan 23, 2025
5e8e051
Merge branch 'main' into patch_off_object
scottcanoe Jan 23, 2025
7a45a49
expound a TODO
scottcanoe Jan 23, 2025
11bab32
Fix copy/paste error
scottcanoe Jan 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions benchmarks/results/montymeetsworld.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
randrot_noise_sim_on_scan_monty_world,80.00%,80.83%,412,0.86,52m,24s
world_image_on_scanned_model,72.92%,83.33%,442,2.15,18m,21s
dark_world_image_on_scanned_model,35.42%,83.33%,430,1.81,16m,20s
bright_world_image_on_scanned_model,43.75%,79.17%,428,1.87,18m,22s
hand_intrusion_world_image_on_scanned_model,39.58%,56.25%,344,2.00,8m,10s
multi_object_world_image_on_scanned_model,43.75%,52.08%,344,1.90,11m,14s
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
randrot_noise_sim_on_scan_monty_world,80.00%,80.83%,412,0.86,39m,18s
world_image_on_scanned_model,72.92%,83.33%,442,2.15,9m,11s
dark_world_image_on_scanned_model,35.42%,83.33%,430,1.81,7m,9s
bright_world_image_on_scanned_model,43.75%,79.17%,428,1.87,10m,11s
hand_intrusion_world_image_on_scanned_model,39.58%,56.25%,344,2.00,6m,7s
multi_object_world_image_on_scanned_model,43.75%,52.08%,344,1.90,7m,8s
26 changes: 13 additions & 13 deletions benchmarks/results/ycb_10objs.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
base_config_10distinctobj_dist_agent,99.29%,3.57%,34,0.24,5m,19s
base_config_10distinctobj_surf_agent,100.00%,0.00%,28,0.18,4m,21s
randrot_noise_10distinctobj_dist_agent,99.00%,6.00%,48,0.46,5m,29s
randrot_noise_10distinctobj_dist_on_distm,100.00%,2.00%,36,0.25,4m,28s
randrot_noise_10distinctobj_surf_agent,100.00%,1.00%,29,0.36,5m,33s
randrot_10distinctobj_surf_agent,100.00%,0.00%,28,0.38,3m,17s
randrot_noise_10distinctobj_5lms_dist_agent,100.00%,6.00%,53,0.84,15m,82s
base_10simobj_surf_agent,93.57%,10.00%,78,0.16,10m,53s
randrot_noise_10simobj_dist_agent,82.00%,42.00%,189,0.59,16m,119s
randrot_noise_10simobj_surf_agent,91.00%,31.00%,162,0.42,23m,192s
randomrot_rawnoise_10distinctobj_surf_agent,72.00%,76.00%,16,1.56,15m,15s
base_10multi_distinctobj_dist_agent,73.57%,40.00%,27,0.73,1h5m,2s
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
base_config_10distinctobj_dist_agent,100.00%,0.00%,37,0.28,5m,19s
base_config_10distinctobj_surf_agent,100.00%,0.00%,28,0.23,4m,18s
randrot_noise_10distinctobj_dist_agent,100.00%,3.00%,55,0.39,5m,33s
randrot_noise_10distinctobj_dist_on_distm,99.00%,1.00%,38,0.21,4m,28s
randrot_noise_10distinctobj_surf_agent,100.00%,0.00%,29,0.39,5m,33s
randrot_10distinctobj_surf_agent,100.00%,1.00%,28,0.32,4m,21s
randrot_noise_10distinctobj_5lms_dist_agent,100.00%,0.00%,57,0.79,10m,84s
base_10simobj_surf_agent,94.29%,10.71%,81,0.24,10m,50s
randrot_noise_10simobj_dist_agent,84.00%,40.00%,237,0.61,18m,147s
randrot_noise_10simobj_surf_agent,93.00%,33.00%,176,0.52,24m,205s
randomrot_rawnoise_10distinctobj_surf_agent,68.00%,78.00%,16,1.78,17m,15s
base_10multi_distinctobj_dist_agent,79.29%,10.71%,31,0.35,43m,1s
12 changes: 6 additions & 6 deletions benchmarks/results/ycb_77objs.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
base_77obj_dist_agent,93.07%,13.85%,86,0.32,56m,178s
base_77obj_surf_agent,98.70%,6.49%,56,0.16,35m,104s
randrot_noise_77obj_dist_agent,87.45%,30.74%,149,0.70,1h17m,264s
randrot_noise_77obj_surf_agent,96.54%,18.61%,103,0.58,1h3m,225s
randrot_noise_77obj_5lms_dist_agent,84.42%,9.09%,66,1.08,39m,843s
Experiment,% Correct,% Used MLH,Num Matching Steps,Rotation Error (radians),Run Time,Episode Run Time (s)
base_77obj_dist_agent,93.51%,12.99%,108,0.28,1h2m,212s
base_77obj_surf_agent,99.13%,6.06%,54,0.19,28m,82s
randrot_noise_77obj_dist_agent,89.61%,22.51%,152,0.65,1h25m,308s
randrot_noise_77obj_surf_agent,92.64%,25.11%,120,0.69,1h5m,227s
randrot_noise_77obj_5lms_dist_agent,88.31%,0.00%,70,1.02,32m,862s
6 changes: 3 additions & 3 deletions benchmarks/results/ycb_unsupervised.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Experiment,%Correct - 1st Epoch,% Correct - >1st Epoch,Mean Objects per Graph,Mean Graphs per Object,Run Time,Episode Run Time (s)
surf_agent_unsupervised_10distinctobj,70.00%,83.33%,1.43,1.11,20m,12s
surf_agent_unsupervised_10distinctobj_noise,70.00%,67.78%,1.19,2.11,24m,15s
Experiment,%Correct - 1st Epoch,% Correct - >1st Epoch,Mean Objects per Graph,Mean Graphs per Object,Run Time,Episode Run Time (s)
surf_agent_unsupervised_10distinctobj,70.00%,83.33%,1.43,1.11,20m,12s
surf_agent_unsupervised_10distinctobj_noise,70.00%,67.78%,1.19,2.11,25m,15s
surf_agent_unsupervised_10simobj,40.00%,86.67%,2.60,1.30,28m,17s
50 changes: 25 additions & 25 deletions docs/overview/benchmark-experiments.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,18 +51,18 @@ The following results are obtained from experiments using the 10-object subsets

| Experiment | % Correct | % Used MLH | Num Matching Steps | Rotation Error (radians) | Run Time | Episode Run Time (s) |
|---------------------------------------------|-----------|------------|--------------------|--------------------------|----------|----------------------|
| base_config_10distinctobj_dist_agent | 99.29% | 3.57% | 34 | 0.24 | 5m | 19s |
| base_config_10distinctobj_surf_agent | 100.00% | 0.00% | 28 | 0.18 | 4m | 21s |
| randrot_noise_10distinctobj_dist_agent | 99.00% | 6.00% | 48 | 0.46 | 5m | 29s |
| randrot_noise_10distinctobj_dist_on_distm | 100.00% | 2.00% | 36 | 0.25 | 4m | 28s |
| randrot_noise_10distinctobj_surf_agent | 100.00% | 1.00% | 29 | 0.36 | 5m | 33s |
| randrot_10distinctobj_surf_agent | 100.00% | 0.00% | 28 | 0.38 | 3m | 17s |
| randrot_noise_10distinctobj_5lms_dist_agent | 100.00% | 6.00% | 53 | 0.84 | 15m | 82s |
| base_10simobj_surf_agent | 93.57% | 10.00% | 78 | 0.16 | 10m | 53s |
| randrot_noise_10simobj_dist_agent | 82.00% | 42.00% | 189 | 0.59 | 16m | 119s |
| randrot_noise_10simobj_surf_agent | 91.00% | 31.00% | 162 | 0.42 | 23m | 192s |
| randomrot_rawnoise_10distinctobj_surf_agent | 72.00% | 76.00% | 16 | 1.56 | 15m | 15s |
| base_10multi_distinctobj_dist_agent | 73.57% | 40.00% | 27 | 0.73 | 1h5m | 2s |
| base_config_10distinctobj_dist_agent | 100.00% | 0.00% | 37 | 0.28 | 5m | 19s |
| base_config_10distinctobj_surf_agent | 100.00% | 0.00% | 28 | 0.23 | 4m | 18s |
| randrot_noise_10distinctobj_dist_agent | 100.00% | 3.00% | 55 | 0.39 | 5m | 33s |
| randrot_noise_10distinctobj_dist_on_distm | 99.00% | 1.00% | 38 | 0.21 | 4m | 28s |
| randrot_noise_10distinctobj_surf_agent | 100.00% | 0.00% | 29 | 0.39 | 5m | 33s |
| randrot_10distinctobj_surf_agent | 100.00% | 1.00% | 28 | 0.32 | 4m | 21s |
| randrot_noise_10distinctobj_5lms_dist_agent | 100.00% | 0.00% | 57 | 0.79 | 10m | 84s |
| base_10simobj_surf_agent | 94.29% | 10.71% | 81 | 0.24 | 10m | 50s |
| randrot_noise_10simobj_dist_agent | 84.00% | 40.00% | 237 | 0.61 | 18m | 147s |
| randrot_noise_10simobj_surf_agent | 93.00% | 33.00% | 176 | 0.52 | 24m | 205s |
| randomrot_rawnoise_10distinctobj_surf_agent | 68.00% | 78.00% | 16 | 1.78 | 17m | 15s |
| base_10multi_distinctobj_dist_agent | 79.29% | 10.71% | 31 | 0.35 | 43m | 1s |

## Longer Experiments With all 77 YCB Objects

Expand All @@ -75,11 +75,11 @@ The following results are obtained from experiments on the entire YCB dataset (7

| Experiment | % Correct | % Used MLH | Num Matching Steps | Rotation Error (radians) | Run Time | Episode Run Time (s) |
|-------------------------------------|-----------|------------|--------------------|--------------------------|----------|----------------------|
| base_77obj_dist_agent | 93.07% | 13.85% | 86 | 0.32 | 56m | 178s |
| base_77obj_surf_agent | 98.70% | 6.49% | 56 | 0.16 | 35m | 104s |
| randrot_noise_77obj_dist_agent | 87.45% | 30.74% | 149 | 0.70 | 1h17m | 264s |
| randrot_noise_77obj_surf_agent | 96.54% | 18.61% | 103 | 0.58 | 1h3m | 225s |
| randrot_noise_77obj_5lms_dist_agent | 84.42% | 9.09% | 66 | 1.08 | 39m | 843s |
| base_77obj_dist_agent | 93.51% | 12.99% | 108 | 0.28 | 1h2m | 212s |
| base_77obj_surf_agent | 99.13% | 6.06% | 54 | 0.19 | 28m | 82s |
| randrot_noise_77obj_dist_agent | 89.61% | 22.51% | 152 | 0.65 | 1h25m | 308s |
| randrot_noise_77obj_surf_agent | 92.64% | 25.11% | 120 | 0.69 | 1h5m | 227s |
| randrot_noise_77obj_5lms_dist_agent | 88.31% | 0.00% | 70 | 1.02 | 32m | 862s |

### Explanation of Some of the Results

Expand Down Expand Up @@ -112,8 +112,8 @@ An object is classified as detected correctly if the detected object ID is in th
| Experiment | %Correct - 1st Epoch | % Correct - >1st Epoch | Mean Objects per Graph | Mean Graphs per Object | Run Time | Episode Run Time (s) |
|---------------------------------------------|----------------------|------------------------|------------------------|------------------------|----------|----------------------|
| surf_agent_unsupervised_10distinctobj | 70.00% | 83.33% | 1.43 | 1.11 | 20m | 12s |
| surf_agent_unsupervised_10distinctobj_noise | 70.00% | 67.78% | 1.19 | 2.11 | 24m | 15s |
| surf_agent_unsupervised_10simobj | 40.00% | 86.67% | 2.60 | 1.30 | 28m | 17s |
| surf_agent_unsupervised_10distinctobj_noise | 70.00% | 67.78% | 1.19 | 2.11 | 25m | 15s |
| surf_agent_unsupervised_10simobj | 40.00% | 86.67% | 2.60 | 1.30 | 28m | 17s |16s |
scottcanoe marked this conversation as resolved.
Show resolved Hide resolved

To obtain these results use `print_unsupervised_stats(train_stats, epoch_len=10)` (wandb logging is currently not written for unsupervised stats). Unsupervised, continual learning can, by definition, not be parallelized across epochs. Therefore these experiments were run without multiprocessing on the laptop (running on cloud CPUs works as well but since these are slower without parallelization these were run on the laptop).

Expand Down Expand Up @@ -159,12 +159,12 @@ See the [monty_lab project folder](https://github.com/thousandbrainsproject/mont

| Experiment | % Correct | % Used MLH | Num Matching Steps | [Rotation Error (radians)] | Run Time | Episode Run Time (s) |
|---------------------------------------------|-----------|------------|--------------------|----------------------------|----------|----------------------|
| randrot_noise_sim_on_scan_monty_world | 80.00% | 80.83% | 412 | 0.86 | 52m | 24s |
| world_image_on_scanned_model | 72.92% | 83.33% | 442 | 2.15 | 18m | 21s |
| dark_world_image_on_scanned_model | 35.42% | 83.33% | 430 | 1.81 | 16m | 20s |
| bright_world_image_on_scanned_model | 43.75% | 79.17% | 428 | 1.87 | 18m | 22s |
| hand_intrusion_world_image_on_scanned_model | 39.58% | 56.25% | 344 | 2.00 | 8m | 10s |
| multi_object_world_image_on_scanned_model | 43.75% | 52.08% | 344 | 1.90 | 11m | 14s |
| randrot_noise_sim_on_scan_monty_world | 80.00% | 80.83% | 412 | 0.86 | 39m | 18s |
| world_image_on_scanned_model | 72.92% | 83.33% | 442 | 2.15 | 9m | 11s |
| dark_world_image_on_scanned_model | 35.42% | 83.33% | 430 | 1.81 | 7m | 9s |
| bright_world_image_on_scanned_model | 43.75% | 79.17% | 428 | 1.87 | 10m | 11s |
| hand_intrusion_world_image_on_scanned_model | 39.58% | 56.25% | 344 | 2.00 | 6m | 7s |
| multi_object_world_image_on_scanned_model | 43.75% | 52.08% | 344 | 1.90 | 7m | 8s |

**Note that rotation errors are meaningless since no ground truth rotation is provided**

Expand Down
Loading
Loading