New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[URGENT] Cannot check compliance for MaskRCNN #6

Open

nvpstr opened this issue Nov 8, 2018 · 0 comments

Contributor

nvpstr commented Nov 8, 2018

MaskRCNN is supposed to reach 2 accuracy values (bbox and mask),
https://github.com/mlperf/policies/blob/master/training_rules.adoc#3-benchmarks
Compliance checker is not ready for that.

If we log 2x eval_accuracy and eval_taget then we cannot distinguish which is for bbox and which is for mask. Additionally following will fail from v0.5.0_common.yaml

TAGS_COUNT_SAME:
- eval_start
- eval_size
- eval_target
- eval_accuracy
- eval_stop

If (what is preferable) we will log 2 values in eval_accuracy and eval_target (e.g. value={"bbox": bbox, "mask":mask} ) then mlp_compliance.py scirpt will fail in get_model_accuracy()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment