Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add cifar10 resnet benchmark #10

Merged
merged 13 commits into from
Jun 15, 2021
Merged

Conversation

tcwangshiqi-columbia
Copy link
Contributor

Please kindly find detailed benchmark descriptions including resnet models information and commands for generating random properties in Readme.

@huanzhang12
Copy link
Contributor

@stanleybak Note that we enlarged the ResNet-4B model to make it around 14K neurons, so we now have a relatively large model (which is also scored) in our benchmark suite (for addressing the issue of lacking large models). To compensate the difficulty caused by the larger network, we reduce epsilon for the ResNet-4B model from 2/255 to 1/255. It actually becomes easier than the previous model (CROWN verified accuracy is higher). if you feel we should stick to the previous smaller model, we can switch it back as well.

BTW, I just found that some PGD attack accuracy numbers reported on the README are outdated. I will update them tonight so don't merge this right now. But all the models and specifications are ready for your review.

@huanzhang12
Copy link
Contributor

@stanleybak I further cleaned up the code and also made the PGD based property generation script usable on CPUs (it was depending on a GPU). I have made necessary changes and the PR is ready for your final review.

@stanleybak
Copy link
Owner

please rename your <shortname>_instances.csv file to be cifar10_resnet_instances.csv

@huanzhang12
Copy link
Contributor

please rename your <shortname>_instances.csv file to be cifar10_resnet_instances.csv

Done.

@stanleybak
Copy link
Owner

you also want to do this in the generation script

@huanzhang12
Copy link
Contributor

you also want to do this in the generation script

Done, sorry for forgetting to change it in the generation script.

@stanleybak stanleybak merged commit 26cd0f6 into stanleybak:main Jun 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants