Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KeyError: 'AlexNet' - CIFAR (Corruptions) #194

Open
OehriSven opened this issue Sep 9, 2024 · 0 comments
Open

KeyError: 'AlexNet' - CIFAR (Corruptions) #194

OehriSven opened this issue Sep 9, 2024 · 0 comments

Comments

@OehriSven
Copy link

Problem:
When running the corruptions benchmark on either of the CIFAR datasets I encounter the following error:

Traceback (most recent call last):
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 191, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 234, in pandas._libs.index.IndexEngine._get_loc_duplicates
  File "index.pyx", line 242, in pandas._libs.index.IndexEngine._maybe_get_bool_indexer
  File "index.pyx", line 134, in pandas._libs.index._unpack_bool_indexer
KeyError: 'AlexNet'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/labor/Schreibtisch/robustbench/claim.py", line 18, in <module>
    clean_acc, robust_acc = benchmark(
  File "/home/labor/Schreibtisch/robustbench/robustbench/eval.py", line 130, in benchmark
    adv_accuracy, adv_mce = corruptions_evaluation(
  File "/home/labor/Schreibtisch/robustbench/robustbench/eval.py", line 211, in corruptions_evaluation
    alexnet_accs_corruption = full_results.loc['AlexNet']
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1191, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1431, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1381, in _get_label
    return self.obj.xs(label, axis=axis)
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/generic.py", line 4301, in xs
    loc = index.get_loc(key)
  File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: 'AlexNet'

This is a result of this commit where adv_mce was introduced for ImageNet in eval.py. Due to the missing results for AlexNet in the unaggregated_results.csv of CIFAR-10 and CIFAR-100 corruptions benchmark a KeyError is raised.

Possible Solution:
As suggested by Hendrycks et al. themselves "any baseline will do (even a baseline with 100% error rates, corresponding to an average of CEs)" for calculating the mCE. Hence, the easiest solution would be to insert a dummy entry for AlexNet in the unaggregated_results.csv of CIFAR-10 and CIFAR-100 corruptions with 0.0 for each corruption and severity. The resulting adv_mce then simply corresponds to 1 - adv_accuracy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant