Illegal instruction on Sherlock

[tahorst]

I was trying to run some sims on Sherlock with qlaunch and ran into a cryptic error message:

Fatal Python error: Illegal instruction

Thread 0x00007fd453fff700 (most recent call first):
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/pymongo/periodic_executor.py", line 131 in _run
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 870 in run
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007fd460d5a700 (most recent call first):
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/pymongo/periodic_executor.py", line 131 in _run
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 870 in run
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007fd461590700 (most recent call first):
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/pymongo/periodic_executor.py", line 131 in _run
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 870 in run
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007fd461d91700 (most recent call first):
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/pymongo/periodic_executor.py", line 131 in _run
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 870 in run
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/groups/mcovert/pyenv/versions/3.8.5/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007fd475c54740 (most recent call first):
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/llvmlite/binding/executionengine.py", line 306 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/llvmlite/binding/__init__.py", line 5 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/numba/core/config.py", line 16 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/numba/__init__.py", line 14 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/home/users/thorst/wcEcoli/wholecell/utils/build_ode.py", line 8 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
  File "/home/users/thorst/wcEcoli/reconstruction/ecoli/dataclasses/process/equilibrium.py", line 15 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/home/users/thorst/wcEcoli/reconstruction/ecoli/dataclasses/process/process.py", line 9 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/home/users/thorst/wcEcoli/reconstruction/ecoli/simulation_data.py", line 23 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/home/users/thorst/wcEcoli/reconstruction/ecoli/fit_sim_data_1.py", line 25 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/home/users/thorst/wcEcoli/wholecell/fireworks/firetasks/fitSimData.py", line 11 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/home/users/thorst/wcEcoli/wholecell/fireworks/firetasks/__init__.py", line 8 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/utilities/fw_serializers.py", line 338 in load_object
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/utilities/fw_serializers.py", line 117 in _recursive_load
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/utilities/fw_serializers.py", line 125 in <listcomp>
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/utilities/fw_serializers.py", line 125 in _recursive_load
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/utilities/fw_serializers.py", line 122 in <dictcomp>
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/utilities/fw_serializers.py", line 122 in _recursive_load
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/utilities/fw_serializers.py", line 161 in <dictcomp>
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/utilities/fw_serializers.py", line 161 in _decorator
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/core/launchpad.py", line 555 in get_fw_by_id
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/core/launchpad.py", line 1221 in _get_a_fw_to_run
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/core/launchpad.py", line 924 in run_exists
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/queue/queue_launcher.py", line 212 in rapidfire
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/scripts/qlaunch_run.py", line 66 in do_launch
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/scripts/qlaunch_run.py", line 224 in qlaunch
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/bin/qlaunch", line 8 in <module>
Illegal instruction

It also happened when trying to run fw_queue.py with a very similar error message. It looks like it is coming from numba because I tried import numba from ipython and receive the illegal instruction message and ipython exits.

I was on sh01-ln01 which is an old node from Sherlock 1. It works fine from a compute node (sh03-02n15). My guess is some compiled piece of code is not playing nicely with the old hardware on sh01 but wanted to make a note of it in case it pops up again.