Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle CPU or GPU via flag? right now using Delta on CPU breaks if CuPy is installed #41

Open
lastephey opened this issue May 18, 2021 · 0 comments

Comments

@lastephey
Copy link
Contributor

Delta right now imports CuPy whether it's running on the CPU or GPU. This is a problem when running on the CPU but in the Delta container where CuPy is installed:

stephey@nid00218:/global/cscratch1/sd/stephey/delta/delta> OMP_NUM_THREADS=16 srun -n 16 -c 2 --cpu-bind=cores shifter python3 -m mpi4py.futures processor.py --config configs/hackathon_test.json --transport transport_tx  --num_ranks_preprocess=1 --num_ranks_analysis=16 --num_queue_threads=1 --run_id=test_25259_GT
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/cupy/__init__.py", line 20, in <module>
    from cupy import core  # NOQA
  File "/usr/local/lib/python3.8/dist-packages/cupy/core/__init__.py", line 1, in <module>
    from cupy.core import core  # NOQA
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/mpi4py/futures/__main__.py", line 72, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/mpi4py/futures/__main__.py", line 60, in main
    run_command_line()
  File "/usr/local/lib/python3.8/dist-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/usr/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "processor.py", line 28, in <module>
    from analysis.task_list import tasklist
  File "/global/cscratch1/sd/stephey/delta/delta/analysis/task_list.py", line 6, in <module>
    from analysis.helpers import get_analysis_task
  File "/global/cscratch1/sd/stephey/delta/delta/analysis/helpers.py", line 10, in <module>
    from analysis.task_spectral_cu import task_coherence_cu, task_crosscorr_cu, task_crossphase_cu, task_crosspower_cu
  File "/global/cscratch1/sd/stephey/delta/delta/analysis/task_spectral_cu.py", line 4, in <module>
    from analysis.kernels_spectral_cu import kernel_crosscorr_cu, kernel_coherence_cu, kernel_crosspower_cu, kernel_crossphase_cu
  File "/global/cscratch1/sd/stephey/delta/delta/analysis/kernels_spectral_cu.py", line 7, in <module>
    import cupy as cp
  File "/usr/local/lib/python3.8/dist-packages/cupy/__init__.py", line 41, in <module>
    raise ImportError(_msg) from e
ImportError: CuPy is not correctly installed.

I suggest that Delta have an --arch or --mode flag that a user can set to CPU or GPU depending on what they need. Then Delta could handle the imports accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant