Unequal consumption of Resources among all GPUs in multi-node execution

[intelligi123]

Hi,

I am running a quantum volume circuit on a multi-node environment where there are two nodes and each has two GPUs.

I have kept total 4 processes and when I look at resource consumption at each GPU, its not uniform. Each node has one GPU which is consumed most and other almost stayed idle (can be seen with RAM usage and GPU Utility). Below is output of nvidia-smi on both nodes:

 NVIDIA-SMI 535.171.04             Driver Version: 535.171.04   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A40                     Off | 00000000:03:00.0 Off |                    0 |
|  0%   40C    P0             161W / 300W |  33511MiB / 46068MiB |    100%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A40                     Off | 00000000:03:01.0 Off |                    0 |
|  0%   33C    P0              72W / 300W |    547MiB / 46068MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1338      G   /usr/lib/xorg/Xorg                            4MiB |
|    0   N/A  N/A      6118      C   ...nda/envs/cuquantum-23.10/bin/python    16742MiB |
|    0   N/A  N/A      6119      C   ...nda/envs/cuquantum-23.10/bin/python    16742MiB |
|    1   N/A  N/A      1338      G   /usr/lib/xorg/Xorg                            4MiB |
|    1   N/A  N/A      6118      C   ...nda/envs/cuquantum-23.10/bin/python      260MiB |
|    1   N/A  N/A      6119      C   ...nda/envs/cuquantum-23.10/bin/python      260MiB |
+---------------------------------------------------------------------------------------+

and this one is second node

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.171.04             Driver Version: 535.171.04   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A40                     Off | 00000000:03:00.0 Off |                    0 |
|  0%   36C    P0              74W / 300W |    547MiB / 46068MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A40                     Off | 00000000:03:01.0 Off |                    0 |
|  0%   42C    P0             138W / 300W |  33511MiB / 46068MiB |    100%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1330      G   /usr/lib/xorg/Xorg                            4MiB |
|    0   N/A  N/A      6255      C   ...nda/envs/cuquantum-23.10/bin/python      260MiB |
|    0   N/A  N/A      6256      C   ...nda/envs/cuquantum-23.10/bin/python      260MiB |
|    1   N/A  N/A      1330      G   /usr/lib/xorg/Xorg                            4MiB |
|    1   N/A  N/A      6255      C   ...nda/envs/cuquantum-23.10/bin/python    16742MiB |
|    1   N/A  N/A      6256      C   ...nda/envs/cuquantum-23.10/bin/python    16742MiB |
+---------------------------------------------------------------------------------------+

The command I am executing along with code is below:
/usr/local/openmpi/bin/mpirun --map-by node -x UCX_TLS=all -x UCX_NET_DEVICES=ens34 -x UCX_LOG_LEVEL=info --hostfile hostfile -n 4 /home/cuquantum/conda/envs/cuquantum-23.10/bin/python quantum_vol.py --nbits 33

Quantum Volume Generation

from qiskit import QuantumCircuit, transpile
from qiskit import Aer
from mpi4py import MPI
from qiskit.circuit.library import QuantumVolume
import argparse


def create_vol_circuit(n_qubits,depth):
    circ = transpile(QuantumVolume(n_qubits, depth, seed = 0))
    return circ


def run(n_qubits, depth, precision, use_cusvaer):
    simulator = Aer.get_backend('aer_simulator_statevector')
    simulator.set_option('cusvaer_enable', use_cusvaer)
    simulator.set_option('precision', precision)
    circuit = create_vol_circuit(n_qubits, depth)
    circuit.measure_all()
    circuit = transpile(circuit, simulator)
    job = simulator.run(circuit)
    result = job.result()

    if MPI.COMM_WORLD.Get_rank() == 0:
        print(f'precision: {precision}')
        print(result.get_counts())
        print(f'backend: {result.backend_name}')


parser = argparse.ArgumentParser(description="Qiskit volume.")
parser.add_argument('--nbits', type=int, default=20, help='the number of qubits')
parser.add_argument('--depth', type=int, default=20, help='the depth of quantum volume circuit')
parser.add_argument('--precision', type=str, default='single', choices=['single', 'double'], help='numerical precision')
parser.add_argument('--disable-cusvaer', default=False, action='store_true', help='disable cusvaer')

args = parser.parse_args()

run(args.nbits, args.depth, args.precision, not args.disable_cusvaer)

[ymagchi]

Hi @intelligi123 ,
I see that MPI process ranks are mapped by node. Would it be possible to remove that option?
cusvaer will assign GPUs by the following scheme: https://docs.nvidia.com/cuda/cuquantum/latest/appliance/cusvaer.html#distributed-state-vector-simulation, assuming that the number of adjacent ranks in MPI_COMM_WORLD on the same host is maximized.

With the mpirun command above, the device index mapping will be as follows:
rank 0 (node 0) -> local device id 0 (= 0 % 2)
rank 1 (node 1) -> local device id 1 (= 1 % 2)
rank 2 (node 0) -> local device id 0 (= 2 % 2)
rank 3 (node 1) -> local device id 1 (= 3 % 2)
Therefore, node 0 device 1, node 1 device 0 will be unused.

[intelligi123]

Creation of substatevectors is failing when I remove --map-by option. I also tried with --map-by ppr:2:node as I want 2 processes on each node (per GPU) so total of 4 processes to use all GPUs to full extent but it also cause errors.

Following happens when I remove --map-by altogether.

(cuquantum-23.10) cuquantum@ubuntu-6-None:~$ /usr/local/openmpi/bin/mpirun  -x UCX_TLS=all -x UCX_NET_DEVICES=ens34 -x UCX_LOG_LEVEL=info --hostfile hostfile -n 4 /home/cuquantum/conda/envs/cuquantum-23.10/bin/python qiskit_ghz.py --nbits 33
[ubuntu-6-None:00730] WARNING: local probe returned unhandled shell:unknown assuming bash
Warning: Permanently added '[192.168.0.34]:2223' (RSA) to the list of known hosts.
[1717656164.914764] [ubuntu-6-None:736  :0]     ucp_context.c:1849 UCX  INFO  Version 1.13.1 (loaded from /usr/local/ucx/lib/libucp.so.0)
[1717656164.914763] [ubuntu-6-None:735  :0]     ucp_context.c:1849 UCX  INFO  Version 1.13.1 (loaded from /usr/local/ucx/lib/libucp.so.0)
[1717656164.793152] [ubuntu-5-None:739  :0]     ucp_context.c:1849 UCX  INFO  Version 1.13.1 (loaded from /usr/local/ucx/lib/libucp.so.0)
[1717656164.800911] [ubuntu-5-None:740  :0]     ucp_context.c:1849 UCX  INFO  Version 1.13.1 (loaded from /usr/local/ucx/lib/libucp.so.0)
[1717656165.095022] [ubuntu-5-None:739  :0]       rdmacm_cm.c:928  UCX  DIAG  rdma_create_event_channel failed: No such device
[1717656165.095054] [ubuntu-5-None:739  :0]      ucp_worker.c:1462 UCX  DIAG  failed to open CM on component rdmacm with status Input/output error
[1717656165.095332] [ubuntu-5-None:739  :0]          parser.c:1916 UCX  INFO  UCX_* env variables: UCX_TLS=all UCX_NET_DEVICES=ens34 UCX_LOG_LEVEL=info
[1717656165.099708] [ubuntu-5-None:740  :0]       rdmacm_cm.c:928  UCX  DIAG  rdma_create_event_channel failed: No such device
[1717656165.099796] [ubuntu-5-None:740  :0]      ucp_worker.c:1462 UCX  DIAG  failed to open CM on component rdmacm with status Input/output error
[1717656165.100134] [ubuntu-5-None:740  :0]          parser.c:1916 UCX  INFO  UCX_* env variables: UCX_TLS=all UCX_NET_DEVICES=ens34 UCX_LOG_LEVEL=info
[1717656165.232302] [ubuntu-6-None:736  :0]       rdmacm_cm.c:928  UCX  DIAG  rdma_create_event_channel failed: No such device
[1717656165.232321] [ubuntu-6-None:736  :0]      ucp_worker.c:1462 UCX  DIAG  failed to open CM on component rdmacm with status Input/output error
[1717656165.232714] [ubuntu-6-None:736  :0]          parser.c:1908 UCX  WARN  unused env variable: UCX_IB_PCI_RELAXED_ORDERING (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
[1717656165.232725] [ubuntu-6-None:736  :0]          parser.c:1916 UCX  INFO  UCX_* env variables: UCX_TLS=all UCX_NET_DEVICES=ens34 UCX_LOG_LEVEL=info UCX_MEMTYPE_CACHE=n UCX_MAX_RNDV_RAILS=1
[1717656165.239310] [ubuntu-6-None:735  :0]       rdmacm_cm.c:928  UCX  DIAG  rdma_create_event_channel failed: No such device
[1717656165.239508] [ubuntu-6-None:735  :0]      ucp_worker.c:1462 UCX  DIAG  failed to open CM on component rdmacm with status Input/output error
[1717656165.240001] [ubuntu-6-None:735  :0]          parser.c:1908 UCX  WARN  unused env variable: UCX_IB_PCI_RELAXED_ORDERING (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
[1717656165.240011] [ubuntu-6-None:735  :0]          parser.c:1916 UCX  INFO  UCX_* env variables: UCX_TLS=all UCX_NET_DEVICES=ens34 UCX_LOG_LEVEL=info UCX_MEMTYPE_CACHE=n UCX_MAX_RNDV_RAILS=1
[1717656165.119596] [ubuntu-5-None:739  :0]      ucp_worker.c:1763 UCX  INFO    ep_cfg[1]: tag(self/memory0 cma/memory cuda_copy/cuda)
[1717656165.119688] [ubuntu-5-None:739  :0]      ucp_worker.c:1763 UCX  INFO    ep_cfg[2]: tag(sysv/memory cma/memory cuda_ipc/cuda)
[1717656165.119821] [ubuntu-5-None:740  :0]      ucp_worker.c:1763 UCX  INFO    ep_cfg[1]: tag(self/memory0 cma/memory cuda_copy/cuda)
[1717656165.119948] [ubuntu-5-None:740  :0]      ucp_worker.c:1763 UCX  INFO    ep_cfg[2]: tag(sysv/memory cma/memory cuda_ipc/cuda)
[1717656165.241897] [ubuntu-6-None:736  :0]      ucp_worker.c:1763 UCX  INFO    ep_cfg[1]: tag(self/memory0 cma/memory cuda_copy/cuda)
[1717656165.242144] [ubuntu-6-None:736  :0]      ucp_worker.c:1763 UCX  INFO    ep_cfg[2]: tag(sysv/memory cma/memory cuda_ipc/cuda)
[1717656165.243290] [ubuntu-6-None:735  :0]      ucp_worker.c:1763 UCX  INFO    ep_cfg[1]: tag(self/memory0 cma/memory cuda_copy/cuda)
[1717656165.244897] [ubuntu-6-None:735  :0]      ucp_worker.c:1763 UCX  INFO    ep_cfg[2]: tag(sysv/memory cma/memory cuda_ipc/cuda)
[1717656167.214916] [ubuntu-6-None:736  :1]      ucp_worker.c:1763 UCX  INFO    ep_cfg[3]: tag(tcp/ens34)
[1717656167.214969] [ubuntu-6-None:735  :1]      ucp_worker.c:1763 UCX  INFO    ep_cfg[3]: tag(tcp/ens34)
[1717656167.140638] [ubuntu-5-None:739  :1]      ucp_worker.c:1763 UCX  INFO    ep_cfg[3]: tag(tcp/ens34)
[1717656167.140775] [ubuntu-5-None:740  :1]      ucp_worker.c:1763 UCX  INFO    ep_cfg[3]: tag(tcp/ens34)
[0/4] notice: cudaErrorPeerAccessUnsupported, failed to get device memory pointer by CUDA IPC
[2/4] notice: cudaErrorPeerAccessUnsupported, failed to get device memory pointer by CUDA IPC
[3/4] notice: cudaErrorPeerAccessUnsupported, failed to get device memory pointer by CUDA IPC
[2/4] error: failed to create sub statevector
[3/4] error: failed to create sub statevector
[0/4] error: failed to create sub statevector
Traceback (most recent call last):
  File "/home/cuquantum/qiskit_ghz.py", line 50, in <module>
    run(args.nbits, args.precision, not args.disable_cusvaer)
  File "/home/cuquantum/qiskit_ghz.py", line 35, in run
    result = job.result()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/jobs/utils.py", line 42, in _wrapper
    return func(self, *args, **kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/jobs/aerjob.py", line 114, in result
[1/4] notice: cudaErrorPeerAccessUnsupported, failed to get device memory pointer by CUDA IPC
Traceback (most recent call last):
  File "/home/cuquantum/qiskit_ghz.py", line 50, in <module>
[1/4] error: failed to create sub statevector
    run(args.nbits, args.precision, not args.disable_cusvaer)
  File "/home/cuquantum/qiskit_ghz.py", line 35, in run
    result = job.result()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/jobs/utils.py", line 42, in _wrapper
Traceback (most recent call last):
  File "/home/cuquantum/qiskit_ghz.py", line 50, in <module>
    run(args.nbits, args.precision, not args.disable_cusvaer)
  File "/home/cuquantum/qiskit_ghz.py", line 35, in run
    result = job.result()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/jobs/utils.py", line 42, in _wrapper
    return func(self, *args, **kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/jobs/aerjob.py", line 114, in result
    return self._future.result(timeout=timeout)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/backends/aerbackend.py", line 404, in _execute_qobj_job
    output = self._cusvaer_execute_qobj(qobj)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/backends/aer_simulator.py", line 1059, in _cusvaer_execute_qobj
    cusvaer_output = cusvaer_sim._run(qobj, cusvaer_options._fields)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 311, in _run
    job = BasicAerJob(self, job_id, self._run_job(job_id, qobj))
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 334, in _run_job
    result_list.append(self.run_experiment(experiment))
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 442, in run_experiment
    self._sub_statevector = self._svops.create_sub_statevector(
  File "cusvaer/binding/backend.pyx", line 636, in cusvaer.binding.backend.StatevectorOperations.create_sub_statevector
  File "cusvaer/binding/backend.pyx", line 267, in cusvaer.binding.backend.statevector_operations_create_sub_statevector
  File "cusvaer/binding/backend.pyx", line 107, in cusvaer.binding.backend.check_status
RuntimeError: CUSTATEVEC_EXECUTION_FAILED Error occurred in native extension
    return func(self, *args, **kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/jobs/aerjob.py", line 114, in result
    return self._future.result(timeout=timeout)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/backends/aerbackend.py", line 404, in _execute_qobj_job
    output = self._cusvaer_execute_qobj(qobj)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/backends/aer_simulator.py", line 1059, in _cusvaer_execute_qobj
    cusvaer_output = cusvaer_sim._run(qobj, cusvaer_options._fields)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 311, in _run
    job = BasicAerJob(self, job_id, self._run_job(job_id, qobj))
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 334, in _run_job
    result_list.append(self.run_experiment(experiment))
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 442, in run_experiment
    self._sub_statevector = self._svops.create_sub_statevector(
  File "cusvaer/binding/backend.pyx", line 636, in cusvaer.binding.backend.StatevectorOperations.create_sub_statevector
  File "cusvaer/binding/backend.pyx", line 267, in cusvaer.binding.backend.statevector_operations_create_sub_statevector
  File "cusvaer/binding/backend.pyx", line 107, in cusvaer.binding.backend.check_status
RuntimeError: CUSTATEVEC_EXECUTION_FAILED Error occurred in native extension
Traceback (most recent call last):
  File "/home/cuquantum/qiskit_ghz.py", line 50, in <module>
    run(args.nbits, args.precision, not args.disable_cusvaer)
  File "/home/cuquantum/qiskit_ghz.py", line 35, in run
    result = job.result()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/jobs/utils.py", line 42, in _wrapper
    return func(self, *args, **kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/jobs/aerjob.py", line 114, in result
    return self._future.result(timeout=timeout)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/backends/aerbackend.py", line 404, in _execute_qobj_job
    output = self._cusvaer_execute_qobj(qobj)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/backends/aer_simulator.py", line 1059, in _cusvaer_execute_qobj
    cusvaer_output = cusvaer_sim._run(qobj, cusvaer_options._fields)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 311, in _run
    job = BasicAerJob(self, job_id, self._run_job(job_id, qobj))
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 334, in _run_job
    result_list.append(self.run_experiment(experiment))
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 442, in run_experiment
    self._sub_statevector = self._svops.create_sub_statevector(
  File "cusvaer/binding/backend.pyx", line 636, in cusvaer.binding.backend.StatevectorOperations.create_sub_statevector
  File "cusvaer/binding/backend.pyx", line 267, in cusvaer.binding.backend.statevector_operations_create_sub_statevector
  File "cusvaer/binding/backend.pyx", line 107, in cusvaer.binding.backend.check_status
RuntimeError: CUSTATEVEC_EXECUTION_FAILED Error occurred in native extension
    return self._future.result(timeout=timeout)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/backends/aerbackend.py", line 404, in _execute_qobj_job
    output = self._cusvaer_execute_qobj(qobj)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/qiskit_aer/backends/aer_simulator.py", line 1059, in _cusvaer_execute_qobj
    cusvaer_output = cusvaer_sim._run(qobj, cusvaer_options._fields)
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 311, in _run
    job = BasicAerJob(self, job_id, self._run_job(job_id, qobj))
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 334, in _run_job
    result_list.append(self.run_experiment(experiment))
  File "/home/cuquantum/conda/envs/cuquantum-23.10/lib/python3.10/site-packages/cusvaer/backends/statevector_simulator.py", line 442, in run_experiment
    self._sub_statevector = self._svops.create_sub_statevector(
  File "cusvaer/binding/backend.pyx", line 636, in cusvaer.binding.backend.StatevectorOperations.create_sub_statevector
  File "cusvaer/binding/backend.pyx", line 267, in cusvaer.binding.backend.statevector_operations_create_sub_statevector
  File "cusvaer/binding/backend.pyx", line 107, in cusvaer.binding.backend.check_status
RuntimeError: CUSTATEVEC_EXECUTION_FAILED Error occurred in native extension
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[52965,1],1]
  Exit code:    1

[ymagchi]

Hi @intelligi123,
The output logs show that peer access has failed.
Could you please share the options you added into simulator = Aer.get_backend('aer_simulator_statevector') (or share your full script)? In addition, could you please share the device network topology on each host by nvidia-smi topo -m?

[intelligi123]

This might seem strange but the next day when I power on both nodes, things worked perfectly fine with all 4 GPUs utilizing to fullest simulating 34 qubits from which i think I cannot go farther as it states following error for --nbits 35

qiskit.providers.basicaer.exceptions.BasicAerError: 'Cannot allocate 35 qubit state vector. The max number of qubits is 34, 4 process(es) / 32 qubits/GPU.'

[ymagchi]

Thank you for the update, noted.

qiskit.providers.basicaer.exceptions.BasicAerError: 'Cannot allocate 35 qubit state vector. The max number of qubits is 34, 4 process(es) / 32 qubits/GPU.'

This is expected output as A40 has 48 GB device memory, and the max number of qubits per A40 is 32 (8 Bytes * 2^32 = 32 GB, in single precision.)