distributed_reset_configuration failed: python: distributed_interfaces/cutensornet_distributed_interface_mpi.c:44: unpackMpiCommunicator: Assertion `sizeof(MPI_Comm) == comm->commSize' failed.

[koichi-tsujino]

Under the following setup.

Hardware: INSPUR NF5488M5 (V100 version)
environments:
Ubuntu 22.04.1 LTS
Python 3.9.15
Nvidia driver: 525.60.13
cuda_12.0.r12.0
mpich-4.0.3
mpi4py 3.1.4
cuquantum 22.11.0

When I run /cuQuantum/python/samples/cutensornet/tensornet_example_mpi.py , I got. It works .

*** Printing is done only from the root process to prevent jumbled messages ***
The number of processes is 1
cuTensorNet-vers: 20000
===== root process device info ======
GPU-name: Tesla V100-SXM3-32GB
GPU-clock: 1597000
GPU-memoryClock: 958000
GPU-nSM: 80
GPU-major: 7
GPU-minor: 0
========================
Include headers and define data types.
Define network, modes, and extents.
Initialize the cuTensorNet library and create a network descriptor.
Process 0 has the path with the lowest FLOP count 4299161600.0.
Find an optimized contraction path with cuTensorNet optimizer.
Allocate workspace.
Create a contraction plan for cuTENSOR and optionally auto-tune it.
Contract the network, each slice uses the same contraction plan.
Check cuTensorNet result against that of cupy.einsum().
num_slices: 1
0.8309440016746521 ms / slice
5173.82831013358 GFLOPS/s
Free resource and exit.

But when I run /cuQuantum/python/samples/cutensornet/tensornet_example_mpi_auto.py I got the following error.

*** Printing is done only from the root process to prevent jumbled messages ***
The number of processes is 1
cuTensorNet-vers: 20000
===== root process device info ======
GPU-name: Tesla V100-SXM3-32GB
GPU-clock: 1597000
GPU-memoryClock: 958000
GPU-nSM: 80
GPU-major: 7
GPU-minor: 0
========================
Include headers and define data types.
Define network, modes, and extents.
Initialize the cuTensorNet library and create a network descriptor.
python: distributed_interfaces/cutensornet_distributed_interface_mpi.c:44: unpackMpiCommunicator: Assertion `sizeof(MPI_Comm) == comm->commSize' failed.
[suneo:06467] *** Process received signal ***
[suneo:06467] Signal: Aborted (6)
[suneo:06467] Signal code:  (-6)
[suneo:06467] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f55bbd22520]
[suneo:06467] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f55bbd76a7c]
[suneo:06467] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f55bbd22476]
[suneo:06467] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f55bbd087f3]
[suneo:06467] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x2871b)[0x7f55bbd0871b]
[suneo:06467] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x39e96)[0x7f55bbd19e96]
[suneo:06467] [ 6] /home/tsujino/anaconda3/envs/cu/lib/libcutensornet_distributed_interface_mpi.so(+0x123c)[0x7f553de1223c]
[suneo:06467] [ 7] /home/tsujino/anaconda3/envs/cu/lib/libcutensornet_distributed_interface_mpi.so(cutensornetMpiCommRank+0x23)[0x7f553de122ae]
[suneo:06467] [ 8] /home/tsujino/anaconda3/envs/cu/lib/python3.9/site-packages/cuquantum/cutensornet/../../../../libcutensornet.so.2(+0x105462)[0x7f554c705462]
[suneo:06467] [ 9] /home/tsujino/anaconda3/envs/cu/lib/python3.9/site-packages/cuquantum/cutensornet/../../../../libcutensornet.so.2(+0x1056bd)[0x7f554c7056bd]
[suneo:06467] [10] /home/tsujino/anaconda3/envs/cu/lib/python3.9/site-packages/cuquantum/cutensornet/../../../../libcutensornet.so.2(+0x1058ed)[0x7f554c7058ed]
[suneo:06467] [11] /home/tsujino/anaconda3/envs/cu/lib/python3.9/site-packages/cuquantum/cutensornet/../../../../libcutensornet.so.2(cutensornetDistributedResetConfiguration+0xd3)[0x7f554c703633]
[suneo:06467] [12] /home/tsujino/anaconda3/envs/cu/lib/python3.9/site-packages/cuquantum/cutensornet/cutensornet.cpython-39-x86_64-linux-gnu.so(+0x26063)[0x7f554e65c063]
[suneo:06467] [13] python[0x507457]
[suneo:06467] [14] python(_PyObject_MakeTpCall+0x2ec)[0x4f068c]
[suneo:06467] [15] python(_PyEval_EvalFrameDefault+0x525b)[0x4ec9fb]
[suneo:06467] [16] python[0x4e689a]
[suneo:06467] [17] python(_PyEval_EvalCodeWithName+0x47)[0x4e6527]
[suneo:06467] [18] python(PyEval_EvalCodeEx+0x39)[0x4e64d9]
[suneo:06467] [19] python(PyEval_EvalCode+0x1b)[0x59329b]
[suneo:06467] [20] python[0x5c0ad7]
[suneo:06467] [21] python[0x5bcb00]
[suneo:06467] [22] python[0x4566f4]
[suneo:06467] [23] python(PyRun_SimpleFileExFlags+0x1a2)[0x5b67e2]
[suneo:06467] [24] python(Py_RunMain+0x37e)[0x5b3d5e]
[suneo:06467] [25] python(Py_BytesMain+0x39)[0x587349]
[suneo:06467] [26] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f55bbd09d90]
[suneo:06467] [27] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f55bbd09e40]
[suneo:06467] [28] python[0x5871fe]
[suneo:06467] *** End of error message ***
Aborted (core dumped)

I have tried other smaples and those works.

[haidarazzam]

Dear koichi-tsujino,
Thank you very much for testing cuQuantum 22.11 and reporting the issue.
Could you please verify that the environment variables related to setting where the MPI lib is well defined as noted in the docs and if yes, is there are multiple mpi libraries installed on the system such that the wrapper is complied with one while the app is loading the other mpi lib?
Could you please to check verify that you are using openMPI for both the wrapper and the app?
Can you please also set CUTENSORNET_LOG_LEVEL=5 so we can see more details in the output.
Thanks

[DmitryLyakh]

Could you please try building and running the tensornet_example_mpi_auto C sample on your machine (samples inside https://github.com/NVIDIA/cuQuantum/tree/main/samples/cutensornet)? Before running the sample, could you please additionally check the environment variable $CUTENSORNET_COMM_LIB that is supposed to point to the libcutensornet_distributed_interface_mpi.so wrapper library.

[DmitryLyakh]

One possible reason why you observe a crash is that the MPI library linked to by the sample you are running is different from the MPI library used by the MPI wrapper libcutensornet_distributed_interface_mpi.so, in case multiple MPI libraries are present in your system. In the meantime, let me try to reproduce your issue locally ...

[DmitryLyakh]

On our local machine, the C/C++ sampler tensornet_example_mpi_auto works fine with both MPICH and OpenMPI. I would guess the issue could be related to the Python environment setup or something ...

[leofang]

@koichi-tsujino Questions:

1. I noticed you are using a Conda env. How did you install cuQuantum Python?
2. Could you post the output of conda list?
3. You're using CUDA 12 inside the conda env, was it a local copy or installed from the nvidia channel? Asking because
   1. On the conda-forge channel CUDA 12 is not ready yet
   2. As of cuQuantum 22.11 we do not yet support CUDA 12, it's planned for the next release
   3. CuPy (our main Python dependency) is not CUDA-12 ready yet
4. Do you already have a working MPI outside your Conda env? (Test this by deactivating your conda env, and then do which mpiexec)

[koichi-tsujino]

@leofang

1. how to install cuquantum
   conda install -c conda-forge cuquantum-python

2. Here is the condalist

_anaconda_depends         2022.05                  py39_0
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   1.3.0                    pypi_0    pypi
aiohttp                   3.8.3            py39h5eee18b_0
aiosignal                 1.2.0              pyhd3eb1b0_0
alabaster                 0.7.12             pyhd3eb1b0_0
anaconda                  custom                   py39_1
anaconda-client           1.11.0           py39h06a4308_0
anaconda-project          0.11.1           py39h06a4308_0
anyio                     3.5.0            py39h06a4308_0
appdirs                   1.4.4              pyhd3eb1b0_0
argon2-cffi               21.3.0             pyhd3eb1b0_0
argon2-cffi-bindings      21.2.0           py39h7f8727e_0
arrow                     1.2.3            py39h06a4308_0
astroid                   2.11.7           py39h06a4308_0
astropy                   5.1              py39h7deecbd_0
asttokens                 2.0.5              pyhd3eb1b0_0
astunparse                1.6.3                    pypi_0    pypi
async-timeout             4.0.2            py39h06a4308_0
atomicwrites              1.4.0                      py_0
attrs                     20.3.0                   pypi_0    pypi
automat                   20.2.0                     py_0
autopep8                  1.6.0              pyhd3eb1b0_1
babel                     2.9.1              pyhd3eb1b0_0
backcall                  0.2.0              pyhd3eb1b0_0
backports                 1.1                pyhd3eb1b0_0
backports.functools_lru_cache 1.6.4              pyhd3eb1b0_0
backports.tempfile        1.0                pyhd3eb1b0_1
backports.weakref         1.0.post1                  py_1
bcrypt                    3.2.0            py39h5eee18b_1
beautifulsoup4            4.11.1           py39h06a4308_0
binaryornot               0.4.4              pyhd3eb1b0_1
bitarray                  2.5.1            py39h5eee18b_0
bkcharts                  0.2              py39h06a4308_1
black                     22.6.0           py39h06a4308_0
blas                      1.0                         mkl
bleach                    4.1.0              pyhd3eb1b0_0
blosc                     1.21.0               h4ff587b_1
bokeh                     2.4.3            py39h06a4308_0
boto3                     1.24.28          py39h06a4308_0
botocore                  1.27.59          py39h06a4308_0
bottleneck                1.3.5            py39h7deecbd_0
brotli                    1.0.9                h5eee18b_7
brotli-bin                1.0.9                h5eee18b_7
brotlipy                  0.7.0           py39h27cfd23_1003
brunsli                   0.1                  h2531618_0
bzip2                     1.0.8                h7b6447c_0
c-ares                    1.18.1               h7f8727e_0
ca-certificates           2022.10.11           h06a4308_0
cachetools                4.2.2              pyhd3eb1b0_0
certifi                   2022.12.7        py39h06a4308_0
cffi                      1.15.1           py39h5eee18b_3
cfitsio                   3.470                hf0d0db6_6
chardet                   4.0.0           py39h06a4308_1003
charls                    2.2.0                h2531618_0
charset-normalizer        2.0.4              pyhd3eb1b0_0
cirq                      1.1.0                    pypi_0    pypi
cirq-aqt                  1.1.0                    pypi_0    pypi
cirq-core                 1.1.0                    pypi_0    pypi
cirq-google               1.1.0                    pypi_0    pypi
cirq-ionq                 1.1.0                    pypi_0    pypi
cirq-pasqal               1.1.0                    pypi_0    pypi
cirq-rigetti              1.1.0                    pypi_0    pypi
cirq-web                  1.1.0                    pypi_0    pypi
click                     8.0.4            py39h06a4308_0
cloudpickle               2.0.0              pyhd3eb1b0_0
clyent                    1.2.2            py39h06a4308_1
colorama                  0.4.5            py39h06a4308_0
colorcet                  3.0.1            py39h06a4308_0
conda                     22.11.1          py39h06a4308_4
conda-content-trust       0.1.3            py39h06a4308_0
conda-pack                0.6.0              pyhd3eb1b0_0
conda-package-handling    1.9.0            py39h5eee18b_1
conda-token               0.4.0              pyhd3eb1b0_0
constantly                15.1.0             pyh2b92418_0
contourpy                 1.0.5            py39hdb19cb5_0
cookiecutter              1.7.3              pyhd3eb1b0_0
cryptography              38.0.1           py39h9ce1e76_0
cssselect                 1.1.0              pyhd3eb1b0_0
cuda                      11.7.1                        0    nvidia
cuda-cccl                 11.7.91                       0    nvidia
cuda-command-line-tools   11.7.1                        0    nvidia
cuda-compiler             11.7.1                        0    nvidia
cuda-cudart               11.7.99                       0    nvidia
cuda-cudart-dev           11.7.99                       0    nvidia
cuda-cuobjdump            11.7.91                       0    nvidia
cuda-cupti                11.7.101                      0    nvidia
cuda-cuxxfilt             11.7.91                       0    nvidia
cuda-demo-suite           12.0.76                       0    nvidia
cuda-documentation        12.0.76                       0    nvidia
cuda-driver-dev           11.7.99                       0    nvidia
cuda-gdb                  12.0.90                       0    nvidia
cuda-libraries            11.7.1                        0    nvidia
cuda-libraries-dev        11.7.1                        0    nvidia
cuda-memcheck             11.8.86                       0    nvidia
cuda-nsight               12.0.78                       0    nvidia
cuda-nsight-compute       12.0.0                        0    nvidia
cuda-nvcc                 11.7.99                       0    nvidia
cuda-nvdisasm             12.0.76                       0    nvidia
cuda-nvml-dev             11.7.91                       0    nvidia
cuda-nvprof               12.0.90                       0    nvidia
cuda-nvprune              11.7.91                       0    nvidia
cuda-nvrtc                11.7.99                       0    nvidia
cuda-nvrtc-dev            11.7.99                       0    nvidia
cuda-nvtx                 11.7.91                       0    nvidia
cuda-nvvp                 12.0.90                       0    nvidia
cuda-runtime              11.7.1                        0    nvidia
cuda-sanitizer-api        12.0.90                       0    nvidia
cuda-toolkit              11.7.1                        0    nvidia
cuda-tools                11.7.1                        0    nvidia
cuda-visual-tools         11.7.1                        0    nvidia
cudatoolkit               11.3.1               h2bc3f7f_2
cupy                      11.4.0           py39hc3c280e_0    conda-forge
cuquantum-python          22.11.0          py39hcf40d7a_0    conda-forge
curl                      7.86.0               h5eee18b_0
custatevec                1.2.0                h0800d71_0    conda-forge
cutensor                  1.6.2.3              h12f7317_0    conda-forge
cutensornet               2.0.0           mpi_mpich_h2fb0270_0    conda-forge
cycler                    0.11.0             pyhd3eb1b0_0
cython                    0.29.32          py39h6a678d5_0
cytoolz                   0.12.0           py39h5eee18b_0
daal4py                   2023.0.1         py39h4cf92e1_0    conda-forge
dal                       2023.0.0         ha770c72_25396    conda-forge
dask                      2022.5.0         py39h06a4308_0
dask-core                 2022.5.0         py39h06a4308_0
dataclasses               0.8                pyh6d0b6a4_7
datashader                0.14.3           py39h06a4308_0
datashape                 0.5.4            py39h06a4308_1
dbus                      1.13.18              hb2f20db_0
debugpy                   1.5.1            py39h295c915_0
decorator                 5.1.1              pyhd3eb1b0_0
defusedxml                0.7.1              pyhd3eb1b0_0
diff-match-patch          20200713           pyhd3eb1b0_0
dill                      0.3.6            py39h06a4308_0
distributed               2022.5.0         py39h06a4308_0
docutils                  0.18.1           py39h06a4308_3
duet                      0.2.7                    pypi_0    pypi
entrypoints               0.4              py39h06a4308_0
et_xmlfile                1.1.0            py39h06a4308_0
executing                 0.8.3              pyhd3eb1b0_0
expat                     2.4.9                h6a678d5_0
fastrlock                 0.8              py39h5a03fae_3    conda-forge
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.6.0              pyhd3eb1b0_0
flake8                    4.0.1              pyhd3eb1b0_1
flask                     2.2.2            py39h06a4308_0
flatbuffers               22.12.6                  pypi_0    pypi
flit-core                 3.6.0              pyhd3eb1b0_0
fontconfig                2.14.1               h52c9d5c_1
fonttools                 4.25.0             pyhd3eb1b0_0
freetype                  2.12.1               h4a9f257_0
frozenlist                1.3.3            py39h5eee18b_0
fsspec                    2022.11.0        py39h06a4308_0
future                    0.18.2           py39h06a4308_1
gast                      0.4.0                    pypi_0    pypi
gds-tools                 1.5.0.59                      0    nvidia
gensim                    4.1.2            py39h295c915_0
giflib                    5.2.1                h7b6447c_0
glib                      2.69.1               he621ea3_2
glob2                     0.7                pyhd3eb1b0_0
gmp                       6.2.1                h295c915_3
gmpy2                     2.1.2            py39heeb90bb_0
gnutls                    3.6.15               he1e5248_0
google-api-core           1.34.0                   pypi_0    pypi
google-auth               2.6.0              pyhd3eb1b0_0
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-cloud-core         2.3.2            py39h06a4308_0
google-cloud-storage      2.6.0            py39h06a4308_0
google-crc32c             1.5.0            py39h5eee18b_0
google-pasta              0.2.0                    pypi_0    pypi
google-resumable-media    2.4.0            py39h06a4308_0
googleapis-common-protos  1.56.4           py39h06a4308_0
greenlet                  2.0.1            py39h6a678d5_0
grpcio                    1.51.1                   pypi_0    pypi
grpcio-status             1.48.2                   pypi_0    pypi
gst-plugins-base          1.14.0               h8213a91_2
gstreamer                 1.14.0               h28cd5cc_2
h11                       0.14.0                   pypi_0    pypi
h5py                      3.7.0            py39h737f45e_0
hdf5                      1.10.6               hb1b8bf9_0
heapdict                  1.0.1              pyhd3eb1b0_0
holoviews                 1.15.2           py39h06a4308_0
httpcore                  0.16.3                   pypi_0    pypi
httpx                     0.23.1                   pypi_0    pypi
hvplot                    0.8.2            py39h06a4308_0
hyperlink                 21.0.0             pyhd3eb1b0_0
icu                       58.2                 he6710b0_3
idna                      3.4              py39h06a4308_0
imagecodecs               2021.8.26        py39hf0132c2_1
imageio                   2.19.3           py39h06a4308_0
imagesize                 1.4.1            py39h06a4308_0
importlib-metadata        4.11.3           py39h06a4308_0
importlib_metadata        4.11.3               hd3eb1b0_0
incremental               21.3.0             pyhd3eb1b0_0
inflection                0.5.1            py39h06a4308_0
iniconfig                 1.1.1              pyhd3eb1b0_0
intake                    0.6.6            py39h06a4308_0
intel-openmp              2021.4.0          h06a4308_3561
intervaltree              3.1.0              pyhd3eb1b0_0
ipykernel                 6.15.2           py39h06a4308_0
ipython                   7.31.1           py39h06a4308_1
ipython_genutils          0.2.0              pyhd3eb1b0_1
ipywidgets                7.6.5              pyhd3eb1b0_1
iso8601                   1.1.0                    pypi_0    pypi
isort                     5.9.3              pyhd3eb1b0_0
itemadapter               0.3.0              pyhd3eb1b0_0
itemloaders               1.0.4              pyhd3eb1b0_1
itsdangerous              2.0.1              pyhd3eb1b0_0
jdcal                     1.4.1              pyhd3eb1b0_0
jedi                      0.18.1           py39h06a4308_1
jeepney                   0.7.1              pyhd3eb1b0_0
jellyfish                 0.9.0            py39h7f8727e_0
jinja2                    3.1.2            py39h06a4308_0
jinja2-time               0.2.0              pyhd3eb1b0_3
jmespath                  0.10.0             pyhd3eb1b0_0
joblib                    1.1.1            py39h06a4308_0
jpeg                      9e                   h7f8727e_0
jq                        1.6               h27cfd23_1000
json5                     0.9.6              pyhd3eb1b0_0
jsonschema                4.16.0           py39h06a4308_0
jupyter                   1.0.0            py39h06a4308_8
jupyter_client            7.4.7            py39h06a4308_0
jupyter_console           6.4.4            py39h06a4308_0
jupyter_core              4.11.2           py39h06a4308_0
jupyter_server            1.18.1           py39h06a4308_0
jupyterlab                3.5.0            py39h06a4308_0
jupyterlab_pygments       0.1.2                      py_0
jupyterlab_server         2.16.3           py39h06a4308_0
jupyterlab_widgets        1.0.0              pyhd3eb1b0_1
jxrlib                    1.1                  h7b6447c_2
keras                     2.11.0                   pypi_0    pypi
keyring                   23.4.0           py39h06a4308_0
kiwisolver                1.4.2            py39h295c915_0
krb5                      1.19.2               hac12032_0
lame                      3.100                h7b6447c_0
lark                      0.11.3                   pypi_0    pypi
lazy-object-proxy         1.6.0            py39h27cfd23_0
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.38                 h1181459_1
lerc                      3.0                  h295c915_0
libaec                    1.0.4                he6710b0_1
libarchive                3.6.1                hab531cd_0
libbrotlicommon           1.0.9                h5eee18b_7
libbrotlidec              1.0.9                h5eee18b_7
libbrotlienc              1.0.9                h5eee18b_7
libclang                  14.0.6                   pypi_0    pypi
libcrc32c                 1.1.2                h6a678d5_0
libcublas                 11.10.3.66                    0    nvidia
libcublas-dev             11.10.3.66                    0    nvidia
libcufft                  10.7.2.124           h4fbf590_0    nvidia
libcufft-dev              10.7.2.124           h98a8f43_0    nvidia
libcufile                 1.5.0.59                      0    nvidia
libcufile-dev             1.5.0.59                      0    nvidia
libcurand                 10.3.1.50                     0    nvidia
libcurand-dev             10.3.1.50                     0    nvidia
libcurl                   7.86.0               h91b91d3_0
libcusolver               11.4.0.1                      0    nvidia
libcusolver-dev           11.4.0.1                      0    nvidia
libcusparse               11.7.4.91                     0    nvidia
libcusparse-dev           11.7.4.91                     0    nvidia
libdeflate                1.8                  h7f8727e_5
libedit                   3.1.20221030         h5eee18b_0
libev                     4.33                 h7f8727e_1
libevent                  2.1.12               h8f2d780_0
libffi                    3.4.2                h6a678d5_6
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            7.5.0               ha8ba4b0_17
libgfortran4              7.5.0               ha8ba4b0_17
libgomp                   12.2.0              h65d4601_19    conda-forge
libiconv                  1.16                 h7f8727e_2
libidn2                   2.3.2                h7f8727e_0
liblief                   0.12.3               h6a678d5_0
libllvm10                 10.0.1               hbcb73fb_5
libllvm11                 11.1.0               h9e868ea_6
libnghttp2                1.46.0               hce63b2e_0
libnpp                    11.7.4.75                     0    nvidia
libnpp-dev                11.7.4.75                     0    nvidia
libnvjpeg                 11.8.0.2                      0    nvidia
libnvjpeg-dev             11.8.0.2                      0    nvidia
libpng                    1.6.37               hbc83047_0
libpq                     12.9                 h16c4e8d_3
libprotobuf               3.20.1               h4ff587b_0
libsodium                 1.0.18               h7b6447c_0
libspatialindex           1.9.3                h2531618_0
libssh2                   1.10.0               h8f2d780_0
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libtasn1                  4.16.0               h27cfd23_0
libtiff                   4.4.0                hecacb30_2
libunistring              0.9.10               h27cfd23_0
libuuid                   1.41.5               h5eee18b_0
libwebp                   1.2.4                h11a3e52_0
libwebp-base              1.2.4                h5eee18b_0
libxcb                    1.15                 h7f8727e_0
libxkbcommon              1.0.1                hfa300c1_0
libxml2                   2.9.14               h74e7548_0
libxslt                   1.1.35               h4e12654_0
libzopfli                 1.0.3                he6710b0_0
llvmlite                  0.38.0           py39h4ff587b_0
locket                    1.0.0            py39h06a4308_0
lxml                      4.9.1            py39h1edc446_0
lz4                       3.1.3            py39h27cfd23_0
lz4-c                     1.9.4                h6a678d5_0
lzo                       2.10                 h7b6447c_2
markdown                  3.4.1            py39h06a4308_0
markupsafe                2.1.1            py39h7f8727e_0
matplotlib                3.6.2            py39h06a4308_0
matplotlib-base           3.6.2            py39h945d387_0
matplotlib-inline         0.1.6            py39h06a4308_0
mccabe                    0.7.0              pyhd3eb1b0_0
mistune                   0.8.4           py39h27cfd23_1000
mkl                       2021.4.0           h06a4308_640
mkl-service               2.4.0            py39h7f8727e_0
mkl_fft                   1.3.1            py39hd3c417c_0
mkl_random                1.2.2            py39h51133e4_0
mock                      4.0.3              pyhd3eb1b0_0
mpc                       1.1.0                h10f8cd9_1
mpfr                      4.0.2                hb69a4c5_1
mpi                       1.0                       mpich
mpi4py                    3.1.4                    pypi_0    pypi
mpich                     4.0.3                external_0    conda-forge
mpmath                    1.2.1            py39h06a4308_0
msgpack                   0.6.2                    pypi_0    pypi
multidict                 6.0.2            py39h5eee18b_0
multipledispatch          0.6.0            py39h06a4308_0
munkres                   1.1.4                      py_0
mypy_extensions           0.4.3            py39h06a4308_1
nbclassic                 0.4.8            py39h06a4308_0
nbclient                  0.5.13           py39h06a4308_0
nbconvert                 6.5.4            py39h06a4308_0
nbformat                  5.7.0            py39h06a4308_0
ncurses                   6.3                  h5eee18b_3
nest-asyncio              1.5.5            py39h06a4308_0
nettle                    3.7.3                hbbd107a_1
networkx                  2.8.4            py39h06a4308_0
nltk                      3.7                pyhd3eb1b0_0
nose                      1.3.7           pyhd3eb1b0_1008
notebook                  6.5.2            py39h06a4308_0
notebook-shim             0.2.2            py39h06a4308_0
nsight-compute            2022.4.0.15                   0    nvidia
nspr                      4.33                 h295c915_0
nss                       3.74                 h0370c37_0
numba                     0.55.1           py39h51133e4_0
numexpr                   2.8.4            py39he184ba9_0
numpy                     1.21.5           py39h6c91a56_3
numpy-base                1.21.5           py39ha15fc14_3
numpydoc                  1.5.0            py39h06a4308_0
oauthlib                  3.2.2                    pypi_0    pypi
olefile                   0.46               pyhd3eb1b0_0
oniguruma                 6.9.7.1              h27cfd23_0
openh264                  2.1.1                h4ff587b_0
openjpeg                  2.4.0                h3ad879b_0
openpyxl                  3.0.10           py39h5eee18b_0
openssl                   1.1.1s               h7f8727e_0
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3               pyhd3eb1b0_0
pandas                    1.5.2            py39h417a72b_0
pandocfilters             1.5.0              pyhd3eb1b0_0
panel                     0.14.1           py39h06a4308_0
param                     1.12.2           py39h06a4308_0
parsel                    1.6.0            py39h06a4308_0
parso                     0.8.3              pyhd3eb1b0_0
partd                     1.2.0              pyhd3eb1b0_1
patchelf                  0.15.0               h6a678d5_0
pathspec                  0.9.0            py39h06a4308_0
patsy                     0.5.2            py39h06a4308_1
pcre                      8.45                 h295c915_0
pep8                      1.7.1            py39h06a4308_1
pexpect                   4.8.0              pyhd3eb1b0_3
pickleshare               0.7.5           pyhd3eb1b0_1003
pillow                    9.3.0            py39hace64e9_0
pip                       22.3.1           py39h06a4308_0
pkginfo                   1.8.3            py39h06a4308_0
platformdirs              2.5.2            py39h06a4308_0
plotly                    5.9.0            py39h06a4308_0
pluggy                    1.0.0            py39h06a4308_1
ply                       3.11             py39h06a4308_0
poyo                      0.5.0              pyhd3eb1b0_0
prometheus_client         0.14.1           py39h06a4308_0
prompt-toolkit            3.0.20             pyhd3eb1b0_0
prompt_toolkit            3.0.20               hd3eb1b0_0
protego                   0.1.16                     py_0
proto-plus                1.22.1                   pypi_0    pypi
protobuf                  3.19.6                   pypi_0    pypi
psutil                    5.9.0            py39h5eee18b_0
ptyprocess                0.7.0              pyhd3eb1b0_2
pure_eval                 0.2.2              pyhd3eb1b0_0
py                        1.11.0             pyhd3eb1b0_0
py-lief                   0.12.3           py39h6a678d5_0
pyasn1                    0.4.8              pyhd3eb1b0_0
pyasn1-modules            0.2.8                      py_0
pycodestyle               2.8.0              pyhd3eb1b0_0
pycosat                   0.6.4            py39h5eee18b_0
pycparser                 2.21               pyhd3eb1b0_0
pyct                      0.4.8            py39h06a4308_1
pycurl                    7.45.1           py39h8f2d780_0
pydantic                  1.10.2                   pypi_0    pypi
pydispatcher              2.0.5            py39h06a4308_2
pydocstyle                6.1.1              pyhd3eb1b0_0
pyerfa                    2.0.0            py39h27cfd23_0
pyflakes                  2.4.0              pyhd3eb1b0_0
pygments                  2.11.2             pyhd3eb1b0_0
pyhamcrest                2.0.2              pyhd3eb1b0_2
pyjwt                     2.6.0                    pypi_0    pypi
pylint                    2.14.5           py39h06a4308_0
pyls-spyder               0.4.0              pyhd3eb1b0_0
pyodbc                    4.0.34           py39h6a678d5_0
pyopenssl                 22.0.0             pyhd3eb1b0_0
pyparsing                 3.0.9            py39h06a4308_0
pyqt                      5.15.7           py39h6a678d5_1
pyqt5-sip                 12.11.0          py39h6a678d5_1
pyqtwebengine             5.15.7           py39h6a678d5_1
pyquil                    3.3.2                    pypi_0    pypi
pyrsistent                0.18.0           py39heee7806_0
pysocks                   1.7.1            py39h06a4308_0
pytables                  3.6.1            py39h77479fe_1
pytest                    7.1.2            py39h06a4308_0
python                    3.9.15               h7a1cb2a_2
python-dateutil           2.8.2              pyhd3eb1b0_0
python-fastjsonschema     2.16.2           py39h06a4308_0
python-libarchive-c       2.9                pyhd3eb1b0_1
python-lsp-black          1.2.1            py39h06a4308_0
python-lsp-jsonrpc        1.0.0              pyhd3eb1b0_0
python-lsp-server         1.5.0            py39h06a4308_0
python-rapidjson          1.9                      pypi_0    pypi
python-slugify            5.0.2              pyhd3eb1b0_0
python-snappy             0.6.1            py39h6a678d5_0
python_abi                3.9                      2_cp39    conda-forge
pytorch                   1.13.1          py3.9_cuda11.7_cudnn8.5.0_0    pytorch
pytorch-cuda              11.7                 h67b0de4_1    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2022.1           py39h06a4308_0
pyviz_comms               2.0.2              pyhd3eb1b0_0
pywavelets                1.4.1            py39h5eee18b_0
pyxdg                     0.27               pyhd3eb1b0_0
pyyaml                    6.0              py39h5eee18b_1
pyzmq                     23.2.0           py39h6a678d5_0
qcs-api-client            0.21.2                   pypi_0    pypi
qdarkstyle                3.0.2              pyhd3eb1b0_0
qstylizer                 0.1.10             pyhd3eb1b0_0
qt                        5.15.9               h06a4308_0
qt-main                   5.15.2               h327a75a_7
qt-webengine              5.15.9               hd2b0992_4
qtawesome                 1.0.3              pyhd3eb1b0_0
qtconsole                 5.3.2            py39h06a4308_0
qtpy                      2.2.0            py39h06a4308_0
qtwebkit                  5.212                h4eab89a_4
queuelib                  1.5.0            py39h06a4308_0
readline                  8.2                  h5eee18b_0
regex                     2022.7.9         py39h5eee18b_0
requests                  2.28.1           py39h06a4308_0
requests-file             1.5.1              pyhd3eb1b0_0
requests-oauthlib         1.3.1                    pypi_0    pypi
retry                     0.9.2                    pypi_0    pypi
retrying                  1.3.4                    pypi_0    pypi
rfc3339                   6.2                      pypi_0    pypi
rfc3986                   1.5.0                    pypi_0    pypi
ripgrep                   13.0.0               hbdeaff8_0
rope                      0.22.0             pyhd3eb1b0_0
rpcq                      3.10.0                   pypi_0    pypi
rsa                       4.7.2              pyhd3eb1b0_1
rtree                     0.9.7            py39h06a4308_1
ruamel.yaml               0.17.21          py39h5eee18b_0
ruamel.yaml.clib          0.2.6            py39h5eee18b_1
ruamel_yaml               0.17.21          py39h5eee18b_0
s3transfer                0.6.0            py39h06a4308_0
scikit-image              0.19.3           py39h6a678d5_1
scikit-learn              1.1.3            py39h6a678d5_0
scikit-learn-intelex      2023.0.1         py39hf3d152e_0    conda-forge
scipy                     1.7.3            py39hc147768_0
scrapy                    2.6.2            py39h06a4308_0
seaborn                   0.12.1           py39h06a4308_0
secretstorage             3.3.1            py39h06a4308_0
send2trash                1.8.0              pyhd3eb1b0_1
service_identity          18.1.0             pyhd3eb1b0_1
setuptools                65.5.0           py39h06a4308_0
sip                       6.6.2            py39h6a678d5_0
six                       1.16.0             pyhd3eb1b0_1
smart_open                5.2.1            py39h06a4308_0
snappy                    1.1.9                h295c915_0
sniffio                   1.2.0            py39h06a4308_1
snowballstemmer           2.2.0              pyhd3eb1b0_0
sortedcollections         2.1.0              pyhd3eb1b0_0
sortedcontainers          2.4.0              pyhd3eb1b0_0
soupsieve                 2.3.2.post1      py39h06a4308_0
sphinx                    5.0.2            py39h06a4308_0
sphinxcontrib-applehelp   1.0.2              pyhd3eb1b0_0
sphinxcontrib-devhelp     1.0.2              pyhd3eb1b0_0
sphinxcontrib-htmlhelp    2.0.0              pyhd3eb1b0_0
sphinxcontrib-jsmath      1.0.1              pyhd3eb1b0_0
sphinxcontrib-qthelp      1.0.3              pyhd3eb1b0_0
sphinxcontrib-serializinghtml 1.1.5              pyhd3eb1b0_0
spyder                    5.3.3            py39h06a4308_0
spyder-kernels            2.3.3            py39h06a4308_0
sqlalchemy                1.4.39           py39h5eee18b_0
sqlite                    3.40.0               h5082296_0
stack_data                0.2.0              pyhd3eb1b0_0
statsmodels               0.13.2           py39h7f8727e_0
sympy                     1.11.1           py39h06a4308_0
tabulate                  0.8.10           py39h06a4308_0
tbb                       2021.6.0             hdb19cb5_0
tbb4py                    2021.6.0         py39hdb19cb5_0
tblib                     1.7.0              pyhd3eb1b0_0
tenacity                  8.0.1            py39h06a4308_1
tensorboard               2.11.0                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.11.0                   pypi_0    pypi
tensorflow-estimator      2.11.0                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.29.0                   pypi_0    pypi
termcolor                 2.1.1                    pypi_0    pypi
terminado                 0.13.1           py39h06a4308_0
testpath                  0.6.0            py39h06a4308_0
text-unidecode            1.3                pyhd3eb1b0_0
textdistance              4.2.1              pyhd3eb1b0_0
threadpoolctl             2.2.0              pyh0d69192_0
three-merge               0.1.1              pyhd3eb1b0_0
tifffile                  2021.7.2           pyhd3eb1b0_2
tinycss                   0.4             pyhd3eb1b0_1002
tinycss2                  1.2.1            py39h06a4308_0
tk                        8.6.12               h1ccaba5_0
tldextract                3.2.0              pyhd3eb1b0_0
toml                      0.10.2             pyhd3eb1b0_0
tomli                     2.0.1            py39h06a4308_0
tomlkit                   0.11.1           py39h06a4308_0
toolz                     0.12.0           py39h06a4308_0
torchaudio                0.13.1               py39_cu117    pytorch
torchvision               0.14.1               py39_cu117    pytorch
tornado                   6.2              py39h5eee18b_0
tqdm                      4.64.1           py39h06a4308_0
traitlets                 5.7.1            py39h06a4308_0
twisted                   22.2.0           py39h5eee18b_1
typed-ast                 1.4.3            py39h7f8727e_1
types-python-dateutil     2.8.19.5                 pypi_0    pypi
types-retry               0.9.9                    pypi_0    pypi
typing-extensions         4.4.0            py39h06a4308_0
typing_extensions         4.4.0            py39h06a4308_0
tzdata                    2022g                h04d1e81_0
ujson                     5.4.0            py39h6a678d5_0
unidecode                 1.2.0              pyhd3eb1b0_0
unixodbc                  2.3.11               h5eee18b_0
urllib3                   1.26.13          py39h06a4308_0
w3lib                     1.21.0             pyhd3eb1b0_0
watchdog                  2.1.6            py39h06a4308_0
wcwidth                   0.2.5              pyhd3eb1b0_0
webencodings              0.5.1            py39h06a4308_1
websocket-client          0.58.0           py39h06a4308_4
werkzeug                  2.2.2            py39h06a4308_0
wget                      1.21.3               h0b77cf5_0
whatthepatch              1.0.2            py39h06a4308_0
wheel                     0.37.1             pyhd3eb1b0_0
widgetsnbextension        3.5.2            py39h06a4308_0
wrapt                     1.14.1           py39h5eee18b_0
wurlitzer                 3.0.2            py39h06a4308_0
xarray                    2022.11.0        py39h06a4308_0
xlrd                      2.0.1              pyhd3eb1b0_0
xlsxwriter                3.0.3              pyhd3eb1b0_0
xz                        5.2.8                h5eee18b_0
yaml                      0.2.5                h7b6447c_0
yapf                      0.31.0             pyhd3eb1b0_0
yarl                      1.8.1            py39h5eee18b_0
zeromq                    4.3.4                h2531618_0
zfp                       0.5.5                h295c915_6
zict                      2.1.0            py39h06a4308_0
zipp                      3.8.0            py39h06a4308_0
zlib                      1.2.13               h5eee18b_0
zope                      1.0              py39h06a4308_1
zope.interface            5.4.0            py39h7f8727e_0
zstd                      1.5.2                ha4553b6_0

3. I made mistake. Actually I'm using 11.7

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

4. which mpiexec

$conda deactivate
$ which mpiexec
/usr/bin/mpiexec

[leofang]

Thanks, @koichi-tsujino. Could you show the output of /usr/bin/mpiexec --version? I believe it points to Open MPI, but in your environment you have the MPICH support installed:

...
cutensornet               2.0.0           mpi_mpich_h2fb0270_0    conda-forge
...
mpi                       1.0                       mpich
mpi4py                    3.1.4                    pypi_0    pypi
mpich                     4.0.3                external_0    conda-forge
...

which led to binary incompatibility that we were able to detect at runtime.

If it's the case, the easiest way is for you to install openmpi from conda-forge. It'd change the installed cutensornet flavor:

conda install -c conda-forge openmpi

However, please also note that you installed mpi4py via pip. The above would not work unless you uninstall mpi4py and reinstall it from conda-forge:

conda install -c conda-forge openmpi mpi4py

which, depending on your need, may or may not be desirable. Alternatively, you can install the "external" flavor of openmpi, and use your system Open MPI (assuming you have Open MPI 4.x) and your existing mpi4py by following conda-forge's suggestion:

conda install "openmpi=4.*=external_*"

this would trigger a reinstallation of cutensornet to get the correct binary (with build string mpi_openmpi_... instead of mpi_mpich_...).

[leofang]

@koichi-tsujino Switching to my other hat as a long-time conda-forge package maintainer, though, I would like to offer you a tip and suggest you to start a fresh environment and only use packages from conda-forge:

conda create -n my_new_env python=3.X cuquantum-python mpi4py  # pick a Python version
conda activate my_new_env

Currently your environment has a mixed of package sources (conda-forge, nvidia, pypi, default, ...) which would lead to a wide variety of ABI issues that we've strived to avoid over the past few years 🙂

[leofang]

Currently your environment has a mixed of package sources (conda-forge, nvidia, pypi, default, ...)

I just noticed, your environment appears to be corrupted. You have cudatoolkit installed from conda-forge, but cuda-* packages installed from nvidia. They are not compatible.

[koichi-tsujino]

@leofang

Thank you very much for your kind hlep.

I have created a new environment, installed cuquantum-python and mpi4py and it works fine.
conda create -n test python=3.9 cuquantum-python mpi4py -c conda-forge

---

But I have still some questions.

Your wrote:
conda install -c conda-forge openmpi mpi4py

But, here openmpi is not included. I don't need openmpi here?
conda create -n test python=3.9 cuquantum-python mpi4py -c conda-forge

Another situation I have:
On to the environment of anaconda. i.e.
conda create -n test anaconda
and then I tried to insall openmpi and mpi4py, it takes quite long time to checking dependency.
conda install -c conda-forge openmpi mpi4py
I'm wondering why. That's OK for me to deploy an enviroment from python3.9 and install neccesary packege but from anaconda it's quite easy.

---

Another point is,

In the page, https://docs.nvidia.com/cuda/cuquantum/getting_started.html
Install cuQuantum from conda-forge
'conda install -c conda-forge cuquantum'

Note: To enable automatic MPI parallelism for cuTensorNet, you can install cuquantum with an MPI from conda-forge, e.g., conda install -c conda-forge cuquantum openmpi. For detailed guide, please refer to cuTensorNet Guide.

Install cuQuantum Python from conda-forge
conda install -c conda-forge cuquantum-python

Here I find mpi4py (optional, see mpi4py installation guide)
https://docs.nvidia.com/cuda/cuquantum/getting_started.html#cuquantum-python
That's why I have installded mpi4py via pip.

In this page,
https://docs.nvidia.com/cuda/cuquantum/cutensornet/getting_started.html
conda install -c conda-forge cutensornet
Alternatively, you can install cuQuantum, which contains both cuTensorNet and cuStateVec, from the conda-forge channel, via:
conda install -c conda-forge cuquantum
If you install cutensornet or cuquantum with an MPI from conda-forge, e.g.,
conda install -c conda-forge cutensornet openmpi

Now, I understand
cuQuantum included CuTensorNet and cuStateVec and I have to install openmpi.

---

1. What is the diffenced between cuquantum and cuquantum-python
2. What kind of MPI related packege shall I install via conda -c conda-forge?

[leofang]

Let me break it down since it's a lot of questions 🙂

I have created a new environment, installed cuquantum-python and mpi4py and it works fine. conda create -n test python=3.9 cuquantum-python mpi4py -c conda-forge

Glad to know it works now, @koichi-tsujino!

But I have still some questions.

Your wrote: conda install -c conda-forge openmpi mpi4py

But, here openmpi is not included. I don't need openmpi here? conda create -n test python=3.9 cuquantum-python mpi4py -c conda-forge

First of all, I did mention openmpi in my first reply 😉

Second, even if you didn't explicitly request it, if you check the to-be-installed packages when you create the env, you should see openmpi being one of them. The reason why openmpi is favored over mpich in this case is not entirely clear to me (due to complex conda solver behavior), so on the safe side you indeed should always request the preferred MPI library explicitly.

Another situation I have: On to the environment of anaconda. i.e. conda create -n test anaconda and then I tried to insall openmpi and mpi4py, it takes quite long time to checking dependency. conda install -c conda-forge openmpi mpi4py I'm wondering why. That's OK for me to deploy an enviroment from python3.9 and install neccesary packege but from anaconda it's quite easy.

This is unfortunately beyond the scope of this repository. It involves again how the conda solver works, and I'd suggest you to ask in one of Conda's public forums:

• https://conda.discourse.group (Anaconda's new forum)
• https://gitter.im/conda/conda (Anaconda's old forum)
• https://gitter.im/conda-forge/conda-forge.github.io (official forum for conda-forge)

That said, I can share short answers:

• If create first then install, the solver needs to inspect the existing packages in the present env. If you can request for everything you need in one shot (during create), I share the same experience for the UX being slightly more pleasant
• You can use mamba as the alternative solver, which is much faster and already used in production by both Anaconda and conda-forge. See, e.g, https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community.

[leofang]

To you final point:

Another point is,

In the page, https://docs.nvidia.com/cuda/cuquantum/getting_started.html Install cuQuantum from conda-forge 'conda install -c conda-forge cuquantum'

Note: To enable automatic MPI parallelism for cuTensorNet, you can install cuquantum with an MPI from conda-forge, e.g., conda install -c conda-forge cuquantum openmpi. For detailed guide, please refer to cuTensorNet Guide.

Install cuQuantum Python from conda-forge conda install -c conda-forge cuquantum-python

Here I find mpi4py (optional, see mpi4py installation guide) https://docs.nvidia.com/cuda/cuquantum/getting_started.html#cuquantum-python That's why I have installded mpi4py via pip.

It is optional indeed, as cuTensorNet (or cuQuantum Python) can work without it. In fact, the (auto-)MPI support is new in cuQuantum 22.11 / cuTensorNet v2.0.0; previously, to enable MPI parallelism we asked users to do it explicitly (e.g. see this Python example).

It is recommended that don't mix packages from the conda-forge channel with those from other channels (default, anaconda, pip, ...) unless there's good reasons. In practice this may not be fully feasible, but please keep this rule of thumb in mind 🙂

In this page, https://docs.nvidia.com/cuda/cuquantum/cutensornet/getting_started.html conda install -c conda-forge cutensornet Alternatively, you can install cuQuantum, which contains both cuTensorNet and cuStateVec, from the conda-forge channel, via: conda install -c conda-forge cuquantum If you install cutensornet or cuquantum with an MPI from conda-forge, e.g., conda install -c conda-forge cutensornet openmpi

Now, I understand cuQuantum included CuTensorNet and cuStateVec and I have to install openmpi.

1. What is the diffenced between cuquantum and cuquantum-python
2. What kind of MPI related packege shall I install via conda -c conda-forge?

To your questions:

1. cuquantum is a meta package containing both custatevec and cutensornet (our C++ libraries) as you figured out. cuquantum-python is the Python package that depend on custatevec and cutensornet. If you only install cuquantum (or custatevec or cutensornet) you can't do import cuquantum in Python. Think of it as hdf5 vs h5py, or openmpi vs mpi4py, or cudatoolkit vs cuda-python. The analogy is precise.
2. If you wanna use MPI from conda-forge, use openmpi. Currently (as of cuQuantum 22.11) we need CUDA-aware MPI, which conda-forge's mpich package cannot support due to MPICH's design constraint.

[koichi-tsujino]

Thank you for your kind reply.
I'll continue to work with cuQuantum.

[koichi-tsujino]

@leofang

I have encounterd antho error.

When I exutge the following code.

from cupy.cuda.runtime import getDeviceCount
from mpi4py import MPI
from cuquantum import cutensornet as cutn

# bind comm to cuTensorNet handle
handle = cutn.create()
comm = MPI.COMM_WORLD
cutn.distributed_reset_configuration(handle, *cutn.get_mpi_comm_pointer(comm))

# make each process run on different GPU
rank = comm.Get_rank()
device_id = rank % getDeviceCount()
cp.cuda.Device(device_id).use()

# 1. assuming input tensors a, b, and c are created on the right GPU
# 2. passing handle explicitly allows reusing it to reduce the handle creation overhead
r = contract(
    "mhkn,ukh,xuy->mxny", a, b, c,
    options={'device_id' : device_id, 'handle': handle})

I got.

---------------------------------------------------------------------------
cuTensorNetError                          Traceback (most recent call last)
Cell In[5], line 8
      6 handle = cutn.create()
      7 comm = MPI.COMM_WORLD
----> 8 cutn.distributed_reset_configuration(handle, *cutn.get_mpi_comm_pointer(comm))
     10 # make each process run on different GPU
     11 rank = comm.Get_rank()

File /opt/conda/lib/python3.10/site-packages/cuquantum/cutensornet/cutensornet.pyx:2306, in cuquantum.cutensornet.cutensornet.distributed_reset_configuration()

File /opt/conda/lib/python3.10/site-packages/cuquantum/cutensornet/cutensornet.pyx:2328, in cuquantum.cutensornet.cutensornet.distributed_reset_configuration()

File /opt/conda/lib/python3.10/site-packages/cuquantum/cutensornet/cutensornet.pyx:229, in cuquantum.cutensornet.cutensornet.check_status()

cuTensorNetError: CUTENSORNET_STATUS_DISTRIBUTED_FAILURE

Do you have any idea how to check the reason?

mpi4py is working I believe. The code below works.

from mpi4py import MPI 
comm = MPI.COMM_WORLD 
rank = comm.Get_rank() 
size = comm.Get_size() 
name = MPI.Get_processor_name() 
print(f"{name} : {rank} : {size}")
``

[koichi-tsujino]

This code works.

from cupy.cuda.runtime import getDeviceCount
from mpi4py import MPI
import numpy as np

from cuquantum import Network

root = 0
comm = MPI.COMM_WORLD

rank, size = comm.Get_rank(), comm.Get_size()

expr = 'ehl,gj,edhg,bif,d,c,k,iklj,cf,a->ba'
shapes = [(8, 2, 5), (5, 7), (8, 8, 2, 5), (8, 6, 3), (8,), (6,), (5,), (6, 5, 5, 7), (6, 3), (3,)]

# Set the operand data on root.
operands = [np.random.rand(*shape) for shape in shapes] if rank == root else None

# Broadcast the operand data.
operands = comm.bcast(operands, root)
    
# Assign the device for each process.
device_id = rank % getDeviceCount()

# Create network object.
network = Network(expr, *operands, options={'device_id' : device_id})

# Compute the path on all ranks with 8 samples for hyperoptimization. Force slicing to enable parallel contraction.
path, info = network.contract_path(optimize={'samples': 8, 'slicing': {'min_slices': max(16, size)}})

# Select the best path from all ranks.
opt_cost, sender = comm.allreduce(sendobj=(info.opt_cost, rank), op=MPI.MINLOC)
if rank == root:
    print(f"Process {sender} has the path with the lowest FLOP count {opt_cost}.")

# Broadcast info from the sender to all other ranks.
info = comm.bcast(info, sender)

# Set path and slices.
path, info = network.contract_path(optimize={'path': info.path, 'slicing': info.slices})

# Calculate this process's share of the slices.
num_slices = info.num_slices
chunk, extra = num_slices // size, num_slices % size
slice_begin = rank * chunk + min(rank, extra)
slice_end = num_slices if rank == size - 1 else (rank + 1) * chunk + min(rank + 1, extra)
slices = range(slice_begin, slice_end)

print(f"Process {rank} is processing slice range: {slices}.")

# Contract the group of slices the process is responsible for.
result = network.contract(slices=slices)

# Sum the partial contribution from each process on root.
result = comm.reduce(sendobj=result, op=MPI.SUM, root=root)

# Check correctness.
if rank == root:
   result_np = np.einsum(expr, *operands, optimize=True)
   print("Does the cuQuantum parallel contraction result match the numpy.einsum result?", np.allclose(result, result_np))

[DmitryLyakh]

Could you please rerun the example with the environment variable "CUTENSORNET_LOG_LEVEL=5" and paste the log here (or the relevant tail of the log), which should give us a better clue? My immediate suspicion is that there might be multiple MPI libraries present in your environment such that mpi4py and the example are using one MPI library whereas the cuQuantum MPI-provider was built with a different MPI library.

[koichi-tsujino]

Here is the log.

Unable to accept distributed communicator, no MPI library found!

mpi is running and code without "distributed_reset_configuration" is OK. I'm really wondering why?
What else?

---

jovyan@jupyter-admin:~/test$ python mpi_test.py
[2023-01-14 00:26:19][cuTensorNet][126][Api][cutensornetCreate] handle=0X7FFEAA419F10
[2023-01-14 00:26:19][cuTensorNet][126][Info][cutensornetCreate] cuTensorNet version: 20000, cuTENSOR version: 10602
[2023-01-14 00:26:20][cuTensorNet][126][Info][cutensornetCreate] Initializing cuTensorNet distributed communication service interface
[2023-01-14 00:26:20][cuTensorNet][126][Info][cutensornetCreate] WARNING: $CUTENSORNET_COMM_LIB environment variable not set: No distributed communication service in use.
[2023-01-14 00:26:20][cuTensorNet][126][Api][cutensornetDistributedResetConfiguration] handle=0X55BB43112120, commPtr=0X7F34A7CFC7C0, commSize=8
[2023-01-14 00:26:20][cuTensorNet][126][Info][cutensornetDistributedResetConfiguration] Resetting distributed communicator inside cuTensorNet context: 0X7F34A7CFC7C0, 8
[2023-01-14 00:26:20][cuTensorNet][126][Info][cutensornetDistributedResetConfiguration] Synchronizing distributed communicator via barrier
[2023-01-14 00:26:20][cuTensorNet][126][Error][cutensornetDistributedResetConfiguration] Unable to accept distributed communicator, no MPI library found!
[2023-01-14 00:26:20][cuTensorNet][126][Hint][cutensornetDistributedResetConfiguration] Make sure $CUTENSORNET_COMM_LIB points to the cuTensorNet-MPI wrapper library.
[2023-01-14 00:26:20][cuTensorNet][126][Api][cutensornetGetErrorString] error=27
Traceback (most recent call last):
File "/home/jovyan/test/mpi_test.py", line 8, in
cutn.distributed_reset_configuration(
File "cuquantum/cutensornet/cutensornet.pyx", line 2306, in cuquantum.cutensornet.cutensornet.distributed_reset_configuration
File "cuquantum/cutensornet/cutensornet.pyx", line 2328, in cuquantum.cutensornet.cutensornet.distributed_reset_configuration
File "cuquantum/cutensornet/cutensornet.pyx", line 229, in cuquantum.cutensornet.cutensornet.check_status
cuquantum.cutensornet.cutensornet.cuTensorNetError: CUTENSORNET_STATUS_DISTRIBUTED_FAILURE

[haidarazzam]

the issue is here:
[2023-01-14 00:26:20][cuTensorNet][126][Info][cutensornetCreate] WARNING: $CUTENSORNET_COMM_LIB environment variable not set: No distributed communication service in use.
you need to set the variable CUTENSORNET_COMM_LIB to point to the libcutensornet_distributed_interface_mpi.so file
something like this
export CUTENSORNET_COMM_LIB=/home/ahaidar/quantum/cuquantum/tensor_network/mpiwrappers/lib/libcutensornet_distributed_interface_mpi.so

To enable distributed parallelism, cuTensorNet requires users to set an environment variable $CUTENSORNET_COMM_LIB containing the path to a shared library wrapping the communication primitives. For MPI users, we ship a wrapper file cutensornet_distributed_interface_mpi.c that can be compiled against the MPI library. cuTensorNet will use the included function pointers to perform inter-process communication.

Please read
https://docs.nvidia.com/cuda/cuquantum/cutensornet/api/functions.html?highlight=cutensornet_comm_lib#distributed-parallelization-api

[koichi-tsujino]

Thank you for your answer!
Now, it wors!

BTW
The sample programs aprears in https://docs.nvidia.com/cuda/cuquantum/python/overview.html#high-level-pythonic-apis, does not work.

from cupy.cuda.runtime import getDeviceCount
from mpi4py import MPI
from cuquantum import cutensornet as cutn

# bind comm to cuTensorNet handle
handle = cutn.create()
comm = MPI.COMM_WORLD
cutn.distributed_reset_configuration(
    handle, *cutn.get_mpi_comm_pointer(comm))

# make each process run on different GPU
rank = comm.Get_rank()
device_id = rank % getDeviceCount()
cp.cuda.Device(device_id).use()

# 1. assuming input tensors a, b, and c are created on the right GPU
# 2. passing handle explicitly allows reusing it to reduce the handle creation overhead
r = contract(
    "mhkn,ukh,xuy->mxny", a, b, c,
    options={'device_id' : device_id, 'handle': handle}))

---

NameError Traceback (most recent call last)
Cell In[7], line 14
12 rank = comm.Get_rank()
13 device_id = rank % getDeviceCount()
---> 14 cp.cuda.Device(device_id).use()
16 # 1. assuming input tensors a, b, and c are created on the right GPU
17 # 2. passing handle explicitly allows reusing it to reduce the handle creation overhead
18 r = contract(
19 "mhkn,ukh,xuy->mxny", a, b, c,
20 options={'device_id' : device_id, 'handle': handle})

NameError: name 'cp' is not defined