DIST: Improve method of distributing packages to users

[asoplata]

Problem:

The current install method suffices for non-parallel simulation, but in an ideal world, it would be easy for users to install both HNN AND what it needs to use the incredible speed-up offered by MPI. Let's make that happen: I propose we change the default install method and instructions to include MPI support (that's the install method, which is not the same thing as pip install hnn_core).

Cross-platform distribution of MPI binaries (and libraries) can be difficult, but Anaconda (Conda) rectifies this by itself being an MPI binary distribution platform; see https://anaconda.org/conda-forge/openmpi and https://anaconda.org/conda-forge/mpi4py .

Solution:

Therefore, I think the best way to do this is to change our default install method by distributing HNN-Core as its own Conda package. This will allow us to specify the right version of Conda's OpenMPI packages as dependencies, which is not possible using the Python ecosystem alone (such as using wheels).

Required Work:

This is not as straightforward as it sounds. We need to first investigate the dependency network between the primary Conda package we need https://anaconda.org/conda-forge/mpi4py (which depends on Conda-installed OpenMPI binaries) across platforms and against NEURON itself. For example, even though conda-forge ships a version of NEURON ( https://anaconda.org/conda-forge/neuron/files ), it is not guaranteed that the NEURON distributables are built against the same OpenMPI version that a conda-forge mpi4py is. Similarly, conda-forge NEURON only has distributables for x86_64 Linux and Mac, but not Apple Silicon (arm64) or Windows. Our Conda package may require installing the conda-forge version of mpi4py, but also depend on the pip version of NEURON. Similarly, we may not be able to improve the Windows distribution using this method; it may still be required that the user manually install NEURON, then install HNN-Core.

Notes:

Please note that this is a task about what form we use for distributing our code/assets to users (the “user side”). This is distinct from improvements to how do we build our packages using development tools (the “dev side”)., such as here #951 .

[jasmainak]

I thought the issue with mpi4py is that you need to have either OpenMPI or another MPI binary installed on your computer ... hence the need to go through anaconda. The anaconda install gets you both the MPI binary and mpi4py. Can this be done through wheels?

[asoplata]

@jasmainak No, I don't think that distributing wheels allows for a reliable method of simultaneously distributing MPI binaries. I think you're correct: cross-platform MPI binary distribution itself is a difficult situation, but fortunately for us, Anaconda is arguably the current-best way to handle that distribution, e.g. https://anaconda.org/conda-forge/openmpi . I am going to re-write this Issue description for how we should investigate building a better install method that includes MPI by default for our users.