Create a conda/python environment to run this code in

[q-rai]

The tutorial is for complete installs. To keep things nicely separated, I'm trying to build a conda environment for the project.

It's been a couple of attempts and I figure it makes more sense to track the progress in an issue right with the code, rather than completely separately. I'll document my progress here, starting by pasting the documentation I have of my attempts so far.

[q-rai]

First attempt!

    conda create -n hydro python=3.6 numpy matplotlib shapely
    source activate hydro
    conda install matplotlib
    conda install -c conda-forge fiona
    conda install -c conda-forge rasterio
    conda install shapely
    pip install MeshPy
    conda install scipy
    conda install vtk
    conda install ipykernel

To add the environment to jupyter notebooks, run
python -m ipykernel install --user --name hydro --display-name "Python 3 (Hydro)"

So far, so good. But gdal refused to install.
The solution is supposedly this, according to one of many search results I went through:

conda install -c conda-forge fiona "libgdal<2.0"

But this would uninstall rasterio. When trying anyway, I got “An unexpected error has occurred.”

Issues along the way

• Code couldn’t find the workflow library

  • tried adding the ideal_octo_waffle to PYTHONPATH before calling jupyter notebook — no luck
  • tried the .pth solution here: https://stackoverflow.com/a/37008663 — no luck
  • would like to try the setup.py solution, that seems cleanest — later

• Error at from rasterio._base import get_window in workflow/clip.py:

  • ImportError: cannot import name 'get_window'
  • Solution: removed that line and prepended rasterio._base in front of get_window -- seems to work, no more error

• Error at from fiona.ogrext import Iterator, ItemsIterator, KeysIterator in workflow/hilev.py

  • ImportError: dlopen(/Users/vib/anaconda/envs/hydro/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.76.dylib Referenced from: /Users/vib/anaconda/envs/hydro/lib/libgdal.20.dylib Reason: image not found
  • Tried: conda install -c conda-forge fiona "libgdal<2.0" "Reason: image not found" error for OS X 10.11, Anaconda 3.5.1 conda-forge/fiona-feedstock#13 (comment)
    • this would uninstall rasterio
    • tried anyway — no luck (see error below)
    • I don’t even understand why it’s using python 2.7! My prompt says I’m in the hydro env: (hydro) vibmbpro:site-packages vib$ conda install -c conda-forge fiona "libgdal<2.0"

Result:

    Preparing transaction: done
    Verifying transaction: failed
    # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
    Traceback (most recent call last):``` 
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 819, in __call__
        return func(*args, **kwargs)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/main.py", line 78, in _main
        exit_code = do_call(args, p)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/main_install.py", line 11, in execute
        install(args, parser, 'install')
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/install.py", line 253, in install
        handle_txn(unlink_link_transaction, prefix, args, newenv)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/cli/install.py", line 282, in handle_txn
        unlink_link_transaction.execute()
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 223, in execute
        self.verify()
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/common/io.py", line 46, in decorated
        return f(*args, **kwds)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 210, in verify
        exceptions = self._verify(self.prefix_setups, self.prefix_action_groups)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 481, in _verify
        cls._verify_transaction_level(prefix_setups),
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 476, in <genexpr>
        exceptions = tuple(exc for exc in concatv(
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 334, in _verify_individual_level
        error_result = axn.verify()
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/core/path_actions.py", line 292, in verify
        source_sha256 = compute_sha256sum(self.source_full_path)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/gateways/disk/read.py", line 75, in compute_sha256sum
        return _digest_path('sha256', file_full_path)
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/gateways/disk/read.py", line 64, in _digest_path
        with open(path, "rb") as fh:
      File "/Users/vib/anaconda/lib/python2.7/site-packages/conda/common/compat.py", line 121, in open
        errors=errors, newline=newline, closefd=closefd)
    IOError: [Errno 13] Permission denied: '/Users/vib/anaconda/pkgs/vtk-6.3.0-py27_1/bin/vtkHashSource-6.3'

[q-rai]

Attempt 2: Install from YML

Ethan added a YML file for the project which he exported from his anaconda root environment. I tried importing it:

conda env create -n hydro -f water_workflow.yml

Install Issues

• When importing the environment, I got:

    ResolvePackageNotFound:
    libgdal==2.2.4=2

• removed libgdal from file:

    - libnetcdf==4.6.1=2
    - rasterio==0.36.0=py36_3 -> gdal=2.2 -> libgdal[version='>=2.2.4,<2.3.0a0'] -> hdf5[version='>=1.8.18,<1.8.19.0a0']```
  
  

• removed libnetcdf from file:

    - hdf5==1.10.1=ha036c08_1
    - libdap4==3.18.3=2
    - rasterio==0.36.0=py36_3```
  
  

• removed hdf5 (same error including the hdf5 line!?) and libdap4 from file:
  • conda env setup seems to be working
  • trouble with meshpy though, which was to be installed through pip

        running install
        running build
        running build_py
        creating build
        creating build/lib.macosx-10.7-x86_64-3.6
        creating build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/__init__.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/common.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/geometry.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/gmsh.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/gmsh_reader.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/naca.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/ply.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/tet.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/tools.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        copying meshpy/triangle.py -> build/lib.macosx-10.7-x86_64-3.6/meshpy
        running build_ext
        building 'meshpy._triangle' extension
        creating build/temp.macosx-10.7-x86_64-3.6
        creating build/temp.macosx-10.7-x86_64-3.6/src
        creating build/temp.macosx-10.7-x86_64-3.6/src/cpp
        gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fwrapv -Wall -O1 -DNDEBUG -DEXTERNAL_TEST=1 -DANSI_DECLARATORS=1 -DTRILIBRARY=1 -Isrc/cpp -I/Users/vib/anaconda/envs/waterworkflow/include/python3.6m -c src/cpp/wrap_triangle.cpp -o build/temp.macosx-10.7-x86_64-3.6/src/cpp/wrap_triangle.o
        src/cpp/wrap_triangle.cpp:2:10: fatal error: 'boost/python.hpp' file not found
        #include <boost/python.hpp>
                 ^~~~~~~~~~~~~~~~~~
        1 error generated.
        error: command 'gcc' failed with exit status 1

• Re-tried pip install MeshPy later and got no error.

Running Issues

With the environment installed, I tried running the workflow example.

ImportError for Poppler

 ImportError: dlopen(/Users/vib/anaconda/envs/hydro/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.76.dylib   Referenced from: /Users/vib/anaconda/envs/hydro/lib/libgdal.20.dylib   Reason: image not found

Searched for libpoppler ll | grep -n "libpoppler" in /Users/vib/anaconda/envs/hydro/lib

Found similar version /Users/vib/anaconda/envs/hydro/lib/libpoppler.76.dylib

Linked the similar version:
ln -s libpoppler.78.dylib libpoppler.76.dylib got rid of the error

pending testing once other errors are gone

ImportError for Kea

ImportError: dlopen(/Users/vib/anaconda/envs/hydro/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libkea.1.4.7.dylib.   Referenced from: /Users/vib/anaconda/envs/hydro/lib/libgdal.20.dylib   Reason: image not found

Linked similar library:
ln -s libkea.1.4.8.dylib libkea.1.4.7.dylib

Got a new error:

ImportError: dlopen(/Users/vib/anaconda/envs/hydro/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libkea.1.4.7.dylib.   Referenced from: /Users/vib/anaconda/envs/hydro/lib/libgdal.20.dylib   Reason: Incompatible library version: libgdal.20.dylib requires version 1.4.7 or later, but libkea.1.4.7.dylib provides version 1.4.0

— odd, because the installed version is newer than what’s required.. I couldn't really make much sense of this so I'll come back to it later.

Back to Installing

I cycled back to trying to fix things by installs of specific versions.
An issue in the conda forge repository suggested:
conda install 'poppler<0.66'

Yet another error…

ImportError: dlopen(/Users/vib/anaconda/envs/hydro/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.71.dylib
  Referenced from: /Users/vib/anaconda/envs/hydro/lib/libgdal.20.dylib
  Reason: image not found

The comments also talk about how the default channel’s gdal may work better.

[q-rai]

Attempt 3: Using IOOS

Someone recommended using the IOOS versions of geography libraries in a Continuum IO issue, so I went ahead and tried that:

conda create -n hydro2 python=3.6 numpy matplotlib shapely ipykernel shapely scipy vtk
conda install -c ioos fiona geopandas rasterio
pip install MeshPy
python -m ipykernel install --user --name hydro2 --display-name "Python 3 (Hydro2)"

And again, the dreaded

ImportError: dlopen(/Users/vib/anaconda/envs/hydro2/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.76.dylib
  Referenced from: /Users/vib/anaconda/envs/hydro2/lib/libgdal.20.dylib
  Reason: image not found

Tried conda install 'poppler<0.66'

And got the other dreaded

ImportError: dlopen(/Users/vib/anaconda/envs/hydro2/lib/python3.6/site-packages/fiona/ogrext.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libpoppler.71.dylib
  Referenced from: /Users/vib/anaconda/envs/hydro2/lib/libgdal.20.dylib
  Reason: image not found

[ecoon]

Ok, I believe I have a reasonable solution. conda-forge appears to be caught up with reasonable versions now.

Note this is with anaconda 5.3.0, and requires a .condarc with conda-forge PRECEEDING defaults (the first line below this ensures this is the case.

$> conda config --prepend channels conda-forge
$> conda create -n ideal_octo_waffle fiona attrs rasterio matplotlib scipy nb_conda pyproj requests
$> conda activate ideal_octo_waffle
$> python -c "import fiona; print(fiona.__version__); import rasterio; print(rasterio.__version__)"
1.8a3
1.0.8

Updating the README now