bindings generated with Clang.jl (#49)

* setup Clang.jl for p4est Co-authored-by: Michael Schlottke-Lakemper <[email protected]> * switch to Clang generated bindings * update docstring * fix type of array in sc_array * add test of p4est_qcoord_to_vertex * update docs build setup * add P4est.version * WIP: add Preferences; update README * semi-hacky way to set preferences * P4est.package_id() * prepare new CI setup * WIP: update CI * rename deps folder * try to fix CI * try to fix CI * try to fix CI * fix test/configure_packages.jl * WIP: try fixing CI * WIP: try fixing CI * Revert "WIP: try fixing CI" This reverts commit 12421fd. * Revert "WIP: try fixing CI" This reverts commit 7110a68. * WIP: new approach trying to fix CI * Revert "WIP: new approach trying to fix CI" This reverts commit ec87982. * test also OpenMPI.jll * show MPIPreferences setting in tests * WIP: debugging CI failures * WIP: move preference loading to main module * disable P4EST_JLL_MPI_OPENMPIJLL since those binaries do not seem to exist * use P4est as environment for test/configure_packages.jl * exclude custom p4est/MPi CI test on Julia 1.6 * disable P4EST_JLL_MPI_CUSTOM * P4EST_TEST -> JULIA_P4EST_TEST, P4EST_TEST_LIBP4EST -> JULIA_P4EST_TEST_LIBP4EST * remove debugging statements * clean up * clean up test dependencies * debugging docs * add MPIPreferences back to test project * switch to Julia v1.8 for docs * set strict = false for Documenter * update docs build * fix some missing links in docs * copy README.md as docs/src/index.md * fix copied files in docs * update README.md * add NEWS.md * write translation rules * improve API reference page * run tests with and without MPI * change order of tests * add index to docs and set strict = true * some clean-up * strict = false again * sum up custom installation * test suggested by Michael * parallel smoke test in README.md * P4est.init * do not initialize MPI automatically * fix doc index * include Manifest.toml in dev for reproducibility * update MPI and P4est_jll in dev/Manifest.toml * use latest working version of Clang.jl in dev/Manifest.toml * fix t::Cint -> t::MPI_datatype * add compat bounds to dev/Project.toml * use Artifacts to retrieve p4est headers in dev * remove deps_disabled * update parallel example in the README.md * temporarily revert using change of P4est_jll * make explicit what we are using from P4est_jll (#53) * disable unused SC MPI macros * bug in Clang.jl ixed in v0.17 * Add more tests (#55) * make explicit what we are using from P4est_jll * .gitignore LocalPreferences.toml, add UUID to Project.toml * add some more basic tests * add p4est_balance test * add more tests * Apply suggestions from code review Co-authored-by: Hendrik Ranocha <[email protected]> * additional tests for global_num_quadrants * correct coarsen_fn Co-authored-by: Hendrik Ranocha <[email protected]> * do not store redundant dev/LibP4est.jl * fix more MPI_Comm types * more notes on translation rules etc. in docs * API Reference as title * Update docs/src/reference.md Co-authored-by: Michael Schlottke-Lakemper <[email protected]> * docstring of uses_mpi * Update src/P4est.jl Co-authored-by: Michael Schlottke-Lakemper <[email protected]> * Update docs/src/introduction.md Co-authored-by: Michael Schlottke-Lakemper <[email protected]> * Update dev/fixes.sh Co-authored-by: Michael Schlottke-Lakemper <[email protected]> * remove unused variable in dev/generator.jl * Update README.md Co-authored-by: Michael Schlottke-Lakemper <[email protected]> * comment on ARtifacts.toml * Translate some function-like C macros (#56) * make explicit what we are using from P4est_jll * .gitignore LocalPreferences.toml, add UUID to Project.toml * add some more basic tests * add p4est_balance test * add more tests * Apply suggestions from code review Co-authored-by: Hendrik Ranocha <[email protected]> * additional tests for global_num_quadrants * correct coarsen_fn * translate some function-like C macros * remove some function-like macros again Co-authored-by: Hendrik Ranocha <[email protected]> * Update dev/fixes.sh * fix capitalization of API reference header * remove index * add dev/LibP4est.jl to .gitignore * src/LibP4est_v2.jl -> src/LibP4est.jl * comment on older version of p4est for custom build * remove P4EST_CUSTOM_MPI_DEFAULT * remove P4EST_JLL_MPI_OPENMPIJLL * translation rules -> guidelines * add MPI to docs/Project.toml * Apply suggestions from code review Co-authored-by: Michael Schlottke-Lakemper <[email protected]> * Update README.md Co-authored-by: Michael Schlottke-Lakemper <[email protected]> * link to usage section Co-authored-by: Michael Schlottke-Lakemper <[email protected]> Co-authored-by: Joshua Lampert <[email protected]>
trixi-framework · Dec 23, 2022 · b2858a4 · b2858a4
1 parent b729031
commit b2858a4
Show file tree

Hide file tree

Showing 36 changed files with 14,940 additions and 768 deletions.
diff --git a/.ci_install_p4est.sh b/.ci_install_p4est.sh
@@ -4,35 +4,13 @@ if [ -z "$P4EST_RELEASE" ]; then
   P4EST_RELEASE="2.8"
 fi
 
-if [ "${P4EST_TEST}" = "P4EST_JLL_USES_MPI_PRE_GENERATED_BINDINGS" ]; then
-  echo "Found 'P4EST_TEST=${P4EST_TEST}'. Nothing to do here."
+if [ "${JULIA_P4EST_TEST}" = "P4EST_JLL_MPI_DEFAULT" ]; then
+  echo "Found 'JULIA_P4EST_TEST=${JULIA_P4EST_TEST}'. Nothing to do here."
 fi
-if [ "${P4EST_TEST}" = "P4EST_JLL_USES_MPI" ]; then
-  echo "Found 'P4EST_TEST=${P4EST_TEST}'. Nothing to do here."
-fi
-if [ "${P4EST_TEST}" = "P4EST_CUSTOM_NON_MPI" ]; then
-  echo "Found 'P4EST_TEST=P4EST_CUSTOM_NON_MPI'. Installing custom p4est *without* MPI support..."
-  pushd `pwd`
-  export P4EST_TMP=`pwd`/libp4est_tmp_non_mpi
-  mkdir -p $P4EST_TMP
-  cd $P4EST_TMP/
-  wget https://p4est.github.io/release/p4est-${P4EST_RELEASE}.tar.gz
-  tar xf p4est-${P4EST_RELEASE}.tar.gz
-  mkdir build
-  cd build/
-  $P4EST_TMP/p4est-${P4EST_RELEASE}/configure --prefix=$P4EST_TMP/prefix
-  make -j 2
-  make install
-  ls -l $P4EST_TMP/prefix/lib/libp4est.so
-  mkdir test_tmp
-  cd test_tmp
-  $P4EST_TMP/prefix/bin/p4est_step1
-  popd
-fi
-if [ "${P4EST_TEST}" = "P4EST_CUSTOM_USES_MPI" ]; then
-  echo "Found 'P4EST_TEST=P4EST_CUSTOM_USES_MPI'. Installing custom p4est *with* MPI support..."
+if [ "${JULIA_P4EST_TEST}" = "P4EST_CUSTOM_MPI_CUSTOM" ]; then
+  echo "Found 'JULIA_P4EST_TEST=${JULIA_P4EST_TEST}'. Installing custom `p4est` with MPI support from the default MPI installation on the system."
   pushd `pwd`
-  export P4EST_TMP=`pwd`/libp4est_tmp_uses_mpi
+  export P4EST_TMP=`pwd`/libp4est_tmp
   mkdir -p $P4EST_TMP
   cd $P4EST_TMP/
   wget https://p4est.github.io/release/p4est-${P4EST_RELEASE}.tar.gz

diff --git a/.github/workflows/Documenter.yml b/.github/workflows/Documenter.yml
@@ -20,10 +20,10 @@ jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
       - uses: julia-actions/setup-julia@v1
         with:
-          version: '1.6'
+          version: '1'
           show-versioninfo: true
       - uses: julia-actions/julia-buildpkg@v1
         env:

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -27,126 +27,100 @@ concurrency:
 jobs:
   test:
     if: "!contains(github.event.head_commit.message, 'skip ci')"
-    name: ${{ matrix.os }} - ${{ matrix.p4est_test }} - Julia ${{ matrix.version }}
+    name: ${{ matrix.os }} - ${{ matrix.JULIA_P4EST_TEST }} - Julia ${{ matrix.version }}
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
         version:
           - '1.6'
+          - '1'
           # - 'nightly'
         os:
           - ubuntu-latest
-          # - macos-latest
-          # - windows-latest
+          - macos-latest
+          - windows-latest
         arch:
           - x64
-        p4est_test:
-          - P4EST_JLL_USES_MPI_PRE_GENERATED_BINDINGS
-          - P4EST_JLL_USES_MPI
-          - P4EST_CUSTOM_NON_MPI
-          - P4EST_CUSTOM_USES_MPI
-        # Only selected tests on macOS and Windows, since not everything works everywhere yet
-        include:
+        JULIA_P4EST_TEST:
+          - P4EST_JLL_MPI_DEFAULT
+          - P4EST_CUSTOM_MPI_CUSTOM
+        exclude:
+          # Some preferences are not loaded in package tests in older versions
+          # of Julia, see
+          # https://github.com/JuliaPackaging/Preferences.jl/issues/40
+          # Thus, we need to disable this setup in CI.
           - version: '1.6'
-            os: macOS-latest
-            arch: x64
-            p4est_test: P4EST_JLL_USES_MPI_PRE_GENERATED_BINDINGS
-          - version: '1.6'
-            os: macOS-latest
-            arch: x64
-            p4est_test: P4EST_JLL_USES_MPI
-          - version: '1.6'
-            os: windows-latest
-            arch: x64
-            p4est_test: P4EST_JLL_USES_MPI_PRE_GENERATED_BINDINGS
-          # Additional tests on Julia v1.7
-          - version: '^1.7.0-0'
-            os: ubuntu-latest
-            arch: x64
-            p4est_test: P4EST_JLL_USES_MPI_PRE_GENERATED_BINDINGS
-          - version: '^1.7.0-0'
-            os: macOS-latest
-            arch: x64
-            p4est_test: P4EST_JLL_USES_MPI_PRE_GENERATED_BINDINGS
-          - version: '^1.7.0-0'
-            os: windows-latest
-            arch: x64
-            p4est_test: P4EST_JLL_USES_MPI_PRE_GENERATED_BINDINGS
+            JULIA_P4EST_TEST: P4EST_CUSTOM_MPI_CUSTOM
+          # Run only selected tests on macOS and Windows, since not everything
+          # works everywhere yet and we are most familiar with setting up custom
+          # p4est and MPI builds on Linux.
+          - os: macos-latest
+            JULIA_P4EST_TEST: P4EST_CUSTOM_MPI_CUSTOM
+          - os: windows-latest
+            JULIA_P4EST_TEST: P4EST_CUSTOM_MPI_CUSTOM
     steps:
       - uses: actions/checkout@v3
       - name: Set p4est release to run tests against
+        # We use an older release of `p4est` on purpose than the version
+        # provided by P4est_jll.jl to check whether everything still works
+        # in another version of `p4est` than the version used to generate the
+        # bindings.
         run: |
           P4EST_RELEASE=2.3.1
           echo "P4EST_RELEASE=$P4EST_RELEASE" >> $GITHUB_ENV
           echo "P4EST_RELEASE=$P4EST_RELEASE"
         shell: bash
       - name: Set environment variables
         run: |
-          echo "Setting variables for '${{ matrix.p4est_test }}'..."
-          echo "P4EST_TEST=${{ matrix.p4est_test }}" >> $GITHUB_ENV
-          echo "P4EST_TEST=${{ matrix.p4est_test }}"
-          if [[ ${{ matrix.p4est_test }} == "P4EST_JLL_USES_MPI_PRE_GENERATED_BINDINGS" ]]; then
+          echo "Setting variables for '${{ matrix.JULIA_P4EST_TEST }}'..."
+          echo "JULIA_P4EST_TEST=${{ matrix.JULIA_P4EST_TEST }}" >> $GITHUB_ENV
+          echo "JULIA_P4EST_TEST=${{ matrix.JULIA_P4EST_TEST }}"
+          if [[ ${{ matrix.JULIA_P4EST_TEST }} == "P4EST_JLL_MPI_DEFAULT" ]]
+          then
             echo "(no additional environment variables required)"
-          elif [[ ${{ matrix.p4est_test }} == "P4EST_JLL_USES_MPI" ]]; then
-            echo "JULIA_P4EST_GENERATE_BINDINGS=yes" >> $GITHUB_ENV
-            echo "JULIA_P4EST_GENERATE_BINDINGS=yes"
-          elif [[ ${{ matrix.p4est_test }} == "P4EST_CUSTOM_NON_MPI" ]]; then
-            echo "JULIA_P4EST_GENERATE_BINDINGS=yes" >> $GITHUB_ENV
-            echo "JULIA_P4EST_GENERATE_BINDINGS=yes"
-            P4EST_TMP=`pwd`/libp4est_tmp_non_mpi
-            echo "P4EST_TMP=$P4EST_TMP" >> $GITHUB_ENV
-            echo "P4EST_TMP=$P4EST_TMP"
-            echo "JULIA_P4EST_LIBRARY=$P4EST_TMP/prefix/lib/libp4est.so" >> $GITHUB_ENV
-            echo "JULIA_P4EST_LIBRARY=$P4EST_TMP/prefix/lib/libp4est.so"
-            echo "JULIA_P4EST_INCLUDE=$P4EST_TMP/prefix/include" >> $GITHUB_ENV
-            echo "JULIA_P4EST_INCLUDE=$P4EST_TMP/prefix/include"
-          elif [[ ${{ matrix.p4est_test }} == "P4EST_CUSTOM_USES_MPI" ]]; then
-            echo "JULIA_P4EST_GENERATE_BINDINGS=yes" >> $GITHUB_ENV
-            echo "JULIA_P4EST_GENERATE_BINDINGS=yes"
-            P4EST_TMP=`pwd`/libp4est_tmp_uses_mpi
+          elif [[ ${{ matrix.JULIA_P4EST_TEST }} == "P4EST_CUSTOM_MPI_CUSTOM" ]]
+          then
+            P4EST_TMP=`pwd`/libp4est_tmp
             echo "P4EST_TMP=$P4EST_TMP" >> $GITHUB_ENV
             echo "P4EST_TMP=$P4EST_TMP"
-            echo "JULIA_P4EST_LIBRARY=$P4EST_TMP/prefix/lib/libp4est.so" >> $GITHUB_ENV
-            echo "JULIA_P4EST_LIBRARY=$P4EST_TMP/prefix/lib/libp4est.so"
-            echo "JULIA_P4EST_INCLUDE=$P4EST_TMP/prefix/include" >> $GITHUB_ENV
-            echo "JULIA_P4EST_INCLUDE=$P4EST_TMP/prefix/include"
-            echo "JULIA_P4EST_USES_MPI=yes" >> $GITHUB_ENV
-            echo "JULIA_P4EST_USES_MPI=yes"
-            echo "JULIA_P4EST_MPI_INCLUDE=/usr/include/x86_64-linux-gnu/mpich" >> $GITHUB_ENV
-            echo "JULIA_P4EST_MPI_INCLUDE=/usr/include/x86_64-linux-gnu/mpich"
+            echo "JULIA_P4EST_TEST_LIBP4EST=$P4EST_TMP/prefix/lib/libp4est.so" >> $GITHUB_ENV
+            echo "$JULIA_P4EST_TEST_LIBP4EST=$P4EST_TMP/prefix/lib/libp4est.so"
           else
-            echo "ERROR: unknown test type ${{ matrix.p4est_test }}" >&2
+            echo "ERROR: unknown test type ${{ matrix.JULIA_P4EST_TEST }}" >&2
             exit 1
           fi
           echo "Done."
         shell: bash
-      - name: Install custom p4est libraries for testing
-        run: |
-          if [[ ${{ matrix.p4est_test }} == "P4EST_CUSTOM_USES_MPI" ]]; then
-            sudo apt-get install -y mpich libmpich-dev
-          fi
-          ./.ci_install_p4est.sh
-        shell: bash
       - uses: julia-actions/setup-julia@v1
         with:
           version: ${{ matrix.version }}
           arch: ${{ matrix.arch }}
           show-versioninfo: true
       - uses: julia-actions/cache@v1
+      - name: Install custom MPI library for testing
+        run: |
+          if [[ ${{ matrix.JULIA_P4EST_TEST }} == "P4EST_CUSTOM_MPI_CUSTOM" ]]
+          then
+            sudo apt-get install -y openmpi-bin libopenmpi-dev
+          fi
+        shell: bash
+      - name: Install custom p4est library for testing
+        run: |
+          ./.ci_install_p4est.sh
+        shell: bash
+      - name: Configure MPI.jl
+        shell: julia --color=yes {0}
+        run: |
+          @show pwd()
+          include(joinpath(pwd(), "test", "configure_packages.jl"))
       - uses: julia-actions/julia-buildpkg@v1
         env:
           PYTHON: ""
       - uses: julia-actions/julia-runtest@v1
         env:
           PYTHON: ""
       - uses: julia-actions/julia-processcoverage@v1
-      - uses: coverallsapp/github-action@master
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          flag-name: run-${{ matrix.os }}-${{ matrix.version }}-${{ matrix.arch }}
-          parallel: true
-          path-to-lcov: ./lcov.info
       - uses: codecov/codecov-action@v3
         with:
           token: ${{ secrets.CODECOV_TOKEN }} # not required for public repos
@@ -155,13 +129,73 @@ jobs:
           name: codecov-umbrella # optional
           fail_ci_if_error: true # optional (default = false)
           # verbose: true # optional (default = false)
+      # The standard setup of Coveralls is just annoying for parallel builds, see, e.g.,
+      # https://github.com/trixi-framework/Trixi.jl/issues/691
+      # https://github.com/coverallsapp/github-action/issues/47
+      # https://github.com/coverallsapp/github-action/issues/67
+      # This standard setup is reproduced below for completeness.
+      # - uses: coverallsapp/github-action@master
+      #   with:
+      #     github-token: ${{ secrets.GITHUB_TOKEN }}
+      #     flag-name: run-${{ matrix.JULIA_P4EST_TEST }}-${{ matrix.os }}-${{ matrix.version }}-${{ matrix.arch }}-${{ github.run_id }}
+      #     parallel: true
+      #     path-to-lcov: ./lcov.info
+      # Instead, we use a more tedious approach:
+      # - Store all individual coverage files as artifacts (directly below)
+      # - Download and merge individual coverage reports in another step
+      # - Upload only the merged coverage report to Coveralls
+      - shell: bash
+        run: |
+          cp ./lcov.info ./lcov-${{ matrix.JULIA_P4EST_TEST }}-${{ matrix.os }}-${{ matrix.version }}-${{ matrix.arch }}.info
+      - uses: actions/upload-artifact@v3
+        with:
+          name: lcov-${{ matrix.JULIA_P4EST_TEST }}-${{ matrix.os }}-${{ matrix.version }}-${{ matrix.arch }}
+          path: ./lcov-${{ matrix.JULIA_P4EST_TEST }}-${{ matrix.os }}-${{ matrix.version }}-${{ matrix.arch }}.info
 
   finish:
     needs: test
     runs-on: ubuntu-latest
     steps:
-    - name: Coveralls Finished
-      uses: coverallsapp/github-action@master
-      with:
-        github-token: ${{ secrets.GITHUB_TOKEN }}
-        parallel-finished: true
+      # The standard setup of Coveralls is just annoying for parallel builds, see, e.g.,
+      # https://github.com/trixi-framework/Trixi.jl/issues/691
+      # https://github.com/coverallsapp/github-action/issues/47
+      # https://github.com/coverallsapp/github-action/issues/67
+      # This standard setup is reproduced below for completeness.
+      # - name: Coveralls Finished
+      #   uses: coverallsapp/github-action@master
+      #   with:
+      #     github-token: ${{ secrets.GITHUB_TOKEN }}
+      #     parallel-finished: true
+      # Instead, we use the more tedious approach described above.
+      # At first, we check out the repository and download all artifacts
+      # (and list files for debugging).
+      - uses: actions/checkout@v3
+      - uses: actions/download-artifact@v3
+      - run: ls -R
+      # Next, we merge the individual coverage files and upload
+      # the combined results to Coveralls.
+      - name: Merge lcov files using Coverage.jl
+        shell: julia --color=yes {0}
+        run: |
+          using Pkg
+          Pkg.activate(temp=true)
+          Pkg.add("Coverage")
+          using Coverage
+          coverage = LCOV.readfolder(".")
+          for cov in coverage
+            cov.filename = replace(cov.filename, "\\" => "/")
+          end
+          coverage = merge_coverage_counts(coverage)
+          @show covered_lines, total_lines = get_summary(coverage)
+          LCOV.writefile("./lcov.info", coverage)
+      - uses: coverallsapp/github-action@master
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          path-to-lcov: ./lcov.info
+      # Upload merged coverage data as artifact for debugging
+      - uses: actions/upload-artifact@v3
+        with:
+          name: lcov
+          path: ./lcov.info
+      # That's it
+      - run: echo "Finished testing P4est"
diff --git a/.gitignore b/.gitignore
@@ -1,9 +1,15 @@
 *.mem
 **/Manifest.toml
+dev/p4est_include
+dev/LibP4est.jl
 docs/build
+docs/src/authors.md
+docs/src/contributing.md
+docs/src/index.md
+docs/src/license.md
 public/
 coverage/
 coverage_report/
 **/*.jl.*.cov
-deps/libp4est.jl
-deps/*.log
+LocalPreferences.toml
+
diff --git a/AUTHORS.md b/AUTHORS.md
@@ -0,0 +1,22 @@
+# Authors
+
+The development of P4est.jl is coordinated by a group of *principal developers*,
+who are also its main contributors and who can be contacted in case of
+questions about P4est.jl. In addition, there are *contributors* who have
+provided substantial additions or modifications. Together, these two groups form
+"The P4est.jl Authors" as mentioned in the [LICENSE.md](LICENSE.md) file.
+
+## Principal Developers
+
+- [Michael Schlottke-Lakemper](https://lakemper.eu)
+- [Hendrik Ranocha](https://ranocha.de)
+
+## Contributors
+
+The following people contributed major additions or modifications to Trixi and
+are listed in alphabetical order:
+
+- Alexander Astanin
+- Lars Christmann
+- Joshua Lampert
+- Lucas C Wilcox
diff --git a/Artifacts.toml b/Artifacts.toml
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -4,7 +4,7 @@ P4est.jl is an open-source project and we are very happy to accept contributions
 from the community. Please feel free to open issues or submit patches (preferably
 as merge requests) any time. For planned larger contributions, it is often
 beneficial to get in contact with one of the principal developers first (see
-[README.md](README.md)).
+[AUTHORS.md](AUTHORS.md)).
 
 P4est.jl and its contributions are licensed under the MIT license (see
 [LICENSE.md](LICENSE.md)). As a contributor, you certify that all your