Skip to content

Commit

Permalink
Update README for building
Browse files Browse the repository at this point in the history
  • Loading branch information
@mattjala
mattjala committed Aug 30, 2023
1 parent 797e5f1 commit 46f16ef
Show file tree
Hide file tree
Showing 2 changed files with 50 additions and 213 deletions.
4 changes: 4 additions & 0 deletions .github/workflows/main.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,10 @@ jobs:
- name: Install HDF + REST VOL
run: make install
working-directory: ${{ github.workspace }}/hdf5/build

- name: Install valgrind
run: sudo apt install valgrind
working-directory: ${{ github.workspace }}

- name: Start HSDS
if: ${{ matrix.endpoint != 'http://127.0.0.1:5101'}}
Expand Down
259 changes: 46 additions & 213 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,9 @@ HDF5 REST VOL connector - currently under development
- [II.A.i. External Libraries](#iiai-external-libraries)
- [II.A.ii. HDF5 REST API server access](#iiaii-hdf5-rest-api-server-access)
- [II.B. Building the REST VOL connector](#iib-building-the-rest-vol-connector)
- [II.B.i. Obtaining the Source](#iibi-obtaining-the-source)
- [II.B.ii. One-step Build](#iibii-one-step-build)
- [II.B.ii.a. Build Script Options](#iibiia-build-script-options)
- [II.B.iii. Manual Build](#iibiii-manual-build)
- [Autotools](#autotools)
- [CMake](#cmake)
- [II.B.iii.a. Options for `configure`](#iibiiia-options-for-configure)
- [II.B.iii.b. Options for CMake](#iibiiib-options-for-cmake)
- [II.B.iv. Build Results](#iibiv-build-results)
- [II.B.i. Building with CMake](#iibi-building-with-cmake)
- [II.B.ii. Options for CMake](#iibii-options-for-cmake)
- [II.B.iii. Build Results](#iibiii-build-results)
- [III. Using/Testing the REST VOL connector](#iii-usingtesting-the-rest-vol-connector)
- [IV. More Information](#iv-more-information)

Expand Down Expand Up @@ -59,12 +53,10 @@ Before building and using the HDF5 REST VOL connector, a few requirements must b
To build the REST VOL connector, the following libraries are required:

+ libhdf5 - The [HDF5](https://www.hdfgroup.org/downloads/hdf5/) library. The HDF5 library
used must be at least version 1.12.0; for convenience, a source distribution of
HDF5 has been included with the REST VOL connector and can be used during the
build process. If a pre-built HDF5 distribution is being used instead, it should
be built as a shared library only for maximal compatibility with the REST VOL
connector. Using statically-built HDF5 libraries can cause issues with the REST
VOL connector under certain circumstances.
used must be at least version 1.14.0. The REST VOL connector must be built
at the same time as the HDF5 library itself, as described in section II.B.
The HDF5 library must also be built as a shared library, as statically-built
HDF5 libraries can cause issues with the REST VOL connector under certain circumstances.

+ libcurl - networking support
+ https://curl.haxx.se/
Expand All @@ -76,233 +68,77 @@ Compiled libraries must either exist in the system's library paths or must be
pointed to during the REST VOL connector build process. Refer to section II.B.ii.
below for more information.


### II.A.ii. HDF5 REST API server access

Additionally, the HDF5 REST VOL connector requires access to a server which implements
The HDF5 REST VOL connector requires access to a server which implements
the HDF5 REST API.

For more information on The HDF Group's officially supported service, please see
https://www.hdfgroup.org/hdf-kita.

https://github.com/HDFGroup/hsds and https://www.hdfgroup.org/hdf-kita.

## II.B. Building the REST VOL connector

### II.B.i. Obtaining the Source
### II.B.i. Building with CMake

The latest and most up-to-date REST VOL connector code can be viewed at:

https://github.com/HDFGroup/vol-rest

and can directly be obtained from:

`git clone https://github.com/HDFGroup/vol-rest`

For building with the 1.12 or later version of the HDF5 library, use the hdf5_1_12_update branch of this repository.

A source distribution of the HDF5 library has been included in the REST VOL connector
source in the `/src/hdf5` directory.

### II.B.ii. One-step Build

Use one of the supplied Autotools or CMake build scripts, depending on preference or
system support.

+ Autotools

Run `build_vol_autotools.sh`. See section II.B.ii.a for configuration options.


+ CMake

Run `build_vol_cmake.sh` (Linux or OS X) or `build_vol_cmake.bat` (Windows).
See section II.B.ii.a for configuration options.


By default, these build scripts will compile and link with the provided HDF5 source
distribution. However, if you wish to use a manually built version of the HDF5 library,
include the flag `-H <dir>` where `dir` is the path to the HDF5 install prefix.

NOTE: For those who are capable of using both build systems, the autotools build currently
does not support out-of-tree builds. If the REST VOL source directory is used for an autotools
build, it is important not to re-use the source directory for a later build using CMake.
This can cause build conflicts and result in strange and unexpected behavior.

In order to clone and build the REST VOL connector at the same time as the HDF5 library, follow the steps listed here:

#### II.B.ii.a. Build Script Options
https://github.com/HDFGroup/hdf5/blob/develop/doc/cmake-vols-fetchcontent.md

The following configuration options are available to all of the build scripts:
while providing the following CMake variables to HDF5 at configuration time:

-h Prints out a help message indicating script usage and available options.
```HDF5_VOL_URL01=https://github.com/HDFGroup/vol-rest
HDF5_VOL_VOL-REST_BRANCH="master"
HDF5_VOL_VOL-REST_NAME="REST"
-d Enables debugging information printouts within the REST VOL connector.
HDF5_ALLOW_EXTERNAL_SUPPORT="GIT"
HDF5_VOL_ALLOW_EXTERNAL=ON
HDF5_TEST_API=ON
-c Enables debugging information printouts from cURL within the REST VOL connector.

-m Enables memory usage tracking within the REST VOL connector. This option is
mostly useful in helping to diagnose any possible memory leaks or other
memory errors within the connector.

-g Enables symbolic debugging of the REST VOL code. (Only available for
`build_vol_autotools.sh`)

-P DIR Specifies where the REST VOL connector should be installed. The default
installation prefix is `rest_vol_build` inside the REST VOL connector source
root directory.

-H DIR Prevents building of the provided HDF5 source. Instead, uses the compiled
library found at directory `DIR`, where `DIR` is the path used as the
installation prefix when building HDF5 manually.

-C DIR Specifies the top-level directory where cURL is installed. Used if cURL is
not installed to a system path or used to override

-Y DIR Specifies the top-level directory where YAJL is installed. Used if YAJL is
not installed to a system path or used to override

Additionally, the CMake build scripts have the following configuration options:

-B DIR Specifies the directory that CMake should use as the build tree location.
The default build tree location is `rest_vol_cmake_build_files` inside the
REST VOL connector source root directory. Note that the REST VOL does not
support in-source CMake builds.

-G DIR Specifies the CMake Generator to use when generating the build files
for the project. On Unix systems, the default is "Unix Makefiles" and if
this is not changed, the build script will automatically attempt to build
the project after generating the Makefiles. If the generator is changed, the
build script will only generate the build files and the build command to
build the project will have to be run manually.

### II.B.iii. Manual Build

In general, the process for building the REST VOL connector involves either obtaining a VOL-enabled
HDF5 distribution or building one from source. Then, the REST VOL connector is built using that
HDF5 distribution by including the appropriate header files and linking against the HDF5 library.
If you wish to manually build HDF5 from the included source distribution, first run the following
commands from the root directory of the REST VOL connector source code in order to checkout the git
submodule and then proceed to build HDF5 as normal.

```bash
$ git submodule init
$ git submodule update
HDF5_ENABLE_SZIP_SUPPORT=OFF
HDF5_ENABLE_Z_LIB_SUPPORT=OFF
HDF5_VOL_VOL-REST_TEST_PARALLEL=OFF
```

Once you have a VOL-enabled HDF5 distribution available, follow the instructions below for your
respective build system in order to build the REST VOL connector against the HDF5 distribution.

#### Autotools

```bash
$ cd rest-vol
$ ./autogen.sh
$ ./configure --prefix=INSTALL_DIR --with-hdf5=HDF5_DIR [options]
$ make
$ make check (requires HDF5 REST API server access -- see section II.A.ii.)
$ make install
```
Note that CMake requires variables to be prefixed with `-D`, such that the command would look like `cmake -DHDF5_VOL_URL01=https://github.com/HDFGroup/vol-rest -DHDF5_VOL_VOL-REST_BRANCH="master" ....`

#### CMake

First, create a build directory within the source tree:

```bash
$ cd rest-vol
$ mkdir build
$ cd build
```

Then, if all of the required components (HDF5, cURL and YAJL) are located within the system path,
building the connector should be as simple as running the following two commands to first have CMake
generate the build files to use and then to build the connector. If the required components are
located somewhere other than the system path, refer to section II.B.iii.b. for information on how to
If the required components (cURL, YAJL) are located somewhere other than the system path, refer to section II.B.ii. for information on how to
point to their locations.

```bash
$ cmake -DPREBUILT_HDF5_DIR=HDF5_DIR [options] ..
$ make && make install (command may differ depending on platform and cmake generator used)
```

and, optionally, run the following to generate a system package for the REST VOL connector:

```bash
$ cpack
```

The options that can be specified to control the build process are covered in section II.B.iii.b.
The options that can be specified to control the build process are covered in section II.B.ii.
Note that by default CMake will generate Unix Makefiles for the build, but other build files can
be generated by specifying the `-G` option for the `cmake` command;
see [CMake Generators](https://cmake.org/cmake/help/v3.16/manual/cmake-generators.7.html) for more
information.

#### II.B.iii.a. Options for `configure`

When building the REST VOL connector manually using Autotools, the following options are
available to `configure`.

The options in the supplied Autotools build script are mapped to the corresponding options here:

-h, --help Prints out a help message indicating script usage and available
options.

--prefix=DIR Specifies the location for the resulting files. The default location
is `rest_vol_build` in the same directory as configure.

--enable-build-mode=(production|debug)
Sets the build mode to be used.
Debug - enable debugging printouts within the REST VOL connector.
Production - Focus more on optimization.

--enable-curl-debug
Enables debugging information printouts from cURL within the
REST VOL connector.
#### II.B.ii. Options for CMake

--enable-mem-tracking
Enables memory tracking within the REST VOL connector. This option is
mostly useful in helping to diagnose any possible memory leaks or
other memory errors within the connector.

--enable-tests
Enables/Disables building of the REST VOL connector tests.

--enable-examples
Enables/Disables building of the REST VOL HDF5 examples.

--with-hdf5=DIR Used to specify the directory where an HDF5 distribution that uses
the VOL layer has already been built. This is to help the REST VOL
connector locate the HDF5 header files that it needs to include.

--with-curl=DIR Used to specify the top-level directory where cURL is installed, if
cURL is not installed to a system path.

--with-yajl=DIR Used to specify the top-level directory where YAJL is installed, if
YAJL is not installed to a system path.


#### II.B.iii.b. Options for CMake

When building the REST VOL connector manually using CMake, the following CMake variables are
When building HDF5 and the REST VOL connector with CMake, the following CMake variables are
available for controlling the build process. These can be supplied to the `cmake` command by
prepending them with `-D`. Some of these options may be needed if, for example, the required
components mentioned previously cannot be found within the system path.

CMake-specific options:

* `CMAKE_INSTALL_PREFIX` - This option controls the install directory that the resulting output files are written to. The default value is `/usr/local`.
* `CMAKE_BUILD_TYPE` - This option controls the type of build used for the VOL connector. Valid values are Release, Debug, RelWithDebInfo and MinSizeRel; the default build type is RelWithDebInfo.
* `CMAKE_BUILD_TYPE` - This option controls the type of build used for HDF5 and the VOL connector. Valid values are Release, Debug, RelWithDebInfo and MinSizeRel; the default build type is RelWithDebInfo.

HDF5 + REST VOL options:

* `BUILD_TESTING` - This option is used to enable/disable building of HDF5 and the REST VOL connector's tests. The default value is `ON`.
* `BUILD_SHARED_LIBS` - This option is used to enable/disable building HDF5 and the REST VOL connector's shared library. The default value is `ON`.
* `BUILD_STATIC_LIBS` - This option is used to enable/disable building HDF5 and the REST VOL connector's static library. The default value is `OFF`.
* `BUILD_STATIC_EXECS` - This option is used to enable/disable building HDF5 and the REST VOL connector's static executables. The default value is `OFF`.

HDF5-specific options:

* `HDF5_USE_STATIC_LIBRARIES` - Indicate if the static HDF5 libraries should be used for linking. The default value is `OFF`.

REST VOL Connector-specific options:

* `PREBUILT_HDF5_DIR` - Specifies a directory which contains a pre-built HDF5 distribution which uses the VOL abstraction layer. By default, the REST VOL connector's CMake build will attempt to build the included HDF5 source distribution, then use that to build the connector itself. However, if a VOL-enabled HDF5 distribution is already available, this option can be set to point to the directory of the HDF5 distribution. In this case, CMake will use that HDF5 distribution to build the REST VOL connector and will not attempt to build HDF5 again.
* `BUILD_TESTING` - This option is used to enable/disable building of the REST VOL connector's tests. The default value is `ON`.
* `BUILD_EXAMPLES` - This option is used to enable/disable building of the REST VOL connector's HDF5 examples. The default value is `ON`.
* `BUILD_SHARED_LIBS` - This option is used to enable/disable building the REST VOL connector's shared library. The default value is `ON`.
* `BUILD_STATIC_LIBS` - This option is used to enable/disable building the REST VOL connector's static library. The default value is `OFF`.
* `BUILD_STATIC_EXECS` - This option is used to enable/disable building the REST VOL connector's static executables. The default value is `OFF`.
* `HDF5_VOL_REST_ENABLE_COVERAGE` - Enables/Disables code coverage for HDF5 REST VOL connector libraries and programs. The default value is `OFF`.
* `HDF5_VOL_REST_ENABLE_DEBUG` - Enables/Disables debugging printouts within the REST VOL connector. The default value is `OFF`.
* `HDF5_VOL_REST_ENABLE_EXAMPLES` - Indicate that building of the examples should be enabled. The default value is `ON`.
Expand All @@ -311,13 +147,11 @@ REST VOL Connector-specific options:
* `HDF5_VOL_REST_THREAD_SAFE` - Enables/Disables linking to HDF5 statically compiled with thread safe option. The default value is `OFF`.
* `YAJL_USE_STATIC_LIBRARIES` - Indicate if the static YAJL libraries should be used for linking. The default value is `OFF`.

Note, when setting BUILD_SHARED_LIBS=ON and YAJL_USE_STATIC_LIBRARIES=ON, the static YAJL libraries have be build with the position independent code (PIC) option enabled. In the static YAJL build,
this PIC option has been turned off by default.
Note, when setting BUILD_SHARED_LIBS=ON and YAJL_USE_STATIC_LIBRARIES=ON, the static YAJL libraries have be build with the position independent code (PIC) option enabled. In the static YAJL build, this PIC option has been turned off by default.

### II.B.iii. Build Results

### II.B.iv. Build Results

If the build is successful, the following files will be written into the installation directory:
If the build is successful, the following files will be written into the installation directory, in addition to the usual files for an HDF5 source build:

```
bin/
Expand All @@ -336,16 +170,15 @@ lib/
share/
cmake/
FindYAJL.cmake
hdf5_vol_rest/
hdf5_vol_rest-config.cmake
hdf5_vol_rest-config-version.cmake
hdf5_vol_rest-targets.cmake
hdf5_vol_rest-targets-<build mode>.cmake
```

If the REST VOL connector was built using one of the included build scripts, all of the usual files
from an HDF5 source build should appear in the respective `bin`, `include`, `lib` and `share`
directories in the install directory. Notable among these is `bin/h5cc`, a special-purpose compiler wrapper script that streamlines the process of building HDF5 applications.
Notable among the files produced by the HDF library build is `bin/h5cc`, a special-purpose compiler wrapper script that streamlines the process of building HDF5 applications.


--------------------------------------------------------------------------------
Expand All @@ -361,6 +194,12 @@ to the REST VOL User's Guide under `docs/users_guide.pdf`.

# IV. More Information

+ Highly Scalable Data Service (HSDS) - A python-based implementation of the HDF5 REST API which
can send and receive HDF5 data through the use of HTTP requests
+ https://github.com/HDFGroup/hsds
+ https://portal.hdfgroup.org/display/KITA/Highly+Scalable+Data+Service
+ http://s3.amazonaws.com/hdfgroup/docs/hdf_data_services_scipy2017.pdf

+ HDF in the Cloud
+ https://www.hdfgroup.org/hdf-kita
+ https://www.hdfgroup.org/solutions/hdf-cloud
Expand All @@ -372,9 +211,3 @@ to the REST VOL User's Guide under `docs/users_guide.pdf`.

+ HDF5-JSON - A specification of and tools for representing HDF5 in JSON
+ http://hdf5-json.readthedocs.io/en/latest/

+ HDF Server (h5serv) - A python-based implementation of the HDF5 REST API which
can send and receive HDF5 data through the use of HTTP requests
+ https://github.com/HDFGroup/h5serv
+ https://support.hdfgroup.org/projects/hdfserver/
+ https://s3.amazonaws.com/hdfgroup/docs/HDFServer_SciPy2015.pdf

0 comments on commit 46f16ef

Please sign in to comment.