-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Major Changes
+ Several breaking changes to the C++ API to make it more native to C++.
The C api is uneffected.
+ new required `prefix` api added to return the prefix of the
compressor using in options.
+ get_metrics_results(), get_metrics(), set_options(), get_options(),
check_options(), get_configuration(), pressio_data_for_each(),
and associated `_impl` methods now use references or values instead
of pointers to be more consistent with C++ style.
+ New operator bool() for `pressio_compressor` to check if the pointer
is occupied.
+ Refactored compressor plugins to use the C++ interface.
+ Changed `BUILD_TESTS` to `BUILD_TESTING` to match the CMake standard.
+ Marked all of the `std_compat.h` functions as internal. Use them
outside of libpressio at your own risk.
+ BREAKING CHANGE `pressio_compressor_check_options` default
implementation now only considers options in the same prefix as
potentially conflicting, this allows more usages than previously
allowed.
Minor Changes
+ Documented how to write a compressor and metrics plugin. Added test
cases for this use.
+ Documented that libpressio now compiles on the gcc 4.8.5
+ Documented the list of builtin plugins
Bug Fixes
+ Correctly pass valgrind options to CTest. Previously the wrong
file name and option names were passed.
+ Fixed a memory leak in `pressio_data::operator=(pressio_data&&)`- Loading branch information
Showing
34 changed files
with
935 additions
and
345 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| set(MEMORYCHECK_COMMAND_OPTIONS "--leak-check=full --show-leak-kinds=all --track-origins=yes") | ||
| set(MEMORYCHECK_SUPPRESSIONS_FILE "${CMAKE_SOURCE_DIR}/test/valgrind.supp") |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # Built-in Modules {#builtins} | ||
|
|
||
| Libpression provides a number of builtin compressor and metrics modules. | ||
|
|
||
| ## Compressor Plugins | ||
|
|
||
| + `sz` -- the SZ error bounded lossy compressor | ||
| + `zfp` -- the ZFP error bounded lossy compressor | ||
| + `mgard` -- the MGARD error bounded lossy compressor | ||
| + `blosc` -- the blosc lossless compressor | ||
| + `magick` -- the ImageMagick image compression/decompression library | ||
|
|
||
| ## Metrics Plugins | ||
|
|
||
| + `time` -- time information on each compressor API | ||
| + `error_stat` -- statistics on the difference between the uncompressed and decompressed values that can be computed in one pass in linear time. | ||
| + `size` -- information on the size of the compressed and decompressed data |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,166 @@ | ||
| # Writing a Compressor Plugin {#writingacompressor} | ||
|
|
||
| Libpressio supports adding custom compressor plugins. | ||
|
|
||
| To create a compressor plugin, simply create a subclass of `libpressio_compressor_plugin` that implements the compressor functions that you are interested in, and register the plugin with libpressio. | ||
|
|
||
| For example, let's create a compressor plugin that preforms a log transform as a preprocessing step to compression, and does a exponential transformation as a post processing step to decompression. Some scientific data compresses much better in the log domain than in the normal domain. | ||
|
|
||
| First you will need to include several headers. | ||
|
|
||
| ```cpp | ||
| #include <cmath> //for exp and log | ||
| #include "libpressio_ext/cpp/data.h" //for access to pressio_data structures | ||
| #include "libpressio_ext/cpp/compressor.h" //for the libpressio_compressor_plugin class | ||
| #include "libpressio_ext/cpp/options.h" // for access to pressio_options | ||
| #include "libpressio_ext/cpp/pressio.h" //for the plugin registries | ||
| ``` | ||
|
|
||
| First we are going to delegate most of the methods to the underlying compressor. | ||
| To do so, we are going to do this in one of the most naive was possible and accept the compressor as a constructor argument. | ||
| Then we are going to provide overrides that pass each of the methods to the underlying compressor. | ||
|
|
||
| Notice the function `check_error`. | ||
| Since we are using composition rather than inheritance, we need to propagate error messages to our wrapper. | ||
| The `check_error` function simply check for the error condition in the underlying function and propagates the error if these is one. | ||
|
|
||
| Also notice that we are using the `_impl` versions of the various methods. | ||
| This is because the base class is doing some work to provide metrics functionality and basic error checking that we want to use. | ||
|
|
||
| ```cpp | ||
| class log_transform : public libpressio_compressor_plugin { | ||
| public: | ||
| log_transform(): compressor(nullptr) {} | ||
| log_transform(pressio_compressor&& comp): compressor(std::move(comp)) {} | ||
|
|
||
|
|
||
| //TODO compress and decompress, see below | ||
|
|
||
|
|
||
| //getting and setting options/configuration | ||
| pressio_options get_options_impl() const override { | ||
| auto options = compressor.plugin->get_options(); | ||
| options.set("log:compressor", (void*)&compressor); | ||
| return options; | ||
| } | ||
| int set_options_impl(pressio_options const& options) override { | ||
| //if the user hasn't set the compressor yet, fail | ||
| if(!compressor) return invalid_compressor(); | ||
| int rc = check_error(compressor.plugin->set_options(options)); | ||
|
|
||
| //try to extract the compressor plugin from the option | ||
| void* tmp; | ||
| if(options.get("log:compressor", &tmp) == pressio_options_key_set) { | ||
| compressor = std::move(*(pressio_compressor*)tmp); | ||
| } | ||
|
|
||
| return rc; | ||
| } | ||
| pressio_options get_configuration_impl() const override { | ||
| //if the user hasn't set the compressor yet, fail | ||
| if(!compressor) return pressio_options(); | ||
| return compressor.plugin->get_configuration(); | ||
| } | ||
| int check_options_impl(pressio_options const& options) override { | ||
| //if the user hasn't set the compressor yet, fail | ||
| if(!compressor) return invalid_compressor(); | ||
| return check_error(compressor.plugin->check_options(options)); | ||
| } | ||
|
|
||
| //getting version information | ||
| const char* prefix() const override { | ||
| return "log"; | ||
| } | ||
| const char* version() const override { | ||
| if(!compressor) return ""; | ||
| return compressor.plugin->version(); | ||
| } | ||
| int major_version() const override { | ||
| if(!compressor) return -1; | ||
| return compressor.plugin->major_version(); | ||
| } | ||
| int minor_version() const override { | ||
| if(!compressor) return -1; | ||
| return compressor.plugin->minor_version(); | ||
| } | ||
| int patch_version() const override { | ||
| if(!compressor) return -1; | ||
| return compressor.plugin->patch_version(); | ||
| } | ||
|
|
||
| private: | ||
| int check_error(int rc) { | ||
| if(rc) { | ||
| set_error( | ||
| compressor.plugin->error_code(), | ||
| compressor.plugin->error_msg() | ||
| ); | ||
| } | ||
| return rc; | ||
| } | ||
| int invalid_compressor() { return set_error(-1, "compressor must be set"); }; | ||
| pressio_compressor compressor; | ||
| }; | ||
| ``` | ||
| Now to the harder part, writing compress and decompress. | ||
|
|
||
|
|
||
| First we can write our log and exponential transform functions. | ||
|
|
||
| ```cpp | ||
| namespace { | ||
| struct log_fn{ | ||
| template <class T> | ||
| pressio_data operator()(T* begin, T* end) { | ||
| pressio_data log_data = pressio_data::clone(*input); | ||
| auto output_it = reinterpret_cast<T*>(log_data.data()); | ||
| std::transform(begin, end, output_it, [](T i){ return std::log(i); }); | ||
| return log_data; | ||
| } | ||
| pressio_data const* input; | ||
| }; | ||
|
|
||
| struct exp_fn{ | ||
| template <class T> | ||
| pressio_data operator()(T* begin, T* end) { | ||
| pressio_data log_data = pressio_data::clone(*output); | ||
| auto output_it = reinterpret_cast<T*>(log_data.data()); | ||
| std::transform(begin, end, output_it, [](T i){ return std::exp(i); }); | ||
| return log_data; | ||
| } | ||
| pressio_data const* output; | ||
| }; | ||
| } | ||
| ``` | ||
|
|
||
| We will use these methods using the `pressio_data_for_each` API to preform the transform | ||
| Now, we can write compress and decompress: | ||
|
|
||
|
|
||
| ```cpp | ||
| //compress and decompress | ||
| int compress_impl(pressio_data const* input, pressio_data* output) override { | ||
| if(!compressor) return invalid_compressor(); | ||
| pressio_data log_input = pressio_data_for_each<pressio_data>(*input, log_fn{input}); | ||
| return check_error(compressor.plugin->compress(&log_input, output)); | ||
| } | ||
|
|
||
| int decompress_impl(pressio_data const* input, pressio_data* output) override { | ||
| if(!compressor) return invalid_compressor(); | ||
| int rc = compressor.plugin->decompress(input, output); | ||
| *output = pressio_data_for_each<pressio_data>(*output, exp_fn{output}); | ||
| return check_error(rc); | ||
| } | ||
| ``` | ||
| We finally register the library with libpressio: | ||
| ```cpp | ||
| static pressio_register X(compressor_plugins(), "log", [](){ return std::make_unique<log_transform>();}); | ||
| ``` | ||
|
|
||
| High quality compressor modules may be accepted into libpressio. Contributed modules should be placed in to | ||
| the `src/plugins/compressors/` directory and added to the main CMakeLists.txt. If the compressor plugin | ||
| requires an external dependency, it should be hidden behind a configuration option. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| # Writing a User Metrics Plugin {#writingametric} | ||
|
|
||
| libpressio supports adding custom metrics plugins. | ||
|
|
||
|
|
||
| To create a metrics plugin, simply create a subclass of `libpressio_metrics_plugin` that implements the check that you are interested in, and register the plugin with libpressio. | ||
|
|
||
| For example, let's create a plugin that counts the number of compressions of each data type that the user preforms. | ||
|
|
||
| First we will include a number of required headers: | ||
|
|
||
| ```cpp | ||
| //required for all plugins | ||
| #include "libpressio_ext/cpp/metrics.h" // provides the libpressio_metrics_plugin | ||
| #include "libpressio_ext/cpp/options.h" // provides pressio_options and related methods | ||
| #include "libpressio_ext/cpp/pressio.h" // for the plugin registry | ||
|
|
||
| //required for this plugin | ||
| #include "libpressio_ext/cpp/data.h" // to be able to access ->dtype() | ||
| #include <memory> //for make_unique (in c++14 or later) | ||
| #include <map> //for the map that counts what data type was used | ||
|
|
||
| ``` | ||
|
|
||
| Next, we need to write our class. Since we want to count the number of data buffers of each type that we compress, we will hook the `begin_compress` method. Alternatively we could also hook the `end_compress` method. Once we have the counts, we report them out using the `get_metrics_results` function. We do this like so: | ||
|
|
||
| ```cpp | ||
| class counting_metric: public libpressio_metrics_plugin { | ||
| counting_metric() { | ||
| //operator[] is non-const, so explicits instantiate each of the values we need | ||
| counts[pressio_int8_dtype] = 0; | ||
| counts[pressio_int16_dtype] = 0; | ||
| counts[pressio_int32_dtype] = 0; | ||
| counts[pressio_int64_dtype] = 0; | ||
| counts[pressio_uint8_dtype] = 0; | ||
| counts[pressio_uint16_dtype] = 0; | ||
| counts[pressio_uint32_dtype] = 0; | ||
| counts[pressio_uint64_dtype] = 0; | ||
| counts[pressio_float_dtype] = 0; | ||
| counts[pressio_double_dtype] = 0; | ||
| counts[pressio_byte_dtype] = 0; | ||
| } | ||
|
|
||
| //increment the count at the beginning of each compress | ||
| void begin_compress(pressio_data const* input, pressio_data const*) override { | ||
| counts[input->dtype()]++; | ||
| } | ||
|
|
||
| //return an options structure containing the metrics we care about | ||
| //notice how the namespaced matches the name of the plugin that is registered below | ||
| pressio_options get_metrics_results() const override { | ||
| pressio_options opts; | ||
| opts.set("counts:int8", counts.at(pressio_int8_dtype)); | ||
| opts.set("counts:int16", counts.at(pressio_int16_dtype)); | ||
| opts.set("counts:int32", counts.at(pressio_int32_dtype)); | ||
| opts.set("counts:int64", counts.at(pressio_int64_dtype)); | ||
| opts.set("counts:uint8", counts.at(pressio_uint8_dtype)); | ||
| opts.set("counts:uint16", counts.at(pressio_uint16_dtype)); | ||
| opts.set("counts:uint32", counts.at(pressio_uint32_dtype)); | ||
| opts.set("counts:uint64", counts.at(pressio_uint64_dtype)); | ||
| opts.set("counts:float", counts.at(pressio_float_dtype)); | ||
| opts.set("counts:double", counts.at(pressio_double_dtype)); | ||
| opts.set("counts:byte", counts.at(pressio_byte_dtype)); | ||
| return opts; | ||
| } | ||
|
|
||
| std::map<pressio_dtype, unsigned int> counts; | ||
| }; | ||
| ``` | ||
| Finally, we will register the plugin in the under the names "counts" in the metrics plugging registry | ||
| ```cpp | ||
| static pressio_register X(metrics_plugins(), "counts", [](){ return std::make_unique<counting_metric>(); }); | ||
| ``` | ||
|
|
||
| Then a user of the library can then ask libpressio to construct their new plugin as normal. | ||
| But what if our metrics modules takes arguments? | ||
| Instead of registering it directly, the user can instantiate it manually and combine it with others from the library using the `make_m_composite` method. | ||
|
|
||
| If your metric is suitably general and well implemented, you can contribute it back to libpressio. | ||
| Metrics should be added to the `src/plugins/metrics/` directory and to the main `CMakeLists.txt`. If the metric requires additional dependencies beyond the standard library, it should be hidden behind a configuration flag that enables its use and disabled by default. | ||
|
|
||
| This full example can be found in the `test/test_regsiter_metrics.cc` file, and the full documentation of the allowed hooks and their arguments can be found in the documentation for `libpressio_metrics_plugin`. |
Oops, something went wrong.