Skip to content

Commit

Permalink
libpressio version 0.32.0
Browse files Browse the repository at this point in the history
Major Changes

+ Introduced the concept of Generic IO as a new concept and extension
  point.  The existing HDF5 and POSIX formats are supported in Generic
  IO. Test cases and documentation were added to support this use case.
  At this point, General IO will not replace the existing IO functions.
  Existing IO functions MAY be deprecated in a upcoming release once the
  Generic IO features have had time to stabilize.
+ External Metrics version 2 now supports using generic IO to output the
  files for analysis.
+ `pressio_data` and `pressio_options` now take a `std::initalizer_list` as a
  constructor to make it easier to construct these types in testing and
  script-like code.  This means that these data structures are no longer
  trivially constructable.

Minor Changes

+ Some internal code was refactored to use newer interfaces.
  • Loading branch information
robertu94 committed Feb 14, 2020
1 parent 07d4f8e commit c7d6736
Show file tree
Hide file tree
Showing 17 changed files with 877 additions and 24 deletions.
5 changes: 4 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
cmake_minimum_required(VERSION 3.13 FATAL_ERROR)
project(libpressio VERSION "0.31.1" LANGUAGES CXX C)
project(libpressio VERSION "0.32.0" LANGUAGES CXX C)

#correct was to set a default build type
# https://blog.kitware.com/cmake-and-the-default-build-type/
Expand Down Expand Up @@ -86,6 +86,7 @@ add_library(libpressio
./src/plugins/metrics/error_stat.cc
./src/plugins/metrics/pearsons.cc
./src/plugins/io/posix.cc
./src/plugins/io/io.cc

#public headers
include/libpressio.h
Expand All @@ -96,7 +97,9 @@ add_library(libpressio
include/libpressio_ext/cpp/options.h
include/libpressio_ext/cpp/pressio.h
include/libpressio_ext/cpp/printers.h
include/libpressio_ext/cpp/io.h
include/libpressio_ext/io/posix.h
include/libpressio_ext/io/pressio_io.h
include/pressio.h
include/pressio_compressor.h
include/pressio_data.h
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ main(int argc, char* argv[])
## Getting Started
After skimming the example, LibPressio has 5 major headers that you will need to use:
After skimming the example, LibPressio has 6 major headers that you will need to use:
Type | Use
----------------------|-------------------
Expand All @@ -112,6 +112,7 @@ Type | Use
`pressio_data.h` | Represents data and associated metadata (size, type, dimentionality, memory ownership)
`pressio_options.h` | Maps between names and values, used for options for compressors and metrics results
`pressio_metrics.h` | A set of metrics to run while compressors run
`pressio_io.h` | An extension header that provides methods to load or store data from/to persistent storage
All of these are included by the convience header `libpressio.h`.
Expand Down
38 changes: 38 additions & 0 deletions docs/PressioOptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,41 @@ The external metrics module allows running 3rd party metrics without having to p
option | type | description
------------------------|-------------|------------
`external:command` | char* | the command to run

## IO Modules

As of LibPressio version 0.32.0, LibPressio gained support for Generic file IO plugins as an extension feature. This allows applications to easily load data using a variety of formats in a generic way.

Eventually the `pressio_io_data_*` methods may be deprecated in favor of these methods` methods may be deprecated in favor of these methods.

### Generic IO

IO Modules MAY support these "standard" options in addition to their own specific options. If a specific option is given, the generic option should be overwritten.

option | type | description
-----------------------|---------------|-----------------------------------------------------------------------------------
`io:path` | const char* | path accessible via the `open` system call on your platform to be read or written
`io:file_pointer` | FILE* | A POSIX FILE* pointer that was returned from `fopen()` to be read or written
`io:file_descriptor` | int32 | a POSIX file descriptor that was returned from `open()` to be read or written

### POSIX

A module to load or store data using the POSIX.1-2001 standard.

option | type | description
-----------------------|---------------|-----------------------------------------------------------------------------------
`io:path` | const char* | see "Generic IO" above
`io:file_pointer` | FILE* | see "Generic IO" above
`io:file_descriptor` | int32 | see "Generic IO" above


### HDF5

A module to load or store data using the HDF file format.

option | type | description
-----------------------|---------------|-----------------------------------------------------------------------------------
`io:path` | const char* | see "Generic IO" above
`hdf5:dataset` | const char* | path to the dataset within the hdf5 file


17 changes: 13 additions & 4 deletions docs/UsingTheExternalMetric.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,24 @@ In order to correctly communicate between LibPressio and an External script, str
The external module is not intended to replace writing metrics modules in C/C++.
Pull requests for such modules will not be accepted.

The `external` plugin will provide the following command line arguments to the script.

## Configuration Options

`external:command` -- the command to execute, the options passed by the module will be appended to this string
`external:io_format` -- the format to write the data to disk. It can be any format supported by `pressio_supported_io_modules`

Additionally any options passed to this metric will be passed to the IO format module.

## Command line Arguments

`--api` the maximum API version number the external module supports, begins at 1
The `external` plugin will provide the following command line arguments to the script.
These may change from version to version.

`--api` the maximum API version number the external module supports, begins at 1. The current version is 2

`--input` path to a temporary file containing the input data prior to compression. It will be a binary blob in row-major order.
`--input` path to a temporary file containing the input data prior to compression. (new in version 2) It will be according to the `external:io_format` option

`--decompressed` path to a temporary file containing the input data prior to compression.
`--decompressed` path to a temporary file containing the input data prior to compression. (new in version 2) It will be according to the `external:io_format` option

`--dim` dimension the dimensions of the dataset from low to high. This argument may be passed more than once. If passed more than once, the dimensions are given in order same order as the `pressio_data_new` functions.

Expand Down Expand Up @@ -70,6 +75,10 @@ ssim=.64
my_analysis=1.03e-3
```

### version 2:

No changes were made to the output format since version 1.


## Expected Standard Error

Expand Down
19 changes: 18 additions & 1 deletion include/libpressio_ext/cpp/data.h
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
#include <cstring>
#include <utility>
#include "pressio_data.h"
#include "pressio_dtype.h"
#include "libpressio_ext/cpp/dtype.h"
#include "libpressio_ext/compat/std_compat.h"

/**
Expand Down Expand Up @@ -273,6 +273,23 @@ struct pressio_data {
return *this;
}

/**
* construct a literal pressio_data object from a initializer list
*
* \param[in] il initializer list to use to create the data object
*/
template <class T>
pressio_data(std::initializer_list<T> il):
data_dtype(pressio_dtype_from_type<T>()),
data_ptr(malloc(il.size() * sizeof(T))),
metadata_ptr(nullptr),
deleter(pressio_data_libc_free_fn),
dims({il.size()})
{
std::copy(std::begin(il), std::end(il), static_cast<T*>(data_ptr));
}


/**
* \returns a non-owning pointer to the data
*/
Expand Down
6 changes: 6 additions & 0 deletions include/libpressio_ext/cpp/dtype.h
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
#ifndef LIBPRESSIO_DTYPE_CPP
#define LIBPRESSIO_DTYPE_CPP
#include <pressio_dtype.h>
#include <cstdint>

/**
* \file
* \brief C++ interface to data types
*/

/**
* Convert types to pressio_dtypes
Expand Down
186 changes: 186 additions & 0 deletions include/libpressio_ext/cpp/io.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
#ifndef PRESSIO_IO_PLUGIN_H
#define PRESSIO_IO_PLUGIN_H
#include <string>
#include <memory>

/**
* \file
* \brief C++ interface to read and write files
*/

struct pressio_data;
struct pressio_options;

/**
* plugin extension base class for io modules
*/
struct libpressio_io_plugin {
public:

virtual ~libpressio_io_plugin()=default;

/** reads a pressio_data buffer from some persistent storage. Modules should override read_impl instead.
* \param[in] data data object to populate, or nullptr to allocate it from
* the file if supported. callers should treat this buffer as if it is moved
* in a C++11 sense
*
* \returns a new read pressio data buffer containing the read information
* \see pressio_io_read for the semantics this function should obey
*/
struct pressio_data* read(struct pressio_data* data);

/** writes a pressio_data buffer to some persistent storage. Modules should override write_impl instead.
* \param[in] data data to write
* \see pressio_io_write for the semantics this function should obey
*/
int write(struct pressio_data const* data);

/** checks for extra arguments set for the io module. Modules should override check_options_impl instead.
* the default verison just checks for unknown options passed in.
*
* \see pressio_io_check_options for semantics this function obeys
* */
int check_options(struct pressio_options const&);

/** sets a set of options for the io_module
* \param[in] options to set for configuration of the io_module
* \see pressio_io_set_options for the semantics this function should obey
*/
int set_options(struct pressio_options const& options);
/** get the compile time configuration of a io module. Modules should override get_configuration_impl instead
*
* \see pressio_io_get_configuration for the semantics this function should obey
*/
struct pressio_options get_configuration() const;
/** get a set of options available for the io module. Modules should override get_options_impl instead
*
* The io module should set a value if they have been set as default
* The io module should set a "reset" value if they are required to be set, but don't have a meaningful default
*
* \see pressio_options.h for how to configure options
*/
struct pressio_options get_options() const;

/** get a implementation specific version string for the io module
* \see pressio_io_version for the semantics this function should obey
*/
virtual const char* version() const=0;
/** get the major version, default version returns 0
* \see pressio_io_major_version for the semantics this function should obey
*/
virtual int major_version() const;
/** get the minor version, default version returns 0
* \see pressio_io_minor_version for the semantics this function should obey
*/
virtual int minor_version() const;
/** get the patch version, default version returns 0
* \see pressio_io_patch_version for the semantics this function should obey
*/
virtual int patch_version() const;
/** get the error message for the last error
* \returns an implementation specific c-style error message for the last error
*/
const char* error_msg() const;
/** get the error code for the last error
* \returns an implementation specific integer error code for the last error, 0 is reserved for no error
*/
int error_code() const;
protected:
/**
* Should be used by implementing plug-ins to provide error codes
* \param[in] code a implementation specific code for the last error
* \param[in] msg a implementation specific message for the last error
* \returns the value passed to code
*/
int set_error(int code, std::string const& msg);

/** checks for extra arguments set for the io module.
* the default verison just checks for unknown options passed in.
*
* \see pressio_io_check_options for semantics this function obeys
* */
virtual int check_options_impl(struct pressio_options const&);

/** reads a pressio_data buffer from some persistent storage
* \param[in] data data object to populate, or nullptr to allocate it from the file if supported
* \see pressio_io_read for the semantics this function should obey
*/
virtual pressio_data* read_impl(struct pressio_data* data)=0;

/** writes a pressio_data buffer to some persistent storage
* \param[in] data data to write
* \see pressio_io_write for the semantics this function should obey
*/
virtual int write_impl(struct pressio_data const* data)=0;

/** get the compile time configuration of a io module
*
* \see pressio_io_get_configuration for the semantics this function should obey
*/
virtual struct pressio_options get_configuration_impl() const=0;

/** sets a set of options for the io_module
* \param[in] options to set for configuration of the io_module
* \see pressio_io_set_options for the semantics this function should obey
*/
virtual int set_options_impl(struct pressio_options const& options)=0;

/** get a set of options available for the io module
*
* The io module should set a value if they have been set as default
* The io module should set a "reset" value if they are required to be set, but don't have a meaningful default
*
* \see pressio_options.h for how to configure options
*/
virtual struct pressio_options get_options_impl() const=0;

private:
struct {
int code;
std::string msg;
} error;
};

/**
* wrapper for the io module to use in C
*/
struct pressio_io {
/**
* \param[in] impl a newly constructed io plugin
*/
pressio_io(std::shared_ptr<libpressio_io_plugin>&& impl): plugin(std::forward<std::shared_ptr<libpressio_io_plugin>>(impl)) {}
/**
* defaults constructs a io with a nullptr
*/
pressio_io()=default;
/**
* move constructs a io from another pointer
*/
pressio_io(pressio_io&& io)=default;
/**
* move assigns a io from another pointer
*/
pressio_io& operator=(pressio_io&& io)=default;

/** \returns true if the plugin is set */
operator bool() const {
return bool(plugin);
}

/** make libpressio_io_plugin behave like a shared_ptr */
libpressio_io_plugin& operator*() const noexcept {
return *plugin;
}

/** make libpressio_io_plugin behave like a shared_ptr */
libpressio_io_plugin* operator->() const noexcept {
return plugin.get();
}

/**
* pointer to the implementation
*/
std::shared_ptr<libpressio_io_plugin> plugin;
};

#endif /* end of include guard: PRESSIO_IO_PLUGIN_H */
Loading

0 comments on commit c7d6736

Please sign in to comment.