Requirements for dataframe generators

[steinnymir]

As this package will concentrate on the number-crunching and handling of the dataframe, we need to define what the dataframes we want to work with should look like.

In this discussion, we should collect all ideas and suggestions to what might be relevant to the construction of the dataframes from the raw data of the different experiments which might make use of the sed.

Make your suggestions here, and we will pin to the top message those we agree upon to be the most important. This can then be used by those developing dataframe generation routines as a guideline to ensure the best compatibility with sed.

[steinnymir]

Dataframe Guidelines so far:

As described above, this is a work-in-progress list of requirements for the dataframe. Once this has grown to a significantly complete set of rules, we could conisder publishing it in the documentation of the sed package.
Add comments below to suggest additions/changes.

Data format

dataframe should be readable as a dask.DataFrame. Supported formats might be:

• Apache Parquet
  • how do we then standardize passing metadata?
• HDF5
  • what should the group structure be like?

Metadata

How do we handle exchange of metadata between the dataframe generation process and the sed?

Structure

How should the data be stored in order to be parsed by the SED?

• nested structure, as groups in HDF5 or nested dictionaries in json?

Content

which metadata entries must/should/can be passed to sed, together with the data?

• All the metadata accumulated together with the acquired data
• statistical information about the dataframe columns
  • min, max, mean, std, minimum step (for digital channels)... other?
  • these are quantities based on the values of the dataframe. Accessing them when anyway reading the data might be cheaper than computing them from the structured dataframe
  • total number of events

[steinnymir]

@rettigl @zainsohail04 @daviddoji throw in your ideas!

[zain-sohail]

The mpes-nexus format might be the way to go for including data + metadata together.

[steinnymir]

Does this allow a tabular format as input? or is it designed for volumetric data?

[zain-sohail]

Nexus itself is built on top of hdf5, with the mpes-nexus just making it domain specific (https://manual.nexusformat.org/introduction.html)