HDF5 vs native DAOS features

[mpokorny]

Hello,
I've just begun a process of evaluating DAOS for storage and access for large data sets in radio astronomy. Primarily I'm thinking about a programming interface to these data sets, and it seems that there are two natural choices that might work: either native libdaos or the HDF5 DAOS VOL. I've got experience programming with HDF5, and the data sets I'm considering do have a native HDF5 file format (optional, but not the usual choice), so the HDF5 option has several points in its favor right from the start. However, DAOS has some attractive features that aren't obviously exposed by the VOL API, such as redundancy, transactions and snapshots, to name a few. I'd like to know whether any of these features might show up in the VOL API in the future in some form (or perhaps already exist, and I've missed them). None of the features I mentioned would I need right from the start, but it would be nice to know where this project is heading; perhaps I've missed it, but is there a development roadmap available?

[mpokorny]

Gonna partially answer my own question :-) I found H5daos_set_object_class() in daos_vol_public.h, which should be sufficient to get me redundancy, striping, etc. Also see there, conditionally, H5daos_snap_create() and H5Pset_daos_snap_open(), which don't have any documentation, but sound like they are related to snapshots. I haven't found any documentation outside of source code on any of these functions; should I just keep digging through code to find out how to use them? (I don't mind doing that!)

As for transactions, is there any hope of supporting that feature? Would it be possible to handle transactions in the application code, or would that break the HDF5 data model and/or library?

[fortnern]

Thanks for your interest in HDF5 and the DAOS VOL connector! Snapshots and transactions are both features we'd like to support in the DAOS VOL, but neither are yet implemented. The #ifdef'ed out code referring to snapshots is out of date, and some work would be needed before it could be enabled. These features would be enabled through DAOS VOL connector-specific functions, not through the standard HDF5 API.

I'm not sure how it would be possible to handle transactions in application code. Currently the connector creates and commits an internal transaction for each HDF5 operation. To support transactions at the HDF5 API level we will need to write the code to have that transaction span across multiple HDF5 operations, and potentially retry all those operations if the transaction fails.

[mpokorny]

Thanks for the info. I appreciate having some idea of where the project is heading, even if it will take a while to get there. Moving ahead -- as I indicated, I'm only getting started -- I have begun a proper translation of my target data definition to the HDF5 data model. It might take a few weeks to get something usable together, and then only after I figure out how to install and use a DAOS system will I be able to start trying out DAOS together with HDF5.