POC/Arrow evaluation

[chrisbc]

as we delve deeper into the EPIC #50 it becomes apparent that maybe dig0data tech like arrow can help. So, can we do this...

basic questions

• convert THS objects into a arrow/parquet dataset that can be worked on easily using just regular FileSystemLike storage (including local and S3)
• compare performance querying and process large task (eg hazard aggregation in THP
• enumerate the pros/cons
• is parquet the preferred serialisation format

Sub questions:

• can we use arrows in-memory features and/or IPC techniques to boost performance and minimise file IO Plasma?
• can we do partitioning in arrow (not just parquet) how does that work see
• can we easily reshape datasets to optimise for different use-cases (3rd party , internal heavy compute)
• can we use SQL-like querys
  also here SELECT ...

Future possibly

• can we use memory-mapped files
• polars ?? https://medium.com/@matteo.arellano/partitioning-polars-dataframe-on-s3-with-apache-arrow-1fa6518d23f1

[chrisbc]

completed in #62