Consider using partitioned parquet files

[tomalrussell]

Parquet allows readers/writers to treat a set of files under a directory as a single dataset, with the option to specify partitions based on data values that are then encoded into the directory structure, for example: roads.parquet/highway=primary/slice=1/part0.parquet

For Python, the key docs are:

• Reading and Writing the Apache Parquet Format
• Tabular Datasets

Some questions:

• what would be the benefits for open-gira intermediate or results datasets? At the least, it could reduce particularly large file sizes and avoid (or simplify) the concatenation steps as currently implemented.
• how would the file/directory structure interact with snakemake? Would we need additional flags or workarounds?