You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Compression should be split into chunk compression and layout compression.
Chunk Compression
The chunk compressor operates over a single chunk of (canonical?) data. It may use external state from previous chunks to prune the search space for compression strategies. Currently it has a naive stats-based search space, there are ways to improve and speed this up.
Layout Compression
Layout compression occurs in LayoutWriter and is configured as part of a layout strategy. ChunkedLayoutWriter should buffer and repartition data into reasonable chunks, and then provide compressor state across chunks to assist with compression.
DictLayoutWriter should attempt to accumuluate a dictionary of values, if this values, fallback to writing regular chunked array.
We may even want to re-use compression state across columns if the dtype is the same and the stats are similar, e.g. for 100k columns of model weights.
The VortexLayoutStrategy should also be configurable per FieldPath if the user wishes to override these things.
The text was updated successfully, but these errors were encountered:
Compression should be split into chunk compression and layout compression.
Chunk Compression
The chunk compressor operates over a single chunk of (canonical?) data. It may use external state from previous chunks to prune the search space for compression strategies. Currently it has a naive stats-based search space, there are ways to improve and speed this up.
Layout Compression
Layout compression occurs in LayoutWriter and is configured as part of a layout strategy. ChunkedLayoutWriter should buffer and repartition data into reasonable chunks, and then provide compressor state across chunks to assist with compression.
DictLayoutWriter should attempt to accumuluate a dictionary of values, if this values, fallback to writing regular chunked array.
We may even want to re-use compression state across columns if the dtype is the same and the stats are similar, e.g. for 100k columns of model weights.
The VortexLayoutStrategy should also be configurable per FieldPath if the user wishes to override these things.
The text was updated successfully, but these errors were encountered: