Added offset index based parquet reading support #4844

malhotrashivam · 2023-11-15T21:40:08Z

Currently, while reading a particular row from a parquet file,

We first check if the row is in any of the pages read so far.
If not, we keep on extending one page till we reach the required page containing the row. Extending the page means computing the number of rows and setting up some internal data structures for each page.
All of this happens inside VariablePageSizeColumnChunkPageStore.

Some of the above operations can be optimized using OffsetIndex data structure.
OffsetIndex is an optional field in Parquet which gives us three things for each page in a parquet file:

Byte offset
Compressed page size
Index of first row (<- Most useful for this PR)

So for locating a page containing a row, we can directly binary search inside the offset index and don't need to extend pages up to that page. Doing this helps speed up sparse reads and improves the parallelization.

This PR makes the change and also fixes a bug in writing offset index for array and vector columns. Because of this bug fix, we can not use the old code to read the parquet files generated with the new code. So the PR is backwards compatible but not forwards compatible.

Also, as part of this PR, we are deleting the FixedPageSizeColumnChunkPageStore because to detect if page sizes are fixed, we need offset index. So fixed page size support is added in the new OffsetIndexBasedColumnChunkPageStore.

Benchmarking results: https://docs.google.com/document/d/19AIp6UpHTWpbmxo0ldkBYyuRKOIhlGLP2NHU26jOeOM/edit?usp=sharing

Closes #4717, #4718
Related to #4879

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ColumnChunkReader.java

...le/src/main/java/io/deephaven/parquet/table/pagestore/FixedPageSizeColumnChunkPageStore.java

...src/main/java/io/deephaven/parquet/table/pagestore/OffsetIndexBasedColumnChunkPageStore.java

devinrsmith

This is not my full review. Just took a quick look at the tests for now. I'll wait until after Ryan's review to dig down deeper.

...nsions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableReadWriteTest.java

rcaudy

Mostly high-level review pass. Be sure that we keep enterprise in the loop regarding forwards compatibility issue, and look for more external validation of our new approach to writing and interpreting offset indexes.

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ColumnChunkReader.java

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ColumnChunkReaderImpl.java

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ColumnPageReaderImpl.java

...le/src/main/java/io/deephaven/parquet/table/pagestore/FixedPageSizeColumnChunkPageStore.java

...src/main/java/io/deephaven/parquet/table/pagestore/OffsetIndexBasedColumnChunkPageStore.java

devinrsmith

Will take another pass soon; need to take a break.

...s/parquet/table/src/main/java/io/deephaven/parquet/table/pagestore/ColumnChunkPageStore.java

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ColumnChunkReader.java

extensions/parquet/table/src/main/java/io/deephaven/parquet/table/metadata/TableInfo.java

...src/main/java/io/deephaven/parquet/table/pagestore/OffsetIndexBasedColumnChunkPageStore.java

...le/src/main/java/io/deephaven/parquet/table/pagestore/FixedPageSizeColumnChunkPageStore.java

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/ColumnChunkReader.java

devinrsmith · 2023-11-16T20:37:15Z

...s/parquet/table/src/main/java/io/deephaven/parquet/table/pagestore/ColumnChunkPageStore.java

+            @NotNull final ToPage<ATTR, ?> toPage,
+            @NotNull final ColumnDefinition<?> columnDefinition) throws IOException {


This is technically a "breaking" change; another reason I don't like how our implementation is structured right now. Not your issue, but just noting. #4850

We don't expect anyone to use our parquet libraries at this level but us. Java does not make it easy to specify that.