Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump parquet and use arrow-js-ffi #341

Closed
wants to merge 1 commit into from
Closed

Conversation

kylebarron
Copy link
Member

@kylebarron kylebarron commented Jan 31, 2024

arrow-js-ffi is a much more memory-efficient option for moving Arrow data from Rust to JS because it doesn't require a full buffer copy on the Wasm side to construct an Arrow IPC buffer.

This is pretty much blocked on apache/arrow-rs#5356 or some workaround to it. Because of the way deck.gl handles GPU-based picking, it can have a maximum of 256 layers because their indices are packed into a single channel. We take great care on the Python side to choose a record batch size that limits the number of pickable layers to a max of 256. But because arrow-rs does its own rechunking, we're left with tons of tiny record batches of 1024 rows each, which blows up deck.gl (this is from the parquet-wasm bump, not related to using arrow-js-ffi).

Also blocked on fixing parquet package exports (ref kylebarron/parquet-wasm#414)

@kylebarron
Copy link
Member Author

Note that we only need a param for the record batch size. In Lonboard we already send separate Parquet files that each contain a single logical Arrow batch. So we really just want to set the record batch size to a very large number.

@kylebarron
Copy link
Member Author

I published parquet-wasm 0.6.0 and it would be good to test this again. We should bump parquet-wasm regardless of whether we enable arrow-js-ffi.

@kylebarron
Copy link
Member Author

Superseded by #477

@kylebarron kylebarron closed this Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant