Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce buffered data for multipart upload #10

Open
artttt opened this issue Sep 1, 2023 · 0 comments
Open

Reduce buffered data for multipart upload #10

artttt opened this issue Sep 1, 2023 · 0 comments

Comments

@artttt
Copy link
Collaborator

artttt commented Sep 1, 2023

Currently the distributed multipart upload keeps whole input bytes objects as the "first part's" buffer.
An additional step could be added where 5000 rather then 10000 partitions are created but then the first 5MiB of each partition is split off into its own partition.
This means only 5MiB is kept in memory while waiting for more data to continue the reduction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

1 participant