Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

5GiB limit for Multipart upload parts #11

Open
artttt opened this issue Sep 2, 2023 · 0 comments
Open

5GiB limit for Multipart upload parts #11

artttt opened this issue Sep 2, 2023 · 0 comments

Comments

@artttt
Copy link
Collaborator

artttt commented Sep 2, 2023

This is currently not handled and is only likely an issue with very large input chunk sizes. However it would be good to be able to handle this nicely.

At the moment this would result in a failed upload.

First option is just a check and throw an error sooner.

Another option is in #10 the buffer part could take some excess data and this would expand the limit to about 10 GiB.
Could also ensure to split parts up so there is 10000.

Beyond that there would need to be a part size estimate from the user, and I don't think that is needed for writing COGs.

The main problem is that the size of parts is unknown when building the graph so large parts can't be handled appropriately in the graph and when partitioning the data. The way around this would be to use futures to construct the writing process on the fly. At this stage I would rather avoid doing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant