You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the distributed multipart upload keeps whole input bytes objects as the "first part's" buffer.
An additional step could be added where 5000 rather then 10000 partitions are created but then the first 5MiB of each partition is split off into its own partition.
This means only 5MiB is kept in memory while waiting for more data to continue the reduction.
The text was updated successfully, but these errors were encountered:
Currently the distributed multipart upload keeps whole input bytes objects as the "first part's" buffer.
An additional step could be added where 5000 rather then 10000 partitions are created but then the first 5MiB of each partition is split off into its own partition.
This means only 5MiB is kept in memory while waiting for more data to continue the reduction.
The text was updated successfully, but these errors were encountered: