Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider thawing loop in same distributed map as copy #13

Open
andrewpatto opened this issue Jan 28, 2025 · 0 comments
Open

Consider thawing loop in same distributed map as copy #13

andrewpatto opened this issue Jan 28, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@andrewpatto
Copy link
Member

The thawing steps is currently in its own distributed map - which waits for all thawing to finish (across all files). It then proceeds to the copy phase which is its own distributed map. This was done because the scale-out factor of the two are different (one is scaling out thousands of tiny lambdas to do thawing, the other is scaling out a limited set of fargate jobs).

We could consider refactoring this

  • so that objects that are already thawed can copy straight away
  • copies could immediately proceed after thawing without waiting for the slowest
@andrewpatto andrewpatto added the enhancement New feature or request label Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant