Implement parallelization across broadcasted dimensions #376
+331
−281
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This builds on #374 to add a possible mode where if we are dealing with arrays with extra/broadcasted dimensions, we can split up the input array into chunks along the broadcasted dimensions which means we don't have to use the workaround of writing the whole dask array to a memmap.
For example, if we are reprojecting an array with dimensions (100, 2048, 2048) to (100, 1024, 1024), reprojecting only the celestial part of the WCS (the second and third dimensions of the array), we can specify e.g.
block_size=(5, 1024, 1024)
and this will cause the input to also be split up into chunks of(5, 2048, 2048)
and each chunk of length 5 along the extra dimension will be handled separately.One thing I'm not sure about is that at the moment we rely on the block size to fulfill certain conditions to toggle between the two main reprojection modes - but would it be cleaner to have a kwarg to do this?
@svank - your input would be appreciated too since you added the broadcasting functionality!