-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Revamp Dask Array creation logic to be fully blockwise #105
Comments
Thinking about this a bit more, we may want to stick with the current approach (or a form of it). Currently, we make one initial array of Then we make an array of When you broadcast those two arrays together, you get the The nice things are:
If we used |
Addressed in #116 |
When dask/dask#7417 gets in, it may both break the current dask-array-creation hacks, and open the door for a much, much simpler approach: just use
da.from_array
.We didn't use
da.from_array
originally and went through all the current rigamarole becausefrom_array
generates a low-level graph, which can be enormous (and slow) for the large datasets we load. But oncefrom_array
uses Blockwise, it will be far simpler and more efficient.We'll just switch to having an array-like class that wraps the asset table and other parameters, and whose
__getitem__
basically callsfetch_raster_window
. However, it's likely worth thinking about just combining all this into the Reader protocol in some way.This will also make it easier to do #62 (and may even inadvertently solve it).
The text was updated successfully, but these errors were encountered: