-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use HighLevelGraph
layers in dask-mongo.read_mongo
#3
Comments
To clarify: Bag.dask is already a HighLevelGraph. This issue is about replacing the basic MaterializedLayer, which is automatically generated by the Bag constructor and contains 1 key per partition, to a bag-specific I/O class. |
@crusaderky I believe you are correct, this was open by James when we were implementing things to work with writing and reading into dataframes. @jrbourbeau Should we close this issue? |
@ncclementi it's still valid - it's just missing the prerequisite infrastructure in the dask package. |
Yeah, let's keep it open, but know that it is low priority |
As a follow-up to #2, we should update
dask-mongo.read_mongo
to useHighLevelGraph
layers (specificallyDataFrameIOLayer
). This should look very similar to what's done indask-snowflake
.cc @ncclementi
The text was updated successfully, but these errors were encountered: