-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CI Testing for Example Notebooks and Python Scripts #190
Conversation
Deploying datachain-documentation with Cloudflare Pages
|
for more information, see https://pre-commit.ci
I have some suggestions regarding the setup:
Additionally, I have a question: Is running the script and relying on the exit code sufficient, or should we implement some snapshot testing as well with syrupy? |
@mnrozhkov you have clear and specific expectations for these examples and I no longer believe this repository is the correct place for them. IMO they would be better placed in a separate customer-workshop repository which you can tune to your specific needs. Throwback to https://github.com/iterative/dvcx/issues/1659#issuecomment-2197406631 as I think the sentiment was similar there. With that said, I fully agree with @skshetry that we should be building something that fits in with the current testing approach instead of something bespoke and different that sits outside of it. Something closer to #188 would be better. FWIW - Right now we have 24 scripts and 6 notebooks for examples and they are (obviously) an integral part of our documentation strategy. We do need however to balance out a few things. Each example should have a decent UX and be real enough to be meaningful BUT every example will come with a maintenance cost and for that reason, they each need a valid and specific reason to exist. A good example of where I don't think we currently have this trade-off right is with the 3 Maybe #146 was the right answer after all. |
@skshetry @mattseddon thank you for your thoughtful comments and suggestions. They're very helpful! Your suggestion to move customer-faced examples to a separate repository makes sense, given their unique purpose and details. Let's explore ways to:
My concerns for this setup:
|
I agree on using regular pytest if possible. @mnrozhkov how long would it take to run (in parallel) all these tests on each PR / commit as a regular CI? Let's say using some smaller dataset(s)? If it's doable, I would strongly prefer the same repo, always run them. Everything else immediately leads to "things are broken for a while, someone (even can be an author of the code change) is fixing is w/o a fresh context".
is it relevant to tests? (just trying to understand)
agreed
also, agreed A few additional points. Same repo or different repo
|
For "fashion-product-images" it takes ~4min now. But it's possible to reduce it at least twice.
No |
@mnrozhkov First of all - great kick-off! My two cents:
|
even 4 mins is totally fine if runs as a separate step in the CI (similar to our E2E tests in Studio, or the way we split datachain server tests). @0x2b3bfa0 might help with this I think. |
@shcheklein I have a PR open at #199 for testing scripts. Looks like we are going to have issues with anything that isn't a toy example. |
We should consider moving tutorials and notebooks outside of the repo to prevent it from growing too large. Currently, I hear that it's difficult to fit notebooks to the 0.5MB file size limit, suggesting we may need to relax this restriction- grow repo even more. From Slack:
|
18Mb and 1 minutes to clone - it's a bit too much for 1 month old project 😅 $ time git clone https://github.com/iterative/datachain/
remote: Counting objects: 100% (1584/1584), done.
remote: Compressing objects: 100% (867/867), done.
Receiving objects: 100% (2182/2182), 12.00 MiB | 247.00 KiB/s, done.
Resolving deltas: 100% (1321/1321), done.
real 0m52.035s
user 0m0.533s
sys 0m0.432s $ du -sh datachain/
18M datachain/ |
Discussed with @dmpetrov and the team yesterday. We agreed to move the notebooks to a separate repo (but run CI, ideally on each PR) still. |
Overview:
This PR aims to open a discussion on an approach to automated tests and quality checks for examples (in Jupyter notebooks and Python scripts in the
examples/
directory), ensuring compatibility with the latestdatachain
library.Approach:
papermill
to ruin examples in Jupyter Notebooks (and commit updates - TBD)datachain query scriptname.py
to test scripts designed for StudioContext:
Right now, most efforts are focused on examples of
open source
libraries.However, we also have clients and need to keep examples consistent with Studio, which is simple to copy-paste into Studio (along with additional imported modules). It seems reasonable to keep them closer to "notebooks" and reuse code and update altogether. Both notebooks and scripts are used for customer workshops.
These considerations were taken into account during the work on
examples/computer_vision/fashion_product_images
:scripts/
dir are designed specifically for Studio!__name__ ....
and with a query at the last line)datachain query ...
CLI, notpython ...
How It Works:
The CI workflow
.github/workflows/examples.yaml
tests examples as follows:examples/
or the workflow file.matrix
strategy to test multiple directories concurrently..ipynb
and.py
filespapermill
using atests/examples/test_notebooks.sh
scriptdatachain query
This approach ensures comprehensive testing of all specified example directories, maintaining their functionality with the latest
datachain
version.Pros:
Cons:
requirements.txt
files for each example.Requirements for Example directory:
To comply with this testing framework, each example must adhere to the following standards:
example
directory.scripts/
)requirements.txt
file in the same directory as the notebooks.datachain query
command.I hope this draft makes sense to some extent and helps us to design an appropriate approach. Looking for your feedback and alternatives!
cc @shcheklein @mattseddon @dberenbaum @volkfox @tibor-mach