Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom dataloader registry support #2932

Open
wants to merge 96 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
7088e4b
copying CZI custom dataloader into our repo
ori-kron-wis Jul 28, 2024
cc72b05
added some fixes to the custom dataloader stuff
ori-kron-wis Jul 30, 2024
46048e3
Some suggestions
canergen Jul 30, 2024
14f343d
Changes to datamodule pipeline
canergen Jul 31, 2024
17282cd
Fixed attr_dict
canergen Jul 31, 2024
a4143f5
added some fixes based on custom data loader test
ori-kron-wis Aug 1, 2024
69abc47
Changes to dataloader
canergen Aug 6, 2024
dc21a3d
copying CZI custom dataloader into our repo
ori-kron-wis Jul 28, 2024
a1098b3
added some fixes to the custom dataloader stuff
ori-kron-wis Jul 30, 2024
b07216b
Some suggestions
canergen Jul 30, 2024
a578af1
Changes to datamodule pipeline
canergen Jul 31, 2024
42434ec
Fixed attr_dict
canergen Jul 31, 2024
3d0c890
added some fixes based on custom data loader test
ori-kron-wis Aug 1, 2024
eff5b1e
Changes to dataloader
canergen Aug 6, 2024
cbdc26e
Merge remote-tracking branch 'origin/ori-2907-custom-dataloader-regis…
ori-kron-wis Aug 7, 2024
18d65a6
add changes to tests and some merging with main following custom data…
ori-kron-wis Aug 7, 2024
4fe3ee1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 7, 2024
1110966
just put the cutom dataloder2 test under remarks so hook tests will r…
ori-kron-wis Aug 7, 2024
7972bdc
fixes
ori-kron-wis Aug 7, 2024
2d86c43
additional external models fixes once there is a registry
ori-kron-wis Aug 7, 2024
3c44d86
fixed a few failed tests
ori-kron-wis Aug 11, 2024
c0889d8
fix archesmixin init and added new custom dataloader test and github …
ori-kron-wis Aug 11, 2024
8fe043c
fix again for from __future__ import annotations
ori-kron-wis Aug 11, 2024
d8cf0f6
fix for run custom dataloader in github action
ori-kron-wis Aug 11, 2024
c41e8b2
rollback
ori-kron-wis Aug 11, 2024
6ec5d4d
added label to the new githubaction for custom dataloader
ori-kron-wis Aug 11, 2024
6bce317
fix for github action for custom dataloaders
ori-kron-wis Aug 12, 2024
1f4ae9d
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
de1f30b
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
49fa01e
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
e33a935
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
48627d9
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
609094d
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
8cf3517
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
ba5a028
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
a7dc3fe
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
f3ff0f8
another fix to custom dataloder test and github action
ori-kron-wis Aug 12, 2024
083c76e
Merge branch 'main' into ori-2907-custom-dataloader-registry
ori-kron-wis Sep 9, 2024
70bba69
Merge branch 'main' into ori-2907-custom-dataloader-registry
ori-kron-wis Sep 15, 2024
8c75662
Merge branch 'main' into ori-2907-custom-dataloader-registry
ori-kron-wis Sep 16, 2024
b6eb2f1
Returned REGISTRY_KEYS for import, after was drop in recent merges
ori-kron-wis Sep 16, 2024
2979ea2
It is ok to drop it after scarches categorial covariates fix
ori-kron-wis Sep 16, 2024
67e9b34
Merge branch 'main' into ori-2907-custom-dataloader-registry
ori-kron-wis Sep 17, 2024
11fe33a
Merge branch 'main' into ori-2907-custom-dataloader-registry
ori-kron-wis Sep 17, 2024
4a648ff
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 17, 2024
e3831cb
moved to type checking blocks beucase of ruff updates
ori-kron-wis Sep 17, 2024
e1837bd
Merge branch 'main' into ori-2907-custom-dataloader-registry
ori-kron-wis Sep 26, 2024
bf4d3bf
Merge remote-tracking branch 'origin/main' into ori-2907-custom-datal…
ori-kron-wis Oct 7, 2024
2cc8ff9
updated for CZI custom dataloader test and backend
ori-kron-wis Oct 9, 2024
e62dc3a
Merge branch 'main' into ori-2907-custom-dataloader-registry
ori-kron-wis Oct 9, 2024
41fd877
added cellxgene-census folder as well for debug (will not be merged)
ori-kron-wis Oct 9, 2024
10ada9c
added cellxgene-census packge to run test
ori-kron-wis Oct 9, 2024
dd3649c
added torchdata packge to run test
ori-kron-wis Oct 9, 2024
c6acb5a
fixed the test workwflow
ori-kron-wis Oct 9, 2024
b35c6eb
adding the lamindb as well
ori-kron-wis Oct 10, 2024
1801604
fix the c.dataloders test
ori-kron-wis Oct 10, 2024
ed77a65
fix the c.dataloders test
ori-kron-wis Oct 10, 2024
fc831d5
fix the c.dataloders test
ori-kron-wis Oct 10, 2024
7400621
fix the c.dataloders test
ori-kron-wis Oct 10, 2024
47376ca
fix the c.dataloders test
ori-kron-wis Oct 10, 2024
f94f7fa
removed redundat functions in code base
ori-kron-wis Oct 13, 2024
962f043
Added scanvi support, including CZI datamodule fix for it
ori-kron-wis Oct 15, 2024
5c21d71
Merge remote-tracking branch 'origin/main' into ori-2907-custom-datal…
ori-kron-wis Oct 20, 2024
a8aeffe
updates from main
ori-kron-wis Dec 25, 2024
1283616
more updates from main
ori-kron-wis Dec 25, 2024
624ee72
Merge branch 'main' into ori-2907-custom-dataloader-registry
ori-kron-wis Dec 25, 2024
6d4f368
Merge remote-tracking branch 'origin/ori-2907-custom-dataloader-regis…
ori-kron-wis Dec 25, 2024
8ab01a4
updated related to tests
ori-kron-wis Dec 25, 2024
31e1d44
updated related to tests
ori-kron-wis Dec 25, 2024
93666fa
Running DataLoader MappedCollection
canergen Dec 30, 2024
1d1d6d3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 30, 2024
7695a8a
Fixed LaminDB dataloader
canergen Dec 31, 2024
e4d732a
Merge branch 'ori-2907-custom-dataloader-registry' of https://github.…
canergen Dec 31, 2024
a651442
LaminDB dataloader test.
canergen Dec 31, 2024
9767b8c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 31, 2024
719e740
Merge branch 'main' into ori-2907-custom-dataloader-registry
ori-kron-wis Dec 31, 2024
1a4c796
Merge remote-tracking branch 'origin/main' into ori-2907-custom-datal…
ori-kron-wis Jan 8, 2025
5666558
Changes for MappedCollection.
canergen Jan 8, 2025
c740dd2
Merge branch 'ori-2907-custom-dataloader-registry' of https://github.…
canergen Jan 8, 2025
61f2e27
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 8, 2025
874935b
Add other notebook for testing new dataloader
canergen Jan 9, 2025
f2c63bd
Merge branch 'ori-2907-custom-dataloader-registry' of https://github.…
canergen Jan 9, 2025
35d45c8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 9, 2025
38c670f
Merge remote-tracking branch 'origin/main' into ori-2907-custom-datal…
ori-kron-wis Jan 16, 2025
c93fc97
updates to test script
ori-kron-wis Jan 16, 2025
5045fc3
remove old test nb
ori-kron-wis Jan 16, 2025
55775f9
update test
ori-kron-wis Jan 16, 2025
7ccdf8d
update test
ori-kron-wis Jan 16, 2025
f88dc50
updated czi cdl
ori-kron-wis Jan 16, 2025
1f3ea11
updated czi cdl
ori-kron-wis Jan 16, 2025
d0ec46f
Merge remote-tracking branch 'origin/main' into ori-2907-custom-datal…
ori-kron-wis Jan 20, 2025
e304922
merge with main + updates
ori-kron-wis Feb 9, 2025
5ccd1ed
more updates
ori-kron-wis Feb 9, 2025
96a09d8
more updates
ori-kron-wis Feb 9, 2025
601d86f
more updates
ori-kron-wis Feb 9, 2025
2485bb6
pyproject update
ori-kron-wis Feb 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix for run custom dataloader in github action
  • Loading branch information
ori-kron-wis committed Aug 11, 2024
commit d8cf0f6c10661f3555f6f2dd90eb69a460a4491e
2 changes: 1 addition & 1 deletion .github/workflows/test_linux.yml
Original file line number Diff line number Diff line change
@@ -53,7 +53,7 @@ jobs:
DISPLAY: :42
COLUMNS: 120
run: |
coverage run -m pytest -v --color=yes
coverage run -m pytest "not custom.dataloader" -v --color=yes
coverage report

- uses: codecov/codecov-action@v4
3 changes: 1 addition & 2 deletions .github/workflows/test_linux_custom_dataloader.yml
Original file line number Diff line number Diff line change
@@ -51,7 +51,7 @@ jobs:
GH_TOKEN: ${{ secrets.GH_TOKEN }}
run: |
git config --global url."https://${GH_TOKEN}:x-oauth-basic@github.com/".insteadOf "https://github.com/"
pip install git+https://github.com/ebezzi/cellxgene-census.git@census-scvi-datamodule
pip install git+https://github.com/ebezzi/chanzuckerberg/cellxgene-census.git@census-scvi-datamodule

- name: Run specific custom dataloader pytest
env:
@@ -61,7 +61,6 @@ jobs:
COLUMNS: 120
run: |
pytest -m custom.dataloader -v --color=yes
coverage report

- uses: codecov/codecov-action@v4
with:
2 changes: 1 addition & 1 deletion tests/dataloaders/test_custom_dataloader.py
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@


@pytest.custom.dataloader
def custom_dataloader_test(save_path):
def test_custom_dataloader(save_path):
# this test checks the local custom dataloder made by CZI and run several tests with it
census = cellxgene_census.open_soma(census_version="stable")