Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add testing for example folder #287

Closed
wants to merge 41 commits into from
Closed

add testing for example folder #287

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
24c77f9
add testing for example folder
baptistecolle Oct 9, 2024
197c721
add testing for example folder
baptistecolle Oct 9, 2024
9cc5264
add testing for example folder
baptistecolle Oct 9, 2024
a2ee6b4
add testing for example folder
baptistecolle Oct 9, 2024
5a04419
fix typo
baptistecolle Oct 14, 2024
bb37c05
fix typo
baptistecolle Oct 14, 2024
0f22db1
fix typo
baptistecolle Oct 14, 2024
0616bf2
refactor to decrease number of workflows
baptistecolle Oct 14, 2024
c19915b
refactor to decrease number of workflows
baptistecolle Oct 14, 2024
b273a08
refactor to decrease number of workflows
baptistecolle Oct 14, 2024
be34484
refactor to decrease number of workflows
baptistecolle Oct 14, 2024
cc3b6dc
fix bugs
baptistecolle Oct 14, 2024
dcc1197
fix bugs
baptistecolle Oct 14, 2024
c45ffa1
fix bugs
baptistecolle Oct 14, 2024
709cbfd
fix bugs
baptistecolle Oct 14, 2024
738c125
fix bugs
baptistecolle Oct 14, 2024
f8ac631
fix bugs
baptistecolle Oct 14, 2024
f61ed47
fix bugs
baptistecolle Oct 14, 2024
ebdc70a
fix bugs
baptistecolle Oct 14, 2024
83c4c9f
fix bugs
baptistecolle Oct 14, 2024
2fb106b
fix bugs
baptistecolle Oct 14, 2024
755c19d
fix bugs
baptistecolle Oct 14, 2024
49a5f70
fix bugs
baptistecolle Oct 14, 2024
32fb3ed
fix bugs
baptistecolle Oct 14, 2024
db02e45
fix bugs
baptistecolle Oct 14, 2024
5efae05
fix bugs
baptistecolle Oct 14, 2024
5d98048
remove broken energy tracker reset method
baptistecolle Oct 14, 2024
b7b8dbb
remove broken energy tracker reset method
baptistecolle Oct 14, 2024
9fdb506
remove broken energy tracker reset method
baptistecolle Oct 14, 2024
0ab2c0e
remove broken energy tracker reset method
baptistecolle Oct 14, 2024
9bcf170
debug cuda andd rocm
baptistecolle Oct 14, 2024
7c5fe6e
debug cuda andd rocm
baptistecolle Oct 14, 2024
4e025cf
debug cuda
baptistecolle Oct 14, 2024
abbfbd6
debug cuda
baptistecolle Oct 14, 2024
002a48a
debug cuda
baptistecolle Oct 14, 2024
5122098
debug cuda
baptistecolle Oct 14, 2024
dbbed90
debug cuda
baptistecolle Oct 14, 2024
3cd0519
fix failling tests
baptistecolle Oct 15, 2024
de90f7e
fix tests
baptistecolle Oct 15, 2024
0faf284
fix tests
baptistecolle Oct 15, 2024
fc7246c
test openvino
baptistecolle Oct 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/test_cli_cpu_ipex.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cpu') ||
contains( github.event.pull_request.labels.*.name, 'ipex') ||
contains( github.event.pull_request.labels.*.name, 'cli_cpu_ipex')
Expand All @@ -47,5 +48,10 @@ jobs:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -e .[testing,ipex,diffusers,timm]

- name: Run tests
run: pytest tests/test_cli.py -s -k "cli and cpu and ipex"
- name: Run cli tests
run: |
pytest tests/test_cli.py -s -k "cli and cpu and ipex"

- name: Run tests from example folder
run: |
pytest tests/test_examples.py -s -k "cpu and ipex"
6 changes: 4 additions & 2 deletions .github/workflows/test_cli_cpu_llama_cpp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cpu') ||
contains( github.event.pull_request.labels.*.name, 'llama_cpp') ||
contains( github.event.pull_request.labels.*.name, 'cli_cpu_llama_cpp')
Expand All @@ -47,5 +48,6 @@ jobs:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -e .[testing,llama-cpp]

- name: Run tests
run: pytest tests/test_cli.py -s -k "llama_cpp"
- name: Run cli tests
run: |
pytest tests/test_cli.py -s -k "llama_cpp"
10 changes: 8 additions & 2 deletions .github/workflows/test_cli_cpu_neural_compressor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cpu') ||
contains( github.event.pull_request.labels.*.name, 'neural_compressor') ||
contains( github.event.pull_request.labels.*.name, 'cli_cpu_neural_compressor')
Expand All @@ -47,5 +48,10 @@ jobs:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -e .[testing,neural-compressor,diffusers,timm]

- name: Run tests
run: pytest tests/test_cli.py -s -k "cli and cpu and neural_compressor"
- name: Run cli tests
run: |
pytest tests/test_cli.py -s -k "cli and cpu and neural_compressor"

- name: Run tests from example folder
run: |
pytest tests/test_examples.py -s -k "cpu and neural_compressor"
10 changes: 8 additions & 2 deletions .github/workflows/test_cli_cpu_onnxruntime.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cpu') ||
contains( github.event.pull_request.labels.*.name, 'onnxruntime') ||
contains( github.event.pull_request.labels.*.name, 'cli_cpu_onnxruntime')
Expand All @@ -47,5 +48,10 @@ jobs:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -e .[testing,onnxruntime,diffusers,timm]

- name: Run tests
run: pytest tests/test_cli.py -s -k "cli and cpu and onnxruntime"
- name: Run cli tests
run: |
pytest tests/test_cli.py -s -k "cli and cpu and onnxruntime"

- name: Run tests from example folder
run: |
pytest tests/test_examples.py -s -k "cpu and onnxruntime"
10 changes: 8 additions & 2 deletions .github/workflows/test_cli_cpu_openvino.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cpu') ||
contains( github.event.pull_request.labels.*.name, 'openvino') ||
contains( github.event.pull_request.labels.*.name, 'cli_cpu_openvino')
Expand All @@ -47,5 +48,10 @@ jobs:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -e .[testing,openvino,diffusers,timm]

- name: Run tests
run: pytest tests/test_cli.py -s -k "cli and cpu and openvino"
- name: Run cli tests
run: |
pytest tests/test_cli.py -s -k "cli and cpu and openvino"

- name: Run tests from example folder
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This takes forever because the CPU for testing ubuntu-latest is not the most optimal for the CI, I think. Maybe we should also update the machine that run those tests

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about this, let's only run the example test if "example" is in the tags

run: |
pytest tests/test_examples.py -s -k "cpu and openvino"
10 changes: 8 additions & 2 deletions .github/workflows/test_cli_cpu_py_txi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cpu') ||
contains( github.event.pull_request.labels.*.name, 'py_txi') ||
contains( github.event.pull_request.labels.*.name, 'cli_cpu_py_txi')
Expand All @@ -47,5 +48,10 @@ jobs:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -e .[testing,py-txi]

- name: Run tests
run: pytest tests/test_cli.py -s -k "cli and cpu and py_txi"
- name: Run cli tests
run: |
pytest tests/test_cli.py -s -k "cli and cpu and py_txi"

- name: Run tests from example folder
run: |
pytest tests/test_examples.py -s -k "examples and cpu and py_txi"
6 changes: 4 additions & 2 deletions .github/workflows/test_cli_cpu_pytorch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cpu') ||
contains( github.event.pull_request.labels.*.name, 'pytorch') ||
contains( github.event.pull_request.labels.*.name, 'cli_cpu_pytorch')
Expand All @@ -47,5 +48,6 @@ jobs:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -e .[testing,diffusers,timm,peft]

- name: Run tests
run: pytest tests/test_cli.py -s -k "cli and cpu and pytorch"
- name: Run cli tests
run: |
pytest tests/test_cli.py -s -k "cli and cpu and pytorch"
3 changes: 2 additions & 1 deletion .github/workflows/test_cli_cuda_onnxruntime.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cuda') ||
contains( github.event.pull_request.labels.*.name, 'onnxruntime') ||
contains( github.event.pull_request.labels.*.name, 'cli_cuda_onnxruntime')
Expand All @@ -45,6 +46,6 @@ jobs:
run: |
pip install -e .[testing,onnxruntime-gpu,diffusers,timm]

- name: Run tests
- name: Run cli tests
run: |
pytest tests/test_cli.py -x -s -k "cli and cuda and onnxruntime"
10 changes: 8 additions & 2 deletions .github/workflows/test_cli_cuda_py_txi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cuda') ||
contains( github.event.pull_request.labels.*.name, 'py_txi') ||
contains( github.event.pull_request.labels.*.name, 'cli_cuda_py_txi')
Expand All @@ -47,5 +48,10 @@ jobs:
pip install --upgrade pip
pip install -e .[testing,py-txi]

- name: Run tests
run: pytest tests/test_cli.py -x -s -k "cli and cuda and py_txi"
- name: Run cli tests
run: |
pytest tests/test_cli.py -x -s -k "cli and cuda and py_txi"

- name: Run tests from example folder
run: |
pytest tests/test_examples.py -x -s -k "cuda and py_txi"
23 changes: 19 additions & 4 deletions .github/workflows/test_cli_cuda_pytorch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cuda') ||
contains( github.event.pull_request.labels.*.name, 'pytorch') ||
contains( github.event.pull_request.labels.*.name, 'single_gpu') ||
Expand All @@ -43,13 +44,21 @@ jobs:
uses: actions/checkout@v4

- name: Install dependencies
env:
LOG_LEVEL: ERROR
run: |
pip install -e .[testing,diffusers,timm,peft,bitsandbytes,autoawq,auto-gptq]
pip install -e .[testing,diffusers,timm,peft,bitsandbytes,autoawq,auto-gptq,codecarbon]

- name: Run tests
- name: Run cli tests
run: |
pytest tests/test_cli.py -x -s -k "cli and cuda and pytorch and not (dp or ddp or device_map or deepspeed)"

- name: Run tests from example folder
env:
LOG_LEVEL: ERROR
run: |
pytest tests/test_examples.py -x -s -k "cuda and pytorch"

run_cli_cuda_pytorch_multi_gpu_tests:
if: ${{
(github.event_name == 'push') ||
Expand All @@ -76,10 +85,16 @@ jobs:
run: |
pip install -e .[testing,diffusers,timm,peft,deepspeed]

- name: Run tests (parallel)
- name: Run cli tests (parallel)
run: |
pytest tests/test_cli.py -x -s -k "cli and cuda and pytorch and (dp or ddp or device_map)"

- name: Run tests (sequential)
- name: Run cli tests (sequential)
run: |
FORCE_SEQUENTIAL=1 pytest tests/test_cli.py -x -s -k "cli and cuda and pytorch and (deepspeed_inference)"

- name: Run tests from example folder
env:
LOG_LEVEL: ERROR
run: |
FORCE_SEQUENTIAL=1 pytest tests/test_examples.py -x -s -k "cuda and pytorch"
10 changes: 8 additions & 2 deletions .github/workflows/test_cli_cuda_tensorrt_llm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cuda') ||
contains( github.event.pull_request.labels.*.name, 'tensorrt_llm') ||
contains( github.event.pull_request.labels.*.name, 'single_gpu') ||
Expand All @@ -46,10 +47,14 @@ jobs:
run: |
pip install -e .[testing]

- name: Run tests
- name: Run cli tests
run: |
pytest tests/test_cli.py -x -s -k "cli and cuda and tensorrt_llm and not (tp or pp)"

- name: Run tests from example folder
run: |
pytest tests/test_examples.py -x -s -k "cuda and tensorrt_llm"

cli_cuda_tensorrt_llm_multi_gpu_tests:
if: ${{
(github.event_name == 'push') ||
Expand All @@ -76,6 +81,7 @@ jobs:
run: |
pip install -e .[testing]

- name: Run tests (sequential)
- name: Run cli tests (sequential)
run: |
FORCE_SEQUENTIAL=1 pytest tests/test_cli.py -x -s -k "cli and cuda and tensorrt_llm and (tp or pp)"

5 changes: 3 additions & 2 deletions .github/workflows/test_cli_cuda_torch_ort.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cuda') ||
contains( github.event.pull_request.labels.*.name, 'torch_ort') ||
contains( github.event.pull_request.labels.*.name, 'multi_gpu') ||
Expand All @@ -78,6 +79,6 @@ jobs:
pip install -e .[testing,torch-ort,peft]
pip install optimum@git+https://github.com/huggingface/optimum.git

- name: Run tests
- name: Run cli tests
run: |
pytest tests/test_cli.py -x -s -k "cli and cuda and torch_ort and (dp or ddp or device_map)"
pytest tests/test_cli.py -x -s -k "cli and cuda and torch_ort and (dp or ddp or device_map)"
12 changes: 11 additions & 1 deletion .github/workflows/test_cli_cuda_vllm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cuda') ||
contains( github.event.pull_request.labels.*.name, 'vllm') ||
contains( github.event.pull_request.labels.*.name, 'single_gpu') ||
Expand All @@ -49,12 +50,17 @@ jobs:
- name: Run tests (sequential)
run: |
FORCE_SEQUENTIAL=1 pytest tests/test_cli.py -x -s -k "cli and cuda and vllm and not (tp or pp)"

- name: Run tests from example folder
run: |
pytest tests/test_examples.py -x -s -k "cuda and vllm"

run_cli_cuda_vllm_multi_gpu_tests:
if: ${{
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'cuda') ||
contains( github.event.pull_request.labels.*.name, 'vllm') ||
contains( github.event.pull_request.labels.*.name, 'multi_gpu') ||
Expand All @@ -76,6 +82,10 @@ jobs:
run: |
pip install -e .[testing]

- name: Run tests (sequential)
- name: Run cli tests (sequential)
run: |
FORCE_SEQUENTIAL=1 pytest tests/test_cli.py -x -s -k "cli and cuda and vllm and (tp or pp)"

- name: Run tests from example folder
run: |
FORCE_SEQUENTIAL=1 pytest tests/test_examples.py -x -s -k "cuda and vllm"
4 changes: 2 additions & 2 deletions .github/workflows/test_cli_misc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,6 @@ jobs:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -e .[testing]

- name: Run tests
- name: Run cli tests
run: |
pytest tests/test_cli.py -s -k "cli and not (cpu or cuda or rocm or mps)"
pytest tests/test_cli.py -s -k "cli and not (cpu or cuda or rocm or mps)"
8 changes: 5 additions & 3 deletions .github/workflows/test_cli_rocm_pytorch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'rocm') ||
contains( github.event.pull_request.labels.*.name, 'pytorch') ||
contains( github.event.pull_request.labels.*.name, 'single_gpu') ||
Expand All @@ -49,7 +50,7 @@ jobs:
run: |
pip install -e .[testing,diffusers,timm,peft,autoawq,auto-gptq]

- name: Run tests
- name: Run cli tests
run: |
pytest tests/test_cli.py -x -s -k "cli and cuda and pytorch and not (dp or ddp or device_map or deepspeed) and not bnb"

Expand All @@ -58,6 +59,7 @@ jobs:
(github.event_name == 'push') ||
(github.event_name == 'workflow_dispatch') ||
contains( github.event.pull_request.labels.*.name, 'cli') ||
contains( github.event.pull_request.labels.*.name, 'examples') ||
contains( github.event.pull_request.labels.*.name, 'rocm') ||
contains( github.event.pull_request.labels.*.name, 'pytorch') ||
contains( github.event.pull_request.labels.*.name, 'multi_gpu') ||
Expand All @@ -82,10 +84,10 @@ jobs:
run: |
pip install -e .[testing,diffusers,timm,deepspeed,peft,autoawq,auto-gptq] "deepspeed<0.15"

- name: Run tests (parallel)
- name: Run cli tests (parallel)
run: |
pytest tests/test_cli.py -x -s -k "cli and cuda and pytorch and (dp or ddp or device_map)"

- name: Run tests (sequential)
- name: Run cli tests (sequential)
run: |
FORCE_SEQUENTIAL=1 pytest tests/test_cli.py -x -s -k "cli and cuda and pytorch and (deepspeed_inference)"
Loading
Loading