Diff code coverage of different branchs using codecov app #2310

actione · 2024-10-22T09:29:16Z

Description

Since recently the codecov app has been allowed to be installed into the project.
This PR aims to enable the viewing of code coverage for newly added code directly within the PR.
The following results are from my testing of this feature in my forked CUDA quantum project:
actione#3
It allows for the simultaneous display of C/C++ and Python code coverage changes within a PR submission.

Note: Since this PR has not been tested on the CICD of this project, there may still be problems

copy-pr-bot · 2024-10-22T09:29:20Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

bmhowe23 · 2024-10-22T15:38:30Z

Command Bot: Processing...

schweitzpgi · 2024-10-23T14:50:36Z

Bot doesn't appear to be making progress.

Signed-off-by: huaweil <[email protected]>

bettinaheim · 2024-10-28T14:31:16Z

Command Bot: Processing...

bettinaheim · 2024-10-28T14:32:44Z

The issue is the workflow file:
Invalid workflow file: .github/workflows/ci.yml#L132
The workflow is not valid. .github/workflows/ci.yml (Line: 132, Col: 11): Secret DOCKERHUB_USERNAME is required, but not provided while calling. .github/workflows/ci.yml (Line: 132, Col: 11): Secret DOCKERHUB_READONLY_TOKEN is required, but not provided while calling.

bettinaheim · 2024-10-28T14:36:30Z

Command Bot: Processing...

DCO Remediation Commit for Bettina Heim <[email protected]> I, Bettina Heim <[email protected]>, hereby add my Signed-off-by to this commit: 2ffbeda Signed-off-by: Bettina Heim <[email protected]>

Signed-off-by: Bettina Heim <[email protected]>

bettinaheim

I took the liberty of making the edits to load an existing image, if available, rather than expecting a tar cache.

.github/actions/run-in-docker/action.yml

.github/workflows/ci.yml

Signed-off-by: Bettina Heim <[email protected]>

bettinaheim · 2024-10-28T15:37:27Z

Command Bot: Processing...

actione · 2024-10-29T05:26:17Z

Hi @bettinaheim
Thank you very much for helping to correct the errors. I checked the CICD logs for Gen code coverage, and currently, we only have the last step left: "Upload C/C++ & Python Coverage To Codecov" is failing. This requires the project administrator to make some additional settings about codecov.
error log: Error: Codecov token not found. Please provide Codecov token with -t flag.
my yml setting:

      - name: Upload C/C++ & Python Coverage To Codecov
        uses: codecov/codecov-action@v4
        with:
          fail_ci_if_error: false
          files: ${{ github.workspace }}/shared/coverage.txt,${{ github.workspace }}/shared/coverage.xml
          token: ${{ secrets.CODECOV_TOKEN }}
          verbose: true

btw: The CICD failure is not shown here because I set fail_ci_if_error: false at this step, which means that even if the code coverage upload fails, it won't cause a CICD error.
The solution to this error is to set a CODECOV_TOKEN for the project on the codecov website. Here is the introduction to this token: https://docs.codecov.com/docs/codecov-tokens
you can find the token here (https://app.codecov.io/gh/NVIDIA/cuda-quantum/config/general), you should be able to access the link.
Then set a CODECOV_TOKEN for the project. I can't set the variable. You should have permission to access it here (https://github.com/NVIDIA/cuda-quantum/settings/secrets/actions) and add the variable.
thanks.

Signed-off-by: Bettina Heim <[email protected]>

bettinaheim · 2024-10-31T11:18:10Z

Command Bot: Processing...

bettinaheim · 2024-10-31T11:46:44Z

@actione I added the token and updated the workflows to pass the secret, but it looks like something is still missing:

debug - 2024-10-31 07:43:26,366 -- Starting upload processing --- {"verbose": true, "auto_load_params_from": null, "codecov_yml_path": null, "enterprise_url": null, "version": "cli-0.8.0", "command": "do-upload", "git_service": "github", "report_code": "default", "network_root_folder": "/home/runner/_work/cuda-quantum/cuda-quantum", "files_search_root_folder": "/home/runner/_work/cuda-quantum/cuda-quantum", "files_search_exclude_folders": [], "disable_search": false, "disable_file_fixes": false, "build_code": "11610951619", "build_url": "https://github.com/NVIDIA/cuda-quantum/actions/runs/11610951619", "job_code": "CI", "name": "11610951619", "branch": "pull-request/2310", "pull_request_number": null, "env_vars": {}, "flags": [], "plugin_names": ["xcode", "gcov", "pycoverage"], "dry_run": false, "use_legacy_uploader": false, "handle_no_reports_found": false, "report_type": "coverage", "network_filter": null, "network_prefix": null, "gcov_args": null, "gcov_ignore": null, "gcov_include": null, "gcov_executable
debug - 2024-10-31 07:43:26,367 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"], "cli_config": {}}
debug - 2024-10-31 07:43:26,367 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-10-31 07:43:26,367 -- Running xcode plugin...
warning - 2024-10-31 07:43:26,367 -- xcrun is not installed or can't be found.
debug - 2024-10-31 07:43:26,367 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-10-31 07:43:26,367 -- Running gcov plugin...
warning - 2024-10-31 07:43:26,378 -- No gcov data found.
debug - 2024-10-31 07:43:26,378 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
warning - 2024-10-31 07:43:26,3[78](https://github.com/NVIDIA/cuda-quantum/actions/runs/11610951619/job/32331818780?pr=2310#step:11:79) -- coverage.py is not installed or can't be found.
debug - 2024-10-31 07:43:26,378 -- Collecting relevant files
warning - 2024-10-31 07:43:26,394 -- Some files were not found --- {"not_found_files": ["/home/runner/_work/cuda-quantum/cuda-quantum/shared/coverage.txt", "/home/runner/_work/cuda-quantum/cuda-quantum/shared/coverage.xml"]}
info - 2024-10-31 07:43:26,412 -- Found 0 coverage files to report
Error: No coverage reports found. Please make sure you're generating reports successfully.
Warning: Codecov:
                        Failed to properly upload report: The process '/home/runner/_work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1

Signed-off-by: huaweil <[email protected]>

actione · 2024-10-31T13:38:17Z

@actione I added the token and updated the workflows to pass the secret, but it looks like something is still missing:

debug - 2024-10-31 07:43:26,366 -- Starting upload processing --- {"verbose": true, "auto_load_params_from": null, "codecov_yml_path": null, "enterprise_url": null, "version": "cli-0.8.0", "command": "do-upload", "git_service": "github", "report_code": "default", "network_root_folder": "/home/runner/_work/cuda-quantum/cuda-quantum", "files_search_root_folder": "/home/runner/_work/cuda-quantum/cuda-quantum", "files_search_exclude_folders": [], "disable_search": false, "disable_file_fixes": false, "build_code": "11610951619", "build_url": "https://github.com/NVIDIA/cuda-quantum/actions/runs/11610951619", "job_code": "CI", "name": "11610951619", "branch": "pull-request/2310", "pull_request_number": null, "env_vars": {}, "flags": [], "plugin_names": ["xcode", "gcov", "pycoverage"], "dry_run": false, "use_legacy_uploader": false, "handle_no_reports_found": false, "report_type": "coverage", "network_filter": null, "network_prefix": null, "gcov_args": null, "gcov_ignore": null, "gcov_include": null, "gcov_executable
debug - 2024-10-31 07:43:26,367 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"], "cli_config": {}}
debug - 2024-10-31 07:43:26,367 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-10-31 07:43:26,367 -- Running xcode plugin...
warning - 2024-10-31 07:43:26,367 -- xcrun is not installed or can't be found.
debug - 2024-10-31 07:43:26,367 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-10-31 07:43:26,367 -- Running gcov plugin...
warning - 2024-10-31 07:43:26,378 -- No gcov data found.
debug - 2024-10-31 07:43:26,378 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
warning - 2024-10-31 07:43:26,3[78](https://github.com/NVIDIA/cuda-quantum/actions/runs/11610951619/job/32331818780?pr=2310#step:11:79) -- coverage.py is not installed or can't be found.
debug - 2024-10-31 07:43:26,378 -- Collecting relevant files
warning - 2024-10-31 07:43:26,394 -- Some files were not found --- {"not_found_files": ["/home/runner/_work/cuda-quantum/cuda-quantum/shared/coverage.txt", "/home/runner/_work/cuda-quantum/cuda-quantum/shared/coverage.xml"]}
info - 2024-10-31 07:43:26,412 -- Found 0 coverage files to report
Error: No coverage reports found. Please make sure you're generating reports successfully.
Warning: Codecov:
                        Failed to properly upload report: The process '/home/runner/_work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1

Hi @bettinaheim This may be because, compared to the last time we updated clang16-main image, I have the following check in the generate_cc.sh script:

    toolchain_contents=$(cat /usr/local/llvm/bootstrap/cc)
    if [[ $toolchain_contents == *"/usr/bin/clang-16"* ]]; then
        use_llvm_cov=true
    else
        echo "Currently not supported, running tests using llvm-lit fails"
        exit 1
    fi

But now the content of the file cat /usr/local/llvm/bootstrap/cc in the container being used is：

#!/bin/bash
"/usr/lib/llvm-16/bin/clang-16" "$@"

now I have removed this restriction; can you re-trigger the tests?

bettinaheim · 2024-10-31T15:16:04Z

Command Bot: Processing...

bettinaheim · 2024-10-31T15:50:28Z

@actione It looks like there is still nothing found to upload.

Signed-off-by: huaweil <[email protected]>

1tnguyen · 2024-11-01T01:40:55Z

Command Bot: Processing...

actione · 2024-11-01T02:49:21Z

Now the code coverage report has been successfully uploaded. refer here: https://app.codecov.io/gh/NVIDIA/cuda-quantum/tree/pull-request%2F2310/?displayType=tree
The current reason why the codecov app cannot display code coverage diff in PR is because the main branch has not yet generated comparable code coverage

actione requested review from bettinaheim, amccaskey and bmhowe23 as code owners October 22, 2024 09:29

Enable code coverage diff

3cd37b1

Signed-off-by: huaweil <[email protected]>

actione force-pushed the enable_coverage_diff branch from ef966ab to 3cd37b1 Compare October 28, 2024 08:27

Update ci.yml

2ffbeda

bettinaheim added 2 commits October 28, 2024 16:24

Allow to get image instead of tar cache for code cov workflow

ebf01f4

DCO Remediation Commit for Bettina Heim <[email protected]> I, Bettina Heim <[email protected]>, hereby add my Signed-off-by to this commit: 2ffbeda Signed-off-by: Bettina Heim <[email protected]>

Update ci.yml

3aa038e

Signed-off-by: Bettina Heim <[email protected]>

bettinaheim reviewed Oct 28, 2024

View reviewed changes

.github/actions/run-in-docker/action.yml Outdated Show resolved Hide resolved

.github/workflows/ci.yml Outdated Show resolved Hide resolved

.github/workflows/ci.yml Show resolved Hide resolved

minor edit

33c299b

Signed-off-by: Bettina Heim <[email protected]>

Passing codecov secret

5288bb5

Signed-off-by: Bettina Heim <[email protected]>

bettinaheim requested a review from 1tnguyen as a code owner October 31, 2024 11:14

bettinaheim added 2 commits October 31, 2024 12:15

Matching update for the workflow

cc036f0

Signed-off-by: Bettina Heim <[email protected]>

Merge branch 'main' into enable_coverage_diff

84f7a29

Adapting to the new docker image

ec32a33

Signed-off-by: huaweil <[email protected]>

bettinaheim added 2 commits October 31, 2024 14:49

Merge branch 'main' into enable_coverage_diff

77226ac

Merge branch 'main' into enable_coverage_diff

30eca6f

Make clang_rt library visible

c8aa19d

Signed-off-by: huaweil <[email protected]>

bettinaheim approved these changes Nov 1, 2024

View reviewed changes

bettinaheim merged commit 1ca93a3 into NVIDIA:main Nov 1, 2024
201 checks passed

bettinaheim added the maintenance Work items to update and improve the code base label Nov 19, 2024

caiyunh mentioned this pull request Nov 29, 2024

Generate coverage data when pusing to main #2442

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Diff code coverage of different branchs using codecov app #2310

Diff code coverage of different branchs using codecov app #2310

actione commented Oct 22, 2024

copy-pr-bot bot commented Oct 22, 2024

bmhowe23 commented Oct 22, 2024 •

edited by github-actions bot

Loading

schweitzpgi commented Oct 23, 2024

bettinaheim commented Oct 28, 2024 •

edited by github-actions bot

Loading

bettinaheim commented Oct 28, 2024

bettinaheim commented Oct 28, 2024 •

edited by github-actions bot

Loading

bettinaheim left a comment

bettinaheim commented Oct 28, 2024 •

edited by github-actions bot

Loading

actione commented Oct 29, 2024

bettinaheim commented Oct 31, 2024 •

edited by github-actions bot

Loading

bettinaheim commented Oct 31, 2024

actione commented Oct 31, 2024

bettinaheim commented Oct 31, 2024 •

edited by github-actions bot

Loading

bettinaheim commented Oct 31, 2024

1tnguyen commented Nov 1, 2024 •

edited by github-actions bot

Loading

actione commented Nov 1, 2024

Diff code coverage of different branchs using codecov app #2310

Diff code coverage of different branchs using codecov app #2310

Conversation

actione commented Oct 22, 2024

Description

copy-pr-bot bot commented Oct 22, 2024

bmhowe23 commented Oct 22, 2024 • edited by github-actions bot Loading

schweitzpgi commented Oct 23, 2024

bettinaheim commented Oct 28, 2024 • edited by github-actions bot Loading

bettinaheim commented Oct 28, 2024

bettinaheim commented Oct 28, 2024 • edited by github-actions bot Loading

bettinaheim left a comment

Choose a reason for hiding this comment

bettinaheim commented Oct 28, 2024 • edited by github-actions bot Loading

actione commented Oct 29, 2024

bettinaheim commented Oct 31, 2024 • edited by github-actions bot Loading

bettinaheim commented Oct 31, 2024

actione commented Oct 31, 2024

bettinaheim commented Oct 31, 2024 • edited by github-actions bot Loading

bettinaheim commented Oct 31, 2024

1tnguyen commented Nov 1, 2024 • edited by github-actions bot Loading

actione commented Nov 1, 2024

bmhowe23 commented Oct 22, 2024 •

edited by github-actions bot

Loading

bettinaheim commented Oct 28, 2024 •

edited by github-actions bot

Loading

bettinaheim commented Oct 28, 2024 •

edited by github-actions bot

Loading

bettinaheim commented Oct 28, 2024 •

edited by github-actions bot

Loading

bettinaheim commented Oct 31, 2024 •

edited by github-actions bot

Loading

bettinaheim commented Oct 31, 2024 •

edited by github-actions bot

Loading

1tnguyen commented Nov 1, 2024 •

edited by github-actions bot

Loading