ci: add codecov

[FrankYang0529]

resolve #367

Need to setup CODECOV_TOKEN. Thanks

[vdice]

Thanks @FrankYang0529!

Re: the CODECOV_TOKEN... it looks like a separate token won't be necessary for this public spin repo, if I'm understanding correctly: https://github.com/codecov/codecov-action#usage

Have you been able to test this on your fork, by chance?

[kate-goldenring]

Once we have the token figured out/removed, I am curious whether installing Tarpaulin on the GitHub runner or using the Tarpaulin containers to execute the tests (like here: https://github.com/project-akri/akri/blob/main/.github/workflows/run-tarpaulin.yml) is faster. Either way it can take a long time to run

[FrankYang0529]

Re: the CODECOV_TOKEN... it looks like a separate token won't be necessary for this public spin repo, if I'm understanding correctly: https://github.com/codecov/codecov-action#usage

Yeah, it looks like we don't need it. I updated a version for it.

Have you been able to test this on your fork, by chance?

Yes, this one https://github.com/FrankYang0529/spin/actions/runs/3369212681/jobs/5588606673

[FrankYang0529]

Once we have the token figured out/removed, I am curious whether installing Tarpaulin on the GitHub runner or using the Tarpaulin containers to execute the tests (like here: https://github.com/project-akri/akri/blob/main/.github/workflows/run-tarpaulin.yml) is faster. Either way it can take a long time to run

I am not sure whether it's faster, but I would prefer not to use containers here, because:

• We need some dependencies like bindle, nomad, and hippo to run e2e-tests.
• Default runners only have 14G SSD. If we use volume with containers, it may take extra space on the host.

[vdice]

Thanks for the updates @FrankYang0529. I see we upload the codecov report as an artifact on the workflow run; what do we envision the current flow for tracking these reports over time?

Would it be possible to have the codecov results be a separate check in CI? I'm thinking it would be handy to see it alongside the other checks (currently the Rust / Build Spin and DCO checks). And maybe eventually we'd have a threshold where if code coverage dips below a certain amount, the check would fail. Perhaps this would involve authorizing a Codecov GH App? Assuming it would take a non-trivial amount of config/work, it can certainly be a follow-up (if it's even something the Spin team wants).

[FrankYang0529]

Would it be possible to have the codecov results be a separate check in CI?

Yes, I think it's good to separate these in another check so that we can unify test commands in the Build Spin check. I will update it tomorrow.

And maybe eventually we'd have a threshold where if code coverage dips below a certain amount, the check would fail.

I would suggest we don't do this now because the result from the Ptrace engine with --follow-exec is not very accurate. We have a test case for spin deploy, and I added e2e-tests in the --features, but the coverage rate for deploy.rs is 0. We can do that after llvm engine supports WASM taiki-e/cargo-llvm-cov#221.

Perhaps this would involve authorizing a Codecov GH App?

I will do some surveys and test in my repo first.

[vdice]

Hi @FrankYang0529, checking in to see if I can help on this one. There may be a bit of refactoring needed now that the build.yml workflow has been broken up into multiple jobs. If it isn't too complex, it would still be great to have the codecov checks run as their own job/check (ref #866 (comment)). Otherwise, I'm sure we could also get a version of this in and start refining/interating from there.

[FrankYang0529]

Hi @vdice, sorry for the late update. I will update the PR this weekend. Thanks.

[FrankYang0529]

Hi @vdice, I separated code-coverage to another job. I think we can merge this PR first and add Codecov GH App in a follow-up PR.

[vdice]

Thank you @FrankYang0529!

Added a few comments/questions for the broader team but otherwise the automation looks great to me.

[vdice]

ping @lann or @radu-matei on if this cargo tarpaulin command suits our needs per the original issue #367

[lann]

Yep that's what I use.

[vdice]

Are we only interested in codecov results for main at this time? cc @lann @radu-matei

[lann]

I would imagine we'd like it for PRs too, but its easy enough to change 🤷

[vdice]

Great, we can see how it works for us for main and then potentially create a follow-up.

[vdice]

Thanks again for this contribution @FrankYang0529!