Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Kill CI Test after 2 hours #14849

Merged
merged 1 commit into from
Nov 19, 2024
Merged

Conversation

lupyuen
Copy link
Member

@lupyuen lupyuen commented Nov 19, 2024

Summary

CI Test will sometimes run for 6 hours (before getting auto-terminated by GitHub):

This is a problem because:

  • It will increase our usage of GitHub Runners. Which may overrun the GitHub Actions Budget allocated by ASF.
  • Suppose right after CI Test there's another build. If CI Test runs for all 6 hours, then the build after CI Test will never run.

For this PR: We assume that Every CI Job (e.g. risc-v-05) will complete normally within 2 hours. If any CI Job exceeds 2 hours: This PR will kill the CI Test Process pytest and allow the next build to run.

Impact

CI Test will no longer run beyond 2 hours. When the CI Test is killed after 2 hours: The CI Job will be reported as a Failed Job, after the subsequent builds have completed.

Testing

We tested at the NuttX Mirror Repo for CI Job risc-v-05. rv-virt:citest terminates after 2 hours, and allows the next build (rv-virt:citest64) to proceed. GitHub Actions reports CI Job risc-v-05 as a Failed Job.

https://github.com/NuttX/nuttx/actions/runs/11900221153/job/33160708989#step:7:306

Configuration/Tool: rv-virt/citest
20:51:20
...
test_open_posix/test_openposix_.py::test_ltp_interfaces_sigpending_1_1 FAILED [ 15%]
test_open_posix/test_openposix_.py::test_ltp_interfaces_sigismember_3_1 FAILED [ 15%]
test_open_posix/test_openposix_.py::test_ltp_interfaces_sem_wait_11_1 FAILED [ 15%]
test_open_posix/test_openposix_.py::test_ltp_interfaces_pthread_rwlockattr_destroy_2_1 FAILED [ 16%]
Killing pytest after timeout...
Terminated
...
Configuration/Tool: rv-virt/citest64
22:51:28

CI Test will sometimes run for 6 hours (before getting auto-terminated by GitHub):
- apache#14808
- apache#14680

This is a problem because:
- It will increase our usage of GitHub Runners. Which may overrun the [GitHub Actions Budget](https://infra.apache.org/github-actions-policy.html) allocated by ASF.
- Suppose right after CI Test there's another build. If CI Test runs for all 6 hours, then the build after CI Test will never run.

For this PR: We assume that Every CI Job (e.g. risc-v-05) will complete normally within 2 hours. If any CI Job exceeds 2 hours: This PR will kill the CI Test Process `pytest` and allow the next build to run.
@lupyuen lupyuen linked an issue Nov 19, 2024 that may be closed by this pull request
1 task
@github-actions github-actions bot added Area: CI Size: XS The size of the change in this PR is very small labels Nov 19, 2024
@xiaoxiang781216 xiaoxiang781216 merged commit 4cfdaa0 into apache:master Nov 19, 2024
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: CI Size: XS The size of the change in this PR is very small
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] CI Test should always terminate after 1 hour
3 participants