CI: Improve test providers coverage and prepare for 98.0 release #3458
Conversation
There was a problem hiding this comment.
Hi @clebergnu, thanks for this. I have one comment for discussion about updating tp-qemu and tp-libvirt in CI.
| # These versions can be bumped at any time, but provide better understanding | ||
| # of regressions or version update requirements on the test providers side. | ||
| - python3 -c "from virttest.asset import ConfigLoader; cl = ConfigLoader('virttest/test-providers.d/io-github-autotest-qemu.ini'); cl.set('provider', 'ref', '33ed9b0ac6e4255877ff109a1d91fa84c9097dc5'); cl.save()" | ||
| - python3 -c "from virttest.asset import ConfigLoader; cl = ConfigLoader('virttest/test-providers.d/io-github-autotest-libvirt.ini'); cl.set('provider', 'ref', 'e6af7700f879fe1547bfb1421439121d35c919bd'); cl.save()" |
There was a problem hiding this comment.
I agree with you that it is important to pin the versions of tp-qemu and tp-libvirt, so we will be changing only one component at the time. But IMO we still need to update these version once in a while, and I am missing some well-defined rules how this will be done. Something like release check that, the tp-qemu and tp-libvirt were updated to the newest version in every release.
There was a problem hiding this comment.
The exact release steps need to be better ironed out indeed on the Avocado-VT side. I can follow this up with a documentation change proposal, or open a discussion about it.
But I think both would take a lot more time (because of the discussion) than what is needed for a release related change.
Let me know if you are thinking of something simpler that can be addressed here.
There was a problem hiding this comment.
No, I agree with you that this needs to be well discuss, so I am ok with doing this after the release. Can you please open an issue or discussion about it, so we don't forget about this? Thanks
There was a problem hiding this comment.
Sure. The issue tracking this is #3462 .
Up until now, the bootstrap process run on CI will fetch the HEAD of the repository URIs given. In testing, it's a good practice to only change one component at a time. By changing only one component at time, we are able to quickly point at compatibility regressions (say a tp-qemu test needs something that is being dropped or changing behavior in Avocado-VT), or, be required to update the reference on the test provider to a version that can make use of a new or improved Avocado-VT API. In order to expand the integration coverage of the test providers and Avocado-VT, it's my understanding that this locking of versions is needed so that maintainers can keep their sanity. Signed-off-by: Cleber Rosa <[email protected]>
By expanding the number of tests from test providers that we run, we increase the level of confidence that Avocado-VT API changes are not breaking tests. The list of tests added was curated by attempting to add tests that are known to be stable on other testing scenarios, but adapted to the Cirrus CI environment (based on JeOS, different KVM host environment, etc). This list is not set in stone, and can and should be continually curated to represent a better sampling of tests and thus a better integration coverage. Signed-off-by: Cleber Rosa <[email protected]>
This is a workaround for the fact that current Avocado will look for the runner script while doing resolution (creating Runnables) because it will populate the Runnable configuration with the "used configuration" for that kind of runner. If a change in Avocado will be needed is TBD, but this takes care of the broken resolution in the CI environment. Reference: avocado-framework#3457 Signed-off-by: Cleber Rosa <[email protected]>
By actually running actual tests, and not just a dry-run, we can increase the compatibility coverage between latest Avocado and Avocado-VT. Signed-off-by: Cleber Rosa <[email protected]>
This marks the point where Avocado-VT and Avocado are compatible (to
the extent that is being tested here). With this, an Avocado-VT
release matching an Avocado release will be produced.
In short, from the latest Avocado (tested with here) there will be a
release 98.0, and from the latest Avocado-VT commit there will be
produced a release 98.0 too. Then both Avocado and Avocado-VT 98.0
will be compatible releases between themselves.
There are two ways to deal with the waiver after the releases are
made:
1) Reinstante the waiver right away
2) Reinstante the waiver when a compatibility with the post Avocado
98.0 release (the development version of Avocado) is found.
I'd suggest going with 2, and creating issues to track the
compatibility problems found and that need to be addressed before the
next compatible release.
Signed-off-by: Cleber Rosa <[email protected]>
|
I'm sending a rebased version of this (with no other changes) just for the sake of a final CI test with what is going to be Avocado 98.0. |
clebergnu
force-pushed
the
improve_ci_test_providers_coverage
branch
from
2064bae to
53e8f1e
Compare
|
All CI checks OK, merging to proceed with release. |
This is a preparation for the Avocado-VT 98.0 tag/release, which is going to be a release compatible with the latest Avocado release (98.0). Included here is:
More explanation about the last point can be found in the respective commit.