CI adjustments #3535

apostasie · 2024-10-14T00:33:26Z

~~NOTE: this is pending #3572, #3573, #3574, #3575~~

This is the next part of the test rework effort.

The most important change here

Is that we will now have two separate steps for every integration test run:

without retry (this is new) - running all tests that are deemed safe ONCE, and hard failing if one of them fails
with retry (this is the previous behavior) - running any other test

Note that the above split does NOT apply to ipv6, nor kube, tests. These are considered safe.

Right now, the balance is about 286 (rootful, no retry) vs. 500 (with retry).
Hopefully, as we fix and migrate more tests, and address bugs in nerdctl, it will shift.

Note that running legacy tests without retries is currently very painful and does surface a lot of conditions that have been previously unseen / ignored for a long time. As a lot of bugs have been recently fixed with regards to concurrency, underlying issues are also starting to crop up, and these very likely run deep / are involved to fix.
These are chiefly:

This is the reason why we are - conservatively - leaving the pre-existing, legacy tests in the "with retries" bucket.

There is now a helper script, test-integration.sh.

When ran without any argument, if will run both steps in order:

If you want to run ONLY the first step (aka "safe tests"), call:
test-integration.sh -test.only-flaky=false

If you want to run ONLY the second step, call:
test-integration.sh -test.only-flaky=true

That works as well with gotestsum, or go test, evidently:

Note though that the default in that case is -test.only-flaky=false:

go test ./cmd/nerdctl/image
go test ./cmd/nerdctl/image -test.only-flaky
# etc...

Other CI changes

cosmetic efforts on test display titles (more compact, more consistent)
removal of all nick/retries
minor tweaks to the Dockerfile to adapt for new script

.github/workflows/lint.yml

.github/workflows/test-canary.yml

.github/workflows/test.yml

cmd/nerdctl/issues/issues_linux_test.go

cmd/nerdctl/network/network_create_linux_test.go

hack/build-integration-kubernetes.sh

hack/test-integration.sh

pkg/mountutil/mountutil_test.go

pkg/netutil/netutil.go

pkg/testutil/test/helpers.go

pkg/testutil/testregistry/testregistry_linux.go

pkg/testutil/testutil.go

cmd/nerdctl/image/image_inspect_test.go

apostasie · 2024-10-18T22:32:29Z

.github/workflows/test.yml


 jobs:
  test-unit:
+    # Supposed to work: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/evaluate-expressions-in-workflows-and-actions#example-returning-a-json-data-type
+    # Apparently does not
+    # timeout-minutes: ${{ fromJSON(env.SHORT_TIMEOUT) }}


I wish that worked. Anyone has insight?

apostasie · 2024-10-18T22:32:45Z

.github/workflows/test.yml

-          max_attempts: 2
-          retry_on: error
-          command: docker run -t --rm --privileged test-integration
+        run: docker run -t --rm --privileged test-integration ./hack/test-integration.sh -test.only-flaky=false


Split the two steps.

apostasie · 2024-10-18T22:33:12Z

.github/workflows/test.yml

@@ -129,7 +133,7 @@ jobs:
          echo '{"ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64", "experimental": true, "ip6tables": true}' | sudo tee /etc/docker/daemon.json
          sudo systemctl restart docker
      - name: "Prepare integration test environment"
-        run: docker build -t test-integration-ipv6 --target test-integration-ipv6 --build-arg UBUNTU_VERSION=${UBUNTU_VERSION} --build-arg CONTAINERD_VERSION=${CONTAINERD_VERSION} .


No need for the extra stage that has nothing specific. It is just an extra arg.

apostasie · 2024-10-18T22:33:23Z

.github/workflows/test.yml

-          max_attempts: 2
-          retry_on: error
-          command: docker run --network host -t --rm --privileged test-integration-ipv6
+        run: docker run --network host -t --rm --privileged test-integration ./hack/test-integration.sh -test.only-ipv6


nick should go

apostasie · 2024-10-18T22:34:02Z

.github/workflows/test.yml

@@ -248,12 +246,13 @@ jobs:
          go-version: ${{ matrix.go-version }}
          cache: true
          check-latest: true
-      - name: "Cross"
+      - name: "build"


Better display. This is not "cross-compilation" it is building for different versions of go.

apostasie · 2024-10-18T22:34:46Z

pkg/testutil/testutil.go

@@ -787,6 +787,9 @@ func newBase(t *testing.T, ns string, ipv6Compatible bool, kubernetesCompatible
 	} else if !base.EnableKubernetes && base.KubernetesCompatible {
 		t.Skip("runner skips Kubernetes compatible tests in the non-Kubernetes environment")
 	}
+	if !GetFlakyEnvironment() && !GetEnableKubernetes() && !GetEnableIPv6() {


This is the part that keep the legacy tests into the "retry" part.

apostasie · 2024-10-18T23:09:41Z

Oh docker, I...

apostasie · 2024-10-19T22:27:45Z

That was quite a trek.

@AkihiroSuda @djdongjin at your convenience.
(See ticket about CI status / data for more info on where we stand)

AkihiroSuda · 2024-10-20T21:47:59Z

hack/test-integration.sh

+  fi
+done
+
+if [ "$needsudo" != "" ]; then


This can cause a confusion when WITH_SUDO is set to "0", "false", etc.

Fair. I'll restrict it.

Signed-off-by: apostasie <[email protected]>

AkihiroSuda

Thanks

apostasie force-pushed the part5 branch from 7d83bdd to d71d69a Compare October 14, 2024 00:38

apostasie changed the title ~~Test rework, part 5~~ [WIP] Test rework, part 5 Oct 14, 2024

apostasie force-pushed the part5 branch from d71d69a to b9bfb1b Compare October 14, 2024 01:20