Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): run lint in a separate build before running the tests #2876

Open
wants to merge 71 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
c9c63bd
chore: run lint in a separate build before running the tests
mdelapenya Nov 5, 2024
4bab0fd
fix: lint work dir
mdelapenya Nov 5, 2024
2f6d74e
Revert "fix: lint work dir"
mdelapenya Nov 6, 2024
77b8c25
Revert "chore: run lint in a separate build before running the tests"
mdelapenya Nov 6, 2024
eaef8e1
chore: detect modified files and select which module to run the tests…
mdelapenya Nov 7, 2024
cfcec39
fix: read env var properly
mdelapenya Nov 7, 2024
4a4482f
fix: simplify
mdelapenya Nov 7, 2024
6550d8e
fix: single quotes
mdelapenya Nov 7, 2024
93934ac
chore: remove job
mdelapenya Nov 7, 2024
2e35ae8
fix: proper all_modified_files quotes
mdelapenya Nov 7, 2024
87d9718
chore: print out modules to run
mdelapenya Nov 7, 2024
4b63312
fix: do not send an array with all modules, but a comma-separated lis…
mdelapenya Nov 7, 2024
1c17f59
fix: do not prepend the root_dir for the modules
mdelapenya Nov 7, 2024
f68e6af
chore: proper runner definition
mdelapenya Nov 7, 2024
3272c43
fix: use runner in the right place
mdelapenya Nov 7, 2024
5e1f13a
chore: separate lint to a different workflow
mdelapenya Nov 7, 2024
8a51fe6
fix: use fromJson
mdelapenya Nov 7, 2024
e9e62cc
fix: proper jobs chain
mdelapenya Nov 7, 2024
d0cf9ab
fix: no need to lint twice
mdelapenya Nov 7, 2024
afedcd6
fix: remove non-existing input
mdelapenya Nov 7, 2024
2352d19
fix: sanitise path
mdelapenya Nov 7, 2024
ff1a5cf
chore: only run the reaper if the core module is modified
mdelapenya Nov 7, 2024
e30d9a0
revert: simple lint
mdelapenya Nov 7, 2024
84d2fd9
Revert "revert: simple lint"
mdelapenya Nov 7, 2024
d66f266
revet: remove lint
mdelapenya Nov 7, 2024
e6636ac
Revert "revet: remove lint"
mdelapenya Nov 7, 2024
02fe23f
chore: try with input platform
mdelapenya Nov 7, 2024
c5db8ab
fix: forgotten platform
mdelapenya Nov 7, 2024
49f7473
fix: pass a string to the reusable workflow
mdelapenya Nov 7, 2024
b4480a8
fix: remove missing parenthesis
mdelapenya Nov 7, 2024
2638f2a
fix: proper input value as string
mdelapenya Nov 7, 2024
c7e5e7e
fix: order properly
mdelapenya Nov 7, 2024
5514edd
chore: simplify merging scripts
mdelapenya Nov 7, 2024
718e0bc
fix: proper array syntax
mdelapenya Nov 8, 2024
a6b0e53
fix: outputs are consumed by downstream jobs that needs the producer
mdelapenya Nov 8, 2024
3314f8d
fix: remove extra element
mdelapenya Nov 8, 2024
459de9f
chore: detemine platform in downstream job
mdelapenya Nov 8, 2024
e23c1cd
fix: read env properly
mdelapenya Nov 8, 2024
c7476de
chore: define runner in a previos step
mdelapenya Nov 8, 2024
9a3f6f9
fix: proper output format
mdelapenya Nov 8, 2024
b75cf65
Revert "fix: proper output format"
mdelapenya Nov 8, 2024
607fb1f
Revert "chore: define runner in a previos step"
mdelapenya Nov 8, 2024
f8dd7b9
chore: define matrix for the downstream job
mdelapenya Nov 8, 2024
f782846
fix: try with string
mdelapenya Nov 8, 2024
9bc5a52
fix: use fromJSON
mdelapenya Nov 8, 2024
963de87
fix: items
mdelapenya Nov 8, 2024
fb33924
chore: default as string
mdelapenya Nov 8, 2024
7bc38cc
chore: pass platforms in the parent workflow
mdelapenya Nov 8, 2024
c12c7a6
fix: single platforms must follow the array-instring format
mdelapenya Nov 8, 2024
bfddbc6
chore: exclude certain locations from the root dir
mdelapenya Nov 8, 2024
4b8d2fa
fix: use jq to get modules count
mdelapenya Nov 8, 2024
8e5691c
chore: do not run sonarcloud for zero modules
mdelapenya Nov 8, 2024
7a2fe26
chore: separate concerns
mdelapenya Nov 8, 2024
0a387e9
chore: format variable name
mdelapenya Nov 8, 2024
b638baa
chore: no need to generate the ci.yml anymore
mdelapenya Nov 8, 2024
d4adfd5
chore: merge lint and test workflow files again
mdelapenya Nov 11, 2024
f9682df
fix: missing lint job in workflow
mdelapenya Nov 11, 2024
730f3e6
chore: run lint first
mdelapenya Nov 11, 2024
b939d62
fix: update name
mdelapenya Nov 11, 2024
0aab655
Merge branch 'main' into run-lint-first
mdelapenya Nov 12, 2024
b8920ef
Merge branch 'main' into run-lint-first
mdelapenya Nov 13, 2024
d8d62d7
fix: pin the action commit
mdelapenya Nov 13, 2024
a63796b
Merge branch 'main' into run-lint-first
mdelapenya Nov 18, 2024
da8b161
chore: matrix.platform
mdelapenya Nov 18, 2024
25c1a90
Merge branch 'main' into run-lint-first
mdelapenya Nov 18, 2024
0b714eb
Merge branch 'main' into run-lint-first
mdelapenya Nov 18, 2024
12a90ab
chore: run lint stage before the test one
mdelapenya Nov 19, 2024
1f63457
Merge branch 'main' into run-lint-first
mdelapenya Nov 27, 2024
b1ce85f
Merge branch 'main' into run-lint-first
mdelapenya Dec 5, 2024
c0a4ecf
Merge branch 'main' into run-lint-first
mdelapenya Dec 11, 2024
f4845d2
Merge branch 'main' into run-lint-first
mdelapenya Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions .github/workflows/ci-lint-go.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: Run lint for a Go project
run-name: "${{ inputs.project-directory }}"

on:
workflow_call:
inputs:
project-directory:
required: true
type: string
default: "."
description: "The directory where the Go project is located."

permissions:
contents: read
# Optional: allow read access to pull request. Use with `only-new-issues` option.
# pull-requests: read

jobs:
lint-go-project:
name: "lint: ${{ inputs.project-directory }}"
runs-on: 'ubuntu-latest'
continue-on-error: false
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4

- name: Set up Go
uses: actions/setup-go@0c52d547c9bc32b1aa3301fd7a9cb496313a4491 # v5
with:
go-version-file: "${{ inputs.project-directory == '' && '.' || inputs.project-directory }}/go.mod"
cache-dependency-path: "${{ inputs.project-directory == '' && '.' || inputs.project-directory }}/go.sum"
id: go

- name: golangci-lint
uses: golangci/golangci-lint-action@971e284b6050e8a5849b72094c50ab08da042db8 # v6.1.1
with:
# Optional: version of golangci-lint to use in form of v1.2 or v1.2.3 or `latest` to use the latest version
version: v1.61.0
# Optional: working directory, useful for monorepos
working-directory: ${{ inputs.project-directory }}
# Optional: golangci-lint command line arguments.
args: --verbose
# Optional: if set to true then the all caching functionality will be complete disabled,
# takes precedence over all other caching options.
skip-cache: true

- name: generate
working-directory: ./${{ inputs.project-directory }}
shell: bash
run: |
make generate
git --no-pager diff && [[ 0 -eq $(git status --porcelain | wc -l) ]]

- name: modVerify
working-directory: ./${{ inputs.project-directory }}
run: go mod verify

- name: modTidy
working-directory: ./${{ inputs.project-directory }}
shell: bash
run: |
make tidy
git --no-pager diff && [[ 0 -eq $(git status --porcelain | wc -l) ]]
65 changes: 13 additions & 52 deletions .github/workflows/ci-test-go.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: Run tests for a Go project
run-name: "${{ inputs.project-directory }} ${{ inputs.go-version }} ${{ inputs.platform }}"
run-name: "${{ inputs.project-directory }} ${{ inputs.go-version }} ${{ inputs.platforms }}"

on:
workflow_call:
Expand All @@ -8,15 +8,16 @@ on:
required: true
type: string
description: "The version of Go to use for the test."
platform:
required: true
type: string
description: "The platform to run the test on."
fail-fast:
required: false
type: boolean
default: true
description: "Fail the workflow if any of the jobs fail."
platforms:
required: true
type: string
default: "ubuntu-latest"
description: "The platforms in which the project will be run"
project-directory:
required: true
type: string
Expand All @@ -27,11 +28,6 @@ on:
type: boolean
default: false
description: "Run the test with rootless docker."
run-tests:
required: false
type: boolean
default: true
description: "Run the tests under conditions controlled by the caller workflow."
ryuk-disabled:
required: false
type: boolean
Expand All @@ -45,13 +41,17 @@ permissions:

jobs:
test-go-project:
name: "${{ inputs.project-directory }}/${{ inputs.platform }}/${{ inputs.go-version }}"
runs-on: ${{ inputs.platform }}
name: "test: ${{ inputs.project-directory }}/${{ inputs.go-version }}"
# Modulegen can run the tests on all platforms
continue-on-error: ${{ !inputs.fail-fast }}
env:
TESTCONTAINERS_RYUK_DISABLED: "${{ inputs.ryuk-disabled }}"
RYUK_CONNECTION_TIMEOUT: "${{ inputs.project-directory == 'modules/compose' && '5m' || '60s' }}"
RYUK_RECONNECTION_TIMEOUT: "${{ inputs.project-directory == 'modules/compose' && '30s' || '10s' }}"
strategy:
matrix:
platform: ${{ fromJSON(inputs.platforms) }}
runs-on: ${{ matrix.platform }}
steps:
- name: Setup rootless Docker
if: ${{ inputs.rootless-docker }}
Expand All @@ -69,50 +69,11 @@ jobs:
cache-dependency-path: '${{ inputs.project-directory }}/go.sum'
id: go

- name: golangci-lint
if: ${{ inputs.platform == 'ubuntu-latest' }}
uses: golangci/golangci-lint-action@971e284b6050e8a5849b72094c50ab08da042db8 # v6.1.1
with:
# Optional: version of golangci-lint to use in form of v1.2 or v1.2.3 or `latest` to use the latest version
version: v1.61.0
# Optional: working directory, useful for monorepos
working-directory: ${{ inputs.project-directory }}
# Optional: golangci-lint command line arguments.
args: --verbose
# Optional: if set to true then the all caching functionality will be complete disabled,
# takes precedence over all other caching options.
skip-cache: true

- name: generate
if: ${{ inputs.platform == 'ubuntu-latest' }}
working-directory: ./${{ inputs.project-directory }}
shell: bash
run: |
make generate
git --no-pager diff && [[ 0 -eq $(git status --porcelain | wc -l) ]]

- name: modVerify
working-directory: ./${{ inputs.project-directory }}
run: go mod verify

- name: modTidy
if: ${{ inputs.platform == 'ubuntu-latest' }}
working-directory: ./${{ inputs.project-directory }}
shell: bash
run: |
make tidy
git --no-pager diff && [[ 0 -eq $(git status --porcelain | wc -l) ]]

- name: ensure compilation
working-directory: ./${{ inputs.project-directory }}
run: go build

- name: go test
# only run tests on linux, there are a number of things that won't allow the tests to run on anything else
# many (maybe, all?) images used can only be build on Linux, they don't have Windows in their manifest, and
# we can't put Windows Server in "Linux Mode" in Github actions
# another, host mode is only available on Linux, and we have tests around that, do we skip them?
if: ${{ inputs.run-tests }}
working-directory: ./${{ inputs.project-directory }}
timeout-minutes: 30
run: make test-unit
Expand All @@ -125,7 +86,7 @@ jobs:
echo "ARTIFACT_NAME=$(basename ${{ inputs.project-directory == '.' && 'core' || inputs.project-directory }})-${{ inputs.go-version }}-${{ inputs.platform }}" >> $GITHUB_ENV

- name: Upload SonarCloud files
if: ${{ github.ref_name == 'main' && github.repository_owner == 'testcontainers' && inputs.platform == 'ubuntu-latest' && inputs.run-tests && !inputs.rootless-docker && !inputs.ryuk-disabled }}
if: ${{ github.ref_name == 'main' && github.repository_owner == 'testcontainers' && matrix.platform == 'ubuntu-latest' && !inputs.rootless-docker && !inputs.ryuk-disabled }}
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
with:
name: sonarcloud-${{ env.ARTIFACT_NAME }}
Expand Down
134 changes: 64 additions & 70 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,46 +1,83 @@
# This file is autogenerated by the 'modulegen' tool.
# Please update the 'ci.yml' template instead.
name: Main pipeline

on:
push:
branches:
- main
paths-ignore:
- '.vscode/**'
- 'mkdocs.yml'
- 'docs/**'
- 'README.md'
pull_request:
paths-ignore:
- '.vscode/**'
- 'mkdocs.yml'
- 'docs/**'
- 'README.md'

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.sha }}
cancel-in-progress: true

jobs:
detect-modules:
runs-on: ubuntu-latest
outputs:
modules: ${{ steps.set-modified-modules.outputs.modules }}
modules_count: ${{ steps.set-modified-modules-count.outputs.modules_count }}
steps:
- name: Check out code into the Go module directory
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4

- id: changed-files
name: Get changed files
uses: tj-actions/changed-files@4edd678ac3f81e2dc578756871e4d00c19191daf # v45.0.4

- id: set-modified-modules
name: Set all modified modules
env:
ALL_CHANGED_FILES: "${{ steps.changed-files.outputs.all_changed_files }}"
run: echo "modules=$(./scripts/changed-modules.sh)" >> $GITHUB_OUTPUT

- id: set-modified-modules-count
name: Set all modified modules count
run: echo "modules_count=$(echo ${{ toJSON(steps.set-modified-modules.outputs.modules) }} | jq '. | length')" >> $GITHUB_OUTPUT

- name: Print out the modules to be used
run: |
echo "${{ steps.set-modified-modules-count.outputs.modules_count }} modules in the build"
echo "${{ steps.set-modified-modules.outputs.modules }}"

lint:
# only run if there are modules to lint
if: ${{ needs.detect-modules.outputs.modules_count > 0 }}
needs:
- detect-modules
strategy:
matrix:
module: ${{ fromJSON(needs.detect-modules.outputs.modules) }}
uses: ./.github/workflows/ci-lint-go.yml
with:
project-directory: "${{ matrix.module }}"

test:
# only run if there are modules to test
if: ${{ needs.detect-modules.outputs.modules_count > 0 }}
needs:
- detect-modules
- lint
strategy:
matrix:
go-version: [1.22.x, 1.x]
platform: [ubuntu-latest, macos-latest]
module: ${{ fromJSON(needs.detect-modules.outputs.modules) }}
uses: ./.github/workflows/ci-test-go.yml
with:
go-version: ${{ matrix.go-version }}
fail-fast: true
platform: ${{ matrix.platform }}
project-directory: "."
platforms: ${{ matrix.module == 'modulegen' && '["ubuntu-latest", "macos-latest", "windows-latest"]' || '["ubuntu-latest"]' }}
project-directory: "${{ matrix.module }}"
rootless-docker: false
run-tests: ${{ matrix.platform == 'ubuntu-latest' }}
ryuk-disabled: false

# The job below is a copy of the job above, but with ryuk disabled.
# It's executed in the first stage to avoid concurrency issues.
test-reaper-off:
# the core module is identified by the empty string (the root path)
if: ${{ contains(fromJSON(needs.detect-modules.outputs.modules), '') }}
needs:
- detect-modules
- lint
name: "Test with reaper off"
strategy:
matrix:
Expand All @@ -49,83 +86,40 @@ jobs:
with:
go-version: ${{ matrix.go-version }}
fail-fast: false
platform: "ubuntu-latest"
platforms: '["ubuntu-latest"]'
project-directory: "."
rootless-docker: false
run-tests: true
ryuk-disabled: true

# The job below is a copy of the job above, but with Docker rootless.
# It's executed in the first stage to avoid concurrency issues.
test-rootless-docker:
# the core module is identified by the empty string (the root path)
if: ${{ contains(fromJSON(needs.detect-modules.outputs.modules), '') }}
needs:
- detect-modules
- lint
name: "Test with Rootless Docker"
strategy:
matrix:
go-version: [1.22.x, 1.x]
platform: [ubuntu-latest]
uses: ./.github/workflows/ci-test-go.yml
with:
go-version: ${{ matrix.go-version }}
fail-fast: false
platform: "ubuntu-latest"
platforms: '["ubuntu-latest"]'
project-directory: "."
rootless-docker: true
run-tests: true
ryuk-disabled: false

test-module-generator:
strategy:
matrix:
go-version: [1.22.x, 1.x]
platform: [ubuntu-latest, macos-latest, windows-latest]
uses: ./.github/workflows/ci-test-go.yml
with:
go-version: ${{ matrix.go-version }}
fail-fast: true
platform: ${{ matrix.platform }}
project-directory: "modulegen"
rootless-docker: false
run-tests: true
ryuk-disabled: false

test-modules:
needs: test
strategy:
matrix:
go-version: [1.22.x, 1.x]
platform: [ubuntu-latest]
module: [artemis, azurite, cassandra, chroma, clickhouse, cockroachdb, compose, consul, couchbase, databend, dolt, dynamodb, elasticsearch, etcd, gcloud, grafana-lgtm, inbucket, influxdb, k3s, k6, kafka, localstack, mariadb, meilisearch, milvus, minio, mockserver, mongodb, mssql, mysql, nats, neo4j, ollama, openfga, openldap, opensearch, postgres, pulsar, qdrant, rabbitmq, redis, redpanda, registry, surrealdb, valkey, vault, vearch, weaviate, yugabytedb]
uses: ./.github/workflows/ci-test-go.yml
with:
go-version: ${{ matrix.go-version }}
fail-fast: false
platform: ${{ matrix.platform }}
project-directory: modules/${{ matrix.module }}
rootless-docker: false
run-tests: ${{ matrix.platform == 'ubuntu-latest' }}
ryuk-disabled: false

test-examples:
needs: test-modules
strategy:
matrix:
module: [nginx, toxiproxy]
uses: ./.github/workflows/ci-test-go.yml
with:
go-version: "1.22.x"
fail-fast: true
platform: 'ubuntu-latest'
project-directory: examples/${{ matrix.module }}
rootless-docker: false
run-tests: true
ryuk-disabled: false

sonarcloud:
permissions:
contents: read # for actions/checkout to fetch code
pull-requests: read # for sonarsource/sonarcloud-github-action to determine which PR to decorate
if: ${{ github.ref_name == 'main' && github.repository_owner == 'testcontainers' }}
needs: test-examples
if: ${{ github.ref_name == 'main' && github.repository_owner == 'testcontainers' && needs.detect-modules.outputs.modules_count > 0 }}
needs:
- detect-modules
- test
runs-on: ubuntu-latest
steps:
- name: Check out code into the Go module directory
Expand Down
Loading
Loading