Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Test matrix #1862

Closed
wants to merge 11 commits into from
Closed

ci: Test matrix #1862

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3f4c940
setup test matrix.
nasdf Sep 8, 2023
25eb1cc
add code coverage to run-tests workflow. remove code-test-coverage wo…
nasdf Sep 11, 2023
95200e6
use makefile for run-tests workflow. use a single step for codecov up…
nasdf Sep 12, 2023
a6c6876
add missing runs-on property to run-tests workflow
nasdf Sep 12, 2023
4f4bb35
set retention for coverage files. add coverpkg flag to test:ci Make t…
nasdf Sep 12, 2023
9cfbbb7
fix incorrect make command in run-tests workflow
nasdf Sep 12, 2023
73eddb2
Merge branch 'develop' into nasdf/ci/test-matrix
nasdf Sep 20, 2023
a349561
replace go-acc with go 1.20 aggregate test coverage. add change detec…
nasdf Sep 20, 2023
6662cf3
fix missing Makefile target
nasdf Sep 20, 2023
80784f6
remove detect-change workflow. update run-tests matrix to include cha…
nasdf Sep 20, 2023
d9be82d
add change-detector matrix variable to run-tests workflow
nasdf Sep 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 0 additions & 76 deletions .github/workflows/code-test-coverage.yml
View file
Open in desktop

This file was deleted.

55 changes: 0 additions & 55 deletions .github/workflows/detect-change.yml
View file
Open in desktop

This file was deleted.

62 changes: 54 additions & 8 deletions .github/workflows/run-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,29 @@ on:

jobs:
run-tests:
name: Run tests job
name: Run tests job matrix

runs-on: ubuntu-latest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: We could also include this in the matrix at somepoint if we want (I would like this).


strategy:
matrix:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: 😍 😍 😍 I did not know we could do this. And I really want it. Thank you very much for opening this draft.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: This will increase the visibility of any test flakiness. This could be extremely annoying, but will force us to sort out flakiness.

client-type: [go, http]
database-type: [badger-file, badger-memory]
mutation-type: [gql, collection-named, collection-save]
detect-changes: [false]
include:
- client-type: go
database-type: badger-memory
mutation-type: collection-save
detect-changes: true

Copy link
Contributor

@AndrewSisley AndrewSisley Sep 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: Once we remove the -p 1 limitation, we can include the change detector in this file using an include, I think the below might work:

 include:
      - client-type: go
        database-type: badger-memory
        mutation-type: collection-save
        detect-changes: true

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added both the change detector and non change detector to the include so both will now run.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: I think we should also include the codecov stuff in this file, and delete the codecov workflow. I don't want to mark this as a todo though, as I'm still playing around with it and it can be done after easily enough.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to try and get that working.

env:
DEFRA_CLIENT_GO: ${{ matrix.client-type == 'go' }}
DEFRA_CLIENT_HTTP: ${{ matrix.client-type == 'http' }}
DEFRA_BADGER_MEMORY: ${{ matrix.database-type == 'badger-memory' }}
DEFRA_BADGER_FILE: ${{ matrix.database-type == 'badger-file' }}
DEFRA_MUTATION_TYPE: ${{ matrix.mutation-type }}

steps:
- name: Checkout code into the directory
uses: actions/checkout@v3
Expand All @@ -39,12 +58,39 @@ jobs:
make deps:modules
make deps:test

- name: Build binary
run: make build
- name: Run integration tests
if: ${{ !matrix.detect-changes }}
run: make test:ci

- name: Run change detector tests
if: ${{ matrix.detect-changes }}
run: make test:changes

# This is to ensure tests pass with a running server.
- name: Start server from binary
run: ./build/defradb start &
- name: Upload coverage artifact
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.client-type }}_${{ matrix.database-type }}_${{ matrix.mutation-type }}
path: coverage.txt
if-no-files-found: error
retention-days: 1

- name: Run the tests, showing name of each test
run: make test:ci
upload-coverage:
name: Upload test code coverage

runs-on: ubuntu-latest

needs: run-tests

steps:
- name: Download coverage reports
uses: actions/download-artifact@v3

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
name: defradb-codecov
flags: all-tests
os: 'linux'
fail_ci_if_error: true
verbose: true
54 changes: 0 additions & 54 deletions .github/workflows/test-collection-named.yml
View file
Open in desktop

This file was deleted.

48 changes: 0 additions & 48 deletions .github/workflows/test-gql-mutations.yml
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ cmd/defradb/defradb
cmd/genclidocs/genclidocs
cmd/genmanpages/genmanpages
coverage.txt
coverage
tests/bench/*.log
tests/bench/*.svg

Expand Down
Loading