E2E sharding part 2 #90

rylew1 · 2024-11-08T19:20:05Z

Ticket

Resolves navapbc/template-infra#720

Changes

wrap playwright merge cmd in make target e2e-merge-reports
set matrix fail-fast to false so reports can be generated if one of the shards fails
update docs for sharding

Context for reviewers

Sharding part 1 PR Run e2e tests in parallel shards template-infra#776
Took a pass at uploading a saved docker image to github actions artifact - which means we avoid rebuilding image in each shard - however this ended up taking longer to upload and download from artifacts. It's faster just to build in each concurrent shard run with make e2e-test
- see a prior run with docker/buildx for how that worked - https://github.com/navapbc/platform-test-nextjs/actions/runs/11808432981/job/32897151032
- And the comment - E2E sharding part 2 #90 (comment)

Testing

Local sharding and report demo:

make e2e-test APP_NAME=app BASE_URL=http://host.docker.internal:3001 TOTAL_SHARDS=3 CURRENT_SHARD=1 CI=true && \
make e2e-test APP_NAME=app BASE_URL=http://host.docker.internal:3001 TOTAL_SHARDS=3 CURRENT_SHARD=2 CI=true && \
make e2e-test APP_NAME=app BASE_URL=http://host.docker.internal:3001 TOTAL_SHARDS=3 CURRENT_SHARD=3 CI=true

make e2e-merge-reports REPORT_PATH=blob-report
make e2e-show-report
make e2e-clean-report

the above commands create a combined report locally:

Demo of one shard failing (fail-fast demo) - html report is still created:

output.mov

Demo of nodejs caching - skips make e2e-setup-ci step on cache hit:

Preview environment

Service endpoint: http://p-90-app-dev-1757748058.us-east-1.elb.amazonaws.com
Deployed commit: f24a51a

.github/workflows/e2e-tests.yml

rylew1 · 2024-11-13T12:19:14Z

.github/workflows/e2e-tests.yml

-        matrix:
-          shard: [1, 2, 3]
-          total_shards: [3] # Github Actions doesn't have a built-in method to get the length of an array
+      fail-fast: false


add a fail-fast to false on the matrix strategy here so even if one shard fails, we still can generate a report out of othershard runs that have succeeded. See PR description for a demo of this working

lorenyu · 2024-11-14T21:46:29Z

is this PR ready for review? i'm confused it looks like there are some changes in here that were already in the previous PR related to e2e tests. did you merge main into this branch

rylew1 · 2024-11-14T23:54:22Z

I'm confused it looks like there are some changes in here that were already in the previous PR related to e2e tests. did you merge main into this branch

@lorenyu There was some shuffle due to the template deploy workflow not working for a couple days - I ended up branching this PR of the sharding part 1 PR just to get started before the template-deploy was fixed - but I'm not sure what parts you're thinking of that are older.

I think the download blob reports step may not have been updated to use ./e2e/blob-report on the PR merge to template-infra - so updating them in this PR
I had to update the package-lock.json to fix native run of e2e tests

I've updated more today, including removing the docker setup cache because it was taking too long - I think it's ready for review

rylew1 · 2024-11-15T01:34:55Z

e2e/package-lock.json

@@ -4,231 +4,231 @@
    "lockfileVersion": 2,
    "requires": true,
    "packages": {
-      "": {


Had to update the lock file to run tests natively/locally

lorenyu

Looking good but let's resolve the issue with some previous changes not appearing on main first

lorenyu · 2024-11-15T16:51:45Z

.github/workflows/e2e-tests.yml

-            BASE_URL=${{ inputs.service_endpoint }} \
-            CURRENT_SHARD=${{ matrix.shard }} \
-            TOTAL_SHARDS=${{ matrix.total_shards }}
+            make e2e-test \


why the extra indent

lorenyu · 2024-11-15T16:52:51Z

.github/workflows/e2e-tests.yml

+        id: cache-node
+        uses: actions/cache@v4
+        with:
+            path: e2e/node_modules


indenting too much here let's be consistent. do you need to check editor settings? i would have imagined the editor would autoformat on save

Are we using actionlint for this? What formatter are you using locally for these yml files

Makefile

.github/workflows/e2e-tests.yml

Makefile

docs/e2e/e2e-checks.md

Co-authored-by: Loren Yu <[email protected]>

rylew1 · 2024-11-18T02:16:57Z

Merged on template-infra navapbc/template-infra#782

rylew1 and others added 30 commits October 27, 2024 21:16

try sharding

20eb003

update shard workflow name and report

5fa0f7a

try merge reports

bfb6601

fix output folder

e1a16e6

revert to example

7e44eae

add verify blob dir step

ff755f7

try ci true

91cd9b0

remove env block in favor of passing args

7eafd22

Merge branch 'main' into rylew/e2e-shard

7c880b1

remove older makefile updates

884b604

try updated package-lock

b9a6e04

try updated workflow file

5ed2fe8

allow e2e-test to pass in shard config

e662964

print all env vars

77e2cc0

updates

086e85b

update path on blob-report

6d2d48e

map blob-report dir on e2e-test cmd

d32452d

docker cp

381aa81

docker cp in e2e-test

a239337

remove -rm on e2e-test

a03ca1e

try w/ updated permissions

3b294f4

checkk blob report folder in container

d09633d

wait for blob-report script

f737805

copy wait script over in dockerfile

dffa14f

call shell script from e2e folder

defffea

fix path

d7ade2a

updates

64f6374

remove extra step

01595b3

fix shell script

b508422

check playwright-report folder in script

d712b04

rylew1 added 2 commits November 12, 2024 19:39

try with make command

2aebd72

revert to docker run

75505dd

rylew1 commented Nov 13, 2024

View reviewed changes

.github/workflows/e2e-tests.yml Outdated Show resolved Hide resolved

rylew1 commented Nov 13, 2024

View reviewed changes

rylew1 requested a review from coilysiren November 13, 2024 16:43

Merge branch 'main' into rylew/e2e-sharding-part-2

7135c9a

rylew1 added 7 commits November 14, 2024 19:11

remove setup from workflow

72a4153

revert package-lock.json

96661f9

update package-lock for local dev

9c9d615

debug cache hit output

310a328

try without cache restore-keys

2bd3c15

remove debug cache hit

7631929

update docs

16f03ef

rylew1 commented Nov 15, 2024

View reviewed changes

rylew1 marked this pull request as ready for review November 15, 2024 01:40

update doc comments

3f3ee3b

lorenyu reviewed Nov 15, 2024

View reviewed changes

rylew1 and others added 7 commits November 15, 2024 16:58

Merge branch 'main' into rylew/e2e-sharding-part-2

607599b

update merge report step name

9b457b3

Co-authored-by: Loren Yu <[email protected]>

comment update

8796227

imperative voice update

44e6b9c

update name on very last step

3be9aab

package-lock space

419d9ca

remove report path param on e2e-merge-reports make target

f24a51a

lorenyu approved these changes Nov 15, 2024

View reviewed changes

rylew1 mentioned this pull request Nov 15, 2024

Finish running e2e tests even when one shard fails navapbc/template-infra#783

Merged

rylew1 closed this Nov 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E sharding part 2 #90

E2E sharding part 2 #90

rylew1 commented Nov 8, 2024 •

edited by github-actions bot

Loading

rylew1 Nov 13, 2024 •

edited

Loading

lorenyu commented Nov 14, 2024

rylew1 commented Nov 14, 2024 •

edited

Loading

rylew1 Nov 15, 2024 •

edited

Loading

lorenyu left a comment

lorenyu Nov 15, 2024

lorenyu Nov 15, 2024

rylew1 Nov 15, 2024

rylew1 commented Nov 18, 2024

E2E sharding part 2 #90

E2E sharding part 2 #90

Conversation

rylew1 commented Nov 8, 2024 • edited by github-actions bot Loading

Ticket

Changes

Context for reviewers

Testing

Preview environment

rylew1 Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

lorenyu commented Nov 14, 2024

rylew1 commented Nov 14, 2024 • edited Loading

rylew1 Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

lorenyu left a comment

Choose a reason for hiding this comment

lorenyu Nov 15, 2024

Choose a reason for hiding this comment

lorenyu Nov 15, 2024

Choose a reason for hiding this comment

rylew1 Nov 15, 2024

Choose a reason for hiding this comment

rylew1 commented Nov 18, 2024

rylew1 commented Nov 8, 2024 •

edited by github-actions bot

Loading

rylew1 Nov 13, 2024 •

edited

Loading

rylew1 commented Nov 14, 2024 •

edited

Loading

rylew1 Nov 15, 2024 •

edited

Loading