ci(containerised-test): add coverage #28

caspiano · 2021-11-10T06:09:25Z

Adds coverage job to the containerised test flow.

This change requires specs in each repository to mount a coverage folder into the test container.

chillfox · 2022-03-16T01:02:43Z

I tested it on staff-api and it runs, but there is this error:

Can't set personality: Operation not permitted
kcov: error: Can't start/attach to /app/app

And it uploads a file saying zero coverage by the looks of it.

chillfox · 2022-03-16T01:07:04Z

SimonKagstrom/kcov#151

docker disables the personality syscall, so need to use --security-opt seccomp=unconfined

docker run -it --rm --security-opt seccomp=unconfined -v $(pwd):/source kcov

caspiano · 2022-03-16T01:11:55Z

@chillfox adding the following to the test container in the docker-compose should solve it

    security_opt:
      - seccomp:unconfined

chillfox · 2022-03-16T01:49:16Z

Timed out after 25 minutes
Error: The action has timed out.

caspiano · 2022-03-16T02:21:18Z

@chillfox I've got an arbitrary timeout of 25 minutes in the job, you could try removing it or bumping it to be higher ~ say 45 minutes.

I'm surprised it's such a long running job

chillfox · 2022-03-16T03:44:53Z

@caspiano I would assume that it is broken, but yeah, might be worth giving it some more time to be sure.

chillfox · 2022-03-16T04:39:59Z

Still times out at 45

caspiano · 2022-03-16T04:43:15Z

It doesn't look like the specs are being run, so it might be worth trying your new images from #55, however we'll need to merge them first

chillfox · 2022-03-16T04:44:02Z

yeah,
will give that a go and if it doesn't work then I guess it will be time to fork https://github.com/flant/kcov-alpine and fix that.

This reverts commit cb05e88.

caspiano · 2022-03-16T08:29:46Z

I added some verbose logging, looks like the crkcov runner is trying to read a file that doesn't exist.

https://github.com/PlaceOS/staff-api/runs/5566730681?check_suite_focus=true#step:3:96

This reverts commit eab1dc5.

chillfox · 2022-05-16T06:45:24Z

so it works locally in Elementary OS (Ubuntu based) but not in Alpine.
In Alpien the covearge.json file is not created

chillfox · 2022-05-16T07:14:05Z

so kcov runs in Alpine and does produce the report. So it is just crkcov that does not...

chillfox · 2022-05-17T03:20:38Z

Status:

kcov works in Alpine (generates the report json).
crkcov does not work in Alpine (the report json is missing).
crkcov does work in Elementary OS (so it will most likely work in Debian/Ubuntu).

Tried:

Messing around with GH actions.
Created a small test app + docker image and script to test locally.
Light modifications to crkcov (disabling everything that deletes files).

Ways forward:

Run crkcov in a Debian/Ubuntu image (this means all our tests would need to be capable of running in Debian/Ubuntu).
Fork crkcov, make its specs run in Alpine (maybe write additional tests to pin down the issue), see how deep the rabbit hole goes.

caspiano · 2022-05-17T03:42:01Z

I reckon trying to get crkcov to work in alpine may be the best way forward. There's could be a small platform based assumption.
Working on an ubuntu image is a fine way to move forward, but it may be a bit of work, albeit straightforward work

chillfox · 2022-06-22T06:54:01Z

finally got it working in alpine

caspiano · 2022-06-22T07:20:02Z

Epic!!!!!! Great work @chillfox

Can't wait to use it

chillfox · 2022-06-23T02:06:54Z

container needs to be run with docker run --security-opt seccomp=unconfined

or in compose

security_opt:
      - seccomp:unconfined

.github/workflows/containerised-test.yml

chillfox · 2022-06-27T07:36:54Z

While this works on a small project it fails on a larger project like the portal.

➜ 5:00PM portal-api (test/coverage) ✔ ./test --coverage
...
Unhandled exception: Error writing file: Broken pipe (IO::Error)
  from /usr/share/crystal/src/gc/boehm.cr:129:5 in 'unbuffered_write'
  from /usr/share/crystal/src/slice.cr:1169:12 in 'write'
  from /usr/share/crystal/src/string.cr:4829:5 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:115:5 in 'main'
  from /lib/ld-musl-x86_64.so.1 in '??'

➜ 5:04PM portal-api (test/coverage) ✗ ./test --shell
...
/app # crystal-coverage 
Unhandled exception: Error executing process: '/bin/sh': Argument list too long (IO::Error)
  from /usr/share/crystal/src/crystal/system/unix/process.cr:225:7 in 'raise_exception_from_errno'
  from /usr/share/crystal/src/io.cr:1183:7 in 'run:shell:input:output:error'
  from /usr/share/crystal/src/process.cr:419:3 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:115:5 in 'main'
  from /lib/ld-musl-x86_64.so.1 in '??'

Likely due to the shell trying to expand some recursive argument like a list of files or something like that.

See https://github.com/place-technology/portal-api/pull/396 for testing.

caspiano · 2022-06-27T07:51:31Z

@chillfox could be how crystal-coverage concatenates the spec files together?

chillfox · 2022-06-27T08:00:08Z

well, the size of data in the pipe should not be a problem, but if it shells out/calls another program (like kcov) with too long of an argument list then that is the issue (I think the argument limit is a kernel limitation)

chillfox · 2022-06-27T08:03:14Z

maybe

if print_only
  puts final_output
else
  system("crystal", ["eval", final_output])
end

I guess I could compile crystal-coverage with some pp in there

chillfox · 2022-06-28T00:03:24Z

➜ 5:36PM code_coverage (master) ✗ grep -rni 'system' crystal-coverage/src   
crystal-coverage/src/coverage/runtime/outputters/html_report.cr:75:    system("rm -rf coverage/")
crystal-coverage/src/coverage/runtime/outputters/coveralls.cr:43:      system("curl -X POST https://coveralls.io/api/v1/jobs -H 'content-type: multipart/form-data'  -F [email protected]")
crystal-coverage/src/coverage/inject/cli.cr:50:        system("crystal", ["eval", final_output])

chillfox · 2022-06-28T00:06:39Z

max argument in alpine container

/app # getconf ARG_MAX
131072

same thing on my desktop (same problem)

➜ 9:34AM code_coverage (master) ✗ getconf ARG_MAX       
2097152

chillfox · 2022-06-28T00:49:20Z

A small test project where it works.

➜ 10:02AM code_coverage (master) ✗ cloc src spec    
       3 text files.
       3 unique files.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.00 s (922.0 files/s, 8605.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Crystal                          3              5              1             22
-------------------------------------------------------------------------------
SUM:                             3              5              1             22
-------------------------------------------------------------------------------
➜ 10:02AM code_coverage (master) ✗ /usr/local/bin/crystal-coverage
.
Lines 80.00% covered

Finished in 1.34 milliseconds
1 examples, 0 failures, 0 errors, 0 pending

A slightly larger project where it fails

➜ 10:01AM boundless-scuttlecoil cloc src spec                                                                     fossil:(  next_state_perf )  ✔ 
      26 text files.
      26 unique files.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.02 s (1245.5 files/s, 158939.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Crystal                         26            418            368           2532
-------------------------------------------------------------------------------
SUM:                            26            418            368           2532
-------------------------------------------------------------------------------
➜ 10:01AM boundless-scuttlecoil /usr/local/bin/crystal-coverage                                                   fossil:(  next_state_perf )  ✔ 
Unhandled exception: Error executing process: '/bin/sh': Argument list too long (IO::Error)
  from /home/mia/.asdf/installs/crystal/1.4.1/share/crystal/src/crystal/system/unix/process.cr:225:7 in 'raise_exception_from_errno'
  from /home/mia/.asdf/installs/crystal/1.4.1/share/crystal/src/io.cr:1183:7 in 'run:shell:input:output:error'
  from /home/mia/.asdf/installs/crystal/1.4.1/share/crystal/src/process.cr:419:3 in '__crystal_main'
  from /home/mia/.asdf/installs/crystal/1.4.1/share/crystal/src/crystal/main.cr:115:5 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /usr/local/bin/crystal-coverage in '_start'
  from ???
➜ 10:01AM boundless-scuttlecoil

Smaller project with another error

➜ 10:11AM boundary-scuttlecoil cloc src spec                                                                                fossil:(  trunk )  ✔ 
       5 text files.
       5 unique files.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.00 s (1065.2 files/s, 231786.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Crystal                          5            146             68            874
-------------------------------------------------------------------------------
SUM:                             5            146             68            874
-------------------------------------------------------------------------------
➜ 10:11AM boundary-scuttlecoil /usr/local/bin/crystal-coverage                                                              fossil:(  trunk )  ✔ 
Showing last frame. Use --error-trace for full trace.

error in line 1
Error: can't find file 'coverage/runtime'

If you're trying to require a shard:
- Did you remember to run `shards install`?
- Did you make sure you're running the compiler in the same directory as your shard.yml?

small project with an error

➜ 10:16AM echo_ip cloc src spec                                                                                             fossil:(  trunk )  ✔ 
       1 text file.
       1 unique file.                              
       0 files ignored.

1 error:
Unable to read:  spec

github.com/AlDanial/cloc v 1.82  T=0.00 s (271.3 files/s, 34726.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Crystal                          1             25             13             90
-------------------------------------------------------------------------------
➜ 10:16AM echo_ip /usr/local/bin/crystal-coverage                                                                           fossil:(  trunk )  ✔ 
Showing last frame. Use --error-trace for full trace.

error in line 1
Error: can't find file 'coverage/runtime'

If you're trying to require a shard:
- Did you remember to run `shards install`?
- Did you make sure you're running the compiler in the same directory as your shard.yml?

chillfox · 2022-06-28T01:42:03Z

This now works

➜ 11:09AM portal-api (test/coverage) ✗ /usr/local/bin/crystal-coverage -p | less

And a new error

➜ 11:11AM portal-api (test/coverage) ✗ /usr/local/bin/crystal-coverage -p | crystal eval 
Showing last frame. Use --error-trace for full trace.

In lib/action-controller/spec/curl_context.cr:35:1

 35 | ::CURL_CONTEXT__ = [] of ActionController::Server
      ^---------------
Error: already initialized constant CURL_CONTEXT__

ci(containerised-test): add coverage

eb2f647

caspiano self-assigned this Nov 10, 2021

caspiano added focus: devops Focus on DevOps/Infra type: enhancement new feature or request labels Nov 16, 2021

jeremyw24 assigned stakach Dec 19, 2021

caspiano requested a review from chillfox March 16, 2022 00:18

Merge branch 'main' into ci/coverage

ad42ddb

test(coverage): increase timeout

cb05e88

chillfox and others added 2 commits March 16, 2022 14:15

Revert "test(coverage): increase timeout"

f253a5f

This reverts commit cb05e88.

fix(containerised-test): verbose kcov output

f7276e1

caspiano unassigned stakach Mar 16, 2022

caspiano assigned chillfox Apr 19, 2022

caspiano added the priority: medium label May 3, 2022

caspiano removed their assignment May 5, 2022

chillfox added 3 commits May 6, 2022 09:34

Merge branch 'main' into ci/coverage

7be49e7

ci: lets try and get normal kcov working first

eab1dc5

Revert "ci: lets try and get normal kcov working first"

e990e52

This reverts commit eab1dc5.

caspiano force-pushed the main branch from f7b572a to bd9fb1b Compare May 20, 2022 02:17

chillfox added 3 commits June 23, 2022 10:16

Merge branch 'main' into ci/coverage

61f549c

feat(coverage): add crystal-coverage to test container

db9a9c3

ci(coverage): upload report?

d2d7e68

refactor(docker): fix lint issues

138fd33

caspiano removed the request for review from chillfox June 27, 2022 06:52

caspiano commented Jun 27, 2022

View reviewed changes

.github/workflows/containerised-test.yml Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(containerised-test): add coverage #28

ci(containerised-test): add coverage #28

caspiano commented Nov 10, 2021

chillfox commented Mar 16, 2022

chillfox commented Mar 16, 2022

caspiano commented Mar 16, 2022

chillfox commented Mar 16, 2022

caspiano commented Mar 16, 2022

chillfox commented Mar 16, 2022

chillfox commented Mar 16, 2022

caspiano commented Mar 16, 2022

chillfox commented Mar 16, 2022

caspiano commented Mar 16, 2022

chillfox commented May 16, 2022

chillfox commented May 16, 2022

chillfox commented May 17, 2022

caspiano commented May 17, 2022

chillfox commented Jun 22, 2022

caspiano commented Jun 22, 2022

chillfox commented Jun 23, 2022

chillfox commented Jun 27, 2022 •

edited

Loading

caspiano commented Jun 27, 2022

chillfox commented Jun 27, 2022

chillfox commented Jun 27, 2022

chillfox commented Jun 28, 2022

chillfox commented Jun 28, 2022 •

edited

Loading

chillfox commented Jun 28, 2022

chillfox commented Jun 28, 2022

ci(containerised-test): add coverage #28

Are you sure you want to change the base?

ci(containerised-test): add coverage #28

Conversation

caspiano commented Nov 10, 2021

chillfox commented Mar 16, 2022

chillfox commented Mar 16, 2022

caspiano commented Mar 16, 2022

chillfox commented Mar 16, 2022

caspiano commented Mar 16, 2022

chillfox commented Mar 16, 2022

chillfox commented Mar 16, 2022

caspiano commented Mar 16, 2022

chillfox commented Mar 16, 2022

caspiano commented Mar 16, 2022

chillfox commented May 16, 2022

chillfox commented May 16, 2022

chillfox commented May 17, 2022

caspiano commented May 17, 2022

chillfox commented Jun 22, 2022

caspiano commented Jun 22, 2022

chillfox commented Jun 23, 2022

chillfox commented Jun 27, 2022 • edited Loading

caspiano commented Jun 27, 2022

chillfox commented Jun 27, 2022

chillfox commented Jun 27, 2022

chillfox commented Jun 28, 2022

chillfox commented Jun 28, 2022 • edited Loading

chillfox commented Jun 28, 2022

chillfox commented Jun 28, 2022

chillfox commented Jun 27, 2022 •

edited

Loading

chillfox commented Jun 28, 2022 •

edited

Loading