Fix GHA macOS to use macos-10.15 #132

ronaldtse · 2021-12-07T04:58:02Z

Metanorma PR checklist

Breaking changes (list related PRs)
Documentation update required (create task for this)
External dependency introduced (documentation update need)
Gem with native library introduced

GHA has just (today) switched the default macos-latest to macos-11, causing failures here: https://github.com/metanorma/packed-mn/runs/4439042422?check_suite_focus=true

macOS-latest workflows will use macOS-11 actions/runner-images#4060

So let's first fix to the older version and address the upgrade.

CAMOBAP · 2021-12-07T21:16:22Z

Well, let me summarize, during 1.5.3 release we faced a couple new issues:

SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError)
Timeout issue for some reason some job (independently on the flavor) takes > 6h
multi_json-1.15.0/lib/multi_json.rb:157: [BUG] Segmentation fault at 0x0000000000000000

1. OpenSSL::SSL::SSLError

We have tried to update embedded certificates, no success. Maybe we need rollback #109

2. Timeout > 6h

I have no idea yet. I had assumptions about some penalties form GHA, so I reduced max-parallel to 6 https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idstrategymax-parallel. Also no success

I have tried to run those flavors locally, no huge delays

3. `multi_json-1.15.0/lib/multi_json.rb:157: [BUG] Segmentation fault at 0x0000000000000000`

Absolutely no idea...

I need to sleep with it

ronaldtse · 2021-12-08T00:59:45Z

#109 is absolutely necessary. Please do not revert it.

I think GHA just has issues with large parallel matrix builds because of throttling reasons per org. Maybe can split the tests into multiple stages? Also we need to cancel old tests.

Look at this: apache/pulsar#9154

ronaldtse · 2021-12-08T01:11:44Z

Also note that GitHub Actions has concurrency limits, especially for macOS:

We should really limit the tests because this also starves our other repos.

ronaldtse · 2021-12-08T03:42:03Z

Builds failing due to installation of Lato font. I've personally experienced this as well, the official site often ends with a timeout.

Invalid URL: https://www.latofonts.com/download/lato2ofl-zip/. Error: #<Down::TimeoutError: timed out while reading data>.
34
C:/Users/RUNNER~1/AppData/Local/Temp/ocr5CE8.tmp/lib/ruby/gems/3.0.0/gems/fontist-1.13.0/lib/fontist/font_installer.rb:68:in `download_file': Invalid URL: https://www.latofonts.com/download/lato2ofl-zip/. Error: #<Down::TimeoutError: timed out while reading data>. (Fontist::Errors::InvalidResourceError)
35
	from C:/Users/RUNNER~1/AppData/Local/Temp/ocr5CE8.tmp/lib/ruby/gems/3.0.0/gems/fontist-1.13.0/lib/fontist/font_installer.rb:41:in `extract'
36
	from C:/Users/RUNNER~1/AppData/Local/Temp/ocr5CE8.tmp/lib/ruby/gems/3.0.0/gems/fontist-1.13.0/lib/fontist/font_installer.rb:24:in `block in install_font'
37
	from C:/Users/RUNNER~1/AppData/Local/Temp/ocr5CE8.tmp/lib/ruby/gems/3.0.0/gems/fontist-1.13.0/lib/fontist/font_installer.rb:32:in `block in run_in_temp_dir'
38

ronaldtse · 2021-12-08T04:20:25Z

Lato font fixed here: fontist/formulas#113

ronaldtse · 2021-12-08T04:22:24Z

Actions restarted.

CAMOBAP · 2021-12-08T07:44:39Z

@ronaldtse I have temporally disabled using mozilla's CA certs, because I don't remember a reason why do we use it

ronaldtse · 2021-12-08T07:46:12Z

@CAMOBAP because the cert that comes with Ruby was outdated when using the old ruby-packer.

We have since fixed this issue here: https://github.com/metanorma/ruby-packer/blob/master/.patches/ruby/0004-ssl-certs.patch

So yes, we can disable the packed-mn override on SSL certs.

CAMOBAP · 2021-12-08T07:47:48Z

@ronaldtse are you sure there ruby-packer/ruby don't use a system ones?

ronaldtse · 2021-12-08T07:48:14Z

@ronaldtse are you sure there ruby-packer/ruby don't use a system ones?

Yes. Old ruby-packer's Ruby does not use system SSL certificates.

ronaldtse · 2021-12-08T07:50:35Z

@CAMOBAP status now:

Windows build OGC is only failing on GHA artifact upload so technically it passed all tests. (https://github.com/metanorma/packed-mn/actions/runs/1552063409)
macOS build OGC failed due to timeout.
Linux (https://github.com/metanorma/packed-mn/runs/4453337573?check_suite_focus=true):
- OGC failed on W3C OpenSSL issue:

#<Thread:0x000000001f0b1868@/__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-1.9.5/lib/relaton/workers_pool.rb:10 run> terminated with exception (report_on_exception is true):
544
/__ruby_packer_memfs__/lib/ruby/2.6.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError)
545
	from /__ruby_packer_memfs__/lib/ruby/2.6.0/net/protocol.rb:44:in `ssl_socket_connect'
546
	from /__ruby_packer_memfs__/lib/ruby/2.6.0/net/http.rb:996:in `connect'
547
	from /__ruby_packer_memfs__/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
548
	from /__ruby_packer_memfs__/lib/ruby/2.6.0/net/http.rb:919:in `start'
549
	from /__ruby_packer_memfs__/lib/ruby/2.6.0/net/http.rb:605:in `start'
550
	from /__ruby_packer_memfs__/lib/ruby/2.6.0/net/http.rb:481:in `get_response'
551
	from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-w3c-1.9.4/lib/relaton_w3c/w3c_bibliography.rb:17:in `search'
552
	from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/relaton-w3c-1.9.4/lib/relaton_w3c/w3c_bibliography.rb:36:in `get'
553

IHO failed on timeout

ronaldtse · 2021-12-08T07:54:37Z

@CAMOBAP my speculation is that we see all these SSL validation failures not because of the SSL certificates -- it may be because:

Relaton added parallel fetch with threads, and that ruby-packer's libsquashfs "read access" is not thread safe:
Reading from memfs file is not thread safe pmq20/ruby-packer#159

@andrew2net is there an easy way to turn off parallel fetches for Relaton?

This reverts commit 8979275.

This reverts commit 7de83a2.

ronaldtse · 2021-12-08T10:28:56Z

7de83a2 didn't work:

https://github.com/metanorma/packed-mn/runs/4455739051?check_suite_focus=true

2021-12-08T10:21:54.1888360Z Downloading font "source" from https://github.com/fontist/source-fonts/releases/download/v1.0/source-fonts-1.0.zip
2021-12-08T10:21:54.1889520Z /__ruby_packer_memfs__/local/metanorma:185: warning: class variable access from toplevel
2021-12-08T10:21:54.1969630Z /__ruby_packer_memfs__/local/metanorma:185: warning: class variable access from toplevel
2021-12-08T10:21:54.2046130Z /__ruby_packer_memfs__/local/metanorma:185: warning: class variable access from toplevel
2021-12-08T10:21:54.2108360Z Invalid URL: https://github.com/fontist/source-fonts/releases/download/v1.0/source-fonts-1.0.zip. Error: #<Down::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)>.
2021-12-08T10:21:54.2147170Z /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/fontist-1.13.0/lib/fontist/font_installer.rb:68:in `download_file': Invalid URL: https://github.com/fontist/source-fonts/releases/download/v1.0/source-fonts-1.0.zip. Error: #<Down::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)>. (Fontist::Errors::InvalidResourceError)
2021-12-08T10:21:54.2150100Z 	from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/fontist-1.13.0/lib/fontist/font_installer.rb:41:in `extract'
2021-12-08T10:21:54.2152330Z 	from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/fontist-1.13.0/lib/fontist/font_installer.rb:24:in `block in install_font'
2021-12-08T10:21:54.2154070Z 	from /__ruby_packer_memfs__/local/vendor/bundle/ruby/2.6.0/gems/fontist-1.13.0/lib/fontist/font_installer.rb:32:in `block in run_in_temp_dir'

ronaldtse · 2021-12-08T14:12:19Z

Finally we have all passing except:

Timeout at linux OGC (https://github.com/metanorma/packed-mn/runs/4457359092?check_suite_focus=true)
GitHub data services intermittently down (Build failure due to download of source-fonts.zip (503 Egress Is Over The Account Limit) #134) (https://github.com/metanorma/packed-mn/runs/4457358397?check_suite_focus=true)
Failure to upload GHA artefacts (Build failure due to unable to upload artifact #135) (https://github.com/metanorma/packed-mn/runs/4456932527?check_suite_focus=true)

I'm going to relax Linux compiles to 45 minutes timeout, and merge this.

ronaldtse and others added 7 commits December 7, 2021 12:01

Lock macOS version to 10.15 for now

655a640

Add ocra/metanorma.ico target to Makefile

61dfd28

Ignore dynamically fetched pem file

aef175f

Try max-parallel restriction to avoid 6h hung

ded3be8

Drop macos-10.15 tests

81927bb

Fix generation of ocra/metanorma.ico

37589de

Do not build random branches

10bcae2

Add 30min timeout to tests and only run 3 jobs in parallel

a5728c2

Avoid using embeded certs

8979275

ronaldtse added 3 commits December 8, 2021 17:47

Revert "Avoid using embeded certs"

9130848

This reverts commit 8979275.

Try using cert_store

7de83a2

Revert "Try using cert_store"

e7f909e

This reverts commit 7de83a2.

Limit Relaton to only fetch 1 at a time

09ffacf

ronaldtse force-pushed the rt-fix-macos-1015 branch from a719af6 to 09ffacf Compare December 8, 2021 11:53

ronaldtse added 2 commits December 8, 2021 22:12

Relax Linux builds to timeout of 45 minutes since it is really necessary

f75918c

Add make all target

2c3ce97

ronaldtse merged commit c5f11e1 into main Dec 8, 2021

ronaldtse deleted the rt-fix-macos-1015 branch December 8, 2021 14:13

ronaldtse mentioned this pull request Dec 8, 2021

Compilation tests running too long #130

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix GHA macOS to use macos-10.15 #132

Fix GHA macOS to use macos-10.15 #132

ronaldtse commented Dec 7, 2021 •

edited

Loading

CAMOBAP commented Dec 7, 2021 •

edited

Loading

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

CAMOBAP commented Dec 8, 2021

ronaldtse commented Dec 8, 2021 •

edited

Loading

CAMOBAP commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

Fix GHA macOS to use macos-10.15 #132

Fix GHA macOS to use macos-10.15 #132

Conversation

ronaldtse commented Dec 7, 2021 • edited Loading

Metanorma PR checklist

CAMOBAP commented Dec 7, 2021 • edited Loading

1. OpenSSL::SSL::SSLError

2. Timeout > 6h

3. multi_json-1.15.0/lib/multi_json.rb:157: [BUG] Segmentation fault at 0x0000000000000000

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

CAMOBAP commented Dec 8, 2021

ronaldtse commented Dec 8, 2021 • edited Loading

CAMOBAP commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 8, 2021

ronaldtse commented Dec 7, 2021 •

edited

Loading

CAMOBAP commented Dec 7, 2021 •

edited

Loading

3. `multi_json-1.15.0/lib/multi_json.rb:157: [BUG] Segmentation fault at 0x0000000000000000`

ronaldtse commented Dec 8, 2021 •

edited

Loading