Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI/CD Workflow update #176

Merged
merged 113 commits into from
May 2, 2024
Merged

CI/CD Workflow update #176

merged 113 commits into from
May 2, 2024

Conversation

maxemiliang
Copy link
Collaborator

@maxemiliang maxemiliang commented Apr 20, 2024 •
edited
Loading

This PR updates the GHA workflow in a number of ways.

  1. Workflows are now run on CSC hosted runners, to allow for consistent performance.
  2. It builds & tests SHOT with both Ipopt/Cbc and proprietary solvers (currently GAMS and Gurobi).
  3. It uploads many of the build/test/configuration files as artifacts to allow for comparison between versions and for local inspection
  4. It runs benchmarks and compares the results to previous commits in the same branch (and same solvers).
  5. Benchmark results are output in a Markdown summary.
  6. It outputs the tests results in a Markdown summary.
    It also adds a way to manually dispatch the different workflows.

It also adds a cloud-init script to allow for smooth setup of the self-hosted runner, documentation for this will follow.

The benchmarker code can be found in this Repo

@maxemiliang
refactor(ci): Refactor the workflow to use self-hosted runners and up…
832824d 
…load the build files as artifacts.
@maxemiliang
fix(ci): Auto accept the dependency installation
d004c65
@maxemiliang
fix(ci): Update cache version and install pkgconfig package
d2b3df4
@maxemiliang
fix(ci): Temporarily remove the build artifact upload
9644bfc
@maxemiliang
refactor(ci): Refactor the build process to use reusable workflows an…
4b1fcbd 
…d readded the artifact storage.
@maxemiliang
fix(ci): Attempt to use the runner workspace as base for the artifact…
b96269b 
… outputs
@maxemiliang
fix(ci): Use correct input for the retention
f8780d6
@maxemiliang
fix(ci): Update checkout version and fix naming for one step
ee1720f
@maxemiliang
feat(ci): Added a step to publish the test results.
087496b
@maxemiliang
feat(ci): Added runner-init.yml script to initialize runners and upda…
cb8445e 
…ted the build script to use docker-runners
@maxemiliang
fix(ci): Fixed so the correct variables are used ✏️
dfdfcac
@maxemiliang
fix: Fixed permission issues with docker and documented the runner-in…
cf8d1e2 
…it.yml a bit better.
@maxemiliang
fix(ci): Attempt to use the correct output
721ba04
@maxemiliang
feat(ci): Allow for disabling artifact output and build args as inputs
cb1c0b4
@maxemiliang
chore(ci): Formatting for the workflow
fdc840a
@maxemiliang
feat(ci): Output test artifact name even if the other artifacts are n…
884d000 
…ot to be uploaded
@maxemiliang
refactor(ci): refactored runner.workspace -> github.workspace
50070ed
@maxemiliang
feat(ci): Added ability to specify proprietary build and added gurobi…
03af112 
… download and license file setup
@maxemiliang
feat(ci): Updated the reusable workflow to correctly get secrets
7a175fb
@maxemiliang
feat(ci): Actually compile with gurobi as a test
a2abadc
@maxemiliang
chore(ci): Updated the job count to match the current runner
c676c11
@maxemiliang
chore(ci): Removed debug from cleanup
0fff51f
@maxemiliang
feat(ci): Added GAMS installation steps and compile using it
1a5ea88
@maxemiliang
feat(ci): Added concurrency, some comments and ability to add suffix …
1e4ae73 
…to artifacts.
@maxemiliang
fix(ci): Fixed the publish test job to use correct outputs
ff5b695
@maxemiliang
fix(ci): Fixed GAMS installation as it always extracts using the same…
b1a7cfe 
… name
@maxemiliang
chore(ci): Formatting the workflow
5192d10
@maxemiliang
fix(ci): Use the correct format for the GAMS library
e55148e
@maxemiliang
feat(ci): Added GAMS path to the runner path
0837d1a
@maxemiliang
fix(ci): Correctly use the gurobi path
fa29e0d
@maxemiliang
chore: add gams/gurobi status to the benchmarker
de28256
@maxemiliang
refactor: Allow for different Gurobi and GAMS versions. Also test the…
8151310 
… new Benchmarker.
@maxemiliang
Merge remote-tracking branch 'origin/feature/ci-cd-improvements' into…
b3207a0 
… feature/ci-cd-improvements
@maxemiliang
feat: Re enable licensed build workflow
aa5a597
@maxemiliang
fix: Fixed an issue with getting the short version of GAMS and Gurobi…
013ed53 
… from the workflow
@maxemiliang
fix: forgot to add $ to variable
9b0dab0
@maxemiliang
fix: Use the correct way to get the version to build the paths
9c175f8
@maxemiliang
fix: bash variable expansion correctness
b37251e
@maxemiliang
fix: rollback to GAMS 42.3.0
947ab13
@maxemiliang
fix: moved shot binary to its own folder in the benchmark step to avo…
75200e5 
…id conflicts.
@maxemiliang
fix: More fixes to the benchmarker, to correctly handle permissions.
531d8c3
@maxemiliang
feat: cleanup workspace before benchmarking, to test if it helps with…
a2a0896 
… SHOT.
@maxemiliang
chore: Added linked libraries output to debug the build
1845a58
@maxemiliang
chore: Added linked libraries output to debug the build
49b8a1b
@maxemiliang
feat: Moved the benchmarking to be done directly after build, to corr…
9053d13 
…ectly used the linked binaries.
@maxemiliang
fix: No longer depend on the benchmarker.
a49fd1e
@maxemiliang
fix: always run benchmarks after build + some inputs to the benchmark…
ed3d84e 
…er and secrets.
@maxemiliang
fix: try to fix the usage of github workspace.
b278758
@maxemiliang
fix: Updated the upload-artifacts workflow to use v4
22447c9
@maxemiliang
fix: Fixed an issue with the benchmarker SHOT path
13d70fe
@maxemiliang
feat: Run proprietary license benchmarks on CI as well.
aafcc59
@maxemiliang
fix: Download artifacts using the correct version of actions/download…
8ccdd22 
…-artifact
@maxemiliang
fix: Correctly set the license file path + license file contents in b…
c6046fd 
…enchmarker
@maxemiliang
fix: Corrected the license file path in the build workflow
c6c01b7
@maxemiliang
fix: Artifact suffix added to benchmarker workflow
970dfe1
@maxemiliang
refactor: benchmarker no longer compares Gurobi/Gams to Ipopt/CBC
4129788
@maxemiliang maxemiliang self-assigned this Apr 20, 2024
@maxemiliang maxemiliang added the enhancement New feature or request label Apr 20, 2024
@andreaslundell andreaslundell merged commit bbe2682 into master May 2, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andreaslundell andreaslundell Awaiting requested review from andreaslundell

Assignees

@maxemiliang maxemiliang

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@maxemiliang @andreaslundell