Setup CI for release #1052

meain · 2022-10-05T13:41:28Z

Description

This takes care of all the CI related to #69 and #571.

Unifies CI and drop all others
Add a CHANGELOG file based on keepachangelog.com/
Remove generated docs from repo
Generated cli docs within the repo for lint
Add releases based on goreleaser
Build/Push docker images bassed on docker action
Build/Push docs to s3 buckets

TODO

Replace release-ci with main
Update reference in file change checker from release-ci to main

Post merge TODO

Validate release build
Validate manual trigger
Validate push to prod docs bucket
Update base_url for docs
Validate docker image tag in release
Validate goreleaser build on tag

%%{
  init: {
    "theme": "dark"
  }
}%%

flowchart LR
START --> gen-cli-docs
START --> lint-code
START --> test-suite

gen-cli-docs --> lint-docs --> lds{state} -->|fail| FAIL
lds -->|success| s
lint-code --> lcs{state} -->|fail| FAIL
lcs -->|success| s(((.)))
test-suite --> tss{state} -->|fail| FAIL
tss -->|success| s

s --> pd(publish-docs) --> pdst{state} -->|fail| FAIL
s --> pb(publish-binary) --> pbst{state} -->|fail| FAIL
s --> pi(publish-image) --> pist{state} -->|fail| FAIL

pdst -->|success| pds{env} -->|test| push-to-test-s3
pds -->|prod| push-to-prod-s3

pbst -->|success| pbs{env} -->|test| push-to-run-artifacts
pbs -->|prod| push-to-release-artifats

pist -->|success| pis{env} -->|test| push-sha-tag
pis -->|prod| push-git_tag-tag

Type of change

Issue(s)

Test Plan

💪 Manual
⚡ Unit test
💚 E2E

meain

This is pretty much complete, let me know if there is any other thoughts/suggestions.

meain · 2022-10-05T13:43:58Z

.github/workflows/ci.yml

-  pull_request:
-    branches: [main]
+    # TODO(meain): update references to release-ci before merge
+    branches: [ release-ci ]


Keeping it as release-ci so that the jobs will be run for "dry run". Once the changes look good, we can change this to main before merge.

meain · 2022-10-05T13:44:38Z

.github/workflows/ci.yml

-  precheck:
-    uses: alcionai/corso/.github/workflows/_filechange_checker.yml@main
+  Precheck:
+    uses: alcionai/corso/.github/workflows/_filechange_checker.yml@release-ci # TODO(meain): post-merge of docs-ci: change this to main


We cannot reference an action without a tag, will change this to main as well before we do a merge. Keeping it like this so that the run will be successful.

meain · 2022-10-05T13:45:13Z

.github/workflows/ci.yml

        uses: ./.github/actions/go-setup-cache
+        if: startsWith(github.ref, 'refs/tags/') || github.ref == 'refs/heads/release-ci' || needs.precheck.outputs.docfileschanged == 'true'


startsWith(github.ref, 'refs/tags/') will be true for releases
github.ref == 'refs/heads/main will be true for commits to main

The fully-formed ref of the branch or tag that triggered the workflow run. For workflows triggered by push, this is the branch or tag ref that was pushed. For workflows triggered by pull_request, this is the pull request merge branch. For workflows triggered by release, this is the release tag created. For other triggers, this is the branch or tag ref that triggered the workflow run. This is only set if a branch or tag is available for the event type. The ref given is fully-formed, meaning that for branches the format is refs/heads/<branch_name>, for pull requests it is refs/pull/<pr_number>/merge, and for tags it is refs/tags/<tag_name>. For example, refs/heads/feature-branch-1.

From docs: https://docs.github.com/en/actions/learn-github-actions/environment-variables#default-environment-variables

meain · 2022-10-05T13:46:12Z

.github/workflows/ci.yml

+          mv ./src/cmd/mdgen/cli_markdown/* ./docs/docs/cli/
+          rm -R ./src/cmd/mdgen/cli_markdown/
+
+      - uses: actions/upload-artifact@master


This is done as we need this data in the publish-docs job.

meain · 2022-10-05T13:46:48Z

CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog


Add a CHANGELOG as discussed in #571 (comment)

meain · 2022-10-05T13:47:12Z

docs/docusaurus.config.js

@@ -9,7 +9,7 @@ const config = {
  title: 'Corso Documentation',
  tagline: 'Free, Secure, and Open-Source Backup for Microsoft 365',
  url: 'https://corsobackup.io',
-  baseUrl: '/',
+  baseUrl: process.env.CORSO_DOCS_BASEURL || '/',


Since we are currently deploy to /preview this is used to set that.

meain · 2022-10-05T13:47:49Z

src/.goreleaser.yaml

+    goos:
+      - linux
+      - windows
+      - darwin
+    goarch: # other options: 386, arm
+      - amd64
+      - arm64


All the different binary versions that we are releasing now.

Let's also add an ignore section so we don't build: windows-arm64 and darwin-amd64

Any reason why we are not building darwin-amd64? All except the latest m1/m2 based macs will be using amd64 if I am not mistaken.

@vkamra Did you mean darwin-arm64?

Even with darwin-arm64, I think it would be a good idea to keep it as that will make sure we can run natively on m1/m2 macs.

I would agree.

I was actually thinking we only build for M1+. If we see asks for others we can add.

I feel like more than 60% developers at least will be using amd64 based machines, but if you think we should drop it, I'll add an ignore entry. The cost for use to include it would be about extra ~15m in the CI.

Let's go with yours and Niraj's recommendation and keep both for darwin.

vkamra · 2022-10-05T21:50:03Z

@meain - this looks right. I wanted to look at the github actions running but they seem to be stuck.

meain · 2022-10-06T03:02:23Z

Not sure what happened there with GH Actions. I just force pushed a timestamp change to trigger a rerun. The total run takes >1.5 hours though, mainly goreleaser taking time to build all the binaries but I guess this is to be expected as we are building {linux,mac,window}*{amd64,arm64} (6 builds) and each build on my machine with 12 cores was taking ~4min.

That said, in most cases we will just be running lints which takes pretty much the same time as before, ie ~30min.
https://github.com/alcionai/corso/actions/runs/3194196558

meain · 2022-10-06T03:38:06Z

The docs will be pushed to https://d2jrnhxpb9sco2.cloudfront.net/preview/index.html

meain · 2022-10-07T10:36:23Z

This should now be ready unless anyone has any comments. Once a review is complete, I'll make the changes mentioned in TODO section in the first comment as that will be needed for this to work once merged. Also there are a few items in Post merge TODO in the first comments which I won't be able to test out until merged.

sonarqubecloud · 2022-10-10T05:30:18Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

* Setup CI for release * Initial commit of CHANGELOG.md * Pin vale and markdownlint-cli to current versions * Update branch name for CI * Pull request workflow job

## Description This sets up the metrics configs using `ldflags`. Keeping it in draft as I wanted to wait till #1052 is merged as that will affect how we use it in the CI. I have currently set the base branch to `release-ci` on GH, the branch for #1052 instead of `main` as the diff would make more sense that way. ## Type of change  - [ ] 🌻 Feature - [ ] 🐛 Bugfix - [ ] 🗺️ Documentation - [ ] 🤖 Test - [x] 💻 CI/Deployment - [ ] 🐹 Trivial/Minor ## Issue(s)  * #1067 ## Test Plan  - [x] 💪 Manual - [ ] ⚡ Unit test - [ ] 💚 E2E

meain requested review from ntolia, vkamra, gmatev and ryanfkeepers October 5, 2022 13:41

meain temporarily deployed to Testing October 5, 2022 13:41 Inactive

meain mentioned this pull request Oct 5, 2022

Add CI for linting docs #998

Closed

9 tasks

meain commented Oct 5, 2022

View reviewed changes

meain temporarily deployed to Testing October 5, 2022 13:53 Inactive

meain had a problem deploying to Testing October 5, 2022 14:11 Failure

meain temporarily deployed to Testing October 5, 2022 14:11 Inactive

meain force-pushed the release-ci branch from 9ab5c88 to 9a6a13a Compare October 6, 2022 02:59

meain temporarily deployed to Testing October 6, 2022 02:59 Inactive

meain temporarily deployed to Testing October 6, 2022 03:00 Inactive

meain temporarily deployed to Testing October 6, 2022 03:11 Inactive

meain temporarily deployed to Testing October 6, 2022 03:27 Inactive

meain had a problem deploying to Testing October 6, 2022 03:27 Failure

meain temporarily deployed to Testing October 6, 2022 04:59 Inactive

meain temporarily deployed to Testing October 7, 2022 04:49 Inactive

meain force-pushed the release-ci branch from 1ae0698 to b3791c8 Compare October 7, 2022 08:21

meain temporarily deployed to Testing October 7, 2022 08:21 Inactive

meain temporarily deployed to Testing October 7, 2022 08:22 Inactive

meain temporarily deployed to Testing October 7, 2022 08:23 Inactive

meain temporarily deployed to Testing October 7, 2022 08:37 Inactive

meain mentioned this pull request Oct 7, 2022

Set Metrics configuration using ldlfags #1088

Merged

9 tasks

vkamra approved these changes Oct 10, 2022

View reviewed changes

meain added 4 commits October 10, 2022 10:42

Setup CI for release

79bc4d1

Initial commit of CHANGELOG.md

4d10eae

Pin vale and markdownlint-cli to current versions

7472b22

Update branch name for CI

8a770c1

meain force-pushed the release-ci branch from b3791c8 to 8a770c1 Compare October 10, 2022 05:12

meain marked this pull request as ready for review October 10, 2022 05:22

Pull request workflow job

564130e

meain temporarily deployed to Testing October 10, 2022 05:29 Inactive

meain merged commit 1793cc3 into main Oct 10, 2022

meain deleted the release-ci branch October 10, 2022 05:30

meain mentioned this pull request Oct 10, 2022

Fix issue with docs lint #1102

Merged

9 tasks

ryanfkeepers pushed a commit that referenced this pull request Oct 10, 2022

Setup CI for release (#1052)

999c752

* Setup CI for release * Initial commit of CHANGELOG.md * Pin vale and markdownlint-cli to current versions * Update branch name for CI * Pull request workflow job

ryanfkeepers mentioned this pull request Oct 12, 2022

fix docs buildimage bug due to inline comments #1154

Merged

2 tasks

gmatev mentioned this pull request Oct 12, 2022

Revert changes that broke docs Dockerfile built #1155

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup CI for release #1052

Setup CI for release #1052

meain commented Oct 5, 2022 •

edited

Loading

meain left a comment

meain Oct 5, 2022

meain Oct 5, 2022

meain Oct 5, 2022

meain Oct 6, 2022

meain Oct 5, 2022

meain Oct 5, 2022

meain Oct 5, 2022

meain Oct 5, 2022

vkamra Oct 6, 2022

meain Oct 6, 2022 •

edited

Loading

ntolia Oct 6, 2022

meain Oct 6, 2022

ntolia Oct 6, 2022

vkamra Oct 6, 2022

meain Oct 6, 2022

vkamra Oct 6, 2022

vkamra commented Oct 5, 2022

meain commented Oct 6, 2022 •

edited

Loading

meain commented Oct 6, 2022

meain commented Oct 7, 2022

sonarqubecloud bot commented Oct 10, 2022

		uses: ./.github/actions/go-setup-cache
		if: startsWith(github.ref, 'refs/tags/') \|\| github.ref == 'refs/heads/release-ci' \|\| needs.precheck.outputs.docfileschanged == 'true'

Setup CI for release #1052

Setup CI for release #1052

Conversation

meain commented Oct 5, 2022 • edited Loading

Description

Type of change

Issue(s)

Test Plan

meain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meain Oct 6, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vkamra commented Oct 5, 2022

meain commented Oct 6, 2022 • edited Loading

meain commented Oct 6, 2022

meain commented Oct 7, 2022

sonarqubecloud bot commented Oct 10, 2022

meain commented Oct 5, 2022 •

edited

Loading

meain Oct 6, 2022 •

edited

Loading

meain commented Oct 6, 2022 •

edited

Loading