Skip to content
This repository has been archived by the owner on Nov 20, 2024. It is now read-only.

Removed deployment job, refactored tests and updated CI image. #23

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 12 additions & 124 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -1,140 +1,28 @@
version: 2.1
parameters:
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleaned up CI as we are only generating content manually. once there is a better automated process - the config can be revert. otherwise - this is a dead code.

trigger-project:
type: string
default: ""
trigger-url:
type: string
default: ""
trigger-sha:
type: string
default: ""
trigger-description:
type: string
default: "A Satis build was triggered for an unknown reason."

jobs:
build:
test: &job-test
docker:
- image: composer/satis
environment:
SATIS: /satis/bin/satis
SATIS_BUILD: /tmp/satis-build/app
- image: docker.io/govcms/govcms-ci
steps:
- checkout
- run:
name: Build or prepare satis /app
command: |
# If satis build was automated, it would happen here. Currently just copy the committed /app to the build directory.
mkdir -p "${SATIS_BUILD}"
cp -R app/* "${SATIS_BUILD}"
- persist_to_workspace:
root: /tmp/satis-build
paths:
- .
- run: ./.circleci/test.sh

test:
docker:
- image: quay.io/govcms/govcms-ci
test_master:
<<: *job-test
environment:
SATIS_BUILD: /tmp/satis-build/app
GOVCMS_SCAFFOLD: /tmp/govcms-build
steps:
- checkout
- attach_workspace:
at: /tmp/satis-build
- run:
name: Test that GovCMS can be built
command: |
# Get package versions required for testing stable release.
VERSION_GOVCMS=$(cat ./satis-config/govcms-stable.json | jq -r '.require | .["govcms/govcms"]')
VERSION_TOOLING=$(cat ./satis-config/govcms-stable.json | jq -r '.require | .["govcms/scaffold-tooling"]')
VERSION_DEV=$(cat ./satis-config/govcms-stable.json | jq -r '.require | .["govcms/require-dev"]')

echo "--> Starting satis web server on http://localhost:4141"
php -S localhost:4141 -t "${SATIS_BUILD}" > /tmp/phpd.log 2>&1 &
echo "--> Cloning govcms8-scaffold-paas into ${GOVCMS_SCAFFOLD}"
composer create-project --no-install --quiet govcms/govcms8-scaffold-paas "${GOVCMS_SCAFFOLD}"
cd "${GOVCMS_SCAFFOLD}"
composer config secure-http false

for branch in {"","develop","master"}; do
echo
echo "--> --------------------------------------------------------"
echo "--> Test build GovCMS against http://localhost:4141/${branch}"
echo
SATIS_BRANCH: master

rm -fR vendor && rm -f composer.lock
composer config repositories.govcms composer http://localhost:4141/"${branch}"
if [ "${branch}" = "master" ] || [ "${branch}" = "develop" ] ; then
composer require --no-update \
govcms/govcms:1.x \
govcms/scaffold-tooling:dev-"${branch}" \
govcms/require-dev:dev-"${branch}" \
symfony/event-dispatcher:"v4.3.11 as v3.4.35" # @todo: remove once govcms/govcms no longer requires "symfony/event-dispatcher:v4.3.11 as v3.4.35" which only works at the root composer.json level.
else
echo -e "--> Expected stable versions, based on the satis config: \n - govcms/govcms:${VERSION_GOVCMS} \n - govcms/scaffold-tooling:${VERSION_TOOLING} \n - govcms/require-dev:${VERSION_DEV}"
composer require --no-update \
govcms/govcms:"${VERSION_GOVCMS}" \
govcms/scaffold-tooling:"${VERSION_TOOLING}" \
govcms/require-dev:"${VERSION_DEV}" \
symfony/event-dispatcher:"v4.3.11 as v3.4.35" # @todo: remove once govcms/govcms no longer requires "symfony/event-dispatcher:v4.3.11 as v3.4.35" which only works at the root composer.json level.
fi
cat composer.json | jq .require
composer -n --quiet --no-suggest --no-scripts update
composer info | grep ^govcms
done

deploy:
docker:
- image: quay.io/govcms/govcms-ci
test_develop:
<<: *job-test
environment:
SATIS_BUILD: /tmp/satis-build/app
steps:
- checkout
- attach_workspace:
at: /tmp/satis-build
- run:
name: Update github develop branch.
command: |
# If builds were automated, this would push the new satis back to git.
exit 1

# Currently just testing a push to a test branch.
git checkout -b test-"${CIRCLE_SHA1}"
rm -Rf app
cp -Rf "${SATIS_BUILD}" .
git add app
git config --global user.email "[email protected]"
git config --global user.name "GovCMS service account"
# Insecure proof of concept. Key removed between testing.
# @todo use ssh key fingerprint https://github.com/integratedexperts/drupal-dev/blob/8.x/.circleci/config.yml#L127
git remote set-url origin https://simesy:"${GITHUB_READ_WRITE_TOKEN}"@github.com/govCMS/satis.git
git commit -m"[skip ci] test-commit"
git push origin test-"${CIRCLE_SHA1}"
SATIS_BRANCH: develop


workflows:
version: 2.1
update:
jobs:
- build:
filters:
tags:
ignore: /.*/
- test:
filters:
tags:
ignore: /.*/
requires:
- build
- deploy:
filters:
tags:
ignore: /.*/
branches:
only:
- upstream_changes
- develop
requires:
- test
- test
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 parallel jobs allows to reduce build time to ~2 mins and make sure that there are no artifacts from the previous build (clean run)

- test_master
- test_develop
70 changes: 70 additions & 0 deletions .circleci/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
#!/usr/bin/env bash
##
## Test that GovCMS scaffold can be built using Satis.
##

# shellcheck disable=SC2002,SC2015

# Satis application directory.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as usual, the first section is expected variables

APP_DIR="${APP_DIR:-./app}"

# The branch of Satis to test against.
SATIS_BRANCH="${SATIS_BRANCH:-}"

# Directory where scaffold is installed into.
GOVCMS_SCAFFOLD_DIR="${GOVCMS_SCAFFOLD_DIR:-/tmp/govcms-build}"

#-------------------------------------------------------------------------------

echo "--> Starting Satis web server on http://localhost:4141."
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this part allows to kill previously started inbuilt php server from the previous run, start the server, wait for app to be ready and assess that the server was successfully started

killall -9 php > /dev/null 2>&1 || true
php -S localhost:4141 -t "${APP_DIR}" > /tmp/phpd.log 2>&1 &
sleep 4 # Waiting for the server to be ready.
netstat_opts='-tulpn'; [ "$(uname)" == "Darwin" ] && netstat_opts='-anv' || true;
netstat "${netstat_opts[@]}" | grep -q 4141 || (echo "ERROR: Unable to start inbuilt PHP server" && cat /tmp/php.log && exit 1)
curl -s -o /dev/null -w "%{http_code}" -L -I http://localhost:4141 | grep -q 200 || (echo "ERROR: Server is started, but site cannot be served" && exit 1)

echo "--> Cloning govcms8-scaffold-paas into ${GOVCMS_SCAFFOLD_DIR}."
rm -Rf "${GOVCMS_SCAFFOLD_DIR}"
composer create-project --no-install --quiet govcms/govcms8-scaffold-paas "${GOVCMS_SCAFFOLD_DIR}"

composer --working-dir="${GOVCMS_SCAFFOLD_DIR}" config secure-http false

echo
echo "--> Test build GovCMS against http://localhost:4141/${SATIS_BRANCH}."
echo

echo "--> Add http://localhost:4141/${SATIS_BRANCH} as a repository."
composer --working-dir="${GOVCMS_SCAFFOLD_DIR}" config repositories.govcms composer http://localhost:4141/"${SATIS_BRANCH}"

if [ "${SATIS_BRANCH}" = "master" ] || [ "${SATIS_BRANCH}" = "develop" ] ; then
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the logic below has not been changed. only verbose output added

php -d memory_limit=-1 "$(command -v composer)" --working-dir="${GOVCMS_SCAFFOLD_DIR}" require --no-update \
govcms/govcms:1.x \
govcms/scaffold-tooling:dev-"${SATIS_BRANCH}" \
govcms/require-dev:dev-"${SATIS_BRANCH}" \
symfony/event-dispatcher:"v4.3.11 as v3.4.35" # @todo: remove once govcms/govcms no longer requires "symfony/event-dispatcher:v4.3.11 as v3.4.35" which only works at the root composer.json level.
else
# Get package versions required for testing stable release.
version_govcms="$(cat "./satis-config/govcms-stable.json" | jq -r '.require | .["govcms/govcms"]')"
version_tooling="$(cat "./satis-config/govcms-stable.json" | jq -r '.require | .["govcms/scaffold-tooling"]')"
version_require_dev="$(cat "./satis-config/govcms-stable.json" | jq -r '.require | .["govcms/require-dev"]')"

echo "--> Expected stable versions, based on the Satis config:"
echo " - govcms/govcms: ${version_govcms}"
echo " - govcms/scaffold-tooling: ${version_tooling}"
echo " - govcms/require-dev: ${version_require_dev}"
php -d memory_limit=-1 "$(command -v composer)" --working-dir="${GOVCMS_SCAFFOLD_DIR}" require --no-update \
govcms/govcms:"${version_govcms}" \
govcms/scaffold-tooling:"${version_tooling}" \
govcms/require-dev:"${version_require_dev}" \
symfony/event-dispatcher:"v4.3.11 as v3.4.35" # @todo: remove once govcms/govcms no longer requires "symfony/event-dispatcher:v4.3.11 as v3.4.35" which only works at the root composer.json level.
fi

echo "--> Contents of composer.json after dependency resolution."
cat "${GOVCMS_SCAFFOLD_DIR}"/composer.json | jq .require

echo "--> Assert that Composer update works."
php -d memory_limit=-1 "$(command -v composer)" --working-dir="${GOVCMS_SCAFFOLD_DIR}" -n --quiet --no-suggest --no-scripts update

echo "--> Assert that govcms* dependencies present."
composer --working-dir="${GOVCMS_SCAFFOLD_DIR}" info | grep ^govcms
81 changes: 46 additions & 35 deletions docs/MAINTENANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,27 @@

## What is it

There are four satis servers which are built from four config files (in ./satis-config).
These can be used in the "repositories" section of composer.json.
There are four satis servers which are built from four config files (in
`./satis-config`).
These can be used in the "repositories" section of `composer.json`.

| | | |
| --- | --- | --- |
| `STABLE` | govcms-stable.json | https://satis.govcms.gov.au/ |
| `MASTER` | govcms-master.json | https://satis.govcms.gov.au/master |
| `DEVELOP` | govcms-develop.json | https://satis.govcms.gov.au/develop |
| `WHITELIST` | govcms-whitelist.json | https://satis.govcms.gov.au/whitelist |
| `STABLE` | `govcms-stable.json` | https://satis.govcms.gov.au/ |
| `MASTER` | `govcms-master.json` | https://satis.govcms.gov.au/master |
| `DEVELOP` | `govcms-develop.json` | https://satis.govcms.gov.au/develop |
| `WHITELIST` | `govcms-whitelist.json` | https://satis.govcms.gov.au/whitelist |

They are updated by running `ahoy build` which populates the `./app` directory
with static files.

## When to update
## Updating content

Currently, updating is manual. The job to automate it was completed but it got stuck waiting
for deployment tokens. In the meantime it has been found that manual intervention is usually needed.
Updating of the Satis repository content is a manual process that includes
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the only paragraph that was changed to explain what the process is.

The rest of the changes are formatting only.

running a set of command locally and committing regenerated content directly
to the repository.

Updating `STABLE` should be done when there is a new point release of GovCMS.
Updating `STABLE` should be done when there is a new minor release of GovCMS.

Updating `MASTER` and `DEVELOP` should be done when the `master` and `develop`
branches are updated on
Expand All @@ -31,18 +33,20 @@ branches are updated on

## Running the automated scripts

Each of the `ahoy build` `ahoy check-dupes` `ahoy verify` and `ahoy debug` commands can
accept any of the current versions (stable, master or develop) as a parameter. Providing no
parameter is equivalent to `stable`. There is also a `-all` variant to these commands that
will run the process for all branches.
Each of the `ahoy build`, `ahoy check-dupes`, `ahoy verify` and `ahoy debug`
commands can accept any of the current versions (`stable`, `master` or
`develop`) as a parameter. Providing no parameter is equivalent to `stable`.
There is also a `-all` variant to these commands that will run the process for
all branches.

## Update MASTER

This will update the `./app/master` directory.

1. Clean `./satis-config/govcms-master.json` by removing:

* extra packages from `require` - only leave the first three `govcms/*` packages
* extra packages from `require` - only leave the first three `govcms/*`
packages
* the `blacklist` section - remove completely

2. Run `ahoy build master` to update /app.
Expand All @@ -51,16 +55,18 @@ This will update the `./app/master` directory.

4. Run `ahoy debug master` - follow instructions.

5. Re-run `ahoy build master` and `ahoy verify master` (ie. repeat the above steps as needed).
5. Re-run `ahoy build master` and `ahoy verify master` (ie. repeat the above
steps as needed).

## Update DEVELOP

This will update the `./app/develop` directory. Repeat the exact steps you followed to
update MASTER, just replace `master` with `develop`.
This will update the `./app/develop` directory. Repeat the exact steps you
followed to update MASTER, just replace `master` with `develop`.

1. Clean `./satis-config/govcms-develop.json` by removing:

* extra packages from `require` - only leave the first three `govcms/*` packages
* extra packages from `require` - only leave the first three `govcms/*`
packages
* the `blacklist` section - remove completely

2. Run `ahoy build develop` to update /app.
Expand All @@ -69,53 +75,58 @@ update MASTER, just replace `master` with `develop`.

4. Run `ahoy debug develop` - follow instructions.

5. Re-run `ahoy build develop` and `ahoy verify develop` (ie. repeat the above steps as needed).
5. Re-run `ahoy build develop` and `ahoy verify develop` (ie. repeat the above
steps as needed).

## Update STABLE

This will update the `./app` directory. You are repeating the steps you followed to
update MASTER, just replace `master` with `stable`.
This will update the `./app` directory. You are repeating the steps you followed
to update MASTER, just replace `master` with `stable`.

There is only one extra step (step 2).


1. Clean `./satis-config/govcms-stable.json` by removing:

* extra packages from `require` - only leave the first three `govcms/*` packages
* extra packages from `require` - only leave the first three `govcms/*`
packages
* the `blacklist` section - remove completely

2. *ONLY FOR STABLE* update the package versions for the `govcms/*` versions to the latest versions.
2. *ONLY FOR STABLE* update the package versions for the `govcms/*` versions
to the latest versions.

3. Run `ahoy build stable` to update /app.

4. Run `ahoy verify stable` - it will likely fail.

5. Run `ahoy debug stable` - follow instructions.

6. Re-run `ahoy build stable` and `ahoy verify stable` (ie. repeat the above steps as needed).
6. Re-run `ahoy build stable` and `ahoy verify stable` (ie. repeat the above
steps as needed).

## Steps to update WHITELIST

This is a hassle free one because it doesn't calculate dependencies. Run `ahoy build-whitelist`.
This is a hassle free one because it doesn't calculate dependencies.
Run `ahoy build-whitelist`.

## Additional troubleshooting steps

To quickly see if there are any duplicates in your package.json files, `ahoy check-dupes { stable|master|develop }`.

To quickly see if there are any duplicates in your package.json files,
`ahoy check-dupes { stable|master|develop }`.

## Push

Once you have updated all branches, create a PR to https://github.com/govcms/satis. Once this is merged it will trigger
quay.io to rebuild the an image (see docker-compose.yml).
Once you have updated all branches, create a PR to https://github.com/govcms/satis.

Once this is merged it will trigger
quay.io to rebuild an image (see `docker-compose.yml`).

## Technical notes

This project is built on [composer/satis](https://github.com/composer/satis) using the
docker container method. The following settings are important because it forces Satis
to resolve an *ideal* set of package versions.
This project is built on top of [composer/satis](https://github.com/composer/satis)
using the Docker image. The following settings are important because it forces
Satis to resolve an *ideal* set of package versions.

```
"require-dependencies": true,
"require-dev-dependencies": true,
"only-best-candidates": true
```
Loading