Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace container build steps with reusable workflow #732

Merged
merged 38 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
12bd06b
refactor: replace container build steps with reusable workflow
dtrai2 Dec 17, 2024
bfb984c
use actual lists for tags
dtrai2 Dec 17, 2024
10db32b
use actual string for tags
dtrai2 Dec 17, 2024
273ef5e
test for python
dtrai2 Dec 17, 2024
7a6958b
fix type
dtrai2 Dec 17, 2024
a69c877
fix type
dtrai2 Dec 17, 2024
74b9b45
fix type
dtrai2 Dec 17, 2024
30eaae0
remove image.name
dtrai2 Dec 17, 2024
cccdb0f
debug image tag
dtrai2 Dec 17, 2024
71e5158
debug image tag
dtrai2 Dec 17, 2024
34f7b84
debug image tag with two tags
dtrai2 Dec 17, 2024
0b33ef8
debug tag print
dtrai2 Dec 17, 2024
28788c0
fix space and debug print
dtrai2 Dec 17, 2024
5109db2
fix strip
dtrai2 Dec 17, 2024
7ca550f
debug tags
dtrai2 Dec 17, 2024
a60b3bd
fix tags
dtrai2 Dec 17, 2024
f319246
debug tags
dtrai2 Dec 17, 2024
8232941
debug tags
dtrai2 Dec 17, 2024
aadd57c
clean up create image tag
dtrai2 Dec 17, 2024
9793709
fix inputs
dtrai2 Dec 17, 2024
2328e0f
fix output
dtrai2 Dec 17, 2024
1a87060
fix github output
dtrai2 Dec 17, 2024
9928104
fix tag separation
dtrai2 Dec 17, 2024
0e6aa68
fix write to output
dtrai2 Dec 17, 2024
7887822
use comma separation
dtrai2 Dec 17, 2024
b99dc5c
debug docker logprep
dtrai2 Dec 17, 2024
4978593
remove needs
dtrai2 Dec 17, 2024
f3fa2e2
print image digests
dtrai2 Dec 17, 2024
4e84b09
print image digests
dtrai2 Dec 17, 2024
9202a3a
debug docker run
dtrai2 Dec 17, 2024
22195d7
tryout local tag name
dtrai2 Dec 17, 2024
6cd0bc7
replace all image refs with local tag
dtrai2 Dec 17, 2024
b46fe94
debug pipeline
dtrai2 Dec 17, 2024
caac344
fix f-string
dtrai2 Dec 17, 2024
29eb8b0
fix digest expression
dtrai2 Dec 17, 2024
6aaee06
fix sbom filename
dtrai2 Dec 17, 2024
3e8f149
set secrets inherit
dtrai2 Dec 17, 2024
f80d627
fix secrets inherit
dtrai2 Dec 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 5 additions & 94 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,97 +94,8 @@ jobs:
uses: codecov/codecov-action@v2

containerbuild:
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12"]

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build image and export to Docker
uses: docker/build-push-action@v6
with:
context: .
load: true
build-args: |
LOGPREP_VERSION=dev
PYTHON_VERSION=${{ matrix.python-version }}
tags: |
ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}

- name: Ensure logprep is available in image
run: |
docker run --rm ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }} --version

# This step will build the image again, but every layer will already be cached, so it is nearly instantaneous.
- name: Push image
uses: docker/build-push-action@v6
id: build-and-push
with:
context: .
push: true
build-args: |
LOGPREP_VERSION=dev
PYTHON_VERSION=${{ matrix.python-version }}
tags: |
ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}

- name: Install Cosign
uses: sigstore/[email protected]
with:
cosign-release: 'v2.4.1'

- name: Create SBOM of container image
uses: anchore/sbom-action@v0
with:
image: ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}@${{ steps.build-and-push.outputs.digest }}
artifact-name: py${{ matrix.python-version }}-${{ github.head_ref }}.spdx.json
output-file: py${{ matrix.python-version }}-${{ github.head_ref }}.spdx.json

- name: Sign image with a key and add sbom attestation
run: |
cosign sign --yes --key env://COSIGN_PRIVATE_KEY ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}@${DIGEST}
cosign attest --yes --key env://COSIGN_PRIVATE_KEY --predicate py${{ matrix.python-version }}-${{ github.head_ref }}.spdx.json ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}@${DIGEST}
env:
COSIGN_PRIVATE_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }}
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
DIGEST: ${{ steps.build-and-push.outputs.digest }}

# To avoid the trivy-db becoming outdated, we save the cache for one day
- name: Get date
id: date
run: echo "date=$(date +%Y-%m-%d)" >> $GITHUB_OUTPUT

- name: Restore trivy cache
uses: actions/cache@v4
with:
path: cache/db
key: trivy-cache-${{ steps.date.outputs.date }}
restore-keys:
trivy-cache-

- name: Scan image using Trivy
uses: aquasecurity/[email protected]
env:
TRIVY_CACHE_DIR: ./cache
with:
scan-type: image
image-ref: ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}
trivy-config: trivy.yaml

# Trivy-db uses `0600` permissions.
# But `action/cache` use `runner` user by default
# So we need to change the permissions before caching the database.
- name: Change permissions for trivy.db
run: sudo chmod 0644 ./cache/db/trivy.db
uses: ./.github/workflows/container-build.yml
secrets: inherit
with:
build-version: dev
tags: "${{ github.head_ref }}"
129 changes: 129 additions & 0 deletions .github/workflows/container-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
name: Reusable Container Build

on:
workflow_call:
inputs:
build-version:
description: "Version of Logprep to build"
required: true
type: string
tags:
description: "Tags to apply to the image"
required: true
type: string

jobs:
containerbuild:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [ "3.10", "3.11", "3.12" ]

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Generate Image Tags
id: image-tag
run: |
python - <<EOF
import os
image = "ghcr.io/fkie-cad/logprep"
version = "${{ matrix.python-version }}"
tags = "${{ inputs.tags }}".split(",")
full_image = [f"{image}:py{version}-{tag.strip()}" for tag in tags]
with open(os.environ['GITHUB_OUTPUT'], 'a') as file:
file.write(f"tags={', '.join(full_image)}")
EOF

- name: Build image and export to Docker
uses: docker/build-push-action@v6
id: build
with:
context: .
load: true
build-args: |
LOGPREP_VERSION=${{ inputs.build-version }}
PYTHON_VERSION=${{ matrix.python-version }}
tags: ${{ steps.image-tag.outputs.tags }}

- name: Extract image ID from sha256
id: imageid
run: |
python - <<EOF
import os
digest = "${{ steps.build.outputs.ImageID }}"
with open(os.environ['GITHUB_OUTPUT'], 'a') as file:
file.write(f"imageid={digest.split(':')[1]}")
EOF

- name: Ensure logprep is available in image
run: |
docker run --rm ${{ steps.imageid.outputs.imageid }} --version

- name: Push image
uses: docker/build-push-action@v6
id: build-and-push
with:
context: .
push: true
build-args: |
LOGPREP_VERSION=${{ inputs.build-version }}
PYTHON_VERSION=${{ matrix.python-version }}
tags: ${{ steps.image-tag.outputs.tags }}

- name: Install Cosign
uses: sigstore/[email protected]
with:
cosign-release: 'v2.4.1'

- name: Create SBOM of container image
uses: anchore/sbom-action@v0
with:
image: ghcr.io/fkie-cad/logprep@${{ steps.build-and-push.outputs.digest }}
artifact-name: logprep@${{ steps.imageid.outputs.imageid }}.spdx.json
output-file: logprep@${{ steps.imageid.outputs.imageid }}.spdx.json

- name: Sign image with a key and add sbom attestation
run: |
cosign sign --yes --key env://COSIGN_PRIVATE_KEY ghcr.io/fkie-cad/logprep@${{ steps.build-and-push.outputs.digest }}
cosign attest --yes --key env://COSIGN_PRIVATE_KEY --predicate logprep@${{ steps.imageid.outputs.imageid }}.spdx.json ghcr.io/fkie-cad/logprep@${{ steps.build-and-push.outputs.digest }}
env:
COSIGN_PRIVATE_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }}
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}

# To avoid the trivy-db becoming outdated, we save the cache for one day
- name: Get date
id: date
run: echo "date=$(date +%Y-%m-%d)" >> $GITHUB_OUTPUT

- name: Restore trivy cache
uses: actions/cache@v4
with:
path: cache/db
key: trivy-cache-${{ steps.date.outputs.date }}
restore-keys:
trivy-cache-

- name: Scan image using Trivy
uses: aquasecurity/[email protected]
env:
TRIVY_CACHE_DIR: ./cache
with:
scan-type: image
image-ref: ghcr.io/fkie-cad/logprep@${{ steps.build-and-push.outputs.digest }}
trivy-config: trivy.yaml

# Trivy-db uses `0600` permissions.
# But `action/cache` use `runner` user by default
# So we need to change the permissions before caching the database.
- name: Change permissions for trivy.db
run: sudo chmod 0644 ./cache/db/trivy.db
34 changes: 6 additions & 28 deletions .github/workflows/publish-latest-dev-release-to-github.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,31 +47,9 @@ jobs:
LICENSE

containerbuild:
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build images
uses: docker/build-push-action@v3
with:
context: .
push: true # Will only build if this is not here
build-args: |
LOGPREP_VERSION=latest
PYTHON_VERSION=${{ matrix.python-version }}
tags: |
ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-main
ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-latest
uses: ./.github/workflows/container-build.yml
needs: create-github-prerelease
secrets: inherit
with:
build-version: ${{ github.ref_name }}
tags: "main, latest"
83 changes: 5 additions & 78 deletions .github/workflows/publish-release-to-pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,82 +60,9 @@ jobs:
uses: pypa/gh-action-pypi-publish@release/v1

containerbuild:
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]

runs-on: ubuntu-latest
uses: ./.github/workflows/container-build.yml
needs: publish-latest-release-to-pypi
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build images
uses: docker/build-push-action@v3
with:
context: .
push: true # Will only build if this is not here
build-args: |
LOGPREP_VERSION=${{ github.ref_name }}
PYTHON_VERSION=${{ matrix.python-version }}
tags: |
ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.ref_name }}
ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-stable
ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-latest

- name: Install Cosign
uses: sigstore/[email protected]
with:
cosign-release: 'v2.4.1'

- name: Create SBOM of container image
uses: anchore/sbom-action@v0
with:
image: ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}@${{ steps.build-and-push.outputs.digest }}
artifact-name: py${{ matrix.python-version }}-${{ github.head_ref }}.spdx.json
output-file: py${{ matrix.python-version }}-${{ github.head_ref }}.spdx.json

- name: Sign image with a key and add sbom attestation
run: |
cosign sign --yes --key env://COSIGN_PRIVATE_KEY ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}@${DIGEST}
cosign attest --yes --key env://COSIGN_PRIVATE_KEY --predicate py${{ matrix.python-version }}-${{ github.head_ref }}.spdx.json ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}@${DIGEST}
env:
COSIGN_PRIVATE_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }}
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
DIGEST: ${{ steps.build-and-push.outputs.digest }}

# To avoid the trivy-db becoming outdated, we save the cache for one day
- name: Get date
id: date
run: echo "date=$(date +%Y-%m-%d)" >> $GITHUB_OUTPUT

- name: Restore trivy cache
uses: actions/cache@v4
with:
path: cache/db
key: trivy-cache-${{ steps.date.outputs.date }}
restore-keys:
trivy-cache-

- name: Scan image using Trivy
uses: aquasecurity/[email protected]
env:
TRIVY_CACHE_DIR: ./cache
with:
scan-type: image
image-ref: ghcr.io/fkie-cad/logprep:py${{ matrix.python-version }}-${{ github.head_ref }}
trivy-config: trivy.yaml

# Trivy-db uses `0600` permissions.
# But `action/cache` use `runner` user by default
# So we need to change the permissions before caching the database.
- name: Change permissions for trivy.db
run: sudo chmod 0644 ./cache/db/trivy.db
secrets: inherit
with:
build-version: ${{ github.ref_name }}
tags: "${{ github.ref_name }}, stable, latest"
Loading