diff --git a/.github/ISSUE_TEMPLATE/create_release_branch.md b/.github/ISSUE_TEMPLATE/create_release_branch.md index 9c5a8268a..c460a90ed 100644 --- a/.github/ISSUE_TEMPLATE/create_release_branch.md +++ b/.github/ISSUE_TEMPLATE/create_release_branch.md @@ -13,16 +13,16 @@ Make sure to follow the steps below and ensure all actions are completed and sig - **K8s version**: 1.xx -- **Owner**: +- **Owner**: `who plans to do the work` -- **Reviewer**: +- **Reviewer**: `who plans to review the work` -- **PR**: -- +- **PR**: https://github.com/canonical/k8s-snap/pull/`` + -- **PR**: +- **PR**: https://github.com/canonical/k8s-snap/pull/`` #### Actions @@ -53,7 +53,7 @@ The steps are to be followed in-order, each task must be completed by the person - [ ] **Owner**: Create `release-1.xx` branch from latest `master` in k8s-dqlite - `git clone git@github.com:canonical/k8s-dqlite.git ~/tmp/release-1.xx` - `pushd ~/tmp/release-1.xx` - - `git switch main` + - `git switch master` - `git pull` - `git checkout -b release-1.xx` - `git push origin release-1.xx` @@ -89,7 +89,7 @@ The steps are to be followed in-order, each task must be completed by the person - [ ] **Owner**: Create `release-1.xx` branch from latest `main` in rawfile-localpv - `git clone git@github.com:canonical/rawfile-localpv.git ~/tmp/release-1.xx` - `pushd ~/tmp/release-1.xx` - - `git switch main` + - `git switch rockcraft` - `git pull` - `git checkout -b release-1.xx` - `git push origin release-1.xx` @@ -98,7 +98,6 @@ The steps are to be followed in-order, each task must be completed by the person - [ ] **Reviewer**: Ensure `release-1.xx` branch is based on latest changes on `main` at the time of the release cut. - [ ] **Owner**: Create PR to initialize `release-1.xx` branch: - [ ] Update `KUBERNETES_RELEASE_MARKER` to `stable-1.xx` in [/build-scripts/hack/update-component-versions.py][] - - [ ] Update `master` to `release-1.xx` in [/build-scripts/components/k8s-dqlite/version][] - [ ] Update `"main"` to `"release-1.xx"` in [/build-scripts/hack/generate-sbom.py][] - [ ] `git commit -m 'Release 1.xx'` - [ ] Create PR against `release-1.xx` with the changes and request review from **Reviewer**. Make sure to update the issue `Information` section with a link to the PR. @@ -107,43 +106,22 @@ The steps are to be followed in-order, each task must be completed by the person - [ ] Add `release-1.xx` in [.github/workflows/update-components.yaml][] - [ ] Remove unsupported releases from the list (if applicable, consult with **Reviewer**) - [ ] Create PR against `main` with the changes and request review from **Reviewer**. Make sure to update the issue information with a link to the PR. -- [ ] **Reviewer**: On merge, confirm [Auto-update strict branch] action runs to completion and that the `autoupdate/release-1.xx-strict` branch is created. -- [ ] **Owner**: Create launchpad builders for `release-1.xx` - - [ ] Go to [lp:k8s][] and do **Import now** to pick up all latest changes. - - [ ] Under **Branches**, select `release-1.xx`, then **Create snap package** - - [ ] Set **Snap recipe name** to `k8s-snap-1.xx` - - [ ] Set **Owner** to `Canonical Kubernetes (containers)` - - [ ] Set **The project that this Snap is associated with** to `k8s` - - [ ] Set **Series** to Infer from snapcraft.yaml - - [ ] Set **Processors** to `AMD x86-64 (amd64)` and `ARM ARMv8 (arm64)` - - [ ] Enable **Automatically build when branch changes** - - [ ] Enable **Automatically upload to store** - - [ ] Set **Registered store name** to `k8s` - - [ ] In **Store Channels**, set **Track** to `1.xx-classic` and **Risk** to `edge`. Leave **Branch** empty - - [ ] Click **Create snap package** at the bottom of the page. -- [ ] **Owner**: Create launchpad builders for `release-1.xx-strict` - - [ ] Return to [lp:k8s][]. - - [ ] Under **Branches**, select `autoupdate/release-1.xx-strict`, then **Create snap package** - - [ ] Set **Snap recipe name** to `k8s-snap-1.xx-strict` - - [ ] Set **Owner** to `Canonical Kubernetes (containers)` - - [ ] Set **The project that this Snap is associated with** to `k8s` - - [ ] Set **Series** to Infer from snapcraft.yaml - - [ ] Set **Processors** to `AMD x86-64 (amd64)` and `ARM ARMv8 (arm64)` - - [ ] Enable **Automatically build when branch changes** - - [ ] Enable **Automatically upload to store** - - [ ] Set **Registered store name** to `k8s` - - [ ] In **Store Channels**, set **Track** to `1.xx` and **Risk** to `edge`. Leave **Branch** empty - - [ ] Click **Create snap package** at the bottom of the page. +- [ ] **Reviewer**: On merge, confirm [Auto-update strict branch] action runs to completion and that the `autoupdate/release-1.xx-*` flavor branches are created. + - [ ] autoupdate/release-1.xx-strict + - [ ] autoupdate/release-1.xx-moonray +- [ ] **Owner**: Create launchpad builders for `release-1.xx` and flavors + - [ ] Run the [Confirm Snap Builds][] Action - [ ] **Reviewer**: Ensure snap recipes are created in [lp:k8s/+snaps][] - - look for `k8s-snap-1.xx` - - look for `k8s-snap-1.xx-strict` + - [ ] look for `k8s-snap-1.xx-classic` + - [ ] look for `k8s-snap-1.xx-strict` + - [ ] look for `k8s-snap-1.xx-moonray` + - [ ] make sure each is "Authorized for Store Upload" #### After release - [ ] **Owner** follows up with the **Reviewer** and team about things to improve around the process. - [ ] **Owner**: After a few weeks of stable CI, update default track to `1.xx/stable` via - On the snap [releases page][], select `Track` > `1.xx` -- [ ] **Reviewer**: Ensure snap recipes are created in [lp:k8s/+snaps][] @@ -161,6 +139,7 @@ The steps are to be followed in-order, each task must be completed by the person [.github/workflows/update-components.yaml]: ../workflows/update-components.yaml [/build-scripts/components/hack/update-component-versions.py]: ../../build-scripts/components/hack/update-component-versions.py [/build-scripts/components/k8s-dqlite/version]: ../../build-scripts/components/k8s-dqlite/version -[/build-scripts/hack/generate-sbom.py]: ../..//build-scripts/hack/generate-sbom.py +[/build-scripts/hack/generate-sbom.py]: ../../build-scripts/hack/generate-sbom.py [lp:k8s]: https://code.launchpad.net/~cdk8s/k8s/+git/k8s-snap [lp:k8s/+snaps]: https://launchpad.net/k8s/+snaps +[Confirm Snap Builds]: https://github.com/canonical/canonical-kubernetes-release-ci/actions/workflows/create-release-branch.yaml diff --git a/.github/workflows/auto-merge-successful-prs.yaml b/.github/workflows/auto-merge-successful-prs.yaml new file mode 100644 index 000000000..e7f4fc096 --- /dev/null +++ b/.github/workflows/auto-merge-successful-prs.yaml @@ -0,0 +1,29 @@ +name: Auto-merge Successful PRs + +on: + workflow_dispatch: + schedule: + - cron: "0 */4 * * *" # Every 4 hours + +permissions: + contents: read + +jobs: + merge-successful-prs: + runs-on: ubuntu-latest + + steps: + - name: Harden Runner + uses: step-security/harden-runner@v2 + with: + egress-policy: audit + - name: Checking out repo + uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - name: Auto-merge pull requests if all status checks pass + env: + GH_TOKEN: ${{ secrets.BOT_TOKEN }} + run: | + build-scripts/hack/auto-merge-successful-pr.py diff --git a/.github/workflows/automatic-doc-checks.yml b/.github/workflows/automatic-doc-checks.yml index 5472b1662..d3bba4576 100644 --- a/.github/workflows/automatic-doc-checks.yml +++ b/.github/workflows/automatic-doc-checks.yml @@ -3,13 +3,15 @@ name: Core Documentation Checks on: - workflow_dispatch +permissions: + contents: read + concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: documentation-checks: - uses: canonical/documentation-workflows/.github/workflows/documentation-checks.yaml@main - with: - working-directory: 'docs/moonray' - + - uses: canonical/documentation-workflows/.github/workflows/documentation-checks.yaml@main + with: + working-directory: 'docs/moonray' diff --git a/.github/workflows/cron-jobs.yaml b/.github/workflows/cron-jobs.yaml index fc658da51..59a1227a1 100644 --- a/.github/workflows/cron-jobs.yaml +++ b/.github/workflows/cron-jobs.yaml @@ -6,7 +6,7 @@ on: permissions: contents: read - + jobs: TICS: permissions: @@ -27,6 +27,9 @@ jobs: uses: actions/checkout@v4 with: ref: ${{matrix.branch}} + - uses: actions/setup-python@v5 + with: + python-version: '3.12' - name: Install Go uses: actions/setup-go@v5 with: @@ -47,14 +50,14 @@ jobs: # TICS requires us to have the test results in cobertura xml format under the # directory use below - make go.unit + sudo make go.unit go install github.com/boumenot/gocover-cobertura@latest gocover-cobertura < coverage.txt > coverage.xml mkdir .coverage mv ./coverage.xml ./.coverage/ # Install the TICS and staticcheck - go install honnef.co/go/tools/cmd/staticcheck@v0.4.7 + go install honnef.co/go/tools/cmd/staticcheck@v0.5.1 . <(curl --silent --show-error 'https://canonical.tiobe.com/tiobeweb/TICS/api/public/v1/fapi/installtics/Script?cfg=default&platform=linux&url=https://canonical.tiobe.com/tiobeweb/TICS/') # We need to have our project built @@ -62,7 +65,7 @@ jobs: # will try to build parts of the project itself sudo add-apt-repository -y ppa:dqlite/dev sudo apt install dqlite-tools libdqlite-dev -y - make clean + sudo make clean go build -a ./... TICSQServer -project k8s-snap -tmpdir /tmp/tics -branchdir $HOME/work/k8s-snap/k8s-snap/ @@ -79,6 +82,8 @@ jobs: - { branch: main, channel: latest/edge } # Stable branches # Add branches to test here + - { branch: release-1.30, channel: 1.30-classic/edge } + - { branch: release-1.31, channel: 1.31-classic/edge } steps: - name: Harden Runner @@ -103,6 +108,8 @@ jobs: format: "sarif" output: "trivy-k8s-repo-scan--results.sarif" severity: "MEDIUM,HIGH,CRITICAL" + env: + TRIVY_DB_REPOSITORY: "public.ecr.aws/aquasecurity/trivy-db" - name: Gather Trivy repo scan results run: | cp trivy-k8s-repo-scan--results.sarif ./sarifs/ @@ -111,7 +118,10 @@ jobs: snap download k8s --channel ${{ matrix.channel }} mv ./k8s*.snap ./k8s.snap unsquashfs k8s.snap - ./trivy rootfs ./squashfs-root/ --format sarif > sarifs/snap.sarif + for var in $(env | grep -o '^TRIVY_[^=]*'); do + unset "$var" + done + ./trivy --db-repository public.ecr.aws/aquasecurity/trivy-db rootfs ./squashfs-root/ --format sarif > sarifs/snap.sarif - name: Get HEAD sha run: | SHA="$(git rev-parse HEAD)" diff --git a/.github/workflows/docs-spelling-checks.yml b/.github/workflows/docs-spelling-checks.yml new file mode 100644 index 000000000..913ab75a8 --- /dev/null +++ b/.github/workflows/docs-spelling-checks.yml @@ -0,0 +1,32 @@ +name: Documentation Spelling Check + +on: + workflow_dispatch: + # pull_request: + # paths: + # - 'docs/**' +permissions: + contents: read + +jobs: + spell-check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install aspell + run: sudo apt-get install aspell aspell-en + - id: spell-check + name: Spell Check + run: make spelling + working-directory: docs/canonicalk8s + continue-on-error: true + # - if: ${{ github.event_name == 'pull_request' && steps.spell-check.outcome == 'failure' }} + # uses: actions/github-script@v6 + # with: + # script: | + # github.rest.issues.createComment({ + # issue_number: context.issue.number, + # owner: context.repo.owner, + # repo: context.repo.repo, + # body: 'Hi, looks like pyspelling job found some issues, you can check it [here](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})' + # }) diff --git a/.github/workflows/go.yaml b/.github/workflows/go.yaml index c29585c09..56be690a9 100644 --- a/.github/workflows/go.yaml +++ b/.github/workflows/go.yaml @@ -2,6 +2,8 @@ name: Go on: push: + paths-ignore: + - 'docs/**' branches: - main - autoupdate/strict @@ -10,6 +12,8 @@ on: - 'autoupdate/release-[0-9]+.[0-9]+-strict' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read @@ -19,6 +23,7 @@ jobs: permissions: contents: read # for actions/checkout to fetch code pull-requests: write # for marocchino/sticky-pull-request-comment to create or update PR comment + checks: write # for golangci/golangci-lint-action to checks to allow the action to annotate code in the PR. name: Unit Tests & Code Quality runs-on: ubuntu-latest @@ -67,6 +72,19 @@ jobs: # root ownership so the tests must be run as root: run: sudo make go.unit + - name: dqlite-for-golangci-lint + working-directory: src/k8s + run: | + sudo add-apt-repository ppa:dqlite/dev + sudo apt update + sudo apt install dqlite-tools libdqlite-dev + + - name: golangci-lint + uses: golangci/golangci-lint-action@v6 + with: + version: v1.61 + working-directory: src/k8s + test-binary: name: Binaries runs-on: ubuntu-latest diff --git a/.github/workflows/integration-informing.yaml b/.github/workflows/integration-informing.yaml index 708094078..9ade424d0 100644 --- a/.github/workflows/integration-informing.yaml +++ b/.github/workflows/integration-informing.yaml @@ -2,11 +2,15 @@ name: Informing Integration Tests on: push: + paths-ignore: + - 'docs/**' branches: - main - 'release-[0-9]+.[0-9]+' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read @@ -17,7 +21,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - patch: ["strict", "moonray"] + patch: ["moonray"] fail-fast: false steps: - name: Harden Runner @@ -54,16 +58,16 @@ jobs: strategy: matrix: os: ["ubuntu:20.04"] - patch: ["strict", "moonray"] + patch: ["moonray"] fail-fast: false - runs-on: ubuntu-20.04 + runs-on: ["self-hosted", "Linux", "AMD64", "jammy", "large"] steps: - name: Check out code uses: actions/checkout@v4 - name: Setup Python uses: actions/setup-python@v5 with: - python-version: '3.8' + python-version: '3.10' - name: Install tox run: pip install tox - name: Install lxd @@ -72,29 +76,33 @@ jobs: sudo lxd init --auto sudo usermod --append --groups lxd $USER sg lxd -c 'lxc version' + sudo iptables -I DOCKER-USER -i lxdbr0 -j ACCEPT + sudo iptables -I DOCKER-USER -o lxdbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT - name: Download snap uses: actions/download-artifact@v4 with: name: k8s-${{ matrix.patch }}.snap - path: build + path: ${{ github.workspace }}/build - name: Apply ${{ matrix.patch }} patch run: | ./build-scripts/patches/${{ matrix.patch }}/apply - name: Run end to end tests + env: + TEST_SNAP: ${{ github.workspace }}/build/k8s-${{ matrix.patch }}.snap + TEST_SUBSTRATE: lxd + TEST_LXD_IMAGE: ${{ matrix.os }} + TEST_FLAVOR: ${{ matrix.patch }} + TEST_INSPECTION_REPORTS_DIR: ${{ github.workspace }}/inspection-reports run: | - export TEST_SNAP="$PWD/build/k8s-${{ matrix.patch }}.snap" - export TEST_SUBSTRATE=lxd - export TEST_LXD_IMAGE=${{ matrix.os }} - export TEST_INSPECTION_REPORTS_DIR="$HOME/inspection-reports" cd tests/integration && sg lxd -c 'tox -e integration' - name: Prepare inspection reports if: failure() run: | - tar -czvf inspection-reports.tar.gz -C $HOME inspection-reports + tar -czvf inspection-reports.tar.gz -C ${{ github.workspace }} inspection-reports echo "artifact_name=inspection-reports-${{ matrix.os }}-${{ matrix.patch }}" | sed 's/:/-/g' >> $GITHUB_ENV - name: Upload inspection report artifact if: failure() uses: actions/upload-artifact@v4 with: name: ${{ env.artifact_name }} - path: inspection-reports.tar.gz + path: ${{ github.workspace }}/inspection-reports.tar.gz diff --git a/.github/workflows/integration.yaml b/.github/workflows/integration.yaml index 6a58c1194..2aefbb939 100644 --- a/.github/workflows/integration.yaml +++ b/.github/workflows/integration.yaml @@ -2,6 +2,8 @@ name: Integration Tests on: push: + paths-ignore: + - 'docs/**' branches: - main - autoupdate/strict @@ -10,6 +12,8 @@ on: - 'autoupdate/release-[0-9]+.[0-9]+-strict' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read @@ -51,6 +55,8 @@ jobs: steps: - name: Check out code uses: actions/checkout@v4 + with: + fetch-depth: 0 - name: Setup Python uses: actions/setup-python@v5 with: @@ -59,7 +65,7 @@ jobs: run: pip install tox - name: Run branch_management tests run: | - tox -c tests/branch_management -e integration + tox -c tests/branch_management -e test test-integration: name: Test ${{ matrix.os }} @@ -67,7 +73,7 @@ jobs: fail-fast: false matrix: os: ["ubuntu:20.04", "ubuntu:22.04", "ubuntu:24.04"] - runs-on: ubuntu-20.04 + runs-on: ["self-hosted", "Linux", "AMD64", "jammy", "large"] needs: build steps: @@ -76,7 +82,7 @@ jobs: - name: Setup Python uses: actions/setup-python@v5 with: - python-version: '3.8' + python-version: '3.10' - name: Install tox run: pip install tox - name: Install lxd @@ -85,29 +91,38 @@ jobs: sudo lxd init --auto sudo usermod --append --groups lxd $USER sg lxd -c 'lxc version' + sudo iptables -I DOCKER-USER -i lxdbr0 -j ACCEPT + sudo iptables -I DOCKER-USER -o lxdbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT - name: Download snap uses: actions/download-artifact@v4 with: name: k8s.snap - path: build + path: ${{ github.workspace }}/build - name: Run end to end tests + env: + TEST_SNAP: ${{ github.workspace }}/build/k8s.snap + TEST_SUBSTRATE: lxd + TEST_LXD_IMAGE: ${{ matrix.os }} + TEST_INSPECTION_REPORTS_DIR: ${{ github.workspace }}/inspection-reports + # Test the latest (up to) 6 releases for the flavour + # TODO(ben): upgrade nightly to run all flavours + TEST_VERSION_UPGRADE_CHANNELS: "recent 6 classic" + # Upgrading from 1.30 is not supported. + TEST_VERSION_UPGRADE_MIN_RELEASE: "1.31" + TEST_MIRROR_LIST: '[{"name": "ghcr.io", "port": 5000, "remote": "https://ghcr.io", "username": "${{ github.actor }}", "password": "${{ secrets.GITHUB_TOKEN }}"}, {"name": "docker.io", "port": 5001, "remote": "https://registry-1.docker.io", "username": "", "password": ""}]' run: | - export TEST_SNAP="$PWD/build/k8s.snap" - export TEST_SUBSTRATE=lxd - export TEST_LXD_IMAGE=${{ matrix.os }} - export TEST_INSPECTION_REPORTS_DIR="$HOME/inspection-reports" - cd tests/integration && sg lxd -c 'tox -e integration -- -k test_control_plane_nodes' + cd tests/integration && sg lxd -c 'tox -e integration' - name: Prepare inspection reports if: failure() run: | - tar -czvf inspection-reports.tar.gz -C $HOME inspection-reports + tar -czvf inspection-reports.tar.gz -C ${{ github.workspace }} inspection-reports echo "artifact_name=inspection-reports-${{ matrix.os }}" | sed 's/:/-/g' >> $GITHUB_ENV - name: Upload inspection report artifact if: failure() uses: actions/upload-artifact@v4 with: name: ${{ env.artifact_name }} - path: inspection-reports.tar.gz + path: ${{ github.workspace }}/inspection-reports.tar.gz security-scan: permissions: @@ -121,6 +136,13 @@ jobs: uses: step-security/harden-runner@v2 with: egress-policy: audit + - name: Login to GitHub Container Registry + uses: docker/login-action@v3 + with: + # We run into rate limiting issues if we don't authenticate + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} - name: Checking out repo uses: actions/checkout@v4 - name: Fetch snap @@ -130,10 +152,12 @@ jobs: path: build - name: Setup Trivy vulnerability scanner run: | - mkdir -p sarifs + mkdir -p manual-trivy/sarifs + pushd manual-trivy VER=$(curl --silent -qI https://github.com/aquasecurity/trivy/releases/latest | awk -F '/' '/^location/ {print substr($NF, 1, length($NF)-1)}'); wget https://github.com/aquasecurity/trivy/releases/download/${VER}/trivy_${VER#v}_Linux-64bit.tar.gz tar -zxvf ./trivy_${VER#v}_Linux-64bit.tar.gz + popd - name: Run Trivy vulnerability scanner in repo mode uses: aquasecurity/trivy-action@master with: @@ -142,15 +166,20 @@ jobs: format: "sarif" output: "trivy-k8s-repo-scan--results.sarif" severity: "MEDIUM,HIGH,CRITICAL" + env: + TRIVY_DB_REPOSITORY: "public.ecr.aws/aquasecurity/trivy-db" - name: Gather Trivy repo scan results run: | - cp trivy-k8s-repo-scan--results.sarif ./sarifs/ + cp trivy-k8s-repo-scan--results.sarif ./manual-trivy/sarifs/ - name: Run Trivy vulnerability scanner on the snap run: | + for var in $(env | grep -o '^TRIVY_[^=]*'); do + unset "$var" + done cp build/k8s.snap . unsquashfs k8s.snap - ./trivy rootfs ./squashfs-root/ --format sarif > sarifs/snap.sarif + ./manual-trivy/trivy --db-repository public.ecr.aws/aquasecurity/trivy-db rootfs ./squashfs-root/ --format sarif > ./manual-trivy/sarifs/snap.sarif - name: Upload Trivy scan results to GitHub Security tab uses: github/codeql-action/upload-sarif@v3 with: - sarif_file: "sarifs" + sarif_file: "./manual-trivy/sarifs" diff --git a/.github/workflows/nightly-test.yaml b/.github/workflows/nightly-test.yaml index 915d6c3f3..3e7fd8972 100644 --- a/.github/workflows/nightly-test.yaml +++ b/.github/workflows/nightly-test.yaml @@ -2,49 +2,68 @@ name: Nightly Latest/Edge Tests on: schedule: - - cron: '0 0 * * *' # Runs every midnight + - cron: "0 0 * * *" # Runs every midnight permissions: contents: read jobs: test-integration: - name: Integration Test ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.releases }} + name: Integration Test ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.release }} strategy: matrix: os: ["ubuntu:20.04", "ubuntu:22.04", "ubuntu:24.04"] arch: ["amd64", "arm64"] - releases: ["latest/edge"] + release: ["latest/edge"] fail-fast: false # TODO: remove once arm64 works - runs-on: ${{ matrix.arch == 'arm64' && 'Ubuntu_ARM64_4C_16G_01' || 'ubuntu-20.04' }} + runs-on: ${{ matrix.arch == 'arm64' && ["self-hosted", "Linux", "ARM64", "jammy", "large"] || ["self-hosted", "Linux", "AMD64", "jammy", "large"] }} steps: - name: Checking out repo uses: actions/checkout@v4 - - name: Setup Python + - name: Install lxd and tox run: | sudo apt update - sudo apt install -y python3 python3-pip - - name: Install tox - run: | - pip3 install tox==4.13 - - name: Install lxd - run: | - sudo snap refresh lxd --channel 5.19/stable + sudo apt install -y tox + sudo snap refresh lxd --channel 5.21/stable sudo lxd init --auto sudo usermod --append --groups lxd $USER sg lxd -c 'lxc version' - name: Create build directory run: mkdir -p build - - name: Install $${ matrix.releases }} k8s snap + - name: Install ${{ matrix.release }} k8s snap run: | cd build - snap download k8s --channel=${{ matrix.releases }} --basename k8s + snap download k8s --channel=${{ matrix.release }} --basename k8s - name: Run end to end tests # tox path needs to be specified for arm64 + env: + TEST_SNAP: ${{ github.workspace }}/build/k8s.snap + TEST_SUBSTRATE: lxd + TEST_LXD_IMAGE: ${{ matrix.os }} + TEST_INSPECTION_REPORTS_DIR: ${{ github.workspace }}/inspection-reports + # Test the latest (up to) 6 releases for the flavour + # TODO(ben): upgrade nightly to run all flavours + TEST_VERSION_UPGRADE_CHANNELS: "recent 6 classic" + # Upgrading from 1.30 is not supported. + TEST_VERSION_UPGRADE_MIN_RELEASE: "1.31" + TEST_STRICT_INTERFACE_CHANNELS: "recent 6 strict" + TEST_MIRROR_LIST: '[{"name": "ghcr.io", "port": 5000, "remote": "https://ghcr.io", "username": "${{ github.actor }}", "password": "${{ secrets.GITHUB_TOKEN }}"}, {"name": "docker.io", "port": 5001, "remote": "https://registry-1.docker.io", "username": "", "password": ""}]' run: | - export TEST_SNAP="$PWD/build/k8s.snap" - export TEST_SUBSTRATE=lxd - export TEST_LXD_IMAGE="${{ matrix.os }}" export PATH="/home/runner/.local/bin:$PATH" - cd tests/integration && sg lxd -c 'tox -e integration' + cd tests/integration && sg lxd -c 'tox -vve integration' + - name: Prepare inspection reports + if: failure() + run: | + tar -czvf inspection-reports.tar.gz -C ${{ github.workspace }} inspection-reports + echo "artifact_name=inspection-reports-${{ matrix.os }}-${{ matrix.arch }}" | sed 's/:/-/g' >> $GITHUB_ENV + - name: Upload inspection report artifact + if: failure() + uses: actions/upload-artifact@v4 + with: + name: ${{ env.artifact_name }} + path: ${{ github.workspace }}/inspection-reports.tar.gz + - name: Tmate debugging session + if: ${{ failure() && github.event_name == 'pull_request' }} + uses: mxschmitt/action-tmate@v3 + timeout-minutes: 10 diff --git a/.github/workflows/python.yaml b/.github/workflows/python.yaml index 2ccb0979f..0c51d8ecf 100644 --- a/.github/workflows/python.yaml +++ b/.github/workflows/python.yaml @@ -2,6 +2,8 @@ name: Python on: push: + paths-ignore: + - 'docs/**' branches: - main - autoupdate/strict @@ -10,6 +12,8 @@ on: - 'autoupdate/release-[0-9]+.[0-9]+-strict' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read diff --git a/.github/workflows/sbom.yaml b/.github/workflows/sbom.yaml index 846a19e76..2faa27a28 100644 --- a/.github/workflows/sbom.yaml +++ b/.github/workflows/sbom.yaml @@ -2,6 +2,8 @@ name: SBOM on: push: + paths-ignore: + - 'docs/**' branches: - main - autoupdate/strict @@ -10,6 +12,8 @@ on: - 'autoupdate/release-[0-9]+.[0-9]+-strict' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read diff --git a/.github/workflows/sync-images.yaml b/.github/workflows/sync-images.yaml deleted file mode 100644 index 02b143a1b..000000000 --- a/.github/workflows/sync-images.yaml +++ /dev/null @@ -1,23 +0,0 @@ -name: Sync upstream images to ghcr.io - -on: - workflow_dispatch: - -jobs: - publish: - runs-on: ubuntu-latest - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Install Go - uses: actions/setup-go@v5 - with: - go-version: "1.22" - - - name: Sync images - env: - USERNAME: ${{ github.actor }} - PASSWORD: ${{ secrets.GITHUB_TOKEN }} - run: | - ./build-scripts/hack/sync-images.sh diff --git a/.github/workflows/update-branches.yaml b/.github/workflows/update-branches.yaml index 356bbce5b..b6ed8f38e 100644 --- a/.github/workflows/update-branches.yaml +++ b/.github/workflows/update-branches.yaml @@ -41,7 +41,7 @@ jobs: - name: Sync ${{ github.ref }} to ${{ steps.determine.outputs.branch }} uses: actions/checkout@v4 with: - ssh-key: ${{ secrets.DEPLOY_KEY_TO_UPDATE_STRICT_BRANCH }} + ssh-key: ${{ secrets.BOT_SSH_KEY }} - name: Apply ${{ matrix.patch }} patch run: | git checkout -b ${{ steps.determine.outputs.branch }} diff --git a/.github/workflows/update-components.yaml b/.github/workflows/update-components.yaml index e46bd55df..23aa952a4 100644 --- a/.github/workflows/update-components.yaml +++ b/.github/workflows/update-components.yaml @@ -21,6 +21,7 @@ jobs: # Keep main branch up to date - main # Supported stable release branches + - release-1.31 - release-1.30 steps: @@ -32,7 +33,7 @@ jobs: uses: actions/checkout@v4 with: ref: ${{ matrix.branch }} - ssh-key: ${{ secrets.DEPLOY_KEY_TO_UPDATE_STRICT_BRANCH }} + ssh-key: ${{ secrets.BOT_SSH_KEY }} - name: Setup Python uses: actions/setup-python@v5 @@ -50,10 +51,11 @@ jobs: - name: Create pull request uses: peter-evans/create-pull-request@v6 with: - git-token: ${{ secrets.DEPLOY_KEY_TO_UPDATE_STRICT_BRANCH }} commit-message: "[${{ matrix.branch }}] Update component versions" title: "[${{ matrix.branch }}] Update component versions" body: "[${{ matrix.branch }}] Update component versions" branch: "autoupdate/sync/${{ matrix.branch }}" + labels: | + automerge delete-branch: true base: ${{ matrix.branch }} diff --git a/.gitignore b/.gitignore index 8c7172b67..896206d59 100644 --- a/.gitignore +++ b/.gitignore @@ -21,3 +21,5 @@ k8s_*.txt /docs/tools/.sphinx/.wordlist.dic /docs/tools/.sphinx/.doctrees/ /docs/tools/.sphinx/node_modules +/docs/tools/.sphinx/styles/* +/docs/tools/.sphinx/vale.ini diff --git a/.golangci.yml b/.golangci.yml new file mode 100644 index 000000000..ece0c60a6 --- /dev/null +++ b/.golangci.yml @@ -0,0 +1,328 @@ +# This code is licensed under the terms of the MIT license https://opensource.org/license/mit +# Copyright (c) 2021 Marat Reymers + +## Golden config for golangci-lint v1.61.0 +# +# This is the best config for golangci-lint based on my experience and opinion. +# It is very strict, but not extremely strict. +# Feel free to adapt and change it for your needs. +linters: + disable-all: true + enable: + - gosimple + - asasalint + - asciicheck + - bidichk + - bodyclose + - canonicalheader + - copyloopvar + - decorder + - dogsled + - dupword + - durationcheck + # - err113 - TODO(ben): maybe consider later + # - errcheck + # - errchkjson + - errname + - errorlint + - exhaustive + - fatcontext + - forbidigo + - forcetypeassert + # - funlen - TODO(ben): maybe consider later; needs some refactoring + - gci + - ginkgolinter + - gocheckcompilerdirectives + # - gochecknoglobals - TODO(ben): We have some global vars, maybe refactor later as this sounds like a useful linter + - gochecksumtype + # - goconst - TODO(ben): Consider moving consts into separate package + - gocritic + - godot + - gofmt + - gofumpt + + +# TODO(ben): Enable those linters step by step and fix existing issues. +# - goheader +# - goimports +# - gomoddirectives +# - gomodguard +# - goprintffuncname +# - gosec +# - gosmopolitan +# - govet +# - grouper +# - importas +# - inamedparam +# - ineffassign +# - interfacebloat +# - intrange +# - ireturn +# - lll +# - loggercheck +# - maintidx +# - makezero +# - mirror +# - misspell +# - mnd +# - musttag +# - nakedret +# - nestif +# - nilerr +# - nilnil +# - nlreturn +# - noctx +# - nolintlint +# - nonamedreturns +# - nosprintfhostport +# - paralleltest +# - perfsprint +# - prealloc +# - predeclared +# - promlinter +# - protogetter +# - reassign +# - revive +# - rowserrcheck +# - sloglint +# - spancheck +# - sqlclosecheck +# - staticcheck +# - stylecheck +# - tagalign +# - tagliatelle +# - tenv +# - testableexamples +# - testifylint +# - testpackage +# - thelper +# - tparallel +# - unconvert +# - unparam +# - unused +# - usestdlibvars +# - varnamelen +# - wastedassign +# - whitespace +# - wrapcheck +# - wsl +# - zerologlint + +run: + # Timeout for analysis, e.g. 30s, 5m. + # Default: 1m + timeout: 5m + +# This file contains only configs which differ from defaults. +# All possible options can be found here https://github.com/golangci/golangci-lint/blob/master/.golangci.reference.yml +linters-settings: + cyclop: + # The maximal code complexity to report. + # Default: 10 + max-complexity: 30 + # The maximal average package complexity. + # If it's higher than 0.0 (float) the check is enabled + # Default: 0.0 + package-average: 10.0 + + errcheck: + # Report about not checking of errors in type assertions: `a := b.(MyStruct)`. + # Such cases aren't reported by default. + # Default: false + check-type-assertions: true + + exhaustive: + # Program elements to check for exhaustiveness. + # Default: [ switch ] + check: + - switch + - map + + exhaustruct: + # List of regular expressions to exclude struct packages and their names from checks. + # Regular expressions must match complete canonical struct package/name/structname. + # Default: [] + exclude: + # std libs + - "^net/http.Client$" + - "^net/http.Cookie$" + - "^net/http.Request$" + - "^net/http.Response$" + - "^net/http.Server$" + - "^net/http.Transport$" + - "^net/url.URL$" + - "^os/exec.Cmd$" + - "^reflect.StructField$" + # public libs + - "^github.com/Shopify/sarama.Config$" + - "^github.com/Shopify/sarama.ProducerMessage$" + - "^github.com/mitchellh/mapstructure.DecoderConfig$" + - "^github.com/prometheus/client_golang/.+Opts$" + - "^github.com/spf13/cobra.Command$" + - "^github.com/spf13/cobra.CompletionOptions$" + - "^github.com/stretchr/testify/mock.Mock$" + - "^github.com/testcontainers/testcontainers-go.+Request$" + - "^github.com/testcontainers/testcontainers-go.FromDockerfile$" + - "^golang.org/x/tools/go/analysis.Analyzer$" + - "^google.golang.org/protobuf/.+Options$" + - "^gopkg.in/yaml.v3.Node$" + + funlen: + # Checks the number of lines in a function. + # If lower than 0, disable the check. + # Default: 60 + lines: 100 + # Checks the number of statements in a function. + # If lower than 0, disable the check. + # Default: 40 + statements: 50 + # Ignore comments when counting lines. + # Default false + ignore-comments: true + + gocognit: + # Minimal code complexity to report. + # Default: 30 (but we recommend 10-20) + min-complexity: 20 + + gocritic: + # Settings passed to gocritic. + # The settings key is the name of a supported gocritic checker. + # The list of supported checkers can be find in https://go-critic.github.io/overview. + settings: + captLocal: + # Whether to restrict checker to params only. + # Default: true + paramsOnly: false + underef: + # Whether to skip (*x).method() calls where x is a pointer receiver. + # Default: true + skipRecvDeref: false + + gomodguard: + blocked: + # List of blocked modules. + # Default: [] + modules: + - github.com/golang/protobuf: + recommendations: + - google.golang.org/protobuf + reason: "see https://developers.google.com/protocol-buffers/docs/reference/go/faq#modules" + - github.com/satori/go.uuid: + recommendations: + - github.com/google/uuid + reason: "satori's package is not maintained" + - github.com/gofrs/uuid: + recommendations: + - github.com/gofrs/uuid/v5 + reason: "gofrs' package was not go module before v5" + + govet: + # Enable all analyzers. + # Default: false + enable-all: true + # Disable analyzers by name. + # Run `go tool vet help` to see all analyzers. + # Default: [] + disable: + - fieldalignment # too strict + # Settings per analyzer. + settings: + shadow: + # Whether to be strict about shadowing; can be noisy. + # Default: false + strict: true + + inamedparam: + # Skips check for interface methods with only a single parameter. + # Default: false + skip-single-param: true + + mnd: + # List of function patterns to exclude from analysis. + # Values always ignored: `time.Date`, + # `strconv.FormatInt`, `strconv.FormatUint`, `strconv.FormatFloat`, + # `strconv.ParseInt`, `strconv.ParseUint`, `strconv.ParseFloat`. + # Default: [] + ignored-functions: + - args.Error + - flag.Arg + - flag.Duration.* + - flag.Float.* + - flag.Int.* + - flag.Uint.* + - os.Chmod + - os.Mkdir.* + - os.OpenFile + - os.WriteFile + - prometheus.ExponentialBuckets.* + - prometheus.LinearBuckets + + nakedret: + # Make an issue if func has more lines of code than this setting, and it has naked returns. + # Default: 30 + max-func-lines: 0 + + nolintlint: + # Exclude following linters from requiring an explanation. + # Default: [] + allow-no-explanation: [ funlen, gocognit, lll ] + # Enable to require an explanation of nonzero length after each nolint directive. + # Default: false + require-explanation: true + # Enable to require nolint directives to mention the specific linter being suppressed. + # Default: false + require-specific: true + + perfsprint: + # Optimizes into strings concatenation. + # Default: true + strconcat: false + + rowserrcheck: + # database/sql is always checked + # Default: [] + packages: + - github.com/jmoiron/sqlx + + sloglint: + # Enforce not using global loggers. + # Values: + # - "": disabled + # - "all": report all global loggers + # - "default": report only the default slog logger + # https://github.com/go-simpler/sloglint?tab=readme-ov-file#no-global + # Default: "" + no-global: "all" + # Enforce using methods that accept a context. + # Values: + # - "": disabled + # - "all": report all contextless calls + # - "scope": report only if a context exists in the scope of the outermost function + # https://github.com/go-simpler/sloglint?tab=readme-ov-file#context-only + # Default: "" + context: "scope" + + tenv: + # The option `all` will run against whole test files (`_test.go`) regardless of method/function signatures. + # Otherwise, only methods that take `*testing.T`, `*testing.B`, and `testing.TB` as arguments are checked. + # Default: false + all: true + +issues: + # Maximum count of issues with the same text. + # Set to 0 to disable. + # Default: 3 + max-same-issues: 50 + + exclude-rules: + - source: "(noinspection|TODO)" + linters: [ godot ] + - source: "//noinspection" + linters: [ gocritic ] + - path: "_test\\.go" + linters: + - dupword + - err113 + - forcetypeassert + - goconst diff --git a/README.md b/README.md index 587518aa7..0c86511f3 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Canonical Kubernetes Snap [![End to End Tests](https://github.com/canonical/k8s-snap/actions/workflows/integration.yaml/badge.svg)](https://github.com/canonical/k8s-snap/actions/workflows/integration.yaml) -![](https://img.shields.io/badge/Kubernetes-1.30-326de6.svg) +![](https://img.shields.io/badge/Kubernetes-1.31-326de6.svg) [![Get it from the Snap Store](https://snapcraft.io/static/images/badges/en/snap-store-black.svg)](https://snapcraft.io/k8s) diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 000000000..1a3299a16 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,15 @@ +# Security Policy + +## Reporting a Vulnerability + +To report a security issue, please follow the steps below: + +Using GitHub, file a [Private Security Report](https://github.com/canonical/k8s-snap/security/advisories/new) with: +- A description of the issue +- Steps to reproduce the issue +- Affected versions of the `k8s-snap` package +- Any known mitigations for the issue + +The [Ubuntu Security disclosure and embargo policy](https://ubuntu.com/security/disclosure-policy) contains more information about what to expect during this process and our requirements for responsible disclosure. + +Thank you for contributing to the security and integrity of `k8s-snap`! diff --git a/build-scripts/components/cni/version b/build-scripts/components/cni/version index 2e7bd9108..53b5bbb12 100644 --- a/build-scripts/components/cni/version +++ b/build-scripts/components/cni/version @@ -1 +1 @@ -v1.5.0 +v1.5.1 diff --git a/build-scripts/components/kubernetes/version b/build-scripts/components/kubernetes/version index d3aa76971..c46737494 100644 --- a/build-scripts/components/kubernetes/version +++ b/build-scripts/components/kubernetes/version @@ -1 +1 @@ -v1.31.0 +v1.31.3 diff --git a/build-scripts/hack/auto-merge-successful-pr.py b/build-scripts/hack/auto-merge-successful-pr.py new file mode 100755 index 000000000..ea6c98e6b --- /dev/null +++ b/build-scripts/hack/auto-merge-successful-pr.py @@ -0,0 +1,55 @@ +#!/bin/env python3 + +import shlex +import subprocess +import json + +LABEL = "automerge" +APPROVE_MSG = "All status checks passed for PR #{}." + + +def sh(cmd: str) -> str: + """Run a shell command and return its output.""" + _pipe = subprocess.PIPE + result = subprocess.run(shlex.split(cmd), stdout=_pipe, stderr=_pipe, text=True) + if result.returncode != 0: + raise Exception(f"Error running command: {cmd}\nError: {result.stderr}") + return result.stdout.strip() + + +def get_pull_requests() -> list: + """Fetch open pull requests matching some label.""" + prs_json = sh("gh pr list --state open --json number,labels") + prs = json.loads(prs_json) + return [pr for pr in prs if any(label["name"] == LABEL for label in pr["labels"])] + + +def check_pr_passed(pr_number) -> bool: + """Check if all status checks passed for the given PR.""" + checks_json = sh(f"gh pr checks {pr_number} --json bucket") + checks = json.loads(checks_json) + return all(check["bucket"] == "pass" for check in checks) + + +def approve_and_merge_pr(pr_number) -> None: + """Approve and merge the PR.""" + print(APPROVE_MSG.format(pr_number) + "Proceeding with merge...") + sh(f'gh pr review {pr_number} --comment -b "{APPROVE_MSG.format(pr_number)}"') + sh(f"gh pr merge {pr_number} --admin --squash") + + +def process_pull_requests(): + """Process the PRs and merge if checks have passed.""" + prs = get_pull_requests() + + for pr in prs: + pr_number: int = pr["number"] + + if check_pr_passed(pr_number): + approve_and_merge_pr(pr_number) + else: + print(f"Status checks have not passed for PR #{pr_number}. Skipping merge.") + + +if __name__ == "__main__": + process_pull_requests() diff --git a/build-scripts/hack/generate-sbom.py b/build-scripts/hack/generate-sbom.py index e5fc93a8c..3303a23d4 100755 --- a/build-scripts/hack/generate-sbom.py +++ b/build-scripts/hack/generate-sbom.py @@ -139,13 +139,15 @@ def k8s_snap_c_dqlite_components(manifest, extra_files): def rock_cilium(manifest, extra_files): LOG.info("Generating SBOM info for Cilium rocks") + cilium_version = "1.16.3" + with util.git_repo(CILIUM_ROCK_REPO, CILIUM_ROCK_TAG) as d: rock_repo_commit = util.parse_output(["git", "rev-parse", "HEAD"], cwd=d) - rockcraft = (d / "cilium/rockcraft.yaml").read_text() - operator_rockcraft = (d / "cilium-operator-generic/rockcraft.yaml").read_text() + rockcraft = (d / f"{cilium_version}/cilium/rockcraft.yaml").read_text() + operator_rockcraft = (d / f"{cilium_version}/cilium-operator-generic/rockcraft.yaml").read_text() - extra_files["cilium/rockcraft.yaml"] = rockcraft - extra_files["cilium-operator-generic/rockcraft.yaml"] = operator_rockcraft + extra_files[f"{cilium_version}/cilium/rockcraft.yaml"] = rockcraft + extra_files[f"{cilium_version}/cilium-operator-generic/rockcraft.yaml"] = operator_rockcraft rockcraft_yaml = yaml.safe_load(rockcraft) repo_url = rockcraft_yaml["parts"]["cilium"]["source"] @@ -169,10 +171,10 @@ def rock_cilium(manifest, extra_files): }, "language": "go", "details": [ - "cilium/rockcraft.yaml", + f"{cilium_version}/cilium/rockcraft.yaml", "cilium/go.mod", "cilium/go.sum", - "cilium-operator-generic/rockcraft.yaml", + f"{cilium_version}/cilium-operator-generic/rockcraft.yaml", "cilium-operator-generic/go.mod", "cilium-operator-generic/go.sum", ], @@ -190,9 +192,10 @@ def rock_coredns(manifest, extra_files): with util.git_repo(COREDNS_ROCK_REPO, COREDNS_ROCK_TAG) as d: rock_repo_commit = util.parse_output(["git", "rev-parse", "HEAD"], cwd=d) - rockcraft = (d / "rockcraft.yaml").read_text() + # TODO(ben): This should not be hard coded. + rockcraft = (d / "1.11.3/rockcraft.yaml").read_text() - extra_files["coredns/rockcraft.yaml"] = rockcraft + extra_files["coredns/1.11.3/rockcraft.yaml"] = rockcraft rockcraft_yaml = yaml.safe_load(rockcraft) repo_url = rockcraft_yaml["parts"]["coredns"]["source"] @@ -211,7 +214,11 @@ def rock_coredns(manifest, extra_files): "revision": rock_repo_commit, }, "language": "go", - "details": ["coredns/rockcraft.yaml", "coredns/go.mod", "coredns/go.sum"], + "details": [ + "coredns/1.11.3/rockcraft.yaml", + "coredns/go.mod", + "coredns/go.sum", + ], "source": { "type": "git", "repo": repo_url, @@ -226,9 +233,10 @@ def rock_metrics_server(manifest, extra_files): with util.git_repo(METRICS_SERVER_ROCK_REPO, METRICS_SERVER_ROCK_TAG) as d: rock_repo_commit = util.parse_output(["git", "rev-parse", "HEAD"], cwd=d) - rockcraft = (d / "rockcraft.yaml").read_text() + # TODO(ben): This should not be hard coded. + rockcraft = (d / "0.7.2/rockcraft.yaml").read_text() - extra_files["metrics-server/rockcraft.yaml"] = rockcraft + extra_files["metrics-server/0.7.2/rockcraft.yaml"] = rockcraft rockcraft_yaml = yaml.safe_load(rockcraft) repo_url = rockcraft_yaml["parts"]["metrics-server"]["source"] @@ -248,7 +256,7 @@ def rock_metrics_server(manifest, extra_files): }, "language": "go", "details": [ - "metrics-server/rockcraft.yaml", + "metrics-server/0.7.2/rockcraft.yaml", "metrics-server/go.mod", "metrics-server/go.sum", ], diff --git a/build-scripts/hack/sync-images.sh b/build-scripts/hack/sync-images.sh deleted file mode 100755 index 5ff80f959..000000000 --- a/build-scripts/hack/sync-images.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash - -# Description: -# Sync images from upstream repositories under ghcr.io/canonical. -# -# Usage: -# $ USERNAME="$username" PASSWORD="$password" ./sync-images.sh - -DIR="$(realpath "$(dirname "${0}")")" - -"${DIR}/../../src/k8s/tools/regsync.sh" once -c "${DIR}/upstream-images.yaml" diff --git a/build-scripts/hack/sync-images.yaml b/build-scripts/hack/sync-images.yaml deleted file mode 100644 index 8f8b68ca4..000000000 --- a/build-scripts/hack/sync-images.yaml +++ /dev/null @@ -1,31 +0,0 @@ -sync: - - source: ghcr.io/canonical/k8s-snap/pause:3.10 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/pause:3.10' - type: image - - source: ghcr.io/canonical/cilium-operator-generic:1.15.2-ck2 - target: '{{ env "MIRROR" }}/canonical/cilium-operator-generic:1.15.2-ck2' - type: image - - source: ghcr.io/canonical/cilium:1.15.2-ck2 - target: '{{ env "MIRROR" }}/canonical/cilium:1.15.2-ck2' - type: image - - source: ghcr.io/canonical/coredns:1.11.1-ck4 - target: '{{ env "MIRROR" }}/canonical/coredns:1.11.1-ck4' - type: image - - source: ghcr.io/canonical/k8s-snap/sig-storage/csi-node-driver-registrar:v2.10.1 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/sig-storage/csi-node-driver-registrar:v2.10.1' - type: image - - source: ghcr.io/canonical/k8s-snap/sig-storage/csi-provisioner:v5.0.1 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/sig-storage/csi-provisioner:v5.0.1' - type: image - - source: ghcr.io/canonical/k8s-snap/sig-storage/csi-resizer:v1.11.1 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/sig-storage/csi-resizer:v1.11.1' - type: image - - source: ghcr.io/canonical/k8s-snap/sig-storage/csi-snapshotter:v8.0.1 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/sig-storage/csi-snapshotter:v8.0.1' - type: image - - source: ghcr.io/canonical/metrics-server:0.7.0-ck1 - target: '{{ env "MIRROR" }}/canonical/metrics-server:0.7.0-ck1' - type: image - - source: ghcr.io/canonical/rawfile-localpv:0.8.0-ck4 - target: '{{ env "MIRROR" }}/canonical/rawfile-localpv:0.8.0-ck4' - type: image diff --git a/build-scripts/hack/update-component-versions.py b/build-scripts/hack/update-component-versions.py index 71f7b71d7..148cd8da9 100755 --- a/build-scripts/hack/update-component-versions.py +++ b/build-scripts/hack/update-component-versions.py @@ -15,13 +15,17 @@ import sys import yaml from pathlib import Path +import re import util +import urllib.request + logging.basicConfig(level=logging.INFO) LOG = logging.getLogger(__name__) DIR = Path(__file__).absolute().parent +SNAPCRAFT = DIR.parent.parent / "snap/snapcraft.yaml" COMPONENTS = DIR.parent / "components" CHARTS = DIR.parent.parent / "k8s" / "manifests" / "charts" @@ -44,7 +48,7 @@ # MetalLB Helm repository and chart version METALLB_REPO = "https://metallb.github.io/metallb" -METALLB_CHART_VERSION = "0.14.5" +METALLB_CHART_VERSION = "0.14.8" def get_kubernetes_version() -> str: @@ -125,6 +129,8 @@ def update_component_versions(dry_run: bool): if not dry_run: Path(path).write_text(version.strip() + "\n") + update_go_version(dry_run) + for component, pull_helm_chart in [ ("bitnami/contour", pull_contour_chart), ("metallb", pull_metallb_chart), @@ -134,6 +140,25 @@ def update_component_versions(dry_run: bool): pull_helm_chart() +def update_go_version(dry_run: bool): + k8s_version = (COMPONENTS / "kubernetes/version").read_text().strip() + url = f"https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/{k8s_version}/.go-version" + with urllib.request.urlopen(url) as response: + go_version = response.read().decode("utf-8").strip() + + LOG.info("Upstream go version is %s", go_version) + go_snap = f'go/{".".join(go_version.split(".")[:2])}/stable' + snapcraft_yaml = SNAPCRAFT.read_text() + if f"- {go_snap}" in snapcraft_yaml: + LOG.info("snapcraft.yaml already contains go version %s", go_snap) + return + + LOG.info("Update go snap version to %s in %s", go_snap, SNAPCRAFT) + if not dry_run: + updated = re.sub(r"- go/\d+\.\d+/stable", f"- {go_snap}", snapcraft_yaml) + SNAPCRAFT.write_text(updated) + + def main(): parser = argparse.ArgumentParser( "update-component-versions.py", usage=USAGE, description=DESCRIPTION diff --git a/build-scripts/hack/update-coredns-chart.sh b/build-scripts/hack/update-coredns-chart.sh new file mode 100755 index 000000000..04e4550f9 --- /dev/null +++ b/build-scripts/hack/update-coredns-chart.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +VERSION="1.36.0" +DIR=$(realpath $(dirname "${0}")) + +CHARTS_PATH="$DIR/../../k8s/manifests/charts" + +cd "$CHARTS_PATH" + +helm pull --repo https://coredns.github.io/helm coredns --version $VERSION diff --git a/build-scripts/hack/update-gateway-api-chart.sh b/build-scripts/hack/update-gateway-api-chart.sh index eb2b9a91b..40039c7ab 100755 --- a/build-scripts/hack/update-gateway-api-chart.sh +++ b/build-scripts/hack/update-gateway-api-chart.sh @@ -1,6 +1,6 @@ #!/bin/bash -VERSION="v1.0.0" +VERSION="v1.1.0" DIR=`realpath $(dirname "${0}")` CHARTS_PATH="$DIR/../../k8s/components/charts" @@ -16,7 +16,6 @@ rm -rf gateway-api/templates/* rm -rf gateway-api/charts cp gateway-api-src/config/crd/standard/* gateway-api/templates/ cp gateway-api-src/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml gateway-api/templates/ -cp gateway-api-src/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml gateway-api/templates/ sed -i 's/^\(version: \).*$/\1'"${VERSION:1}"'/' gateway-api/Chart.yaml sed -i 's/^\(appVersion: \).*$/\1'"${VERSION:1}"'/' gateway-api/Chart.yaml sed -i 's/^\(description: \).*$/\1'"A Helm Chart containing Gateway API CRDs"'/' gateway-api/Chart.yaml diff --git a/build-scripts/hack/update-metallb-chart.sh b/build-scripts/hack/update-metallb-chart.sh new file mode 100755 index 000000000..5f8feab3e --- /dev/null +++ b/build-scripts/hack/update-metallb-chart.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +VERSION="0.14.8" +DIR=$(realpath $(dirname "${0}")) + +CHARTS_PATH="$DIR/../../k8s/manifests/charts" + +cd "$CHARTS_PATH" + +helm pull --repo https://metallb.github.io/metallb metallb --version $VERSION diff --git a/build-scripts/hack/update-metrics-server-chart.sh b/build-scripts/hack/update-metrics-server-chart.sh index 7d8dc352b..873257a28 100755 --- a/build-scripts/hack/update-metrics-server-chart.sh +++ b/build-scripts/hack/update-metrics-server-chart.sh @@ -1,9 +1,9 @@ #!/bin/bash -VERSION="3.12.0" -DIR=`realpath $(dirname "${0}")` +VERSION="3.12.2" +DIR=$(realpath $(dirname "${0}")) -CHARTS_PATH="$DIR/../../k8s/components/charts" +CHARTS_PATH="$DIR/../../k8s/manifests/charts" cd "$CHARTS_PATH" diff --git a/build-scripts/hack/upstream-images.yaml b/build-scripts/hack/upstream-images.yaml index fbc74a75a..2f8b50f3c 100644 --- a/build-scripts/hack/upstream-images.yaml +++ b/build-scripts/hack/upstream-images.yaml @@ -49,11 +49,11 @@ sync: - source: quay.io/tigera/operator:v1.34.0 target: ghcr.io/canonical/k8s-snap/tigera/operator:v1.34.0 type: image - - source: quay.io/metallb/controller:v0.14.5 - target: ghcr.io/canonical/k8s-snap/metallb/controller:v0.14.5 + - source: quay.io/metallb/controller:v0.14.8 + target: ghcr.io/canonical/k8s-snap/metallb/controller:v0.14.8 type: image - - source: quay.io/metallb/speaker:v0.14.5 - target: ghcr.io/canonical/k8s-snap/metallb/speaker:v0.14.5 + - source: quay.io/metallb/speaker:v0.14.8 + target: ghcr.io/canonical/k8s-snap/metallb/speaker:v0.14.8 type: image - source: quay.io/frrouting/frr:9.0.2 target: ghcr.io/canonical/k8s-snap/frrouting/frr:9.0.2 diff --git a/build-scripts/patches/moonray/apply b/build-scripts/patches/moonray/apply index 1233dae42..32a2f8510 100755 --- a/build-scripts/patches/moonray/apply +++ b/build-scripts/patches/moonray/apply @@ -2,9 +2,12 @@ DIR="$(realpath "$(dirname "${0}")")" -# Configure git author -git config user.email k8s-bot@canonical.com -git config user.name k8s-bot +# Configure git author if unset +git_email=$(git config --default "" user.email) +if [ -z "${git_email}" ]; then + git config user.email k8s-team-ci@canonical.com + git config user.name 'k8s-team-ci (CDK Bot)' +fi # Remove unrelated tests rm "${DIR}/../../../tests/integration/tests/test_cilium_e2e.py" diff --git a/build-scripts/patches/strict/0001-Strict-patch.patch b/build-scripts/patches/strict/0001-Strict-patch.patch index 2ca7c50fa..bed2ed6ff 100644 --- a/build-scripts/patches/strict/0001-Strict-patch.patch +++ b/build-scripts/patches/strict/0001-Strict-patch.patch @@ -1,16 +1,17 @@ -From 3338580f4e22b001615320c40b1c1ad95f8a945e Mon Sep 17 00:00:00 2001 +From 94dadc0e3963e0b01af66e490500c619ec45c019 Mon Sep 17 00:00:00 2001 From: Angelos Kolaitis Date: Fri, 10 May 2024 19:17:55 +0300 Subject: [PATCH] Strict patch --- - k8s/hack/init.sh | 6 +- - k8s/wrappers/services/containerd | 5 - - snap/snapcraft.yaml | 168 ++++++++++++++++++++++++++++++- - 3 files changed, 172 insertions(+), 7 deletions(-) + k8s/hack/init.sh | 6 +- + k8s/wrappers/services/containerd | 5 - + snap/snapcraft.yaml | 171 +++++++++++++++++++++- + tests/integration/tests/test_util/util.py | 38 +++-- + 4 files changed, 198 insertions(+), 22 deletions(-) diff --git a/k8s/hack/init.sh b/k8s/hack/init.sh -index a0b57c7d..d53b528a 100755 +index a0b57c7..d53b528 100755 --- a/k8s/hack/init.sh +++ b/k8s/hack/init.sh @@ -1,3 +1,7 @@ @@ -23,7 +24,7 @@ index a0b57c7d..d53b528a 100755 +"${DIR}/connect-interfaces.sh" +"${DIR}/network-requirements.sh" diff --git a/k8s/wrappers/services/containerd b/k8s/wrappers/services/containerd -index c3f71a01..a82e1c03 100755 +index c3f71a0..a82e1c0 100755 --- a/k8s/wrappers/services/containerd +++ b/k8s/wrappers/services/containerd @@ -21,9 +21,4 @@ You can try to apply the profile manually by running: @@ -37,7 +38,7 @@ index c3f71a01..a82e1c03 100755 - k8s::common::execute_service containerd diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml -index 54b5fc0b..01631684 100644 +index 9d21e55..26f49ad 100644 --- a/snap/snapcraft.yaml +++ b/snap/snapcraft.yaml @@ -7,7 +7,7 @@ description: |- @@ -49,7 +50,7 @@ index 54b5fc0b..01631684 100644 base: core20 environment: REAL_PATH: $PATH -@@ -216,6 +216,20 @@ parts: +@@ -217,6 +217,20 @@ parts: apps: k8s: command: k8s/wrappers/commands/k8s @@ -70,7 +71,7 @@ index 54b5fc0b..01631684 100644 containerd: command: k8s/wrappers/services/containerd daemon: notify -@@ -226,43 +240,195 @@ apps: +@@ -227,43 +241,198 @@ apps: restart-condition: always start-timeout: 5m before: [kubelet] @@ -263,9 +264,61 @@ index 54b5fc0b..01631684 100644 + plugs: + - network + - network-bind -+ - process-control + - network-control ++ - network-observe ++ - process-control + - firewall-control ++ - system-observe ++ - mount-observe +diff --git a/tests/integration/tests/test_util/util.py b/tests/integration/tests/test_util/util.py +index 3e54d68..295c458 100644 +--- a/tests/integration/tests/test_util/util.py ++++ b/tests/integration/tests/test_util/util.py +@@ -191,21 +191,29 @@ def remove_k8s_snap(instance: harness.Instance): + ["snap", "remove", config.SNAP_NAME, "--purge"] + ) + +- LOG.info("Waiting for shims to go away...") +- stubbornly(retries=20, delay_s=5).on(instance).until( +- lambda p: all( +- x not in p.stdout.decode() +- for x in ["containerd-shim", "cilium", "coredns", "/pause"] +- ) +- ).exec(["ps", "-fea"]) +- +- LOG.info("Waiting for kubelet and containerd mounts to go away...") +- stubbornly(retries=20, delay_s=5).on(instance).until( +- lambda p: all( +- x not in p.stdout.decode() +- for x in ["/var/lib/kubelet/pods", "/run/containerd/io.containerd"] +- ) +- ).exec(["mount"]) ++ # NOTE(lpetrut): on "strict", the snap remove hook is unable to: ++ # * terminate processes ++ # * remove network namespaces ++ # * list mounts ++ # ++ # https://paste.ubuntu.com/p/WscCCfnvGH/plain/ ++ # https://paste.ubuntu.com/p/sSnJVvZkrr/plain/ ++ # ++ # LOG.info("Waiting for shims to go away...") ++ # stubbornly(retries=20, delay_s=5).on(instance).until( ++ # lambda p: all( ++ # x not in p.stdout.decode() ++ # for x in ["containerd-shim", "cilium", "coredns", "/pause"] ++ # ) ++ # ).exec(["ps", "-fea"]) ++ # ++ # LOG.info("Waiting for kubelet and containerd mounts to go away...") ++ # stubbornly(retries=20, delay_s=5).on(instance).until( ++ # lambda p: all( ++ # x not in p.stdout.decode() ++ # for x in ["/var/lib/kubelet/pods", "/run/containerd/io.containerd"] ++ # ) ++ # ).exec(["mount"]) + + # NOTE(neoaggelos): Temporarily disable this as it fails on strict. + # For details, `snap changes` then `snap change $remove_k8s_snap_change`. -- -2.34.1 +2.43.0 diff --git a/build-scripts/patches/strict/apply b/build-scripts/patches/strict/apply index 1729742e2..3f6f7de14 100755 --- a/build-scripts/patches/strict/apply +++ b/build-scripts/patches/strict/apply @@ -3,8 +3,11 @@ DIR="$(realpath "$(dirname "${0}")")" # Configure git author -git config user.email k8s-bot@canonical.com -git config user.name k8s-bot +git_email=$(git config --default "" user.email) +if [ -z "${git_email}" ]; then + git config user.email k8s-team-ci@canonical.com + git config user.name 'k8s-team-ci (CDK Bot)' +fi # Apply strict patch git am "${DIR}/0001-Strict-patch.patch" diff --git a/docs/README.md b/docs/README.md index 9c51c5e0f..f1af97305 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,7 @@ # K8s snap documentation -This part of the repository contains the tools and the source for generating documentation for the Canonical Kubernetes snap. +This part of the repository contains the tools and the source for generating +documentation for the Canonical Kubernetes snap. The directories are organised like this: @@ -11,16 +12,20 @@ The directories are organised like this: ├── README.md ├── src │ ├──{source files for the docs} -└── tools - ├──{sphinx build tools for creating the docs} +├── canonicalk8s +│ ├──{sphinx build tools for creating the docs for Canonical K8s} +├── moonray +│ ├──{sphinx build tools for creating the docs for Canonical K8s} ``` ## Building the docs -This documentation uses the /tools/Makefile to generate HTML docs from the sources. -This can also run specific local tests such as spelling and linkchecking. +This documentation uses the /canonicalk8s/Makefile to generate HTML docs from +the sources. This can also run specific local tests such as spelling and +linkchecking. ## Contributing to the docs -Contributions to this documentation are welcome. Generally these follow the same -rules and process as other contributions - modify the docs source and submit a PR. \ No newline at end of file +Contributions to this documentation are welcome. Generally these follow the +same rules and process as other contributions - modify the docs source and +submit a PR. diff --git a/docs/canonicalk8s/.gitignore b/docs/canonicalk8s/.gitignore new file mode 100644 index 000000000..da6f688f1 --- /dev/null +++ b/docs/canonicalk8s/.gitignore @@ -0,0 +1,14 @@ +/*env*/ +.sphinx/venv/ +.sphinx/warnings.txt +.sphinx/.wordlist.dic +.sphinx/.doctrees/ +.sphinx/node_modules/ +package*.json +_build +.DS_Store +__pycache__ +.idea/ +.vscode/ +.sphinx/styles/* +.sphinx/vale.ini \ No newline at end of file diff --git a/docs/canonicalk8s/.markdownlint.json b/docs/canonicalk8s/.markdownlint.json new file mode 100644 index 000000000..7c2088dbd --- /dev/null +++ b/docs/canonicalk8s/.markdownlint.json @@ -0,0 +1,16 @@ +{ + "default": false, + "MD003": { "style": "atx" }, + "MD013": { "code_blocks": false, "tables": false, "stern": true, "line_length": 80}, + "MD014": true, + "MD018": true, + "MD022": true, + "MD023": true, + "MD026": { "punctuation": ".,;。，；"}, + "MD031": { "list_items": false}, + "MD032": true, + "MD035": true, + "MD042": true, + "MD045": true, + "MD052": true +} diff --git a/docs/canonicalk8s/.readthedocs.yaml b/docs/canonicalk8s/.readthedocs.yaml new file mode 100644 index 000000000..6f30701e6 --- /dev/null +++ b/docs/canonicalk8s/.readthedocs.yaml @@ -0,0 +1,43 @@ +# .readthedocs.yaml +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details + +# Required +version: 2 + +# Set the version of Python and other tools you might need +build: + os: ubuntu-22.04 + tools: + python: "3.12" + jobs: + post_checkout: + # Cancel building pull requests when there aren't changed in the docs directory or YAML file. + # + # https://docs.readthedocs.io/en/latest/build-customization.html#cancel-build-based-on-a-condition + # If there are no changes (git diff exits with 0) we force the command to return with 183. + # This is a special exit code on Read the Docs that will cancel the build immediately. + - | + git fetch --unshallow || true + if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet origin/main -- docs/ .readthedocs.yaml; + then + exit 183; + fi + ls + +# Build documentation in the docs/ directory with Sphinx +sphinx: + builder: dirhtml + configuration: docs/canonicalk8s/conf-rtd.py + +# If using Sphinx, optionally build your docs in additional formats such as PDF +# formats: +# - pdf + +# Optionally declare the Python requirements required to build your docs +python: + install: + - requirements: docs/canonicalk8s/.sphinx/requirements.txt + + + diff --git a/docs/canonicalk8s/.sphinx/_static/404.svg b/docs/canonicalk8s/.sphinx/_static/404.svg new file mode 100644 index 000000000..b353cd339 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_static/404.svg @@ -0,0 +1,13 @@ + + + + + + + + + diff --git a/docs/canonicalk8s/.sphinx/_static/custom.css b/docs/canonicalk8s/.sphinx/_static/custom.css new file mode 100644 index 000000000..2b9e81fb1 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_static/custom.css @@ -0,0 +1,392 @@ +/** + Ubuntu variable font definitions. + Based on https://github.com/canonical/vanilla-framework/blob/main/scss/_base_fontfaces.scss + + When font files are updated in Vanilla, the links to font files will need to be updated here as well. +*/ + +/* default font set */ +@font-face { + font-family: 'Ubuntu variable'; + font-stretch: 100%; /* min and max value for the width axis, expressed as percentage */ + font-style: normal; + font-weight: 100 800; /* min and max value for the weight axis */ + src: url('https://assets.ubuntu.com/v1/f1ea362b-Ubuntu%5Bwdth,wght%5D-latin-v0.896a.woff2') format('woff2-variations'); +} + +@font-face { + font-family: 'Ubuntu variable'; + font-stretch: 100%; /* min and max value for the width axis, expressed as percentage */ + font-style: italic; + font-weight: 100 800; /* min and max value for the weight axis */ + src: url('https://assets.ubuntu.com/v1/90b59210-Ubuntu-Italic%5Bwdth,wght%5D-latin-v0.896a.woff2') format('woff2-variations'); +} + +@font-face { + font-family: 'Ubuntu Mono variable'; + font-style: normal; + font-weight: 100 800; /* min and max value for the weight axis */ + src: url('https://assets.ubuntu.com/v1/d5fc1819-UbuntuMono%5Bwght%5D-latin-v0.869.woff2') format('woff2-variations'); +} + +/* cyrillic-ext */ +@font-face { + font-family: 'Ubuntu variable'; + font-stretch: 100%; /* min and max value for the width axis, expressed as percentage */ + font-style: normal; + font-weight: 100 800; /* min and max value for the weight axis */ + src: url('https://assets.ubuntu.com/v1/77cd6650-Ubuntu%5Bwdth,wght%5D-cyrillic-extended-v0.896a.woff2') format('woff2-variations'); + unicode-range: U+0460-052F, U+20B4, U+2DE0-2DFF, U+A640-A69F; +} + +/* cyrillic */ +@font-face { + font-family: 'Ubuntu variable'; + font-stretch: 100%; /* min and max value for the width axis, expressed as percentage */ + font-style: normal; + font-weight: 100 800; /* min and max value for the weight axis */ + src: url('https://assets.ubuntu.com/v1/2702fce5-Ubuntu%5Bwdth,wght%5D-cyrillic-v0.896a.woff2') format('woff2-variations'); + unicode-range: U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116; +} + +/* greek-ext */ +@font-face { + font-family: 'Ubuntu variable'; + font-stretch: 100%; /* min and max value for the width axis, expressed as percentage */ + font-style: normal; + font-weight: 100 800; /* min and max value for the weight axis */ + src: url('https://assets.ubuntu.com/v1/5c108b7d-Ubuntu%5Bwdth,wght%5D-greek-extended-v0.896a.woff2') format('woff2-variations'); + unicode-range: U+1F00-1FFF; +} + +/* greek */ +@font-face { + font-family: 'Ubuntu variable'; + font-stretch: 100%; /* min and max value for the width axis, expressed as percentage */ + font-style: normal; + font-weight: 100 800; /* min and max value for the weight axis */ + src: url('https://assets.ubuntu.com/v1/0a14c405-Ubuntu%5Bwdth,wght%5D-greek-v0.896a.woff2') format('woff2-variations'); + unicode-range: U+0370-03FF; +} + +/* latin-ext */ +@font-face { + font-family: 'Ubuntu variable'; + font-stretch: 100%; /* min and max value for the width axis, expressed as percentage */ + font-style: normal; + font-weight: 100 800; /* min and max value for the weight axis */ + src: url('https://assets.ubuntu.com/v1/19f68eeb-Ubuntu%5Bwdth,wght%5D-latin-extended-v0.896a.woff2') format('woff2-variations'); + unicode-range: U+0100-024F, U+1E00-1EFF, U+20A0-20AB, U+20AD-20CF, U+2C60-2C7F, U+A720-A7FF; +} + + +/** Define font-weights as per Vanilla + Based on: https://github.com/canonical/vanilla-framework/blob/main/scss/_base_typography-definitions.scss + + regular text: 400, + bold: 550, + thin: 300, + + h1: bold, + h2: 180; + h3: bold, + h4: 275, + h5: bold, + h6: regular +*/ + +/* default regular text */ +html { + font-weight: 400; +} + +/* heading specific definitions */ +h1, h3, h5 { font-weight: 550; } +h2 { font-weight: 180; } +h4 { font-weight: 275; } + +/* bold */ +.toc-tree li.scroll-current>.reference, +dl.glossary dt, +dl.simple dt, +dl:not([class]) dt { + font-weight: 550; +} + + +/** Table styling **/ + +th.head { + text-transform: uppercase; + font-size: var(--font-size--small); + text-align: initial; +} + +table.align-center th.head { + text-align: center +} + +table.docutils { + border: 0; + box-shadow: none; + width:100%; +} + +table.docutils td, table.docutils th, table.docutils td:last-child, table.docutils th:last-child, table.docutils td:first-child, table.docutils th:first-child { + border-right: none; + border-left: none; +} + +/* Allow to centre text horizontally in table data cells */ +table.align-center { + text-align: center !important; +} + +/** No rounded corners **/ + +.admonition, code.literal, .sphinx-tabs-tab, .sphinx-tabs-panel, .highlight { + border-radius: 0; +} + +/** Admonition styling **/ + +.admonition { + border-top: 1px solid #d9d9d9; + border-right: 1px solid #d9d9d9; + border-bottom: 1px solid #d9d9d9; +} + +/** Color for the "copy link" symbol next to headings **/ + +a.headerlink { + color: var(--color-brand-primary); +} + +/** Line to the left of the current navigation entry **/ + +.sidebar-tree li.current-page { + border-left: 2px solid var(--color-brand-primary); +} + +/** Some tweaks for Sphinx tabs **/ + +[role="tablist"] { + border-bottom: 1px solid var(--color-sidebar-item-background--hover); +} + +.sphinx-tabs-tab[aria-selected="true"], .sd-tab-set>input:checked+label{ + border: 0; + border-bottom: 2px solid var(--color-brand-primary); + font-weight: 400; + font-size: 1rem; + color: var(--color-brand-primary); +} + +body[data-theme="dark"] .sphinx-tabs-tab[aria-selected="true"] { + background: var(--color-background-primary); + border-bottom: 2px solid var(--color-brand-primary); +} + +button.sphinx-tabs-tab[aria-selected="false"]:hover, .sd-tab-set>input:not(:checked)+label:hover { + border-bottom: 2px solid var(--color-foreground-border); +} + +button.sphinx-tabs-tab[aria-selected="false"]{ + border-bottom: 2px solid var(--color-background-primary); +} + +body[data-theme="dark"] .sphinx-tabs-tab { + background: var(--color-background-primary); +} + +.sphinx-tabs-tab, .sd-tab-set>label{ + color: var(--color-brand-primary); + font-family: var(--font-stack); + font-weight: 400; + font-size: 1rem; + padding: 1em 1.25em .5em +} + +.sphinx-tabs-panel { + border: 0; + border-bottom: 1px solid var(--color-sidebar-item-background--hover); + background: var(--color-background-primary); + padding: 0.75rem 0 0.75rem 0; +} + +body[data-theme="dark"] .sphinx-tabs-panel { + background: var(--color-background-primary); +} + +/** A tweak for issue #190 **/ + +.highlight .hll { + background-color: var(--color-highlighted-background); +} + + +/** Custom classes to fix scrolling in tables by decreasing the + font size or breaking certain columns. + Specify the classes in the Markdown file with, for example: + ```{rst-class} break-col-4 min-width-4-8 + ``` +**/ + +table.dec-font-size { + font-size: smaller; +} +table.break-col-1 td.text-left:first-child { + word-break: break-word; +} +table.break-col-4 td.text-left:nth-child(4) { + word-break: break-word; +} +table.min-width-1-15 td.text-left:first-child { + min-width: 15em; +} +table.min-width-4-8 td.text-left:nth-child(4) { + min-width: 8em; +} + +/** Underline for abbreviations **/ + +abbr[title] { + text-decoration: underline solid #cdcdcd; +} + +/** Use the same style for right-details as for left-details **/ +.bottom-of-page .right-details { + font-size: var(--font-size--small); + display: block; +} + +/** Version switcher */ +button.version_select { + color: var(--color-foreground-primary); + background-color: var(--color-toc-background); + padding: 5px 10px; + border: none; +} + +.version_select:hover, .version_select:focus { + background-color: var(--color-sidebar-item-background--hover); +} + +.version_dropdown { + position: relative; + display: inline-block; + text-align: right; + font-size: var(--sidebar-item-font-size); +} + +.available_versions { + display: none; + position: absolute; + right: 0px; + background-color: var(--color-toc-background); + box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.2); + z-index: 11; +} + +.available_versions a { + color: var(--color-foreground-primary); + padding: 12px 16px; + text-decoration: none; + display: block; +} + +.available_versions a:hover {background-color: var(--color-sidebar-item-background--current)} + +/** Suppress link underlines outside on-hover **/ +a { + text-decoration: none; +} + +a:hover, a:visited:hover { + text-decoration: underline; +} + +.show {display:block;} + +/** Fix for nested numbered list - the nested list is lettered **/ +ol.arabic ol.arabic { + list-style: lower-alpha; +} + +/** Make expandable sections look like links **/ +details summary { + color: var(--color-link); +} + +/** Fix the styling of the version box for readthedocs **/ + +#furo-readthedocs-versions .rst-versions, #furo-readthedocs-versions .rst-current-version, #furo-readthedocs-versions:focus-within .rst-current-version, #furo-readthedocs-versions:hover .rst-current-version { + background: var(--color-sidebar-item-background--hover); +} + +.rst-versions .rst-other-versions dd a { + color: var(--color-link); +} + +#furo-readthedocs-versions:focus-within .rst-current-version .fa-book, #furo-readthedocs-versions:hover .rst-current-version .fa-book, .rst-versions .rst-other-versions { + color: var(--color-sidebar-link-text); +} + +.rst-versions .rst-current-version { + color: var(--color-version-popup); + font-weight: bolder; +} + +/* Code-block copybutton invisible by default + (overriding Furo config to achieve default copybutton setting). */ +.highlight button.copybtn { + opacity: 0; +} + +/* Mimicking the 'Give feedback' button for UX consistency */ +.sidebar-search-container input[type=submit] { + color: #FFFFFF; + border: 2px solid #D6410D; + padding: var(--sidebar-search-input-spacing-vertical) var(--sidebar-search-input-spacing-horizontal); + background: #D6410D; + font-weight: bold; + font-size: var(--font-size--small); + cursor: pointer; +} + +.sidebar-search-container input[type=submit]:hover { + text-decoration: underline; +} + +/* Make inline code the same size as code blocks */ +p code.literal { + border: 0; + font-size: var(--code-font-size); +} + +/* Use the general admonition font size for inline code */ +.admonition p code.literal { + font-size: var(--admonition-font-size); +} + +.highlight .s, .highlight .s1, .highlight .s2 { + color: #3F8100; +} + +.highlight .o { + color: #BB5400; +} + +.rubric > .hclass2 { + display: block; + font-size: 2em; + border-radius: .5rem; + font-weight: 300; + line-height: 1.25; + margin-top: 1.75rem; + margin-right: -0.5rem; + margin-bottom: 0.5rem; + margin-left: -0.5rem; + padding-left: .5rem; + padding-right: .5rem; +} \ No newline at end of file diff --git a/docs/canonicalk8s/.sphinx/_static/favicon.png b/docs/canonicalk8s/.sphinx/_static/favicon.png new file mode 100644 index 000000000..7f175e461 Binary files /dev/null and b/docs/canonicalk8s/.sphinx/_static/favicon.png differ diff --git a/docs/canonicalk8s/.sphinx/_static/footer.css b/docs/canonicalk8s/.sphinx/_static/footer.css new file mode 100644 index 000000000..a0a1db454 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_static/footer.css @@ -0,0 +1,47 @@ +.display-contributors { + color: var(--color-sidebar-link-text); + cursor: pointer; +} +.all-contributors { + display: none; + z-index: 55; + list-style: none; + position: fixed; + top: 0; + bottom: 0; + left: 0; + right: 0; + width: 200px; + height: 200px; + overflow-y: scroll; + margin: auto; + padding: 0; + background: var(--color-background-primary); + scrollbar-color: var(--color-foreground-border) transparent; + scrollbar-width: thin; +} + +.all-contributors li:hover { + background: var(--color-sidebar-item-background--hover); + width: 100%; +} + +.all-contributors li a{ + color: var(--color-sidebar-link-text); + padding: 1rem; + display: inline-block; +} + +#overlay { + position: fixed; + display: none; + width: 100%; + height: 100%; + top: 0; + left: 0; + right: 0; + bottom: 0; + background-color: rgba(0,0,0,0.5); + z-index: 2; + cursor: pointer; +} diff --git a/docs/canonicalk8s/.sphinx/_static/footer.js b/docs/canonicalk8s/.sphinx/_static/footer.js new file mode 100644 index 000000000..9a08b1e99 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_static/footer.js @@ -0,0 +1,12 @@ +$(document).ready(function() { + $(document).on("click", function () { + $(".all-contributors").hide(); + $("#overlay").hide(); + }); + + $('.display-contributors').click(function(event) { + $('.all-contributors').toggle(); + $("#overlay").toggle(); + event.stopPropagation(); + }); +}) diff --git a/docs/canonicalk8s/.sphinx/_static/furo_colors.css b/docs/canonicalk8s/.sphinx/_static/furo_colors.css new file mode 100644 index 000000000..4cfdbe7bf --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_static/furo_colors.css @@ -0,0 +1,89 @@ +body { + --color-code-background: #f8f8f8; + --color-code-foreground: black; + --code-font-size: 1rem; + --font-stack: Ubuntu variable, Ubuntu, -apple-system, Segoe UI, Roboto, Oxygen, Cantarell, Fira Sans, Droid Sans, Helvetica Neue, sans-serif; + --font-stack--monospace: Ubuntu Mono variable, Ubuntu Mono, Consolas, Monaco, Courier, monospace; + --color-foreground-primary: #111; + --color-foreground-secondary: var(--color-foreground-primary); + --color-foreground-muted: #333; + --color-background-secondary: #FFF; + --color-background-hover: #f2f2f2; + --color-brand-primary: #111; + --color-brand-content: #06C; + --color-api-background: #E3E3E3; + --color-inline-code-background: rgba(0,0,0,.03); + --color-sidebar-link-text: #111; + --color-sidebar-item-background--current: #ebebeb; + --color-sidebar-item-background--hover: #f2f2f2; + --toc-font-size: var(--font-size--small); + --color-admonition-title-background--note: var(--color-background-primary); + --color-admonition-title-background--tip: var(--color-background-primary); + --color-admonition-title-background--important: var(--color-background-primary); + --color-admonition-title-background--caution: var(--color-background-primary); + --color-admonition-title--note: #24598F; + --color-admonition-title--tip: #24598F; + --color-admonition-title--important: #C7162B; + --color-admonition-title--caution: #F99B11; + --color-highlighted-background: #EBEBEB; + --color-link-underline: var(--color-link); + --color-link-underline--hover: var(--color-link); + --color-link-underline--visited: var(--color-link--visited); + --color-link-underline--visited--hover: var(--color-link--visited); + --color-version-popup: #772953; +} + +@media not print { + body[data-theme="dark"] { + --color-code-background: #202020; + --color-code-foreground: #d0d0d0; + --color-foreground-secondary: var(--color-foreground-primary); + --color-foreground-muted: #CDCDCD; + --color-background-secondary: var(--color-background-primary); + --color-background-hover: #666; + --color-brand-primary: #fff; + --color-brand-content: #69C; + --color-sidebar-link-text: #f7f7f7; + --color-sidebar-item-background--current: #666; + --color-sidebar-item-background--hover: #333; + --color-admonition-background: transparent; + --color-admonition-title-background--note: var(--color-background-primary); + --color-admonition-title-background--tip: var(--color-background-primary); + --color-admonition-title-background--important: var(--color-background-primary); + --color-admonition-title-background--caution: var(--color-background-primary); + --color-admonition-title--note: #24598F; + --color-admonition-title--tip: #24598F; + --color-admonition-title--important: #C7162B; + --color-admonition-title--caution: #F99B11; + --color-highlighted-background: #666; + --color-version-popup: #F29879; + } + @media (prefers-color-scheme: dark) { + body:not([data-theme="light"]) { + --color-api-background: #A4A4A4; + --color-code-background: #202020; + --color-code-foreground: #d0d0d0; + --color-foreground-secondary: var(--color-foreground-primary); + --color-foreground-muted: #CDCDCD; + --color-background-secondary: var(--color-background-primary); + --color-background-hover: #666; + --color-brand-primary: #fff; + --color-brand-content: #69C; + --color-sidebar-link-text: #f7f7f7; + --color-sidebar-item-background--current: #666; + --color-sidebar-item-background--hover: #333; + --color-admonition-background: transparent; + --color-admonition-title-background--note: var(--color-background-primary); + --color-admonition-title-background--tip: var(--color-background-primary); + --color-admonition-title-background--important: var(--color-background-primary); + --color-admonition-title-background--caution: var(--color-background-primary); + --color-admonition-title--note: #24598F; + --color-admonition-title--tip: #24598F; + --color-admonition-title--important: #C7162B; + --color-admonition-title--caution: #F99B11; + --color-highlighted-background: #666; + --color-link: #F9FCFF; + --color-version-popup: #F29879; + } + } +} diff --git a/docs/canonicalk8s/.sphinx/_static/github_issue_links.css b/docs/canonicalk8s/.sphinx/_static/github_issue_links.css new file mode 100644 index 000000000..db166ed95 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_static/github_issue_links.css @@ -0,0 +1,24 @@ +.github-issue-link-container { + padding-right: 0.5rem; +} +.github-issue-link { + font-size: var(--font-size--small); + font-weight: bold; + background-color: #D6410D; + padding: 13px 23px; + text-decoration: none; +} +.github-issue-link:link { + color: #FFFFFF; +} +.github-issue-link:visited { + color: #FFFFFF +} +.muted-link.github-issue-link:hover { + color: #FFFFFF; + text-decoration: underline; +} +.github-issue-link:active { + color: #FFFFFF; + text-decoration: underline; +} diff --git a/docs/canonicalk8s/.sphinx/_static/github_issue_links.js b/docs/canonicalk8s/.sphinx/_static/github_issue_links.js new file mode 100644 index 000000000..f0706038b --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_static/github_issue_links.js @@ -0,0 +1,34 @@ +// if we already have an onload function, save that one +var prev_handler = window.onload; + +window.onload = function() { + // call the previous onload function + if (prev_handler) { + prev_handler(); + } + + const link = document.createElement("a"); + link.classList.add("muted-link"); + link.classList.add("github-issue-link"); + link.text = "Give feedback"; + link.href = ( + github_url + + "/issues/new?" + + "title=docs%3A+TYPE+YOUR+QUESTION+HERE" + + "&body=*Please describe the question or issue you're facing with " + + `"${document.title}"` + + ".*" + + "%0A%0A%0A%0A%0A" + + "---" + + "%0A" + + `*Reported+from%3A+${location.href}*` + ); + link.target = "_blank"; + + const div = document.createElement("div"); + div.classList.add("github-issue-link-container"); + div.append(link) + + const container = document.querySelector(".article-container > .content-icon-container"); + container.prepend(div); +}; diff --git a/docs/canonicalk8s/.sphinx/_static/header-nav.js b/docs/canonicalk8s/.sphinx/_static/header-nav.js new file mode 100644 index 000000000..3608576e0 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_static/header-nav.js @@ -0,0 +1,10 @@ +$(document).ready(function() { + $(document).on("click", function () { + $(".more-links-dropdown").hide(); + }); + + $('.nav-more-links').click(function(event) { + $('.more-links-dropdown').toggle(); + event.stopPropagation(); + }); +}) diff --git a/docs/canonicalk8s/.sphinx/_static/header.css b/docs/canonicalk8s/.sphinx/_static/header.css new file mode 100644 index 000000000..0b9440903 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_static/header.css @@ -0,0 +1,167 @@ +.p-navigation { + border-bottom: 1px solid var(--color-sidebar-background-border); +} + +.p-navigation__nav { + background: #333333; + display: flex; +} + +.p-logo { + display: flex !important; + padding-top: 0 !important; + text-decoration: none; +} + +.p-logo-image { + height: 44px; + padding-right: 10px; +} + +.p-logo-text { + margin-top: 18px; + color: white; + text-decoration: none; +} + +ul.p-navigation__links { + display: flex; + list-style: none; + margin-left: 0; + margin-top: auto; + margin-bottom: auto; + max-width: 800px; + width: 100%; +} + +ul.p-navigation__links li { + margin: 0 auto; + text-align: center; + width: 100%; +} + +ul.p-navigation__links li a { + background-color: rgba(0, 0, 0, 0); + border: none; + border-radius: 0; + color: var(--color-sidebar-link-text); + display: block; + font-weight: 400; + line-height: 1.5rem; + margin: 0; + overflow: hidden; + padding: 1rem 0; + position: relative; + text-align: left; + text-overflow: ellipsis; + transition-duration: .1s; + transition-property: background-color, color, opacity; + transition-timing-function: cubic-bezier(0.215, 0.61, 0.355, 1); + white-space: nowrap; + width: 100%; +} + +ul.p-navigation__links .p-navigation__link { + color: #ffffff; + font-weight: 300; + text-align: center; + text-decoration: none; +} + +ul.p-navigation__links .p-navigation__link:hover { + background-color: #2b2b2b; +} + +ul.p-navigation__links .p-dropdown__link:hover { + background-color: var(--color-sidebar-item-background--hover); +} + +ul.p-navigation__links .p-navigation__sub-link { + background: var(--color-background-primary); + padding: .5rem 0 .5rem .5rem; + font-weight: 300; +} + +ul.p-navigation__links .more-links-dropdown li a { + border-left: 1px solid var(--color-sidebar-background-border); + border-right: 1px solid var(--color-sidebar-background-border); +} + +ul.p-navigation__links .more-links-dropdown li:first-child a { + border-top: 1px solid var(--color-sidebar-background-border); +} + +ul.p-navigation__links .more-links-dropdown li:last-child a { + border-bottom: 1px solid var(--color-sidebar-background-border); +} + +ul.p-navigation__links .p-navigation__logo { + padding: 0.5rem; +} + +ul.p-navigation__links .p-navigation__logo img { + width: 40px; +} + +ul.more-links-dropdown { + display: none; + overflow-x: visible; + height: 0; + z-index: 55; + padding: 0; + position: relative; + list-style: none; + margin-bottom: 0; + margin-top: 0; +} + +.nav-more-links::after { + background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='16' height='16'%3E%3Cpath fill='%23111' d='M8.187 11.748l6.187-6.187-1.06-1.061-5.127 5.127L3.061 4.5 2 5.561z'/%3E%3C/svg%3E"); + background-position: center; + background-repeat: no-repeat; + background-size: contain; + content: ""; + display: block; + filter: invert(100%); + height: 1rem; + pointer-events: none; + position: absolute; + right: 1rem; + text-indent: calc(100% + 10rem); + top: calc(1rem + 0.25rem); + width: 1rem; +} + +.nav-ubuntu-com { + display: none; +} + +@media only screen and (min-width: 480px) { + ul.p-navigation__links li { + width: 100%; + } + + .nav-ubuntu-com { + display: inherit; + } +} + +@media only screen and (max-width: 800px) { + .nav-more-links { + margin-left: auto !important; + padding-right: 2rem !important; + width: 8rem !important; + } +} + +@media only screen and (min-width: 800px) { + ul.p-navigation__links li { + width: 100% !important; + } +} + +@media only screen and (min-width: 1310px) { + ul.p-navigation__links { + margin-left: calc(50% - 41em); + } +} diff --git a/docs/canonicalk8s/.sphinx/_static/tag.png b/docs/canonicalk8s/.sphinx/_static/tag.png new file mode 100644 index 000000000..f6f6e5aa4 Binary files /dev/null and b/docs/canonicalk8s/.sphinx/_static/tag.png differ diff --git a/docs/canonicalk8s/.sphinx/_templates/404.html b/docs/canonicalk8s/.sphinx/_templates/404.html new file mode 100644 index 000000000..4cb2d50d3 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_templates/404.html @@ -0,0 +1,17 @@ +{% extends "page.html" %} + +{% block content -%} +
+

Page not found

+
+
+
+ {{ body }} +
+
+ Penguin with a question mark +
+
+
+
+{%- endblock content %} diff --git a/docs/canonicalk8s/.sphinx/_templates/base.html b/docs/canonicalk8s/.sphinx/_templates/base.html new file mode 100644 index 000000000..33081547c --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_templates/base.html @@ -0,0 +1,12 @@ +{% extends "furo/base.html" %} + +{% block theme_scripts %} + +{% endblock theme_scripts %} + +{# ru-fu: don't include the color variables from the conf.py file, but use a + separate CSS file to save space #} +{% block theme_styles %} +{% endblock theme_styles %} diff --git a/docs/canonicalk8s/.sphinx/_templates/footer.html b/docs/canonicalk8s/.sphinx/_templates/footer.html new file mode 100644 index 000000000..6839f0154 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_templates/footer.html @@ -0,0 +1,131 @@ +{# ru-fu: copied from Furo, with modifications as stated below. Modifications are marked 'mod:'. #} + +
+ {# mod: Per-page navigation #} + {% if meta %} + {% if 'sequential_nav' in meta %} + {% set sequential_nav = meta.sequential_nav %} + {% endif %} + {% endif %} + {# mod: Conditional wrappers to control page navigation buttons #} + {% if sequential_nav != "none" -%} + {% if next and (sequential_nav == "next" or sequential_nav == "both") -%} + +
+
+ {{ _("Next") }} +
+
{{ next.title }}
+
+ + + {%- endif %} + {% if prev and (sequential_nav == "prev" or sequential_nav == "both") -%} + + +
+
+ {{ _("Previous") }} +
+ {% if prev.link == pathto(root_doc) %} +
{{ _("Home") }}
+ {% else %} +
{{ prev.title }}
+ {% endif %} +
+ + {%- endif %} + {%- endif %} +
+
+
+ {%- if show_copyright %} +
+ {%- if hasdoc('copyright') %} + {% trans path=pathto('copyright'), copyright=copyright|e -%} + Copyright © {{ copyright }} + {%- endtrans %} + {%- else %} + {% trans copyright=copyright|e -%} + Copyright © {{ copyright }} + {%- endtrans %} + {%- endif %} +
+ {%- endif %} + + {# mod: removed "Made with" #} + + {%- if last_updated -%} +
+ {% trans last_updated=last_updated|e -%} + Last updated on {{ last_updated }} + {%- endtrans -%} +
+ {%- endif %} + + {%- if show_source and has_source and sourcename %} +
+ Show source +
+ {%- endif %} +
+
+ {% if github_url and github_folder and pagename and page_source_suffix and display_contributors %} + {% set contributors = get_contribs(github_url, github_folder, pagename, page_source_suffix, display_contributors_since) %} + {% if contributors %} + {% if contributors | length > 1 %} + Thanks to the {{ contributors |length }} contributors! + {% else %} + Thanks to our contributor! + {% endif %} +
+ + {% endif %} + {% endif %} +
+
+ + {# mod: replaced RTD icons with our links #} + + {% if discourse %} +
+ Ask a question on Discourse +
+ {% endif %} + + {% if mattermost %} +
+ Ask a question on Mattermost +
+ {% endif %} + + {% if matrix %} +
+ Ask a question on Matrix +
+ {% endif %} + + {% if github_url and github_version and github_folder %} + + {% if github_issues %} +
+ Open a GitHub issue for this page +
+ {% endif %} + +
+ Edit this page on GitHub +
+ {% endif %} + + +
+
+ diff --git a/docs/canonicalk8s/.sphinx/_templates/header.html b/docs/canonicalk8s/.sphinx/_templates/header.html new file mode 100644 index 000000000..1a128b6f8 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_templates/header.html @@ -0,0 +1,36 @@ + diff --git a/docs/canonicalk8s/.sphinx/_templates/page.html b/docs/canonicalk8s/.sphinx/_templates/page.html new file mode 100644 index 000000000..bda306109 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_templates/page.html @@ -0,0 +1,49 @@ +{% extends "furo/page.html" %} + +{% block footer %} + {% include "footer.html" %} +{% endblock footer %} + +{% block body -%} + {% include "header.html" %} + {{ super() }} +{%- endblock body %} + +{% if meta and ((meta.discourse and discourse_prefix) or meta.relatedlinks) %} + {% set furo_hide_toc_orig = furo_hide_toc %} + {% set furo_hide_toc=false %} +{% endif %} + +{% block right_sidebar %} +
+ {% if not furo_hide_toc_orig %} +
+ + {{ _("Contents") }} + +
+
+
+ {{ toc }} +
+
+ {% endif %} + {% if meta and ((meta.discourse and discourse_prefix) or meta.relatedlinks) %} +
+ + Related links + +
+
+
+ {% if meta.discourse and discourse_prefix %} + {{ discourse_links(meta.discourse) }} + {% endif %} + {% if meta.relatedlinks %} + {{ related_links(meta.relatedlinks) }} + {% endif %} +
+
+ {% endif %} +
+{% endblock right_sidebar %} diff --git a/docs/canonicalk8s/.sphinx/_templates/sidebar/search.html b/docs/canonicalk8s/.sphinx/_templates/sidebar/search.html new file mode 100644 index 000000000..644a5ef6a --- /dev/null +++ b/docs/canonicalk8s/.sphinx/_templates/sidebar/search.html @@ -0,0 +1,7 @@ + + diff --git a/docs/canonicalk8s/.sphinx/build_requirements.py b/docs/canonicalk8s/.sphinx/build_requirements.py new file mode 100644 index 000000000..df6f149b4 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/build_requirements.py @@ -0,0 +1,127 @@ +import sys + +sys.path.append('./') +from custom_conf import * + +# The file contains helper functions and the mechanism to build the +# .sphinx/requirements.txt file that is needed to set up the virtual +# environment. + +# You should not do any modifications to this file. Put your custom +# requirements into the custom_required_modules array in the custom_conf.py +# file. If you need to change this file, contribute the changes upstream. + +legacyCanonicalSphinxExtensionNames = [ + "youtube-links", + "related-links", + "custom-rst-roles", + "terminal-output" + ] + +def IsAnyCanonicalSphinxExtensionUsed(): + for extension in custom_extensions: + if (extension.startswith("canonical.") or + extension in legacyCanonicalSphinxExtensionNames): + return True + + return False + +def IsNotFoundExtensionUsed(): + return "notfound.extension" in custom_extensions + +def IsSphinxTabsUsed(): + for extension in custom_extensions: + if extension.startswith("sphinx_tabs."): + return True + + return False + +def AreRedirectsDefined(): + return ("sphinx_reredirects" in custom_extensions) or ( + ("redirects" in globals()) and \ + (redirects is not None) and \ + (len(redirects) > 0)) + +def IsOpenGraphConfigured(): + if "sphinxext.opengraph" in custom_extensions: + return True + + for global_variable_name in list(globals()): + if global_variable_name.startswith("ogp_"): + return True + + return False + +def IsMyStParserUsed(): + return ("myst_parser" in custom_extensions) or \ + ("custom_myst_extensions" in globals()) + +def DeduplicateExtensions(extensionNames: [str]): + extensionNames = dict.fromkeys(extensionNames) + resultList = [] + encounteredCanonicalExtensions = [] + + for extensionName in extensionNames: + if extensionName in legacyCanonicalSphinxExtensionNames: + extensionName = "canonical." + extensionName + + if extensionName.startswith("canonical."): + if extensionName not in encounteredCanonicalExtensions: + encounteredCanonicalExtensions.append(extensionName) + resultList.append(extensionName) + else: + resultList.append(extensionName) + + return resultList + +if __name__ == "__main__": + requirements = [ + "furo", + "pyspelling", + "sphinx", + "sphinx-autobuild", + "sphinx-copybutton", + "sphinx-design", + "sphinxcontrib-jquery", + "watchfiles", + "GitPython" + + ] + + requirements.extend(custom_required_modules) + + if IsAnyCanonicalSphinxExtensionUsed(): + requirements.append("canonical-sphinx-extensions") + + if IsNotFoundExtensionUsed(): + requirements.append("sphinx-notfound-page") + + if IsSphinxTabsUsed(): + requirements.append("sphinx-tabs") + + if AreRedirectsDefined(): + requirements.append("sphinx-reredirects") + + if IsOpenGraphConfigured(): + requirements.append("sphinxext-opengraph") + + if IsMyStParserUsed(): + requirements.append("myst-parser") + requirements.append("linkify-it-py") + + # removes duplicate entries + requirements = list(dict.fromkeys(requirements)) + requirements.sort() + + with open(".sphinx/requirements.txt", 'w') as requirements_file: + requirements_file.write( + "# DO NOT MODIFY THIS FILE DIRECTLY!\n" + "#\n" + "# This file is generated automatically.\n" + "# Add custom requirements to the custom_required_modules\n" + "# array in the custom_conf.py file and run:\n" + "# make clean && make install\n") + + for requirement in requirements: + requirements_file.write(requirement) + requirements_file.write('\n') diff --git a/docs/canonicalk8s/.sphinx/fonts/Ubuntu-B.ttf b/docs/canonicalk8s/.sphinx/fonts/Ubuntu-B.ttf new file mode 100644 index 000000000..b173da274 Binary files /dev/null and b/docs/canonicalk8s/.sphinx/fonts/Ubuntu-B.ttf differ diff --git a/docs/canonicalk8s/.sphinx/fonts/Ubuntu-R.ttf b/docs/canonicalk8s/.sphinx/fonts/Ubuntu-R.ttf new file mode 100644 index 000000000..d748728a2 Binary files /dev/null and b/docs/canonicalk8s/.sphinx/fonts/Ubuntu-R.ttf differ diff --git a/docs/canonicalk8s/.sphinx/fonts/Ubuntu-RI.ttf b/docs/canonicalk8s/.sphinx/fonts/Ubuntu-RI.ttf new file mode 100644 index 000000000..4f2d2bc7c Binary files /dev/null and b/docs/canonicalk8s/.sphinx/fonts/Ubuntu-RI.ttf differ diff --git a/docs/canonicalk8s/.sphinx/fonts/UbuntuMono-B.ttf b/docs/canonicalk8s/.sphinx/fonts/UbuntuMono-B.ttf new file mode 100644 index 000000000..7bd666576 Binary files /dev/null and b/docs/canonicalk8s/.sphinx/fonts/UbuntuMono-B.ttf differ diff --git a/docs/canonicalk8s/.sphinx/fonts/UbuntuMono-R.ttf b/docs/canonicalk8s/.sphinx/fonts/UbuntuMono-R.ttf new file mode 100644 index 000000000..fdd309d71 Binary files /dev/null and b/docs/canonicalk8s/.sphinx/fonts/UbuntuMono-R.ttf differ diff --git a/docs/canonicalk8s/.sphinx/fonts/UbuntuMono-RI.ttf b/docs/canonicalk8s/.sphinx/fonts/UbuntuMono-RI.ttf new file mode 100644 index 000000000..18f81a292 Binary files /dev/null and b/docs/canonicalk8s/.sphinx/fonts/UbuntuMono-RI.ttf differ diff --git a/docs/canonicalk8s/.sphinx/fonts/ubuntu-font-licence-1.0.txt b/docs/canonicalk8s/.sphinx/fonts/ubuntu-font-licence-1.0.txt new file mode 100644 index 000000000..ae78a8f94 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/fonts/ubuntu-font-licence-1.0.txt @@ -0,0 +1,96 @@ +------------------------------- +UBUNTU FONT LICENCE Version 1.0 +------------------------------- + +PREAMBLE +This licence allows the licensed fonts to be used, studied, modified and +redistributed freely. The fonts, including any derivative works, can be +bundled, embedded, and redistributed provided the terms of this licence +are met. The fonts and derivatives, however, cannot be released under +any other licence. The requirement for fonts to remain under this +licence does not require any document created using the fonts or their +derivatives to be published under this licence, as long as the primary +purpose of the document is not to be a vehicle for the distribution of +the fonts. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this licence and clearly marked as such. This may +include source files, build scripts and documentation. + +"Original Version" refers to the collection of Font Software components +as received under this licence. + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to +a new environment. + +"Copyright Holder(s)" refers to all individuals and companies who have a +copyright ownership of the Font Software. + +"Substantially Changed" refers to Modified Versions which can be easily +identified as dissimilar to the Font Software by users of the Font +Software comparing the Original Version with the Modified Version. + +To "Propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification and with or without charging +a redistribution fee), making available to the public, and in some +countries other activities as well. + +PERMISSION & CONDITIONS +This licence does not grant any rights under trademark law and all such +rights are reserved. + +Permission is hereby granted, free of charge, to any person obtaining a +copy of the Font Software, to propagate the Font Software, subject to +the below conditions: + +1) Each copy of the Font Software must contain the above copyright +notice and this licence. These can be included either as stand-alone +text files, human-readable headers or in the appropriate machine- +readable metadata fields within text or binary files as long as those +fields can be easily viewed by the user. + +2) The font name complies with the following: +(a) The Original Version must retain its name, unmodified. +(b) Modified Versions which are Substantially Changed must be renamed to +avoid use of the name of the Original Version or similar names entirely. +(c) Modified Versions which are not Substantially Changed must be +renamed to both (i) retain the name of the Original Version and (ii) add +additional naming elements to distinguish the Modified Version from the +Original Version. The name of such Modified Versions must be the name of +the Original Version, with "derivative X" where X represents the name of +the new work, appended to that name. + +3) The name(s) of the Copyright Holder(s) and any contributor to the +Font Software shall not be used to promote, endorse or advertise any +Modified Version, except (i) as required by this licence, (ii) to +acknowledge the contribution(s) of the Copyright Holder(s) or (iii) with +their explicit written permission. + +4) The Font Software, modified or unmodified, in part or in whole, must +be distributed entirely under this licence, and must not be distributed +under any other licence. The requirement for fonts to remain under this +licence does not affect any document created using the Font Software, +except any version of the Font Software extracted from a document +created using the Font Software may only be distributed under this +licence. + +TERMINATION +This licence becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF +COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER +DEALINGS IN THE FONT SOFTWARE. diff --git a/docs/canonicalk8s/.sphinx/get_vale_conf.py b/docs/canonicalk8s/.sphinx/get_vale_conf.py new file mode 100644 index 000000000..23d890153 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/get_vale_conf.py @@ -0,0 +1,41 @@ +#! /usr/bin/env python + +import requests +import os + +DIR=os.getcwd() + +def main(): + + if os.path.exists(f"{DIR}/.sphinx/styles/Canonical"): + print("Vale directory exists") + else: + os.makedirs(f"{DIR}/.sphinx/styles/Canonical") + + url = "https://api.github.com/repos/canonical/praecepta/contents/styles/Canonical" + r = requests.get(url) + for item in r.json(): + download = requests.get(item["download_url"]) + file = open(".sphinx/styles/Canonical/" + item["name"], "w") + file.write(download.text) + file.close() + + if os.path.exists(f"{DIR}/.sphinx/styles/config/vocabularies/Canonical"): + print("Vocab directory exists") + else: + os.makedirs(f"{DIR}/.sphinx/styles/config/vocabularies/Canonical") + + url = "https://api.github.com/repos/canonical/praecepta/contents/styles/config/vocabularies/Canonical" + r = requests.get(url) + for item in r.json(): + download = requests.get(item["download_url"]) + file = open(".sphinx/styles/config/vocabularies/Canonical/" + item["name"], "w") + file.write(download.text) + file.close() + config = requests.get("https://raw.githubusercontent.com/canonical/praecepta/main/vale.ini") + file = open(".sphinx/vale.ini", "w") + file.write(config.text) + file.close() + +if __name__ == "__main__": + main() diff --git a/docs/canonicalk8s/.sphinx/images/Canonical-logo-4x.png b/docs/canonicalk8s/.sphinx/images/Canonical-logo-4x.png new file mode 100644 index 000000000..fd75696eb Binary files /dev/null and b/docs/canonicalk8s/.sphinx/images/Canonical-logo-4x.png differ diff --git a/docs/canonicalk8s/.sphinx/images/front-page-light.pdf b/docs/canonicalk8s/.sphinx/images/front-page-light.pdf new file mode 100644 index 000000000..bb68cdf8f Binary files /dev/null and b/docs/canonicalk8s/.sphinx/images/front-page-light.pdf differ diff --git a/docs/canonicalk8s/.sphinx/images/front-page.png b/docs/canonicalk8s/.sphinx/images/front-page.png new file mode 100644 index 000000000..c80e84303 Binary files /dev/null and b/docs/canonicalk8s/.sphinx/images/front-page.png differ diff --git a/docs/canonicalk8s/.sphinx/images/normal-page-footer.pdf b/docs/canonicalk8s/.sphinx/images/normal-page-footer.pdf new file mode 100644 index 000000000..dfd73cbc7 Binary files /dev/null and b/docs/canonicalk8s/.sphinx/images/normal-page-footer.pdf differ diff --git a/docs/canonicalk8s/.sphinx/latex_elements_template.txt b/docs/canonicalk8s/.sphinx/latex_elements_template.txt new file mode 100644 index 000000000..2b13b514a --- /dev/null +++ b/docs/canonicalk8s/.sphinx/latex_elements_template.txt @@ -0,0 +1,119 @@ +{ + 'papersize': 'a4paper', + 'pointsize': '11pt', + 'fncychap': '', + 'preamble': r''' +%\usepackage{charter} +%\usepackage[defaultsans]{lato} +%\usepackage{inconsolata} +\setmainfont[UprightFont = *-R, BoldFont = *-B, ItalicFont=*-RI, Extension = .ttf]{Ubuntu} +\setmonofont[UprightFont = *-R, BoldFont = *-B, ItalicFont=*-RI, Extension = .ttf]{UbuntuMono} +\usepackage[most]{tcolorbox} +\tcbuselibrary{breakable} +\usepackage{lastpage} +\usepackage{tabto} +\usepackage{ifthen} +\usepackage{etoolbox} +\usepackage{fancyhdr} +\usepackage{graphicx} +\usepackage{titlesec} +\usepackage{fontspec} +\usepackage{tikz} +\usepackage{changepage} +\usepackage{array} +\usepackage{tabularx} +\definecolor{yellowgreen}{RGB}{154, 205, 50} +\definecolor{title}{RGB}{76, 17, 48} +\definecolor{subtitle}{RGB}{116, 27, 71} +\definecolor{label}{RGB}{119, 41, 100} +\definecolor{copyright}{RGB}{174, 167, 159} +\makeatletter +\def\tcb@finalize@environment{% + \color{.}% hack for xelatex + \tcb@layer@dec% +} +\makeatother +\newenvironment{sphinxclassprompt}{\color{yellowgreen}\setmonofont[Color = 9ACD32, UprightFont = *-R, Extension = .ttf]{UbuntuMono}}{} +\tcbset{enhanced jigsaw, colback=black, fontupper=\color{white}} +\newtcolorbox{termbox}{use color stack, breakable, colupper=white, halign=flush left} +\newenvironment{sphinxclassterminal}{\setmonofont[Color = white, UprightFont = *-R, Extension = .ttf]{UbuntuMono}\sphinxsetup{VerbatimColor={black}}\begin{termbox}}{\end{termbox}} +\newcommand{\dimtorightedge}{% + \dimexpr\paperwidth-1in-\hoffset-\oddsidemargin\relax} +\newcommand{\dimtotop}{% + \dimexpr\height-1in-\voffset-\topmargin-\headheight-\headsep\relax} +\newtoggle{tpage} +\AtBeginEnvironment{titlepage}{\global\toggletrue{tpage}} +\fancypagestyle{plain}{ + \fancyhf{} + \fancyfoot[R]{\thepage\ of \pageref*{LastPage}} + \renewcommand{\headrulewidth}{0pt} + \renewcommand{\footrulewidth}{0pt} +} +\fancypagestyle{normal}{ + \fancyhf{} + \fancyfoot[R]{\thepage\ of \pageref*{LastPage}} + \renewcommand{\headrulewidth}{0pt} + \renewcommand{\footrulewidth}{0pt} +} +\fancypagestyle{titlepage}{% + \fancyhf{} + \fancyfoot[L]{\footnotesize \textcolor{copyright}{© 2024 Canonical Ltd. All rights reserved.}} +} +\newcommand\sphinxbackoftitlepage{\thispagestyle{titlepage}} +\titleformat{\chapter}[block]{\Huge \color{title} \bfseries\filright}{\thechapter .}{1.5ex}{} +\titlespacing{\chapter}{0pt}{0pt}{0pt} +\titleformat{\section}[block]{\huge \bfseries\filright}{\thesection .}{1.5ex}{} +\titlespacing{\section}{0pt}{0pt}{0pt} +\titleformat{\subsection}[block]{\Large \bfseries\filright}{\thesubsection .}{1.5ex}{} +\titlespacing{\subsection}{0pt}{0pt}{0pt} +\setcounter{tocdepth}{1} +\renewcommand\pagenumbering[1]{} +''', + 'sphinxsetup': 'verbatimwithframe=false, pre_border-radius=0pt, verbatimvisiblespace=\\phantom{}, verbatimcontinued=\\phantom{}', + 'extraclassoptions': 'openany,oneside', + 'maketitle': r''' +\begin{titlepage} +\begin{flushleft} + \begin{tikzpicture}[remember picture,overlay] + \node[anchor=south east, inner sep=0] at (current page.south east) { + \includegraphics[width=\paperwidth, height=\paperheight]{front-page-light} + }; + \end{tikzpicture} +\end{flushleft} + +\vspace*{3cm} + +\begin{adjustwidth}{8cm}{0pt} +\begin{flushleft} + \huge \textcolor{black}{\textbf{}{\raggedright{$PROJECT}}} +\end{flushleft} +\end{adjustwidth} + +\vfill + +\begin{adjustwidth}{8cm}{0pt} +\begin{tabularx}{0.5\textwidth}{ l l } + \textcolor{lightgray}{© 2024 Canonical Ltd.} & \hspace{3cm} \\ + \textcolor{lightgray}{All rights reserved.} & \hspace{3cm} \\ + & \hspace{3cm} \\ + & \hspace{3cm} \\ + +\end{tabularx} +\end{adjustwidth} + +\end{titlepage} +\RemoveFromHook{shipout/background} +\AddToHook{shipout/background}{ + \begin{tikzpicture}[remember picture,overlay] + \node[anchor=south west, align=left, inner sep=0] at (current page.south west) { + \includegraphics[width=\paperwidth]{normal-page-footer} + }; + \end{tikzpicture} + \begin{tikzpicture}[remember picture,overlay] + \node[anchor=north east, opacity=0.5, inner sep=35] at (current page.north east) { + \includegraphics[width=4cm]{Canonical-logo-4x} + }; + \end{tikzpicture} + } +''', +} \ No newline at end of file diff --git a/docs/canonicalk8s/.sphinx/pa11y.json b/docs/canonicalk8s/.sphinx/pa11y.json new file mode 100644 index 000000000..8df0cb9cb --- /dev/null +++ b/docs/canonicalk8s/.sphinx/pa11y.json @@ -0,0 +1,9 @@ +{ + "chromeLaunchConfig": { + "args": [ + "--no-sandbox" + ] + }, + "reporter": "cli", + "standard": "WCAG2AA" +} \ No newline at end of file diff --git a/docs/canonicalk8s/.sphinx/requirements.txt b/docs/canonicalk8s/.sphinx/requirements.txt new file mode 100644 index 000000000..539fce3c8 --- /dev/null +++ b/docs/canonicalk8s/.sphinx/requirements.txt @@ -0,0 +1,22 @@ +# DO NOT MODIFY THIS FILE DIRECTLY! +# +# This file is generated automatically. +# Add custom requirements to the custom_required_modules +# array in the custom_conf.py file and run: +# make clean && make install +GitPython +canonical-sphinx-extensions +furo +linkify-it-py +myst-parser +pyspelling +sphinx +sphinx-autobuild +sphinx-copybutton +sphinx-design +sphinx-notfound-page +sphinx-tabs +sphinxcontrib-jquery +sphinxcontrib-svg2pdfconverter[CairoSVG] +sphinxext-opengraph +watchfiles diff --git a/docs/canonicalk8s/.sphinx/spellingcheck.yaml b/docs/canonicalk8s/.sphinx/spellingcheck.yaml new file mode 100644 index 000000000..648d9d7ee --- /dev/null +++ b/docs/canonicalk8s/.sphinx/spellingcheck.yaml @@ -0,0 +1,30 @@ +matrix: +- name: rST files + aspell: + lang: en + d: en_GB + dictionary: + wordlists: + - src/.wordlist.txt + - src/.custom_wordlist.txt + output: .sphinx/.wordlist.dic + sources: + - ../_build/**/*.html + pipeline: + - pyspelling.filters.html: + comments: false + attributes: + - title + - alt + ignores: + - code + - pre + - spellexception + - link + - title + - div.relatedlinks + - strong.command + - div.visually-hidden + - img + - a.p-navigation__link + - a.contributor diff --git a/docs/canonicalk8s/.wokeignore b/docs/canonicalk8s/.wokeignore new file mode 100644 index 000000000..c64a60376 --- /dev/null +++ b/docs/canonicalk8s/.wokeignore @@ -0,0 +1,4 @@ +# the cheat sheets contain a link to a repository with a block word which we +# cannot avoid for now, ie +# https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html +doc-cheat-sheet* diff --git a/docs/canonicalk8s/LICENSE b/docs/canonicalk8s/LICENSE new file mode 100644 index 000000000..31081ad40 --- /dev/null +++ b/docs/canonicalk8s/LICENSE @@ -0,0 +1,694 @@ +License for Canonical Starter Pack +================================== + +Unless otherwise stated, all code in this repository is licensed under +the following GPL license. +Documentation for this project is licensed under CC-BY-SA 3.0. + +Starter Pack code +----------------- + + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. + +Starter Pack documentation +-------------------------- + +Copyright 2024 Canonical Ltd. + +This work is licensed under the Creative Commons Attribution-Share Alike 3.0 +Unported License. To view a copy of this license, visit +http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative +Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. \ No newline at end of file diff --git a/docs/canonicalk8s/Makefile b/docs/canonicalk8s/Makefile new file mode 100644 index 000000000..72fe7c602 --- /dev/null +++ b/docs/canonicalk8s/Makefile @@ -0,0 +1,31 @@ +# This Makefile stub allows you to customize starter pack (SP) targets. +# Consider this file as a bridge between your project +# and the starter pack's predefined targets that reside in Makefile.sp. +# +# You can add your own, non-SP targets here or override SP targets +# to fit your project's needs. For example, you can define and use targets +# named "install" or "run", but continue to use SP targets like "sp-install" +# or "sp-run" when working on the documentation. + +# Put it first so that "make" without argument is like "make help". +help: + @echo "\n" \ + "------------------------------------------------------------- \n" \ + "* watch, build and serve the documentation: make run \n" \ + "* only build: make html \n" \ + "* only serve: make serve \n" \ + "* clean built doc files: make clean-doc \n" \ + "* clean full environment: make clean \n" \ + "* check links: make linkcheck \n" \ + "* check spelling: make spelling \n" \ + "* check spelling (without building again): make spellcheck \n" \ + "* check inclusive language: make woke \n" \ + "* check accessibility: make pa11y \n" \ + "* check style guide compliance: make vale \n" \ + "* check style guide compliance on target: make vale TARGET=* \n" \ + "* check metrics for documentation: make allmetrics \n" \ + "* other possible targets: make \n" \ + "------------------------------------------------------------- \n" + +%: + $(MAKE) -f Makefile.sp sp-$@ diff --git a/docs/canonicalk8s/Makefile.sp b/docs/canonicalk8s/Makefile.sp new file mode 100644 index 000000000..6b98451f5 --- /dev/null +++ b/docs/canonicalk8s/Makefile.sp @@ -0,0 +1,156 @@ +# Minimal makefile for Sphinx documentation +# +# `Makefile.sp` is from the Sphinx starter pack and should not be +# modified. +# Add your customisation to `Makefile` instead. + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXDIR = .sphinx +SPHINXOPTS ?= -c . -d $(SPHINXDIR)/.doctrees -j auto +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +METRICSDIR = ./metrics +BUILDDIR = ../_build +VENVDIR = $(SPHINXDIR)/venv +PA11Y = $(SPHINXDIR)/node_modules/pa11y/bin/pa11y.js --config $(SPHINXDIR)/pa11y.json +VENV = $(VENVDIR)/bin/activate +TARGET = * +ALLFILES = *.rst **/*.rst +ADDPREREQS ?= +REQPDFPACKS = latexmk fonts-freefont-otf texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended texlive-font-utils texlive-lang-cjk texlive-xetex plantuml xindy tex-gyre dvipng + +.PHONY: sp-full-help sp-woke-install sp-pa11y-install sp-install sp-run sp-html \ + sp-epub sp-serve sp-clean sp-clean-doc sp-spelling sp-spellcheck sp-linkcheck sp-woke \ + sp-allmetrics sp-pa11y sp-pdf-prep-force sp-pdf-prep sp-pdf Makefile.sp sp-vale sp-bash + +sp-full-help: $(VENVDIR) + @. $(VENV); $(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + @echo "\n\033[1;31mNOTE: This help texts shows unsupported targets!\033[0m" + @echo "Run 'make help' to see supported targets." + +# Shouldn't assume that venv is available on Ubuntu by default; discussion here: +# https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1290847 +$(SPHINXDIR)/requirements.txt: + @python3 -c "import venv" || \ + (echo "You must install python3-venv before you can build the documentation."; exit 1) + python3 -m venv $(VENVDIR) + @if [ ! -z "$(ADDPREREQS)" ]; then \ + . $(VENV); pip install \ + $(PIPOPTS) --require-virtualenv $(ADDPREREQS); \ + fi + . $(VENV); python3 $(SPHINXDIR)/build_requirements.py + +# If requirements are updated, venv should be rebuilt and timestamped. +$(VENVDIR): $(SPHINXDIR)/requirements.txt + @echo "... setting up virtualenv" + python3 -m venv $(VENVDIR) + . $(VENV); pip install $(PIPOPTS) --require-virtualenv \ + --upgrade -r $(SPHINXDIR)/requirements.txt \ + --log $(VENVDIR)/pip_install.log + @test ! -f $(VENVDIR)/pip_list.txt || \ + mv $(VENVDIR)/pip_list.txt $(VENVDIR)/pip_list.txt.bak + @. $(VENV); pip list --local --format=freeze > $(VENVDIR)/pip_list.txt + @touch $(VENVDIR) + +sp-woke-install: + @type woke >/dev/null 2>&1 || \ + { echo "Installing \"woke\" snap... \n"; sudo snap install woke; } + +sp-pa11y-install: + @type $(PA11Y) >/dev/null 2>&1 || { \ + echo "Installing \"pa11y\" from npm... \n"; \ + mkdir -p $(SPHINXDIR)/node_modules/ ; \ + npm install --prefix $(SPHINXDIR) pa11y; \ + } + +sp-install: $(VENVDIR) + +sp-run: sp-install + . $(VENV); sphinx-autobuild -b dirhtml "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) + +# Doesn't depend on $(BUILDDIR) to rebuild properly at every run. +sp-html: sp-install + . $(VENV); $(SPHINXBUILD) -W --keep-going -b dirhtml "$(SOURCEDIR)" "$(BUILDDIR)" -w $(SPHINXDIR)/warnings.txt $(SPHINXOPTS) + +sp-epub: sp-install + . $(VENV); $(SPHINXBUILD) -b epub "$(SOURCEDIR)" "$(BUILDDIR)" -w $(SPHINXDIR)/warnings.txt $(SPHINXOPTS) + +sp-serve: sp-html + cd "$(BUILDDIR)"; python3 -m http.server --bind 127.0.0.1 8000 + +sp-clean: sp-clean-doc + @test ! -e "$(VENVDIR)" -o -d "$(VENVDIR)" -a "$(abspath $(VENVDIR))" != "$(VENVDIR)" + rm -rf $(VENVDIR) + rm -f $(SPHINXDIR)/requirements.txt + rm -rf $(SPHINXDIR)/node_modules/ + rm -rf $(SPHINXDIR)/styles + rm -rf $(SPHINXDIR)/vale.ini + +sp-clean-doc: + git clean -fx "$(BUILDDIR)" + rm -rf $(SPHINXDIR)/.doctrees + +sp-spellcheck: + . $(VENV) ; python3 -m pyspelling -c $(SPHINXDIR)/spellingcheck.yaml -j $(shell nproc) + +sp-spelling: sp-html sp-spellcheck + +sp-linkcheck: sp-install + . $(VENV) ; $(SPHINXBUILD) -b linkcheck "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) || { grep --color -F "[broken]" "$(BUILDDIR)/output.txt"; exit 1; } + exit 0 + +sp-woke: sp-woke-install + woke $(ALLFILES) --exit-1-on-failure \ + -c https://github.com/canonical/Inclusive-naming/raw/main/config.yml + +sp-pa11y: sp-pa11y-install sp-html + find $(BUILDDIR) -name *.html -print0 | xargs -n 1 -0 $(PA11Y) + +sp-vale: sp-install + @. $(VENV); test -d $(SPHINXDIR)/venv/lib/python*/site-packages/vale || pip install vale + @. $(VENV); test -f $(SPHINXDIR)/vale.ini || python3 $(SPHINXDIR)/get_vale_conf.py + @. $(VENV); find $(SPHINXDIR)/venv/lib/python*/site-packages/vale/vale_bin -size 195c -exec vale --config "$(SPHINXDIR)/vale.ini" $(TARGET) > /dev/null \; + @cat $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept.txt > $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept_backup.txt + @cat $(SOURCEDIR)/.wordlist.txt $(SOURCEDIR)/.custom_wordlist.txt >> $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept.txt + @echo "" + @echo "Running Vale against $(TARGET). To change target set TARGET= with make command" + @echo "" + @. $(VENV); vale --config "$(SPHINXDIR)/vale.ini" --glob='*.{md,txt,rst}' $(TARGET) || true + @cat $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept_backup.txt > $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept.txt && rm $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept_backup.txt + +sp-pdf-prep: sp-install + @for packageName in $(REQPDFPACKS); do (dpkg-query -W -f='$${Status}' $$packageName 2>/dev/null | \ + grep -c "ok installed" >/dev/null && echo "Package $$packageName is installed") && continue || \ + (echo "\nPDF generation requires the installation of the following packages: $(REQPDFPACKS)" && \ + echo "" && echo "Run sudo make pdf-prep-force to install these packages" && echo "" && echo \ + "Please be aware these packages will be installed to your system") && exit 1 ; done + +sp-pdf-prep-force: + apt-get update + apt-get upgrade -y + apt-get install --no-install-recommends -y $(REQPDFPACKS) \ + +sp-pdf: sp-pdf-prep + @. $(VENV); sphinx-build -M latexpdf "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) + @rm ./$(BUILDDIR)/latex/front-page-light.pdf || true + @rm ./$(BUILDDIR)/latex/normal-page-footer.pdf || true + @find ./$(BUILDDIR)/latex -name "*.pdf" -exec mv -t ./$(BUILDDIR) {} + + @rm -r $(BUILDDIR)/latex + @echo "\nOutput can be found in ./$(BUILDDIR)\n" + +sp-allmetrics: sp-html + @echo "Recording documentation metrics..." + @echo "Checking for existence of vale..." + . $(VENV) + @. $(VENV); test -d $(SPHINXDIR)/venv/lib/python*/site-packages/vale || pip install vale + @. $(VENV); test -f $(SPHINXDIR)/vale.ini || python3 $(SPHINXDIR)/get_vale_conf.py + @. $(VENV); find $(SPHINXDIR)/venv/lib/python*/site-packages/vale/vale_bin -size 195c -exec vale --config "$(SPHINXDIR)/vale.ini" $(TARGET) > /dev/null \; + @eval '$(METRICSDIR)/scripts/source_metrics.sh $(PWD)' + @eval '$(METRICSDIR)/scripts/build_metrics.sh $(PWD) $(METRICSDIR)' + + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile.sp + . $(VENV); $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/canonicalk8s/about.md b/docs/canonicalk8s/about.md new file mode 100644 index 000000000..08309665a --- /dev/null +++ b/docs/canonicalk8s/about.md @@ -0,0 +1,2 @@ +```{include} src/snap/explanation/about.md +``` \ No newline at end of file diff --git a/docs/canonicalk8s/community.md b/docs/canonicalk8s/community.md new file mode 100644 index 000000000..66b0a8c9d --- /dev/null +++ b/docs/canonicalk8s/community.md @@ -0,0 +1,2 @@ +```{include} src/snap/reference/community.md +``` \ No newline at end of file diff --git a/docs/canonicalk8s/conf-rtd.py b/docs/canonicalk8s/conf-rtd.py new file mode 100644 index 000000000..05d127f0d --- /dev/null +++ b/docs/canonicalk8s/conf-rtd.py @@ -0,0 +1,254 @@ +import sys +import os +import requests +from urllib.parse import urlparse +from git import Repo, InvalidGitRepositoryError +import time +import ast +import yaml + +sys.path.append('./') +from custom_conf import * +sys.path.append('.sphinx/') +from build_requirements import * + +# Configuration file for the Sphinx documentation builder. +# You should not do any modifications to this file. Put your custom +# configuration into the custom_conf.py file. +# If you need to change this file, contribute the changes upstream. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +############################################################ +### Extensions +############################################################ + +extensions = [ + 'sphinx_design', + 'sphinx_copybutton', + 'sphinxcontrib.jquery', +] + +# Only add redirects extension if any redirects are specified. +if AreRedirectsDefined(): + extensions.append('sphinx_reredirects') + +# Only add myst extensions if any configuration is present. +if IsMyStParserUsed(): + extensions.append('myst_parser') + + # Additional MyST syntax + myst_enable_extensions = [ + 'substitution', + 'deflist', + 'linkify' + ] + myst_enable_extensions.extend(custom_myst_extensions) + +# Only add Open Graph extension if any configuration is present. +if IsOpenGraphConfigured(): + extensions.append('sphinxext.opengraph') + +extensions.extend(custom_extensions) +extensions = DeduplicateExtensions(extensions) + +### Configuration for extensions + +# Used for related links +if not 'discourse_prefix' in html_context and 'discourse' in html_context: + html_context['discourse_prefix'] = html_context['discourse'] + '/t/' + +# The URL prefix for the notfound extension depends on whether the documentation uses versions. +# For documentation on documentation.ubuntu.com, we also must add the slug. +url_version = '' +url_lang = '' + +# Determine if the URL uses versions and language +if 'READTHEDOCS_CANONICAL_URL' in os.environ and os.environ['READTHEDOCS_CANONICAL_URL']: + url_parts = os.environ['READTHEDOCS_CANONICAL_URL'].split('/') + + if len(url_parts) >= 2 and 'READTHEDOCS_VERSION' in os.environ and os.environ['READTHEDOCS_VERSION'] == url_parts[-2]: + url_version = url_parts[-2] + '/' + + if len(url_parts) >= 3 and 'READTHEDOCS_LANGUAGE' in os.environ and os.environ['READTHEDOCS_LANGUAGE'] == url_parts[-3]: + url_lang = url_parts[-3] + '/' + +# Set notfound_urls_prefix to the slug (if defined) and the version/language affix +if slug: + notfound_urls_prefix = '/' + slug + '/' + url_lang + url_version +elif len(url_lang + url_version) > 0: + notfound_urls_prefix = '/' + url_lang + url_version +else: + notfound_urls_prefix = '' + +notfound_context = { + 'title': 'Page not found', + 'body': '

Sorry, but the documentation page that you are looking for was not found.

\n\n

Documentation changes over time, and pages are moved around. We try to redirect you to the updated content where possible, but unfortunately, that didn\'t work this time (maybe because the content you were looking for does not exist in this version of the documentation).

\n

You can try to use the navigation to locate the content you\'re looking for, or search for a similar page.

\n', +} + +# Default image for OGP (to prevent font errors, see +# https://github.com/canonical/sphinx-docs-starter-pack/pull/54 ) +if not 'ogp_image' in locals(): + ogp_image = 'https://assets.ubuntu.com/v1/253da317-image-document-ubuntudocs.svg' + +############################################################ +### General configuration +############################################################ + +exclude_patterns = [ + '_build', + 'Thumbs.db', + '.DS_Store', + '.sphinx', +] +exclude_patterns.extend(custom_excludes) + +rst_epilog = ''' +.. include:: /reuse/links.txt +''' +if 'custom_rst_epilog' in locals(): + rst_epilog = custom_rst_epilog + +source_suffix = { + '.rst': 'restructuredtext', + '.md': 'markdown', +} + +if not 'conf_py_path' in html_context and 'github_folder' in html_context: + html_context['conf_py_path'] = html_context['github_folder'] + +# For ignoring specific links +linkcheck_anchors_ignore_for_url = [ + r'https://github\.com/.*' +] +linkcheck_anchors_ignore_for_url.extend(custom_linkcheck_anchors_ignore_for_url) + +# Tags cannot be added directly in custom_conf.py, so add them here +for tag in custom_tags: + tags.add(tag) + +# html_context['get_contribs'] is a function and cannot be +# cached (see https://github.com/sphinx-doc/sphinx/issues/12300) +suppress_warnings = ["config.cache"] + +############################################################ +### Styling +############################################################ + +# Find the current builder +builder = 'dirhtml' +if '-b' in sys.argv: + builder = sys.argv[sys.argv.index('-b')+1] + +# Setting templates_path for epub makes the build fail +if builder == 'dirhtml' or builder == 'html': + templates_path = ['.sphinx/_templates'] + notfound_template = '404.html' + +# Theme configuration +html_theme = 'furo' +html_last_updated_fmt = '' +html_permalinks_icon = '¶' + +if html_title == '': + html_theme_options = { + 'sidebar_hide_name': True + } + +############################################################ +### Additional files +############################################################ + +html_static_path = ['.sphinx/_static'] + +html_css_files = [ + 'custom.css', + 'header.css', + 'github_issue_links.css', + 'furo_colors.css', + 'footer.css' +] +html_css_files.extend(custom_html_css_files) + +html_js_files = ['header-nav.js', 'footer.js'] +if 'github_issues' in html_context and html_context['github_issues'] and not disable_feedback_button: + html_js_files.append('github_issue_links.js') +html_js_files.extend(custom_html_js_files) + +############################################################# +# Display the contributors + + +############################################################# +# DISABLED AS IT DOESN'T WORK FOR source not in same dir + +#def get_contributors_for_file(github_url, github_folder, github_source, pagename, page_source_suffix, display_contributors_since=None): +# filename = f"{pagename}{page_source_suffix}" +# paths=html_context['github_source'][1:] + filename +# +# try: +# repo = Repo(".") +# except InvalidGitRepositoryError: +# cwd = os.getcwd() +# ghfolder = html_context['github_source'][:-1] +# +# if ghfolder and cwd.endswith(ghfolder): +# repo = Repo(cwd.rpartition(ghfolder)[0]) +# else: +# print("The local Git repository could not be found.") +# return +# +# since = display_contributors_since if display_contributors_since and display_contributors_since.strip() else None +# +# commits = repo.iter_commits(paths=paths, since=since) +# +# contributors_dict = {} +# for commit in commits: +# contributor = commit.author.name +# if contributor not in contributors_dict or commit.committed_date > contributors_dict[contributor]['date']: +# contributors_dict[contributor] = { +# 'date': commit.committed_date, +# 'sha': commit.hexsha +# } +# # The github_page contains the link to the contributor's latest commit. +# contributors_list = [{'name': name, 'github_page': f"{github_url}/commit/{data['sha']}"} for name, data in contributors_dict.items()] +# sorted_contributors_list = sorted(contributors_list, key=lambda x: x['name']) +# return sorted_contributors_list +# +# html_context['get_contribs'] = get_contributors_for_file + +############################################################ +### Myst configuration +############################################################ +if os.path.exists('./reuse/substitutions.yaml'): + with open('./reuse/substitutions.yaml', 'r') as fd: + myst_substitutions = yaml.safe_load(fd.read()) + + +############################################################ +### PDF configuration +############################################################ + +latex_additional_files = [ + "./.sphinx/fonts/Ubuntu-B.ttf", + "./.sphinx/fonts/Ubuntu-R.ttf", + "./.sphinx/fonts/Ubuntu-RI.ttf", + "./.sphinx/fonts/UbuntuMono-R.ttf", + "./.sphinx/fonts/UbuntuMono-RI.ttf", + "./.sphinx/fonts/UbuntuMono-B.ttf", + "./.sphinx/images/Canonical-logo-4x.png", + "./.sphinx/images/front-page-light.pdf", + "./.sphinx/images/normal-page-footer.pdf", +] + +latex_engine = 'xelatex' +latex_show_pagerefs = True +latex_show_urls = 'footnote' + +with open(".sphinx/latex_elements_template.txt", "rt") as file: + latex_config = file.read() + +latex_elements = ast.literal_eval(latex_config.replace("$PROJECT", project)) + +master_doc = 'index' \ No newline at end of file diff --git a/docs/canonicalk8s/conf.py b/docs/canonicalk8s/conf.py new file mode 100644 index 000000000..c804e78c1 --- /dev/null +++ b/docs/canonicalk8s/conf.py @@ -0,0 +1,255 @@ +import sys +import os +import requests +from urllib.parse import urlparse +from git import Repo, InvalidGitRepositoryError +import time +import ast +import yaml + +sys.path.append('./') +from custom_conf import * +sys.path.append('.sphinx/') +from build_requirements import * + +# Configuration file for the Sphinx documentation builder. +# You should not do any modifications to this file. Put your custom +# configuration into the custom_conf.py file. +# If you need to change this file, contribute the changes upstream. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +############################################################ +### Extensions +############################################################ + +extensions = [ + 'sphinx_design', + 'sphinx_copybutton', + 'sphinxcontrib.jquery', +] + +# Only add redirects extension if any redirects are specified. +if AreRedirectsDefined(): + extensions.append('sphinx_reredirects') + +# Only add myst extensions if any configuration is present. +if IsMyStParserUsed(): + extensions.append('myst_parser') + + # Additional MyST syntax + myst_enable_extensions = [ + 'substitution', + 'deflist', + 'linkify' + ] + myst_enable_extensions.extend(custom_myst_extensions) + +# Only add Open Graph extension if any configuration is present. +if IsOpenGraphConfigured(): + extensions.append('sphinxext.opengraph') + +extensions.extend(custom_extensions) +extensions = DeduplicateExtensions(extensions) + +### Configuration for extensions + +# Used for related links +if not 'discourse_prefix' in html_context and 'discourse' in html_context: + html_context['discourse_prefix'] = html_context['discourse'] + '/t/' + +# The URL prefix for the notfound extension depends on whether the documentation uses versions. +# For documentation on documentation.ubuntu.com, we also must add the slug. +url_version = '' +url_lang = '' + +# Determine if the URL uses versions and language +if 'READTHEDOCS_CANONICAL_URL' in os.environ and os.environ['READTHEDOCS_CANONICAL_URL']: + url_parts = os.environ['READTHEDOCS_CANONICAL_URL'].split('/') + + if len(url_parts) >= 2 and 'READTHEDOCS_VERSION' in os.environ and os.environ['READTHEDOCS_VERSION'] == url_parts[-2]: + url_version = url_parts[-2] + '/' + + if len(url_parts) >= 3 and 'READTHEDOCS_LANGUAGE' in os.environ and os.environ['READTHEDOCS_LANGUAGE'] == url_parts[-3]: + url_lang = url_parts[-3] + '/' + +# Set notfound_urls_prefix to the slug (if defined) and the version/language affix +if slug: + notfound_urls_prefix = '/' + slug + '/' + url_lang + url_version +elif len(url_lang + url_version) > 0: + notfound_urls_prefix = '/' + url_lang + url_version +else: + notfound_urls_prefix = '' + +notfound_context = { + 'title': 'Page not found', + 'body': '

Sorry, but the documentation page that you are looking for was not found.

\n\n

Documentation changes over time, and pages are moved around. We try to redirect you to the updated content where possible, but unfortunately, that didn\'t work this time (maybe because the content you were looking for does not exist in this version of the documentation).

\n

You can try to use the navigation to locate the content you\'re looking for, or search for a similar page.

\n', +} + +# Default image for OGP (to prevent font errors, see +# https://github.com/canonical/sphinx-docs-starter-pack/pull/54 ) +if not 'ogp_image' in locals(): + ogp_image = 'https://assets.ubuntu.com/v1/253da317-image-document-ubuntudocs.svg' + +############################################################ +### General configuration +############################################################ + +exclude_patterns = [ + '_build', + 'Thumbs.db', + '.DS_Store', + '.sphinx', +] +exclude_patterns.extend(custom_excludes) + +rst_epilog = ''' +.. include:: /reuse/links.txt +''' +if 'custom_rst_epilog' in locals(): + rst_epilog = custom_rst_epilog + +source_suffix = { + '.rst': 'restructuredtext', + '.md': 'markdown', +} + +if not 'conf_py_path' in html_context and 'github_folder' in html_context: + html_context['conf_py_path'] = html_context['github_folder'] + +# For ignoring specific links +linkcheck_anchors_ignore_for_url = [ + r'https://github\.com/.*' +] +linkcheck_anchors_ignore_for_url.extend(custom_linkcheck_anchors_ignore_for_url) + +# Tags cannot be added directly in custom_conf.py, so add them here +for tag in custom_tags: + tags.add(tag) + +# html_context['get_contribs'] is a function and cannot be +# cached (see https://github.com/sphinx-doc/sphinx/issues/12300) +suppress_warnings = ["config.cache"] + +############################################################ +### Styling +############################################################ + +# Find the current builder +builder = 'dirhtml' +if '-b' in sys.argv: + builder = sys.argv[sys.argv.index('-b')+1] + +# Setting templates_path for epub makes the build fail +if builder == 'dirhtml' or builder == 'html': + templates_path = ['.sphinx/_templates'] + notfound_template = '404.html' + +# Theme configuration +html_theme = 'furo' +html_last_updated_fmt = '' +html_permalinks_icon = '¶' + +if html_title == '': + html_theme_options = { + 'sidebar_hide_name': True + } + +############################################################ +### Additional files +############################################################ + +html_static_path = ['.sphinx/_static'] + +html_css_files = [ + 'custom.css', + 'header.css', + 'github_issue_links.css', + 'furo_colors.css', + 'footer.css' +] +html_css_files.extend(custom_html_css_files) + +html_js_files = ['header-nav.js', 'footer.js'] +if 'github_issues' in html_context and html_context['github_issues'] and not disable_feedback_button: + html_js_files.append('github_issue_links.js') +html_js_files.extend(custom_html_js_files) + +############################################################# +# Display the contributors + + +############################################################# +# DISABLED AS IT DOESN'T WORK FOR source not in same dir + +#def get_contributors_for_file(github_url, github_folder, github_source, pagename, page_source_suffix, display_contributors_since=None): +# filename = f"{pagename}{page_source_suffix}" +# paths=html_context['github_source'][1:] + filename +# +# try: +# repo = Repo(".") +# except InvalidGitRepositoryError: +# cwd = os.getcwd() +# ghfolder = html_context['github_source'][:-1] +# +# if ghfolder and cwd.endswith(ghfolder): +# repo = Repo(cwd.rpartition(ghfolder)[0]) +# else: +# print("The local Git repository could not be found.") +# return +# +# since = display_contributors_since if display_contributors_since and display_contributors_since.strip() else None +# +# commits = repo.iter_commits(paths=paths, since=since) +# +# contributors_dict = {} +# for commit in commits: +# contributor = commit.author.name +# if contributor not in contributors_dict or commit.committed_date > contributors_dict[contributor]['date']: +# contributors_dict[contributor] = { +# 'date': commit.committed_date, +# 'sha': commit.hexsha +# } +# # The github_page contains the link to the contributor's latest commit. +# contributors_list = [{'name': name, 'github_page': f"{github_url}/commit/{data['sha']}"} for name, data in contributors_dict.items()] +# sorted_contributors_list = sorted(contributors_list, key=lambda x: x['name']) +# return sorted_contributors_list +# +# html_context['get_contribs'] = get_contributors_for_file + +############################################################ +### Myst configuration +############################################################ +if os.path.exists('./reuse/substitutions.yaml'): + with open('./reuse/substitutions.yaml', 'r') as fd: + myst_substitutions = yaml.safe_load(fd.read()) + +suppress_warnings = ["myst.xref_missing", "myst.iref_ambiguous"] + +############################################################ +### PDF configuration +############################################################ + +latex_additional_files = [ + "./.sphinx/fonts/Ubuntu-B.ttf", + "./.sphinx/fonts/Ubuntu-R.ttf", + "./.sphinx/fonts/Ubuntu-RI.ttf", + "./.sphinx/fonts/UbuntuMono-R.ttf", + "./.sphinx/fonts/UbuntuMono-RI.ttf", + "./.sphinx/fonts/UbuntuMono-B.ttf", + "./.sphinx/images/Canonical-logo-4x.png", + "./.sphinx/images/front-page-light.pdf", + "./.sphinx/images/normal-page-footer.pdf", +] + +latex_engine = 'xelatex' +latex_show_pagerefs = True +latex_show_urls = 'footnote' + +with open(".sphinx/latex_elements_template.txt", "rt") as file: + latex_config = file.read() + +latex_elements = ast.literal_eval(latex_config.replace("$PROJECT", project)) + +master_doc = 'index' \ No newline at end of file diff --git a/docs/canonicalk8s/custom_conf.py b/docs/canonicalk8s/custom_conf.py new file mode 100644 index 000000000..e2532892b --- /dev/null +++ b/docs/canonicalk8s/custom_conf.py @@ -0,0 +1,233 @@ +import datetime + +# Custom configuration for the Sphinx documentation builder. +# All configuration specific to your project should be done in this file. +# +# The file is included in the common conf.py configuration file. +# You can modify any of the settings below or add any configuration that +# is not covered by the common conf.py file. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html +# +# If you're not familiar with Sphinx and don't want to use advanced +# features, it is sufficient to update the settings in the "Project +# information" section. + +############################################################ +### Project information +############################################################ + +# Product name +project = 'Canonical Kubernetes' +author = 'Canonical Group Ltd' + +# The title you want to display for the documentation in the sidebar. +# You might want to include a version number here. +# To not display any title, set this option to an empty string. +html_title = '' + +# The default value uses CC-BY-SA as the license and the current year +# as the copyright year. +# +# If your documentation needs a different copyright license, use that +# instead of 'CC-BY-SA'. Also, if your documentation is included as +# part of the code repository of your project, it'll inherit the license +# of the code. So you'll need to specify that license here (instead of +# 'CC-BY-SA'). +# +# For static works, it is common to provide the year of first publication. +# Another option is to give the first year and the current year +# for documentation that is often changed, e.g. 2022–2023 (note the en-dash). +# +# A way to check a GitHub repo's creation date is to obtain a classic GitHub +# token with 'repo' permissions here: https://github.com/settings/tokens +# Next, use 'curl' and 'jq' to extract the date from the GitHub API's output: +# +# curl -H 'Authorization: token ' \ +# -H 'Accept: application/vnd.github.v3.raw' \ +# https://api.github.com/repos/canonical/ | jq '.created_at' + +copyright = '%s CC-BY-SA, %s' % (datetime.date.today().year, author) + +## Open Graph configuration - defines what is displayed as a link preview +## when linking to the documentation from another website (see https://ogp.me/) +# The URL where the documentation will be hosted (leave empty if you +# don't know yet) +# NOTE: If no ogp_* variable is defined (e.g. if you remove this section) the +# sphinxext.opengraph extension will be disabled. +ogp_site_url = 'https://canonical-starter-pack.readthedocs-hosted.com/' +# The documentation website name (usually the same as the product name) +ogp_site_name = project +# The URL of an image or logo that is used in the preview +ogp_image = 'https://assets.ubuntu.com/v1/253da317-image-document-ubuntudocs.svg' + +# Update with the local path to the favicon for your product +# (default is the circle of friends) +html_favicon = '.sphinx/_static/favicon.png' + +html_context = { + + # Change to the link to the website of your product (without "https://") + # For example: "ubuntu.com/lxd" or "microcloud.is" + # If there is no product website, edit the header template to remove the + # link (see the readme for instructions). + 'product_page': 'ubuntu.com/kubernetes', + + # Add your product tag (the orange part of your logo, will be used in the + # header) to ".sphinx/_static" and change the path here (start with "_static") + # (default is the circle of friends) + 'product_tag': '_static/tag.png', + + # Change to the discourse instance you want to be able to link to + # using the :discourse: metadata at the top of a file + # (use an empty value if you don't want to link) + 'discourse': ' https://discourse.ubuntu.com/c/kubernetes/180', + + # Change to the Mattermost channel you want to link to + # (use an empty value if you don't want to link) + 'matrix': 'https://matrix.to/#/#k8s:ubuntu.com', + + # Change to the GitHub URL for your project + 'github_url': 'https://github.com/canonical/k8s-snap', + + # Change to the branch for this version of the documentation + 'github_version': 'main', + + # Change to the folder that contains the documentation + # (usually "/" or "/docs/") + 'github_folder': '/docs/src/', + + # Change to an empty value if your GitHub repo doesn't have issues enabled. + # This will disable the feedback button and the issue link in the footer. + 'github_issues': 'enabled', + + # Controls the existence of Previous / Next buttons at the bottom of pages + # Valid options: none, prev, next, both + # You can override the default setting on a page-by-page basis by specifying + # it as file-wide metadata at the top of the file, see + # https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + 'sequential_nav': "none", + # Controls if to display the contributors of a file or not + "display_contributors": False, + + # Controls time frame for showing the contributors + "display_contributors_since": "" +} + +# If your project is on documentation.ubuntu.com, specify the project +# slug (for example, "lxd") here. +slug = "canonical-kubernetes" + +############################################################ +### Redirects +############################################################ + +# Set up redirects (https://documatt.gitlab.io/sphinx-reredirects/usage.html) +# For example: 'explanation/old-name.html': '../how-to/prettify.html', +# You can also configure redirects in the Read the Docs project dashboard +# (see https://docs.readthedocs.io/en/stable/guides/redirects.html). +# NOTE: If this variable is not defined, set to None, or the dictionary is empty, +# the sphinx_reredirects extension will be disabled. +redirects = {} + +############################################################ +### Link checker exceptions +############################################################ + +# Links to ignore when checking links +linkcheck_ignore = [ + 'http://127.0.0.1:8000' + ] + +# Pages on which to ignore anchors +# (This list will be appended to linkcheck_anchors_ignore_for_url) +custom_linkcheck_anchors_ignore_for_url = [] + +############################################################ +### Additions to default configuration +############################################################ + +## The following settings are appended to the default configuration. +## Use them to extend the default functionality. + +# Remove this variable to disable the MyST parser extensions. +custom_myst_extensions = [] + +# Add custom Sphinx extensions as needed. +# This array contains recommended extensions that should be used. +# NOTE: The following extensions are handled automatically and do +# not need to be added here: myst_parser, sphinx_copybutton, sphinx_design, +# sphinx_reredirects, sphinxcontrib.jquery, sphinxext.opengraph +custom_extensions = [ + 'sphinx_tabs.tabs', + 'canonical.youtube-links', + 'canonical.related-links', + 'canonical.custom-rst-roles', + 'canonical.terminal-output', + 'notfound.extension', + 'sphinxcontrib.cairosvgconverter', + ] + +# Add custom required Python modules that must be added to the +# .sphinx/requirements.txt file. +# NOTE: The following modules are handled automatically and do not need to be +# added here: canonical-sphinx-extensions, furo, linkify-it-py, myst-parser, +# pyspelling, sphinx, sphinx-autobuild, sphinx-copybutton, sphinx-design, +# sphinx-notfound-page, sphinx-reredirects, sphinx-tabs, sphinxcontrib-jquery, +# sphinxext-opengraph +custom_required_modules = [ + 'sphinxcontrib-svg2pdfconverter[CairoSVG]', +] + +# Add files or directories that should be excluded from processing. +custom_excludes = [ + 'doc-cheat-sheet*', + '_parts/*', + 'src/_parts' + ] + +# Add CSS files (located in .sphinx/_static/) +custom_html_css_files = [] + +# Add JavaScript files (located in .sphinx/_static/) +custom_html_js_files = [] + +## The following settings override the default configuration. + +# Specify a reST string that is included at the end of each file. +# If commented out, use the default (which pulls the reuse/links.txt +# file into each reST file). +# custom_rst_epilog = '' + +# By default, the documentation includes a feedback button at the top. +# You can disable it by setting the following configuration to True. +disable_feedback_button = False + +# Add tags that you want to use for conditional inclusion of text +# (https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#tags) +custom_tags = [] + +# If you are using the :manpage: role, set this variable to the URL for the version +# that you want to link to: +# manpages_url = "https://manpages.ubuntu.com/manpages/noble/en/man{section}/{page}.{section}.html" + +############################################################ +### Additional configuration +############################################################ + +## Add any configuration that is not covered by the common conf.py file. + +# Change the default code highlighting to 'none' + +highlight_language = 'none' + + + +# Define a :center: role that can be used to center the content of table cells. +rst_prolog = ''' +.. role:: center + :class: align-center +.. role:: h2 + :class: hclass2 +''' diff --git a/docs/src/index.md b/docs/canonicalk8s/index.md similarity index 60% rename from docs/src/index.md rename to docs/canonicalk8s/index.md index 1635cc3f5..f84017892 100644 --- a/docs/src/index.md +++ b/docs/canonicalk8s/index.md @@ -19,53 +19,29 @@ Home :hidden: :titlesonly: :maxdepth: 6 -:caption: Deploy from Snap package -Overview -snap/tutorial/index -snap/howto/index -snap/explanation/index -snap/reference/index -``` -```{toctree} -:hidden: -:caption: Deploy with Juju -:titlesonly: -:glob: -Overview -charm/tutorial/index -charm/howto/index -charm/explanation/index -charm/reference/index -``` +about.md +Deploy from Snap package +Deploy with Juju +Deploy with Cluster API +Community +Release notes -```{toctree} -:hidden: -:caption: Deploy with Cluster API (WIP) -:titlesonly: -:glob: -Overview -capi/tutorial/index -capi/howto/index -capi/explanation/index -capi/reference/index ``` ---- - ````{grid} 1 1 2 2 ```{grid-item-card} -:link: snap/ -### [Install K8s from a snap ›](snap/index) +:link: src/snap/ +### [Install K8s from a snap ›](src/snap/index) ^^^ Our tutorials, How To guides and other pages will explain how to install, - configure and use the {{product}} 'k8s' snap. + configure and use the {{product}} 'k8s' snap. This is a great option if you are new to Kubernetes. ``` ```{grid-item-card} -:link: charm/ -### [Deploy K8s using Juju ›](charm/index) +:link: src/charm/ +### [Deploy K8s using Juju ›](src/charm/index) ^^^ Our tutorials, How To guides and other pages will explain how to install, configure and use the {{product}} 'k8s' charm. @@ -73,12 +49,20 @@ Our tutorials, How To guides and other pages will explain how to install, ```{grid-item-card} -:link: capi/ -### [Deploy K8s using Cluster API ›](capi/index) +:link: src/capi/ +### [Deploy K8s using Cluster API ›](src/capi/index) ^^^ Our tutorials, guides and explanation pages will explain how to install, configure and use {{product}} through CAPI. ``` + +```{grid-item-card} +:link: about +### [Overview of {{product}} ›](about) +^^^ +Find out more about {{product}}, what services are included and get the +answers to some common questions. +``` ```` --- @@ -101,8 +85,8 @@ and constructive feedback. [Code of Conduct]: https://ubuntu.com/community/ethos/code-of-conduct -[community]: snap/reference/community -[contribute]: snap/howto/contribute -[roadmap]: snap/reference/roadmap -[overview page]: snap/explanation/about -[architecture documentation]: snap/reference/architecture +[community]: src/snap/reference/community +[contribute]: src/snap/howto/contribute +[roadmap]: src/snap/reference/roadmap +[overview page]: about +[architecture documentation]: src/snap/reference/architecture diff --git a/docs/canonicalk8s/make.bat b/docs/canonicalk8s/make.bat new file mode 100644 index 000000000..32bb24529 --- /dev/null +++ b/docs/canonicalk8s/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=_build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/docs/canonicalk8s/metrics/scripts/build_metrics.sh b/docs/canonicalk8s/metrics/scripts/build_metrics.sh new file mode 100755 index 000000000..610724e2a --- /dev/null +++ b/docs/canonicalk8s/metrics/scripts/build_metrics.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +links=0 +images=0 + +# count number of links +links=$(find . -type d -path './.sphinx/src' -prune -o -name '*.html' -exec cat {} + | grep -o " + + +### cluster-config.network +**Type:** `object`
+ +Configuration options for the network feature. + +### cluster-config.network.enabled +**Type:** `bool`
+ +Determines if the feature should be enabled. +If omitted defaults to `true` + +### cluster-config.dns +**Type:** `object`
+ +Configuration options for the dns feature. + +### cluster-config.dns.enabled +**Type:** `bool`
+ +Determines if the feature should be enabled. +If omitted defaults to `true` + +### cluster-config.dns.cluster-domain +**Type:** `string`
+ +Sets the local domain of the cluster. +If omitted defaults to `cluster.local`. + +### cluster-config.dns.service-ip +**Type:** `string`
+ +Sets the IP address of the dns service. If omitted defaults to the IP address +of the Kubernetes service created by the feature. + +Can be used to point to an external dns server when feature is disabled. + +### cluster-config.dns.upstream-nameservers +**Type:** `[]string`
+ +Sets the upstream nameservers used to forward queries for out-of-cluster +endpoints. + +If omitted defaults to `/etc/resolv.conf` and uses the nameservers of the node. + +### cluster-config.ingress +**Type:** `object`
+ +Configuration options for the ingress feature. + +### cluster-config.ingress.enabled +**Type:** `bool`
+ +Determines if the feature should be enabled. +If omitted defaults to `false` + +### cluster-config.ingress.default-tls-secret +**Type:** `string`
+ +Sets the name of the secret to be used for providing default encryption to +ingresses. + +Ingresses can specify another TLS secret in their resource definitions, +in which case the default secret won't be used. + +### cluster-config.ingress.enable-proxy-protocol +**Type:** `bool`
+ +Determines if the proxy protocol should be enabled for ingresses. +If omitted defaults to `false`. + +### cluster-config.load-balancer +**Type:** `object`
+ +Configuration options for the load-balancer feature. + +### cluster-config.load-balancer.enabled +**Type:** `bool`
+ +Determines if the feature should be enabled. +If omitted defaults to `false`. + +### cluster-config.load-balancer.cidrs +**Type:** `[]string`
+ +Sets the CIDRs used for assigning IP addresses to Kubernetes services with type +`LoadBalancer`. + +### cluster-config.load-balancer.l2-mode +**Type:** `bool`
+ +Determines if L2 mode should be enabled. +If omitted defaults to `false`. + +### cluster-config.load-balancer.l2-interfaces +**Type:** `[]string`
+ +Sets the interfaces to be used for announcing IP addresses through ARP. +If omitted all interfaces will be used. + +### cluster-config.load-balancer.bgp-mode +**Type:** `bool`
+ +Determines if BGP mode should be enabled. +If omitted defaults to `false`. + +### cluster-config.load-balancer.bgp-local-asn +**Type:** `int`
+ +Sets the ASN to be used for the local virtual BGP router. +Required if bgp-mode is true. + +### cluster-config.load-balancer.bgp-peer-address +**Type:** `string`
+ +Sets the IP address of the BGP peer. +Required if bgp-mode is true. + +### cluster-config.load-balancer.bgp-peer-asn +**Type:** `int`
+ +Sets the ASN of the BGP peer. +Required if bgp-mode is true. + +### cluster-config.load-balancer.bgp-peer-port +**Type:** `int`
+ +Sets the port of the BGP peer. +Required if bgp-mode is true. + +### cluster-config.local-storage +**Type:** `object`
+ +Configuration options for the local-storage feature. + +### cluster-config.local-storage.enabled +**Type:** `bool`
+ +Determines if the feature should be enabled. +If omitted defaults to `false`. + +### cluster-config.local-storage.local-path +**Type:** `string`
+ +Sets the path to be used for storing volume data. +If omitted defaults to `/var/snap/k8s/common/rawfile-storage` + +### cluster-config.local-storage.reclaim-policy +**Type:** `string`
+ +Sets the reclaim policy of the storage class. +If omitted defaults to `Delete`. +Possible values: `Retain | Recycle | Delete` + +### cluster-config.local-storage.default +**Type:** `bool`
+ +Determines if the storage class should be set as default. +If omitted defaults to `true` + +### cluster-config.gateway +**Type:** `object`
+ +Configuration options for the gateway feature. + +### cluster-config.gateway.enabled +**Type:** `bool`
+ +Determines if the feature should be enabled. +If omitted defaults to `true`. + +### cluster-config.metrics-server +**Type:** `object`
+ +Configuration options for the metric server feature. + +### cluster-config.metrics-server.enabled +**Type:** `bool`
+ +Determines if the feature should be enabled. +If omitted defaults to `true`. + +### cluster-config.cloud-provider +**Type:** `string`
+ +Sets the cloud provider to be used by the cluster. + +When this is set as `external`, node will wait for an external cloud provider to +do cloud specific setup and finish node initialization. + +Possible values: `external`. + +### cluster-config.annotations +**Type:** `map[string]string`
+ +Annotations is a map of strings that can be used to store arbitrary metadata configuration. +Please refer to the annotations reference for further details on these options. + +### control-plane-taints +**Type:** `[]string`
+ +List of taints to be applied to control plane nodes. + +### pod-cidr +**Type:** `string`
+ +The CIDR to be used for assigning pod addresses. +If omitted defaults to `10.1.0.0/16`. + +### service-cidr +**Type:** `string`
+ +The CIDR to be used for assigning service addresses. +If omitted defaults to `10.152.183.0/24`. + +### disable-rbac +**Type:** `bool`
+ +Determines if RBAC should be disabled. +If omitted defaults to `false`. + +### secure-port +**Type:** `int`
+ +The port number for kube-apiserver to use. +If omitted defaults to `6443`. + +### k8s-dqlite-port +**Type:** `int`
+ +The port number for k8s-dqlite to use. +If omitted defaults to `9000`. + +### datastore-type +**Type:** `string`
+ +The type of datastore to be used. +If omitted defaults to `k8s-dqlite`. + +Can be used to point to an external datastore like etcd. + +Possible Values: `k8s-dqlite | external`. + +### datastore-servers +**Type:** `[]string`
+ +The server addresses to be used when `datastore-type` is set to `external`. + +### datastore-ca-crt +**Type:** `string`
+ +The CA certificate to be used when communicating with the external datastore. + +### datastore-client-crt +**Type:** `string`
+ +The client certificate to be used when communicating with the external +datastore. + +### datastore-client-key +**Type:** `string`
+ +The client key to be used when communicating with the external datastore. + +### extra-sans +**Type:** `[]string`
+ +List of extra SANs to be added to certificates. + +### ca-crt +**Type:** `string`
+ +The CA certificate to be used for Kubernetes services. +If omitted defaults to an auto generated certificate. + +### ca-key +**Type:** `string`
+ +The CA key to be used for Kubernetes services. +If omitted defaults to an auto generated key. + +### client-ca-crt +**Type:** `string`
+ +The client CA certificate to be used for Kubernetes services. +If omitted defaults to an auto generated certificate. + +### client-ca-key +**Type:** `string`
+ +The client CA key to be used for Kubernetes services. +If omitted defaults to an auto generated key. + +### front-proxy-ca-crt +**Type:** `string`
+ +The CA certificate to be used for the front proxy. +If omitted defaults to an auto generated certificate. + +### front-proxy-ca-key +**Type:** `string`
+ +The CA key to be used for the front proxy. +If omitted defaults to an auto generated key. + +### front-proxy-client-crt +**Type:** `string`
+ +The client certificate to be used for the front proxy. +If omitted defaults to an auto generated certificate. + +### front-proxy-client-key +**Type:** `string`
+ +The client key to be used for the front proxy. +If omitted defaults to an auto generated key. + +### apiserver-kubelet-client-crt +**Type:** `string`
+ +The client certificate to be used by kubelet for communicating with the kube-apiserver. +If omitted defaults to an auto generated certificate. + +### apiserver-kubelet-client-key +**Type:** `string`
+ +The client key to be used by kubelet for communicating with the kube-apiserver. +If omitted defaults to an auto generated key. + +### admin-client-crt +**Type:** `string`
+ +The admin client certificate to be used for Kubernetes services. +If omitted defaults to an auto generated certificate. + +### admin-client-key +**Type:** `string`
+ +The admin client key to be used for Kubernetes services. +If omitted defaults to an auto generated key. + +### kube-proxy-client-crt +**Type:** `string`
+ +The client certificate to be used for the kube-proxy. +If omitted defaults to an auto generated certificate. + +### kube-proxy-client-key +**Type:** `string`
+ +The client key to be used for the kube-proxy. +If omitted defaults to an auto generated key. + +### kube-scheduler-client-crt +**Type:** `string`
+ +The client certificate to be used for the kube-scheduler. +If omitted defaults to an auto generated certificate. + +### kube-scheduler-client-key +**Type:** `string`
+ +The client key to be used for the kube-scheduler. +If omitted defaults to an auto generated key. + +### kube-controller-manager-client-crt +**Type:** `string`
+ +The client certificate to be used for the Kubernetes controller manager. +If omitted defaults to an auto generated certificate. + +### kube-controller-manager-client-key +**Type:** `string`
+ +The client key to be used for the Kubernetes controller manager. +If omitted defaults to an auto generated key. + +### service-account-key +**Type:** `string`
+ +The key to be used by the default service account. +If omitted defaults to an auto generated key. + +### apiserver-crt +**Type:** `string`
+ +The certificate to be used for the kube-apiserver. +If omitted defaults to an auto generated certificate. + +### apiserver-key +**Type:** `string`
+ +The key to be used for the kube-apiserver. +If omitted defaults to an auto generated key. + +### kubelet-crt +**Type:** `string`
+ +The certificate to be used for the kubelet. +If omitted defaults to an auto generated certificate. + +### kubelet-key +**Type:** `string`
+ +The key to be used for the kubelet. +If omitted defaults to an auto generated key. + +### kubelet-client-crt +**Type:** `string`
+ +The certificate to be used for the kubelet client. +If omitted defaults to an auto generated certificate. + +### kubelet-client-key +**Type:** `string`
+ +The key to be used for the kubelet client. +If omitted defaults to an auto generated key. + +### extra-node-config-files +**Type:** `map[string]string`
+ +Additional files that are uploaded `/var/snap/k8s/common/args/conf.d/` +to a node on bootstrap. These files can then be referenced by Kubernetes +service arguments. + +The format is `map[]`. + +### extra-node-kube-apiserver-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kube-apiserver` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-kube-controller-manager-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kube-controller-manager` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-kube-scheduler-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kube-scheduler` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-kube-proxy-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kube-proxy` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-kubelet-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kubelet` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-containerd-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to `containerd` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-k8s-dqlite-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to `k8s-dqlite` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-containerd-config +**Type:** `apiv1.MapStringAny`
+ +Extra configuration for the containerd config.toml + diff --git a/docs/src/_parts/control_plane_join_config.md b/docs/src/_parts/control_plane_join_config.md new file mode 100644 index 000000000..fa2919e45 --- /dev/null +++ b/docs/src/_parts/control_plane_join_config.md @@ -0,0 +1,152 @@ +### extra-sans +**Type:** `[]string`
+ +List of extra SANs to be added to certificates. + +### front-proxy-client-crt +**Type:** `string`
+ +The client certificate to be used for the front proxy. +If omitted defaults to an auto generated certificate. + +### front-proxy-client-key +**Type:** `string`
+ +The client key to be used for the front proxy. +If omitted defaults to an auto generated key. + +### kube-proxy-client-crt +**Type:** `string`
+ +The client certificate to be used by kubelet for communicating with the kube-apiserver. +If omitted defaults to an auto generated certificate. + +### kube-proxy-client-key +**Type:** `string`
+ +The client key to be used by kubelet for communicating with the kube-apiserver. +If omitted defaults to an auto generated key. + +### kube-scheduler-client-crt +**Type:** `string`
+ +The client certificate to be used for the kube-scheduler. +If omitted defaults to an auto generated certificate. + +### kube-scheduler-client-key +**Type:** `string`
+ +The client key to be used for the kube-scheduler. +If omitted defaults to an auto generated key. + +### kube-controller-manager-client-crt +**Type:** `string`
+ +The client certificate to be used for the Kubernetes controller manager. +If omitted defaults to an auto generated certificate. + +### kube-controller-manager-client-key +**Type:** `string`
+ +The client key to be used for the Kubernetes controller manager. +If omitted defaults to an auto generated key. + +### apiserver-crt +**Type:** `string`
+ +The certificate to be used for the kube-apiserver. +If omitted defaults to an auto generated certificate. + +### apiserver-key +**Type:** `string`
+ +The key to be used for the kube-apiserver. +If omitted defaults to an auto generated key. + +### kubelet-crt +**Type:** `string`
+ +The certificate to be used for the kubelet. +If omitted defaults to an auto generated certificate. + +### kubelet-key +**Type:** `string`
+ +The key to be used for the kubelet. +If omitted defaults to an auto generated key. + +### kubelet-client-crt +**Type:** `string`
+ +The client certificate to be used for the kubelet. +If omitted defaults to an auto generated certificate. + +### kubelet-client-key +**Type:** `string`
+ +The client key to be used for the kubelet. +If omitted defaults to an auto generated key. + +### extra-node-config-files +**Type:** `map[string]string`
+ +Additional files that are uploaded `/var/snap/k8s/common/args/conf.d/` +to a node on bootstrap. These files can then be referenced by Kubernetes +service arguments. + +The format is `map[]`. + +### extra-node-kube-apiserver-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kube-apiserver` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-kube-controller-manager-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kube-controller-manager` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-kube-scheduler-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kube-scheduler` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-kube-proxy-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kube-proxy` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-kubelet-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kubelet` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-containerd-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to `containerd` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-k8s-dqlite-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to `k8s-dqlite` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-containerd-config +**Type:** `apiv1.MapStringAny`
+ +Extra configuration for the containerd config.toml + diff --git a/docs/src/_parts/install.md b/docs/src/_parts/install.md new file mode 100644 index 000000000..be6b47317 --- /dev/null +++ b/docs/src/_parts/install.md @@ -0,0 +1,3 @@ +``` +sudo snap install k8s --classic --channel=1.31/edge +``` \ No newline at end of file diff --git a/docs/src/_parts/template-explanation b/docs/src/_parts/template-explanation index 905a30f07..3ec1911ae 100644 --- a/docs/src/_parts/template-explanation +++ b/docs/src/_parts/template-explanation @@ -15,11 +15,6 @@ The documentation also supports various diagrams-as-code options. We prefer to use UML-style diagrams, but you can also use Mermaid or many other types. -Diagrams like this are processed using the 'kroki' directive: - -```{kroki} ../../assets/ck-cluster.puml -``` - ## Links Explanations frequently include links to other documents. In particular, please diff --git a/docs/src/_parts/template-tutorial b/docs/src/_parts/template-tutorial index 039f0ee52..d1d46591e 100644 --- a/docs/src/_parts/template-tutorial +++ b/docs/src/_parts/template-tutorial @@ -68,7 +68,7 @@ workload and remove everything again! ## Next Steps -- Keep mastering Canonical Kubernetes with kubectl: [How to use kubectl] +- How to control {{product}} with `kubectl`: [How to use kubectl] - Explore Kubernetes commands with our [Command Reference Guide] - Learn how to set up a multi-node environment [Setting up a K8s cluster] - Configure storage options [Storage] diff --git a/docs/src/_parts/worker_join_config.md b/docs/src/_parts/worker_join_config.md new file mode 100644 index 000000000..70a515a8f --- /dev/null +++ b/docs/src/_parts/worker_join_config.md @@ -0,0 +1,78 @@ +### kubelet-crt +**Type:** `string`
+ +The certificate to be used for the kubelet. +If omitted defaults to an auto generated certificate. + +### kubelet-key +**Type:** `string`
+ +The key to be used for the kubelet. +If omitted defaults to an auto generated key. + +### kubelet-client-crt +**Type:** `string`
+ +The client certificate to be used for the kubelet. +If omitted defaults to an auto generated certificate. + +### kubelet-client-key +**Type:** `string`
+ +The client key to be used for the kubelet. +If omitted defaults to an auto generated key. + +### kube-proxy-client-crt +**Type:** `string`
+ +The client certificate to be used for the kube-proxy. +If omitted defaults to an auto generated certificate. + +### kube-proxy-client-key +**Type:** `string`
+ +The client key to be used for the kube-proxy. +If omitted defaults to an auto generated key. + +### extra-node-config-files +**Type:** `map[string]string`
+ +Additional files that are uploaded `/var/snap/k8s/common/args/conf.d/` +to a node on bootstrap. These files can then be referenced by Kubernetes +service arguments. + +The format is `map[]`. + +### extra-node-kube-proxy-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kube-proxy` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-kubelet-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to the `kubelet` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-containerd-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to `containerd` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-k8s-apiserver-proxy-args +**Type:** `map[string]string`
+ +Additional arguments that are passed to `k8s-api-server-proxy` only for that specific node. +A parameter that is explicitly set to `null` is deleted. +The format is `map[<--flag-name>]`. + +### extra-node-containerd-config +**Type:** `apiv1.MapStringAny`
+ +Extra configuration for the containerd config.toml + diff --git a/docs/src/capi/explanation/capi-ck8s.svg b/docs/src/assets/capi-ck8s.svg similarity index 100% rename from docs/src/capi/explanation/capi-ck8s.svg rename to docs/src/assets/capi-ck8s.svg diff --git a/docs/src/assets/how-to-cloud-storage-aws-ccm.yaml b/docs/src/assets/how-to-cloud-storage-aws-ccm.yaml new file mode 100644 index 000000000..fa6dc3cb9 --- /dev/null +++ b/docs/src/assets/how-to-cloud-storage-aws-ccm.yaml @@ -0,0 +1,170 @@ +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: aws-cloud-controller-manager + namespace: kube-system + labels: + k8s-app: aws-cloud-controller-manager +spec: + selector: + matchLabels: + k8s-app: aws-cloud-controller-manager + updateStrategy: + type: RollingUpdate + template: + metadata: + labels: + k8s-app: aws-cloud-controller-manager + spec: + nodeSelector: + node-role.kubernetes.io/control-plane: "" + tolerations: + - key: node.cloudprovider.kubernetes.io/uninitialized + value: "true" + effect: NoSchedule + - effect: NoSchedule + key: node-role.kubernetes.io/control-plane + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/control-plane + operator: Exists + serviceAccountName: cloud-controller-manager + containers: + - name: aws-cloud-controller-manager + image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.28.3 + args: + - --v=2 + - --cloud-provider=aws + - --use-service-account-credentials=true + - --configure-cloud-routes=false + resources: + requests: + cpu: 200m + hostNetwork: true +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: cloud-controller-manager + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: cloud-controller-manager:apiserver-authentication-reader + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: extension-apiserver-authentication-reader +subjects: + - apiGroup: "" + kind: ServiceAccount + name: cloud-controller-manager + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: system:cloud-controller-manager +rules: +- apiGroups: + - "" + resources: + - events + verbs: + - create + - patch + - update +- apiGroups: + - "" + resources: + - nodes + verbs: + - '*' +- apiGroups: + - "" + resources: + - nodes/status + verbs: + - patch +- apiGroups: + - "" + resources: + - services + verbs: + - list + - patch + - update + - watch +- apiGroups: + - "" + resources: + - services/status + verbs: + - list + - patch + - update + - watch +- apiGroups: + - "" + resources: + - serviceaccounts + verbs: + - create + - get + - list + - watch +- apiGroups: + - "" + resources: + - persistentvolumes + verbs: + - get + - list + - update + - watch +- apiGroups: + - "" + resources: + - endpoints + verbs: + - create + - get + - list + - watch + - update +- apiGroups: + - coordination.k8s.io + resources: + - leases + verbs: + - create + - get + - list + - watch + - update +- apiGroups: + - "" + resources: + - serviceaccounts/token + verbs: + - create +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: system:cloud-controller-manager +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:cloud-controller-manager +subjects: + - apiGroup: "" + kind: ServiceAccount + name: cloud-controller-manager + namespace: kube-system diff --git a/docs/src/assets/k8sd-component.puml b/docs/src/assets/k8sd-component.puml index f95cd278c..3e39ce90b 100644 --- a/docs/src/assets/k8sd-component.puml +++ b/docs/src/assets/k8sd-component.puml @@ -16,13 +16,13 @@ Container(K8sSnapDistribution.State, "State", $techn="", $descr="Datastores hold Container(K8sSnapDistribution.KubernetesServices, "Kubernetes Services", $techn="", $descr="API server, kubelet, kube-proxy, scheduler, kube-controller", $tags="", $link="") Container_Boundary("K8sSnapDistribution.K8sd_boundary", "K8sd", $tags="") { - Component(K8sSnapDistribution.K8sd.CLI, "CLI", $techn="CLI", $descr="The CLI the offered", $tags="", $link="") + Component(K8sSnapDistribution.K8sd.CLI, "CLI", $techn="CLI", $descr="The CLI offered", $tags="", $link="") Component(K8sSnapDistribution.K8sd.APIviaHTTP, "API via HTTP", $techn="REST", $descr="The API interface offered", $tags="", $link="") - Component(K8sSnapDistribution.K8sd.CLustermanagement, "CLuster management", $techn="", $descr="Management of the cluster with the help of MicroCluster", $tags="", $link="") + Component(K8sSnapDistribution.K8sd.CLustermanagement, "Cluster management", $techn="", $descr="Management of the cluster with the help of MicroCluster", $tags="", $link="") } Rel(K8sAdmin, K8sSnapDistribution.K8sd.CLI, "Sets up and configured the cluster", $techn="", $tags="", $link="") -Rel(CharmK8s, K8sSnapDistribution.K8sd.APIviaHTTP, "Orchestrates the lifecycle management of K8s", $techn="", $tags="", $link="") +Rel(CharmK8s, K8sSnapDistribution.K8sd.APIviaHTTP, "Orchestrates the lifecycle management of K8s when deployed with Juju", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd.CLustermanagement, K8sSnapDistribution.KubernetesServices, "Configures", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.KubernetesServices, K8sSnapDistribution.State, "Uses by default", $techn="", $tags="", $link="") Rel(K8sSnapDistribution.K8sd.CLustermanagement, K8sSnapDistribution.State, "Keeps state in", $techn="", $tags="", $link="") diff --git a/docs/src/capi/explanation/capi-ck8s.md b/docs/src/capi/explanation/capi-ck8s.md index d75db76ac..5ba3487b3 100644 --- a/docs/src/capi/explanation/capi-ck8s.md +++ b/docs/src/capi/explanation/capi-ck8s.md @@ -1,12 +1,26 @@ # Cluster API - {{product}} -ClusterAPI (CAPI) is an open-source Kubernetes project that provides a declarative API for cluster creation, configuration, and management. It is designed to automate the creation and management of Kubernetes clusters in various environments, including on-premises data centers, public clouds, and edge devices. - -CAPI abstracts away the details of infrastructure provisioning, networking, and other low-level tasks, allowing users to define their desired cluster configuration using simple YAML manifests. This makes it easier to create and manage clusters in a repeatable and consistent manner, regardless of the underlying infrastructure. In this way a wide range of infrastructure providers has been made available, including but not limited to Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), and OpenStack. - -CAPI also abstracts the provisioning and management of Kubernetes clusters allowing for a variety of Kubernetes distributions to be delivered in all of the supported infrastructure providers. {{product}} is one such Kubernetes distribution that seamlessly integrates with Cluster API. +ClusterAPI (CAPI) is an open-source Kubernetes project that provides a +declarative API for cluster creation, configuration, and management. It is +designed to automate the creation and management of Kubernetes clusters in +various environments, including on-premises data centres, public clouds, and +edge devices. + +CAPI abstracts away the details of infrastructure provisioning, networking, and +other low-level tasks, allowing users to define their desired cluster +configuration using simple YAML manifests. This makes it easier to create and +manage clusters in a repeatable and consistent manner, regardless of the +underlying infrastructure. In this way a wide range of infrastructure providers +has been made available, including but not limited to Amazon Web Services +(AWS), Microsoft Azure, Google Cloud Platform (GCP), and OpenStack. + +CAPI also abstracts the provisioning and management of Kubernetes clusters +allowing for a variety of Kubernetes distributions to be delivered in all of +the supported infrastructure providers. {{product}} is one such Kubernetes +distribution that seamlessly integrates with Cluster API. With {{product}} CAPI you can: + - provision a cluster with: - Kubernetes version 1.31 onwards - risk level of the track you want to follow (stable, candidate, beta, edge) @@ -20,21 +34,59 @@ Please refer to the “Tutorial” section for concrete examples on CAPI deploym ## CAPI architecture -Being a cloud-native framework, CAPI implements all its components as controllers that run within a Kubernetes cluster. There is a separate controller, called a ‘provider’, for each supported infrastructure substrate. The infrastructure providers are responsible for provisioning physical or virtual nodes and setting up networking elements such as load balancers and virtual networks. In a similar way, each Kubernetes distribution that integrates with ClusterAPI is managed by two providers: the control plane provider and the bootstrap provider. The bootstrap provider is responsible for delivering and managing Kubernetes on the nodes, while the control plane provider handles the control plane’s specific lifecycle. - -The CAPI providers operate within a Kubernetes cluster known as the management cluster. The administrator is responsible for selecting the desired combination of infrastructure and Kubernetes distribution by instantiating the respective infrastructure, bootstrap, and control plane providers on the management cluster. - -The management cluster functions as the control plane for the ClusterAPI operator, which is responsible for provisioning and managing the infrastructure resources necessary for creating and managing additional Kubernetes clusters. It is important to note that the management cluster is not intended to support any other workload, as the workloads are expected to run on the provisioned clusters. As a result, the provisioned clusters are referred to as workload clusters. - -Typically, the management cluster runs in a separate environment from the clusters it manages, such as a public cloud or an on-premises data center. It serves as a centralized location for managing the configuration, policies, and security of multiple managed clusters. By leveraging the management cluster, users can easily create and manage a fleet of Kubernetes clusters in a consistent and repeatable manner. +Being a cloud-native framework, CAPI implements all its components as +controllers that run within a Kubernetes cluster. There is a separate +controller, called a ‘provider’, for each supported infrastructure substrate. +The infrastructure providers are responsible for provisioning physical or +virtual nodes and setting up networking elements such as load balancers and +virtual networks. In a similar way, each Kubernetes distribution that +integrates with ClusterAPI is managed by two providers: the control plane +provider and the bootstrap provider. The bootstrap provider is responsible for +delivering and managing Kubernetes on the nodes, while the control plane +provider handles the control plane’s specific lifecycle. + +The CAPI providers operate within a Kubernetes cluster known as the management +cluster. The administrator is responsible for selecting the desired combination +of infrastructure and Kubernetes distribution by instantiating the respective +infrastructure, bootstrap, and control plane providers on the management +cluster. + +The management cluster functions as the control plane for the ClusterAPI +operator, which is responsible for provisioning and managing the infrastructure +resources necessary for creating and managing additional Kubernetes clusters. +It is important to note that the management cluster is not intended to support +any other workload, as the workloads are expected to run on the provisioned +clusters. As a result, the provisioned clusters are referred to as workload +clusters. + +Typically, the management cluster runs in a separate environment from the +clusters it manages, such as a public cloud or an on-premises data centre. It +serves as a centralised location for managing the configuration, policies, and +security of multiple managed clusters. By leveraging the management cluster, +users can easily create and manage a fleet of Kubernetes clusters in a +consistent and repeatable manner. The {{product}} team maintains the two providers required for integrating with CAPI: -- The Cluster API Bootstrap Provider {{product}} (**CABPCK**) responsible for provisioning the nodes in the cluster and preparing them to be joined to the Kubernetes control plane. When you use the CABPCK you define a Kubernetes Cluster object that describes the desired state of the new cluster and includes the number and type of nodes in the cluster, as well as any additional configuration settings. The Bootstrap Provider then creates the necessary resources in the Kubernetes API server to bring the cluster up to the desired state. Under the hood, the Bootstrap Provider uses cloud-init to configure the nodes in the cluster. This includes setting up SSH keys, configuring the network, and installing necessary software packages. - -- The Cluster API Control Plane Provider {{product}} (**CACPCK**) enables the creation and management of Kubernetes control planes using {{product}} as the underlying Kubernetes distribution. Its main tasks are to update the machine state and to generate the kubeconfig file used for accessing the cluster. The kubeconfig file is stored as a secret which the user can then retrieve using the `clusterctl` command. - -```{figure} ./capi-ck8s.svg +- The Cluster API Bootstrap Provider {{product}} (**CABPCK**) responsible for + provisioning the nodes in the cluster and preparing them to be joined to the + Kubernetes control plane. When you use the CABPCK you define a Kubernetes + Cluster object that describes the desired state of the new cluster and + includes the number and type of nodes in the cluster, as well as any + additional configuration settings. The Bootstrap Provider then creates the + necessary resources in the Kubernetes API server to bring the cluster up to + the desired state. Under the hood, the Bootstrap Provider uses cloud-init to + configure the nodes in the cluster. This includes setting up SSH keys, + configuring the network, and installing necessary software packages. + +- The Cluster API Control Plane Provider {{product}} (**CACPCK**) enables the + creation and management of Kubernetes control planes using {{product}} as the + underlying Kubernetes distribution. Its main tasks are to update the machine + state and to generate the kubeconfig file used for accessing the cluster. The + kubeconfig file is stored as a secret which the user can then retrieve using + the `clusterctl` command. + +```{figure} ../../assets/capi-ck8s.svg :width: 100% :alt: Deployment of components diff --git a/docs/src/capi/explanation/in-place-upgrades.md b/docs/src/capi/explanation/in-place-upgrades.md new file mode 100644 index 000000000..5196fd7d1 --- /dev/null +++ b/docs/src/capi/explanation/in-place-upgrades.md @@ -0,0 +1,132 @@ +# In-Place Upgrades + +Regularly upgrading the Kubernetes version of the machines in a cluster +is important. While rolling upgrades are a popular strategy, certain +situations will require in-place upgrades: + +- Resource constraints (i.e. cost of additional machines). +- Expensive manual setup process for nodes. + +## Annotations + +CAPI machines are considered immutable. Consequently, machines are replaced +instead of reconfigured. +While CAPI doesn't support in-place upgrades, {{product}} CAPI does +by leveraging annotations for the implementation. +For a deeper understanding of the CAPI design decisions, consider reading about +[machine immutability in CAPI][1], and Kubernetes objects: [`labels`][2], +[`spec` and `status`][3]. + +## Controllers + +In {{product}} CAPI, there are two main types of controllers that handle the +process of performing in-place upgrades: + +- Single Machine In-Place Upgrade Controller +- Orchestrated In-Place Upgrade Controller + +The core component of performing an in-place upgrade is the `Single Machine +Upgrader`. The controller watches for annotations on machines and reconciles +them to ensure the upgrades happen smoothly. + +The `Orchestrator` watches for certain annotations on +machine owners, reconciles them and upgrades groups of owned machines. +It’s responsible for ensuring that all the machines owned by the +reconciled object get upgraded successfully. + +The main annotations that drive the upgrade process are as follows: + +- `v1beta2.k8sd.io/in-place-upgrade-to` --> `upgrade-to` : Instructs +the controller to perform an upgrade with the specified option/method. +- `v1beta2.k8sd.io/in-place-upgrade-status` --> `status` : As soon as the +controller starts the upgrade process, the object will be marked with the +`status` annotation which can either be `in-progress`, `failed` or `done`. +- `v1beta2.k8sd.io/in-place-upgrade-release` --> `release` : When the +upgrade is performed successfully, this annotation will indicate the current +Kubernetes release/version installed on the machine. + +For a complete list of annotations and their values please +refer to the [annotations reference page][4]. This explanation proceeds +to use abbreviations of the mentioned labels. + +### Single Machine In-Place Upgrade Controller + +The Machine objects can be marked with the `upgrade-to` annotation to +trigger an in-place upgrade for that machine. While watching for changes +on the machines, the single machine upgrade controller notices this annotation +and attempts to upgrade the Kubernetes version of that machine to the +specified version. + +Upgrade methods or options can be specified to upgrade to a snap channel, +revision, or a local snap file already placed on the +machine in air-gapped environments. + +A successfully upgraded machine shows the following annotations: + +```yaml +annotations: + v1beta2.k8sd.io/in-place-upgrade-release: "channel=1.31/stable" + v1beta2.k8sd.io/in-place-upgrade-status: "done" +``` + +If the upgrade fails, the controller will mark the machine and retry +the upgrade immediately: + +```yaml +annotations: + # the `upgrade-to` causes the retry to happen + v1beta2.k8sd.io/in-place-upgrade-to: "channel=1.31/stable" + v1beta2.k8sd.io/in-place-upgrade-status: "failed" + + # orchestrator will notice this annotation and knows that the + # upgrade for this machine failed + v1beta2.k8sd.io/in-place-upgrade-last-failed-attempt-at: "Sat, 7 Nov + 2024 13:30:00 +0400" +``` + +By applying and removing annotations, the single machine +upgrader determines the upgrade status of the machine it’s trying to +reconcile and takes necessary actions to successfully complete an +in-place upgrade. The following diagram shows the flow of the in-place +upgrade of a single machine: + +![Diagram][img-single-machine] + +### Machine Upgrade Process + +The {{product}}'s `k8sd` daemon exposes endpoints that can be used to +interact with the cluster. The single machine upgrader calls the +`/snap/refresh` endpoint on the machine to trigger the upgrade +process while checking `/snap/refresh-status` periodically. + +![Diagram][img-k8sd-call] + +### In-place upgrades on large workload clusters + +While the “Single Machine In-Place Upgrade Controller” is responsible +for upgrading individual machines, the "Orchestrated In-Place Upgrade +Controller" ensures that groups of machines will get upgraded. +By applying the `upgrade-to` annotation on an object that owns machines +(e.g. a `MachineDeployment`), this controller will mark the owned machines +one by one which will cause the "Single Machine Upgrader" to pickup those +annotations and upgrade the machines. To avoid undesirable situations + like quorum loss or severe downtime, these upgrades happen in sequence. + +The failures and successes of individual machine upgrades will be reported back +to the orchestrator by the single machine upgrader via annotations. + +The illustrated flow of orchestrated in-place upgrades: + +![Diagram][img-orchestrated] + + + +[img-single-machine]: https://assets.ubuntu.com/v1/1200f040-single-machine.png +[img-k8sd-call]: https://assets.ubuntu.com/v1/518eb73a-k8sd-call.png +[img-orchestrated]: https://assets.ubuntu.com/v1/8f302a00-orchestrated.png + + +[1]: https://cluster-api.sigs.k8s.io/user/concepts#machine-immutability-in-place-upgrade-vs-replace +[2]: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ +[3]: https://kubernetes.io/docs/concepts/overview/working-with-objects/#object-spec-and-status +[4]: ../reference/annotations.md diff --git a/docs/src/capi/explanation/index.md b/docs/src/capi/explanation/index.md index 775dd26a3..d4ad076be 100644 --- a/docs/src/capi/explanation/index.md +++ b/docs/src/capi/explanation/index.md @@ -11,12 +11,12 @@ Overview ```{toctree} :titlesonly: -:globs: +:glob: about security capi-ck8s.md - +in-place-upgrades.md ``` diff --git a/docs/src/capi/explanation/security.md b/docs/src/capi/explanation/security.md index 6c1048a19..002e071d1 100644 --- a/docs/src/capi/explanation/security.md +++ b/docs/src/capi/explanation/security.md @@ -1,2 +1,2 @@ -```{include} /snap/explanation/security.md +```{include} ../../snap/explanation/security.md ``` diff --git a/docs/src/capi/howto/custom-ck8s.md b/docs/src/capi/howto/custom-ck8s.md index d81191980..ba3a1d0fe 100644 --- a/docs/src/capi/howto/custom-ck8s.md +++ b/docs/src/capi/howto/custom-ck8s.md @@ -1,6 +1,6 @@ # Install custom {{product}} on machines -By default, the `version` field in the machine specifications will determine which {{product}} is downloaded from the `stable` rist level. While you can install different versions of the `stable` risk level by changing the `version` field, extra steps should be taken if you're willing to install a specific risk level. +By default, the `version` field in the machine specifications will determine which {{product}} is downloaded from the `stable` risk level. While you can install different versions of the `stable` risk level by changing the `version` field, extra steps should be taken if you're willing to install a specific risk level. This guide walks you through the process of installing custom {{product}} on workload cluster machines. ## Prerequisites @@ -13,7 +13,7 @@ To follow this guide, you will need: Please refer to the [getting-started guide][getting-started] for further details on the required setup. -In this guide we call the generated cluster spec manifrst `cluster.yaml`. +In this guide we call the generated cluster spec manifest `cluster.yaml`. ## Overwrite the existing `install.sh` script diff --git a/docs/src/capi/howto/external-etcd.md b/docs/src/capi/howto/external-etcd.md index f6509fb22..a77600c68 100644 --- a/docs/src/capi/howto/external-etcd.md +++ b/docs/src/capi/howto/external-etcd.md @@ -9,7 +9,7 @@ with an external etcd. To follow this guide, you will need: -- [Clusterctl][clusterctl] installed +- [clusterctl][clusterctl] installed - A CAPI management cluster initialised with the infrastructure, bootstrap and control plane providers of your choice. Please refer to the [getting-started guide][getting-started] for instructions. @@ -78,7 +78,7 @@ kubectl get secrets ## Update etcd cluster template -Please refer to [capi-templates][capi-templates] for the latest templates. +Please refer to [CAPI-templates][CAPI-templates] for the latest templates. Update the control plane resource `CK8sControlPlane` so that it is configured to store the Kubernetes state in etcd. Add the following additional configuration to the cluster template `cluster-template.yaml`: @@ -120,5 +120,5 @@ clusterctl describe cluster peaches ``` [getting-started]: ../tutorial/getting-started.md -[capi-templates]: https://github.com/canonical/cluster-api-k8s/tree/main/templates +[CAPI-templates]: https://github.com/canonical/cluster-api-k8s/tree/main/templates [clusterctl]: https://cluster-api.sigs.k8s.io/clusterctl/overview diff --git a/docs/src/capi/howto/in-place-upgrades.md b/docs/src/capi/howto/in-place-upgrades.md new file mode 100644 index 000000000..7b20c9dab --- /dev/null +++ b/docs/src/capi/howto/in-place-upgrades.md @@ -0,0 +1,98 @@ +# Perform an in-place upgrade for a machine + +This guide walks you through the steps to perform an in-place upgrade for a +Cluster API managed machine. + +## Prerequisites + +To follow this guide, you will need: + +- A Kubernetes management cluster with Cluster API and providers installed + and configured. +- A target workload cluster managed by CAPI. +- `kubectl` installed and configured to access your management cluster. +- The workload cluster kubeconfig. + +Please refer to the [getting-started guide][getting-started] for further +details on the required setup. +This guide refers to the workload cluster as `c1` and its +kubeconfig as `c1-kubeconfig.yaml`. + +## Check the current cluster status + +Prior to the upgrade, ensure that the management cluster is in a healthy +state. + +``` +kubectl get nodes -o wide +``` + +Confirm the Kubernetes version of the workload cluster: + +``` +kubectl --kubeconfig c1-kubeconfig.yaml get nodes -o wide +``` + +## Annotate the machine + +In this first step, annotate the Machine resource with +the in-place upgrade annotation. In this example, the machine +is called `c1-control-plane-xyzbw`. + +``` +kubectl annotate machine c1-control-plane-xyzbw "v1beta2.k8sd.io/in-place-upgrade-to=" +``` + +`` can be one of: + +* `channel=` which refreshes k8s to the given snap channel. + e.g. `channel=1.30-classic/stable` +* `revision=` which refreshes k8s to the given revision. + e.g. `revision=123` +* `localPath=` which refreshes k8s with the snap file from + the given absolute path. e.g. `localPath=full/path/to/k8s.snap` + +Please refer to the [ClusterAPI Annotations Reference][annotations-reference] +for further details on these options. + +## Monitor the in-place upgrade + +Watch the status of the in-place upgrade for the machine, +by running the following command and checking the +`v1beta2.k8sd.io/in-place-upgrade-status` annotation: + +``` +kubectl get machine c1-control-plane-xyzbw -o yaml +``` + +On a successful upgrade: + +* Value of the `v1beta2.k8sd.io/in-place-upgrade-status` annotation + will be changed to `done` +* Value of the `v1beta2.k8sd.io/in-place-upgrade-release` annotation + will be changed to the `` used to perform the upgrade. + +## Cancelling a failing upgrade + +The upgrade is retried periodically if the operation was unsuccessful. + +The upgrade can be cancelled by running the following commands +that remove the annotations: + +``` +kubectl annotate machine c1-control-plane-xyzbw "v1beta2.k8sd.io/in-place-upgrade-to-" +kubectl annotate machine c1-control-plane-xyzbw "v1beta2.k8sd.io/in-place-upgrade-change-id-" +``` + +## Verify the Kubernetes upgrade + +Confirm that the node is healthy and runs on the new Kubernetes version: + +``` +kubectl --kubeconfig c1-kubeconfig.yaml get nodes -o wide +``` + + + +[getting-started]: ../tutorial/getting-started.md +[annotations-reference]: ../reference/annotations.md diff --git a/docs/src/capi/howto/index.md b/docs/src/capi/howto/index.md index 375a5025a..3bf0cca3a 100644 --- a/docs/src/capi/howto/index.md +++ b/docs/src/capi/howto/index.md @@ -14,11 +14,13 @@ Overview :glob: :titlesonly: -external-etcd +Use external etcd rollout-upgrades +in-place-upgrades upgrade-providers migrate-management custom-ck8s +refresh-certs ``` --- diff --git a/docs/src/capi/howto/migrate-management.md b/docs/src/capi/howto/migrate-management.md index 11a1474f3..f902a0731 100644 --- a/docs/src/capi/howto/migrate-management.md +++ b/docs/src/capi/howto/migrate-management.md @@ -1,4 +1,4 @@ -# Migrate the managment cluster +# Migrate the management cluster Management cluster migration is a really powerful operation in the cluster’s lifecycle as it allows admins to move the management cluster in a more reliable substrate or perform maintenance tasks without disruptions. diff --git a/docs/src/capi/howto/refresh-certs.md b/docs/src/capi/howto/refresh-certs.md new file mode 100644 index 000000000..9f8d3347d --- /dev/null +++ b/docs/src/capi/howto/refresh-certs.md @@ -0,0 +1,107 @@ +# Refreshing Workload Cluster Certificates + +This how-to will walk you through the steps to refresh the certificates for +both control plane and worker nodes in your {{product}} Cluster API cluster. + +## Prerequisites + +- A Kubernetes management cluster with Cluster API and Canonical K8s providers + installed and configured. +- A target workload cluster managed by Cluster API. +- `kubectl` installed and configured to access your management cluster. + +Please refer to the [getting-started guide][getting-started] for further +details on the required setup. +This guide refers to the workload cluster as `c1`. + +```{note} To refresh the certificates in your cluster, make sure it was +initially set up with self-signed certificates. You can verify this by +checking the `CK8sConfigTemplate` resource for the cluster to see if a +`BootstrapConfig` value was provided with the necessary certificates. +``` + +### Refresh Control Plane Node Certificates + +To refresh the certificates on control plane nodes, follow these steps for each +control plane node in your workload cluster: + +1. First, check the names of the control plane machines in your cluster: + +``` +clusterctl describe cluster c1 +``` + +2. For each control plane machine, annotate the machine resource with the +`v1beta2.k8sd.io/refresh-certificates` annotation. The value of the annotation +should specify the duration for which the certificates will be valid. For +example, to refresh the certificates for a control plane machine named +`c1-control-plane-nwlss` to expire in 10 years, run the following command: + +``` +kubectl annotate machine c1-control-plane-nwlss v1beta2.k8sd.io/refresh-certificates=10y +``` + +```{note} The value of the annotation can be specified in years (y), months +(mo), (d) days, or any unit accepted by the [ParseDuration] function in +Go. +``` + +The Cluster API provider will automatically refresh the certificates on the +control plane node and restart the necessary services. To track the progress of +the certificate refresh, check the events for the machine resource: + +``` +kubectl get events --field-selector involvedObject.name=c1-control-plane-nwlss +``` + +The machine will be ready once the event `CertificatesRefreshDone` is +displayed. + +3. After the certificate refresh is complete, the new expiration date will be +displayed in the `machine.cluster.x-k8s.io/certificates-expiry` annotation of +the machine resource: + +``` +"machine.cluster.x-k8s.io/certificates-expiry": "2034-10-25T14:25:23-05:00" +``` + +### Refresh Worker Node Certificates + +To refresh the certificates on worker nodes, follow these steps for each worker +node in your workload cluster: + +1. Check the names of the worker machines in your cluster: + +``` +clusterctl describe cluster c1 +``` + +2. Add the `v1beta2.k8sd.io/refresh-certificates` annotation to each worker +machine, specifying the desired certificate validity duration. For example, to +set the certificates for `c1-worker-md-0-4lxb7-msq44` to expire in 10 years: + +``` +kubectl annotate machine c1-worker-md-0-4lxb7-msq44 v1beta2.k8sd.io/refresh-certificates=10y +``` + +The ClusterAPI provider will handle the certificate refresh and restart +necessary services. Track the progress by checking the machine's events: + +``` +kubectl get events --field-selector involvedObject.name=c1-worker-md-0-4lxb7-msq44 +``` + +The machine will be ready once the event `CertificatesRefreshDone` is +displayed. + +3. After the certificate refresh is complete, the new expiration date will be +displayed in the `machine.cluster.x-k8s.io/certificates-expiry` annotation of +the machine resource: + +``` +"machine.cluster.x-k8s.io/certificates-expiry": "2034-10-25T14:33:04-05:00" +``` + + +[getting-started]: ../tutorial/getting-started.md +[ParseDuration]: https://pkg.go.dev/time#ParseDuration diff --git a/docs/src/capi/index.md b/docs/src/capi/index.md index df4102a01..87b56a3ab 100644 --- a/docs/src/capi/index.md +++ b/docs/src/capi/index.md @@ -1,5 +1,21 @@ # Installing {{product}} with Cluster API +```{toctree} +:hidden: +Overview +``` + +```{toctree} +:hidden: +:titlesonly: +:glob: +:caption: Deploy with Cluster API +tutorial/index.md +howto/index.md +explanation/index.md +reference/index.md +``` + Cluster API (CAPI) is a Kubernetes project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. The supporting infrastructure, like virtual machines, networks, load balancers, and VPCs, as well as the cluster configuration are all defined in the same way that cluster operators are already familiar with. {{product}} supports deploying and operating Kubernetes through CAPI. ![Illustration depicting working on components and clouds][logo] @@ -55,10 +71,10 @@ and constructive feedback. [Code of Conduct]: https://ubuntu.com/community/ethos/code-of-conduct -[community]: /charm/reference/community -[contribute]: /snap/howto/contribute -[roadmap]: /snap/reference/roadmap -[overview page]: /charm/explanation/about -[arch]: /charm/reference/architecture +[community]: ../charm/reference/community +[contribute]: ../snap/howto/contribute +[roadmap]: ../snap/reference/roadmap +[overview page]: ../charm/explanation/about +[arch]: ../charm/reference/architecture [Juju]: https://juju.is -[k8s snap package]: /snap/index \ No newline at end of file +[k8s snap package]: ../snap/index \ No newline at end of file diff --git a/docs/src/capi/reference/annotations.md b/docs/src/capi/reference/annotations.md index 8f9e87fa9..3d540b446 100644 --- a/docs/src/capi/reference/annotations.md +++ b/docs/src/capi/reference/annotations.md @@ -7,9 +7,17 @@ pairs that can be used to reflect additional metadata for CAPI resources. The following annotations can be set on CAPI `Machine` resources. +### In-place Upgrade + | Name | Description | Values | Set by user | |-----------------------------------------------|------------------------------------------------------|------------------------------|-------------| | `v1beta2.k8sd.io/in-place-upgrade-to` | Trigger a Kubernetes version upgrade on that machine | snap version e.g.:
- `localPath=/full/path/to/k8s.snap`
- `revision=123`
- `channel=latest/edge` | yes | | `v1beta2.k8sd.io/in-place-upgrade-status` | The status of the version upgrade | in-progress\|done\|failed | no | | `v1beta2.k8sd.io/in-place-upgrade-release` | The current version on the machine | snap version e.g.:
- `localPath=/full/path/to/k8s.snap`
- `revision=123`
- `channel=latest/edge` | no | | `v1beta2.k8sd.io/in-place-upgrade-change-id` | The ID of the currently running upgrade | ID string | no | + +### Refresh Certificates + +| Name | Description | Values | Set by user | +|-----------------------------------------------|------------------------------------------------------|------------------------------|-------------| +| `v1beta2.k8sd.io/refresh-certificates` | The requested duration (TTL) that the refreshed certificates should expire in. | Duration (TTL) string. A number followed by a unit e.g.: `1mo`, `1y`, `90d`
Allowed units: Any unit supported by `time.ParseDuration` as well as `y` (year), `mo` (month) and `d` (day). | yes | diff --git a/docs/src/capi/reference/configs.md b/docs/src/capi/reference/configs.md index 60ce9bebe..870d240f9 100644 --- a/docs/src/capi/reference/configs.md +++ b/docs/src/capi/reference/configs.md @@ -68,6 +68,7 @@ spec: - echo "second-command" ``` +(preruncommands)= ### `preRunCommands` **Type:** `[]string` @@ -107,7 +108,7 @@ spec: **Required:** no -`airGapped` is used to signal that we are deploying to an airgap environment. In this case, the provider will not attempt to install k8s-snap on the machine. The user is expected to install k8s-snap manually with [`preRunCommands`](#preRunCommands), or provide an image with k8s-snap pre-installed. +`airGapped` is used to signal that we are deploying to an air-gapped environment. In this case, the provider will not attempt to install k8s-snap on the machine. The user is expected to install k8s-snap manually with [`preRunCommands`](#preruncommands), or provide an image with k8s-snap pre-installed. **Example Usage:** ```yaml @@ -120,7 +121,7 @@ spec: **Required:** no -`initConfig` is configuration for the initializing the cluster features +`initConfig` is configuration for the initialising the cluster features **Fields:** @@ -192,8 +193,8 @@ spec: | `datastoreType` | `string` | The type of datastore to use for the control plane. | `""` | | `datastoreServersSecretRef` | `struct{name:str, key:str}` | A reference to a secret containing the datastore servers. | `{}` | | `k8sDqlitePort` | `int` | The port to use for k8s-dqlite. If unset, 2379 (etcd) will be used. | `2379` | -| `microclusterAddress` | `string` | The address (or CIDR) to use for microcluster. If unset, the default node interface is chosen. | `""` | -| `microclusterPort` | `int` | The port to use for microcluster. If unset, ":2380" (etcd peer) will be used. | `":2380"` | +| `microclusterAddress` | `string` | The address (or CIDR) to use for MicroCluster. If unset, the default node interface is chosen. | `""` | +| `microclusterPort` | `int` | The port to use for MicroCluster. If unset, ":2380" (etcd peer) will be used. | `":2380"` | | `extraKubeAPIServerArgs` | `map[string]string` | Extra arguments to add to kube-apiserver. | `map[]` | **Example Usage:** diff --git a/docs/src/capi/reference/index.md b/docs/src/capi/reference/index.md index cd239300c..b37f2385f 100644 --- a/docs/src/capi/reference/index.md +++ b/docs/src/capi/reference/index.md @@ -12,7 +12,7 @@ Overview :titlesonly: releases annotations -community +Community configs ``` diff --git a/docs/src/capi/tutorial/getting-started.md b/docs/src/capi/tutorial/getting-started.md index 8f5554b19..fa6aac101 100644 --- a/docs/src/capi/tutorial/getting-started.md +++ b/docs/src/capi/tutorial/getting-started.md @@ -1,4 +1,4 @@ -# Cluster provisioning with CAPI and Canonical K8s +# Cluster provisioning with CAPI and {{product}} This guide covers how to deploy a {{product}} multi-node cluster using Cluster API (CAPI). @@ -16,7 +16,7 @@ curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.7.3/ sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl ``` -### Configure clusterctl +## Configure `clusterctl` `clusterctl` contains a list of default providers. Right now, {{product}} is not yet part of that list. To make `clusterctl` aware of the new @@ -33,10 +33,10 @@ providers: url: "https://github.com/canonical/cluster-api-k8s/releases/latest/control-plane-components.yaml" ``` -### Set up a management cluster +## Set up a management cluster -The management cluster hosts the CAPI providers. You can use Canonical -Kubernetes as a management cluster: +The management cluster hosts the CAPI providers. You can use {{product}} as a +management cluster: ``` sudo snap install k8s --classic --edge @@ -50,7 +50,7 @@ When setting up the management cluster, place its kubeconfig under `~/.kube/config` so other tools such as `clusterctl` can discover and interact with it. -### Prepare the infrastructure provider +## Prepare the infrastructure provider Before generating a cluster, you need to configure the infrastructure provider. Each provider has its own prerequisites. Please follow the instructions @@ -68,7 +68,7 @@ chmod +x clusterawsadm sudo mv clusterawsadm /usr/local/bin ``` -`clusterawsadm` helps you bootstrapping the AWS environment that CAPI will use +`clusterawsadm` helps you bootstrapping the AWS environment that CAPI will use. It will also create the necessary IAM roles for you. Start by setting up environment variables defining the AWS account to use, if @@ -153,7 +153,7 @@ You are now all set to deploy the MAAS CAPI infrastructure provider. ```` ````` -### Initialise the management cluster +## Initialise the management cluster To initialise the management cluster with the latest released version of the providers and the infrastructure of your choice: @@ -162,7 +162,7 @@ providers and the infrastructure of your choice: clusterctl init --bootstrap ck8s --control-plane ck8s -i ``` -### Generate a cluster spec manifest +## Generate a cluster spec manifest Once the bootstrap and control-plane controllers are up and running, you can apply the cluster manifests with the specifications of the cluster you want to @@ -170,7 +170,7 @@ provision. You can generate a cluster manifest for a selected set of commonly used infrastructures via templates provided by the {{product}} team. -Ensure you have initialized the desired infrastructure provider and fetch +Ensure you have initialised the desired infrastructure provider and fetch the {{product}} provider repository: ``` @@ -198,7 +198,7 @@ set the cluster’s properties. Review the available options in the respective definitions file and edit the cluster manifest (`cluster.yaml` above) to match your needs. -### Deploy the cluster +## Deploy the cluster To deploy the cluster, run: @@ -237,7 +237,7 @@ You can then see the workload nodes using: KUBECONFIG=./kubeconfig sudo k8s kubectl get node ``` -### Delete the cluster +## Delete the cluster To delete a cluster: diff --git a/docs/src/charm/explanation/index.md b/docs/src/charm/explanation/index.md index 58409c598..9b4652cf2 100644 --- a/docs/src/charm/explanation/index.md +++ b/docs/src/charm/explanation/index.md @@ -39,4 +39,4 @@ details or information such as the command reference or release notes. [Tutorials section]: ../tutorial/index [How-to guides]: ../howto/index [Reference section]: ../reference/index -[explanation topic]: /snap/explanation/index.md +[explanation topic]: ../../snap/explanation/index.md diff --git a/docs/src/charm/explanation/security.md b/docs/src/charm/explanation/security.md index 6c1048a19..002e071d1 100644 --- a/docs/src/charm/explanation/security.md +++ b/docs/src/charm/explanation/security.md @@ -1,2 +1,2 @@ -```{include} /snap/explanation/security.md +```{include} ../../snap/explanation/security.md ``` diff --git a/docs/src/charm/howto/charm.md b/docs/src/charm/howto/charm.md index 5b85f6b62..06c609c9e 100644 --- a/docs/src/charm/howto/charm.md +++ b/docs/src/charm/howto/charm.md @@ -9,7 +9,7 @@ This guide assumes the following: - The rest of this page assumes you already have Juju installed and have added [credentials] for a cloud and bootstrapped a controller. -- If you still need to do this, please take a look at the quickstart +- If you still need to do this, please take a look at the quick-start instructions, or, for custom clouds (OpenStack, MAAS), please consult the [Juju documentation][juju]. - You are not using the Juju 'localhost' cloud (see [localhost diff --git a/docs/src/charm/howto/contribute.md b/docs/src/charm/howto/contribute.md index eda251301..dff142dca 100644 --- a/docs/src/charm/howto/contribute.md +++ b/docs/src/charm/howto/contribute.md @@ -88,7 +88,7 @@ it on the [Diátaxis website]. In essence though, this guides the way we categorise and write our documentation. You can see there are four main categories of documentation: -- **Tutorials** for guided walkthroughs +- **Tutorials** for guided walk-throughs - **How to** pages for specific tasks and goals - **Explanation** pages which give background reasons and, well, explanations - **Reference**, where you will find the commands, the roadmap, etc. diff --git a/docs/src/charm/howto/cos-lite.md b/docs/src/charm/howto/cos-lite.md index 80efeebe3..97838e42a 100644 --- a/docs/src/charm/howto/cos-lite.md +++ b/docs/src/charm/howto/cos-lite.md @@ -28,7 +28,7 @@ juju add-model --config logging-config='=DEBUG' microk8s-ubuntu We also set the logging level to DEBUG so that helpful debug information is shown when you use `juju debug-log` (see [juju debug-log][juju-debug-log]). -Use the Ubuntu charm to deploy an application named “microk8s”: +Use the Ubuntu charm to deploy an application named `microk8s`: ``` juju deploy ubuntu microk8s --series=focal --constraints="mem=8G cores=4 root-disk=30G" @@ -36,13 +36,13 @@ juju deploy ubuntu microk8s --series=focal --constraints="mem=8G cores=4 root-di Deploy MicroK8s on Ubuntu by accessing the unit you created at the last step with `juju ssh microk8s/0` and following the -[Install Microk8s][how-to-install-microk8s] guide for configuration. +[Install MicroK8s][how-to-install-MicroK8s] guide for configuration. ```{note} Make sure to enable the hostpath-storage and MetalLB addons for -Microk8s. +MicroK8s. ``` -Export the Microk8s kubeconfig file to your current directory after +Export the MicroK8s kubeconfig file to your current directory after configuration: ``` @@ -57,9 +57,9 @@ command): KUBECONFIG=microk8s-config.yaml juju add-k8s microk8s-cloud ``` -## Deploying COS Lite on the Microk8s cloud +## Deploying COS Lite on the MicroK8s cloud -On the Microk8s cloud, create a new model and deploy the `cos-lite` bundle: +On the MicroK8s cloud, create a new model and deploy the `cos-lite` bundle: ``` juju add-model cos-lite microk8s-cloud @@ -145,4 +145,4 @@ you can head over to the [COS Lite documentation][cos-lite-docs]. [juju-models]: https://juju.is/docs/juju/model [juju-debug-log]: https://juju.is/docs/juju/juju-debug-log [cross-model-integration]: https://juju.is/docs/juju/relation#heading--cross-model -[how-to-install-microk8s]: https://microk8s.io/docs/getting-started \ No newline at end of file +[how-to-install-MicroK8s]: https://microk8s.io/docs/getting-started \ No newline at end of file diff --git a/docs/src/charm/howto/custom-registry.md b/docs/src/charm/howto/custom-registry.md new file mode 100644 index 000000000..2db2dcc39 --- /dev/null +++ b/docs/src/charm/howto/custom-registry.md @@ -0,0 +1,76 @@ +# Configure a Custom Registry + +The `k8s` charm can be configured to use a custom container registry for its +container images. This is particularly useful if you have a private registry or +operate in an air-gapped environment where you need to pull images from a +different registry. This guide will walk you through the steps to set up `k8s` +charm to pull images from a custom registry. + +## Prerequisites + +- A running `k8s` charm cluster. +- Access to a custom container registry from the cluster (e.g., docker registry + or Harbor). + +## Configure the Charm + +To configure the charm to use a custom registry, you need to set the +`containerd_custom_registries` configuration option. This options allows +the charm to configure `containerd` to pull images from registries that require +authentication. This configuration option should be a JSON-formatted array of +credential objects. For more details on the `containerd_custom_registries` +option, refer to the [charm configurations] documentation. + +For example, to configure the charm to use a custom registry at +`myregistry.example.com:5000` with the username `myuser` and password +`mypassword`, set the `containerd_custom_registries` configuration option as +follows: + +``` +juju config k8s containerd_custom_registries='[{ + "url": "http://myregistry.example.com:5000", + "host": "myregistry.example.com:5000", + "username": "myuser", + "password": "mypassword" +}]' +``` + +Allow the charm to apply the configuration changes and wait for Juju to +indicate that the changes have been successfully applied. You can monitor the +progress by running: + +``` +juju status --watch 2s +``` + +## Verify the Configuration + +Once the charm is configured and active, verify that the custom registry is +configured correctly by creating a new workload and ensuring that the images +are being pulled from the custom registry. + +For example, to create a new workload using the `nginx:latest` image that you +have previously pushed to the `myregistry.example.com:5000` registry, run the +following command: + +``` +kubectl run nginx --image=myregistry.example.com:5000/nginx:latest +``` + +To confirm that the image has been pulled from the custom registry and that the +workload is running, use the following command: + +``` +kubectl get pod nginx -o jsonpath='{.spec.containers[*].image}{"->"}{.status.containerStatuses[*].ready}' +``` + +The output should indicate that the image was pulled from the custom registry +and that the workload is running. + +``` +myregistry.example.com:5000/nginx:latest->true +``` + + + +[charm configurations]: https://charmhub.io/k8s/configurations diff --git a/docs/src/charm/howto/index.md b/docs/src/charm/howto/index.md index 0f885b73e..618ab666a 100644 --- a/docs/src/charm/howto/index.md +++ b/docs/src/charm/howto/index.md @@ -16,10 +16,11 @@ Overview charm install-lxd -etcd +Integrate with etcd proxy cos-lite contribute +custom-registry ``` diff --git a/docs/src/charm/howto/install-lxd.md b/docs/src/charm/howto/install-lxd.md index 321ce4e2c..99a41b7e1 100644 --- a/docs/src/charm/howto/install-lxd.md +++ b/docs/src/charm/howto/install-lxd.md @@ -24,7 +24,7 @@ profiles by running the command: lxc profile list ``` -For example, suppose we have created a model called 'myk8s'. This will +For example, suppose we have created a model called `myk8s`. This will output a table like this: ``` @@ -73,7 +73,7 @@ lxc profile show juju-myk8s ``` ```{note} For an explanation of the settings in this file, - [see below](explain-rules) + [see below](explain-rules-charm) ``` ## Deploying to a container @@ -81,7 +81,7 @@ lxc profile show juju-myk8s We can now deploy {{product}} into the LXD-based model as described in the [charm][] guide. -(explain-rules)= +(explain-rules-charm)= ## Explanation of custom LXD rules diff --git a/docs/src/charm/howto/proxy.md b/docs/src/charm/howto/proxy.md index 7a514dee9..8a57c4fc7 100644 --- a/docs/src/charm/howto/proxy.md +++ b/docs/src/charm/howto/proxy.md @@ -1,6 +1,6 @@ # Configuring proxy settings for K8s -{{product}} packages a number of utilities (eg curl, helm) which need +{{product}} packages a number of utilities (for example curl, helm) which need to fetch resources they expect to find on the internet. In a constrained network environment, such access is usually controlled through proxies. diff --git a/docs/src/charm/index.md b/docs/src/charm/index.md index 5ebb51296..83f34fe72 100644 --- a/docs/src/charm/index.md +++ b/docs/src/charm/index.md @@ -1,5 +1,20 @@ # {{product}} charm documentation +```{toctree} +:hidden: +Overview +``` + +```{toctree} +:hidden: +:titlesonly: +:caption: Deploy with Juju +tutorial/index.md +howto/index.md +explanation/index.md +reference/index.md +``` + The {{product}} charm, `k8s`, is an operator: software which wraps an application and contains all of the instructions necessary for deploying, configuring, scaling, integrating the application on any cloud supported by @@ -66,10 +81,10 @@ and constructive feedback. [Code of Conduct]: https://ubuntu.com/community/ethos/code-of-conduct -[community]: /charm/reference/community -[contribute]: /snap/howto/contribute -[roadmap]: /snap/reference/roadmap -[overview page]: /charm/explanation/about -[arch]: /charm/reference/architecture +[community]: reference/community +[contribute]: ../snap/howto/contribute +[roadmap]: ../snap/reference/roadmap +[overview page]: explanation/about +[arch]: reference/architecture [Juju]: https://juju.is -[k8s snap package]: /snap/index \ No newline at end of file +[k8s snap package]: ../snap/index \ No newline at end of file diff --git a/docs/src/charm/reference/architecture.md b/docs/src/charm/reference/architecture.md index 28a51c753..2e2696ba5 100644 --- a/docs/src/charm/reference/architecture.md +++ b/docs/src/charm/reference/architecture.md @@ -1,5 +1,5 @@ # K8s charm architecture -```{include} /snap/reference/architecture.md +```{include} ../../snap/reference/architecture.md :start-after: '## Canonical K8s charms' ``` diff --git a/docs/src/charm/reference/charms.md b/docs/src/charm/reference/charms.md index 29450218a..eb889da5e 100644 --- a/docs/src/charm/reference/charms.md +++ b/docs/src/charm/reference/charms.md @@ -24,13 +24,13 @@ The source code for both charms is contained in a single repository: [https://github.com/canonical/k8s-operator][repo] -Please see the [readme file][] there for further specifics of the charm +Please see the [README file][] there for further specifics of the charm implementation. [Juju]: https://juju.is -[explaining channels]: /charm/explanation/channels +[explaining channels]: ../explanation/channels [cs-k8s]: https://charmhub.io/k8s [cs-k8s-worker]: https://charmhub.io/k8s-worker -[readme file]: https://github.com/canonical/k8s-operator#readme +[README file]: https://github.com/canonical/k8s-operator#readme [repo]: https://github.com/canonical/k8s-operator \ No newline at end of file diff --git a/docs/src/charm/reference/index.md b/docs/src/charm/reference/index.md index 606de9a9a..d145e1bfd 100644 --- a/docs/src/charm/reference/index.md +++ b/docs/src/charm/reference/index.md @@ -15,7 +15,7 @@ releases charms proxy architecture -community +Community ``` diff --git a/docs/src/charm/reference/proxy.md b/docs/src/charm/reference/proxy.md index e4dfd0e16..1ebabb32b 100644 --- a/docs/src/charm/reference/proxy.md +++ b/docs/src/charm/reference/proxy.md @@ -1,2 +1,2 @@ -```{include} /snap/reference/proxy.md +```{include} ../../snap/reference/proxy.md ``` \ No newline at end of file diff --git a/docs/src/charm/tutorial/getting-started.md b/docs/src/charm/tutorial/getting-started.md index c43b189e5..222ba1120 100644 --- a/docs/src/charm/tutorial/getting-started.md +++ b/docs/src/charm/tutorial/getting-started.md @@ -7,7 +7,7 @@ instances and also to integrate other operators to enhance or customise your Kubernetes deployment. This tutorial will take you through installing Kubernetes and some common first steps. -## What you will learn +## What will be covered - How to install {{product}} - Making a cluster @@ -41,7 +41,9 @@ The currently available versions of the charm can be discovered by running: ``` juju info k8s ``` + or + ``` juju info k8s-worker ``` @@ -106,13 +108,14 @@ fetched earlier also includes a list of the relations possible, and from this we can see that the k8s-worker requires "cluster: k8s-cluster". To connect these charms and effectively add the worker to our cluster, we use -the 'integrate' command, adding the interface we wish to connect +the 'integrate' command, adding the interface we wish to connect. ``` juju integrate k8s k8s-worker:cluster ``` -After a short time, the worker node will share information with the control plane and be joined to the cluster. +After a short time, the worker node will share information with the control plane +and be joined to the cluster. ## 4. Scale the cluster (Optional) @@ -168,7 +171,8 @@ config file which will just require a bit of editing: juju run k8s/0 get-kubeconfig >> ~/.kube/config ``` -The output includes the root of the YAML, `kubeconfig: |`, so we can just use an editor to remove that line: +The output includes the root of the YAML, `kubeconfig: |`, so we can just use an +editor to remove that line: ``` nano ~/.kube/config @@ -189,6 +193,7 @@ kubectl config show ``` ...which should output something like this: + ``` apiVersion: v1 clusters: @@ -217,15 +222,7 @@ running a simple command such as : kubectl get pods -A ``` -This should return some pods, confirming the command can reach the cluster: - -``` -NAMESPACE NAME READY STATUS RESTARTS AGE -kube-system cilium-4m5xj 1/1 Running 0 35m -kube-system cilium-operator-5ff9ddcfdb-b6qxm 1/1 Running 0 35m -kube-system coredns-7d4dffcffd-tvs6v 1/1 Running 0 35m -kube-system metrics-server-6f66c6cc48-wdxxk 1/1 Running 0 35m -``` +This should return some pods, confirming the command can reach the cluster. ## Next steps @@ -239,5 +236,5 @@ informed of updates. [Juju client]: https://juju.is/docs/juju/install-and-manage-the-client [Juju tutorial]: https://juju.is/docs/juju/tutorial [Kubectl]: https://kubernetes.io/docs/reference/kubectl/ -[the channel explanation page]: /snap/explanation/channels -[releases page]: /charm/reference/releases \ No newline at end of file +[the channel explanation page]: ../../snap/explanation/channels +[releases page]: ../reference/releases \ No newline at end of file diff --git a/docs/src/snap/explanation/certificates.md b/docs/src/snap/explanation/certificates.md index 5417bb13b..63ee334b6 100644 --- a/docs/src/snap/explanation/certificates.md +++ b/docs/src/snap/explanation/certificates.md @@ -3,7 +3,7 @@ Certificates are a crucial part of Kubernetes' security infrastructure, serving to authenticate and secure communication within the cluster. They play a key role in ensuring that communication between various components (such as the -API server, kubelets, and the datastore) is both encrypted and restricted to +API server, kubelet, and the datastore) is both encrypted and restricted to authorised components only. In Kubernetes, [X.509][] certificates are primarily used for diff --git a/docs/src/snap/explanation/cis.md b/docs/src/snap/explanation/cis.md new file mode 100644 index 000000000..527342676 --- /dev/null +++ b/docs/src/snap/explanation/cis.md @@ -0,0 +1,28 @@ +# CIS Hardening + +CIS Hardening refers to the process of implementing security configurations that +align with the benchmarks set forth by the [Center for Internet Security] (CIS). +These [benchmarks] are a set of best practices and guidelines designed to secure +various software and hardware systems, including Kubernetes clusters. The +primary goal of CIS hardening is to reduce the attack surface and enhance the +overall security posture of an environment by enforcing configurations that are +known to protect against common vulnerabilities and threats. + +## Why is CIS Hardening Important for Kubernetes? + +Kubernetes, by its nature, is a complex system with many components interacting +in a distributed environment. This complexity can introduce numerous security +risks if not properly managed such as unauthorised access, data breaches and +service disruption. CIS hardening for Kubernetes focuses on configuring various +components of a Kubernetes cluster to meet the security standards specified in +the [CIS Kubernetes Benchmark]. + +## Apply CIS Hardening to {{product}} + +If you would like to apply CIS hardening to your cluster see our [how-to guide]. + + +[benchmarks]: https://www.cisecurity.org/cis-benchmarks +[Center for Internet Security]: https://www.cisecurity.org/ +[CIS Kubernetes Benchmark]: https://www.cisecurity.org/benchmark/kubernetes +[how-to guide]: ../howto/cis-hardening.md \ No newline at end of file diff --git a/docs/src/snap/explanation/clustering.md b/docs/src/snap/explanation/clustering.md index a185a6f8c..374c46d66 100644 --- a/docs/src/snap/explanation/clustering.md +++ b/docs/src/snap/explanation/clustering.md @@ -18,14 +18,13 @@ and scheduling of workloads. This is the overview of a {{product}} cluster: -```{kroki} ../../assets/ck-cluster.puml -``` +![cluster6][] ## The Role of `k8sd` in Kubernetes Clustering `k8sd` plays a vital role in the {{product}} architecture, enhancing the functionality of both the Control Plane and Worker nodes through the use -of [microcluster]. This component simplifies cluster management tasks, such as +of [MicroCluster]. This component simplifies cluster management tasks, such as adding or removing nodes and integrating them into the cluster. It also manages essential features like DNS and networking within the cluster, streamlining the entire process for a more efficient operation. @@ -69,7 +68,11 @@ entire life-cycle. Their components include: - **Container Runtime**: The software responsible for running containers. In {{product}} the runtime is `containerd`. + + +[cluster6]: https://assets.ubuntu.com/v1/e6d02e9c-cluster6.svg + [Kubernetes Components]: https://kubernetes.io/docs/concepts/overview/components/ -[microcluster]: https://github.com/canonical/microcluster +[MicroCluster]: https://github.com/canonical/microcluster diff --git a/docs/src/snap/explanation/epa.md b/docs/src/snap/explanation/epa.md new file mode 100644 index 000000000..8d3786991 --- /dev/null +++ b/docs/src/snap/explanation/epa.md @@ -0,0 +1,547 @@ +# Enhanced Platform Awareness + +Enhanced Platform Awareness (EPA) is a methodology and a set of enhancements +across various layers of the orchestration stack. + +EPA focuses on discovering, scheduling and isolating server hardware +capabilities. This document provides a detailed guide of how EPA applies to +{{product}}, which centre around the following technologies: + +- **HugePage support**: In GA from Kubernetes v1.14, this feature enables the + discovery, scheduling and allocation of HugePages as a first-class + resource. +- **Real-time kernel**: Ensures that high-priority tasks are run within a + predictable time frame, providing the low latency and high determinism + essential for time-sensitive applications. +- **CPU pinning** (CPU Manager for Kubernetes (CMK)): In GA from Kubernetes + v1.26, provides mechanisms for CPU pinning and isolation of containerised + workloads. +- **NUMA topology awareness**: Ensures that CPU and memory allocation are + aligned according to the NUMA architecture, reducing memory latency and + increasing performance for memory-intensive applications. +- **Single Root I/O Virtualisation (SR-IOV)**: Enhances networking by enabling + virtualisation of a single physical network device into multiple virtual + devices. +- **DPDK (Data Plane Development Kit)**: A set of libraries and drivers for + fast packet processing, designed to run in user space, optimising network + performance and reducing latency. + +This document provides relevant links to detailed instructions for setting up +and installing these technologies. It is designed for developers and architects +who wish to integrate these new technologies into their {{product}}-based +networking solutions. The separate [how to guide][howto-epa] for EPA includes the +necessary steps to implement these features on {{product}}. + +## HugePages + +HugePages are a feature in the Linux kernel which enables the allocation of +larger memory pages. This reduces the overhead of managing large amounts of +memory and can improve performance for applications that require significant +memory access. + +### Key features + +- **Larger memory pages**: HugePages provide larger memory pages (e.g., 2MB or + 1GB) compared to the standard 4KB pages, reducing the number of pages the + system must manage. +- **Reduced overhead**: By using fewer, larger pages, the system reduces the + overhead associated with page table entries, leading to improved memory + management efficiency. +- **Improved TLB performance**: The Translation Lookaside Buffer (TLB) stores + recent translations of virtual memory to physical memory addresses. Using + HugePages increases TLB hit rates, reducing the frequency of memory + translation lookups. +- **Enhanced application performance**: Applications that access large amounts + of memory can benefit from HugePages by experiencing lower latency and + higher throughput due to reduced page faults and better memory access + patterns. +- **Support for high-performance workloads**: Ideal for high-performance + computing (HPC) applications, databases and other memory-intensive + workloads that demand efficient and fast memory access. +- **Native Kubernetes integration**: Starting from Kubernetes v1.14, HugePages + are supported as a native, first-class resource, enabling their + discovery, scheduling and allocation within Kubernetes environments. + +### Application to Kubernetes + +The architecture for HugePages on Kubernetes integrates the management and +allocation of large memory pages into the Kubernetes orchestration system. Here +are the key architectural components and their roles: + +- **Node configuration**: Each Kubernetes node must be configured to reserve + HugePages. This involves setting the number of HugePages in the node's + kernel boot parameters. +- **Kubelet configuration**: The `kubelet` on each node must be configured to + recognise and manage HugePages. This is typically done through the `kubelet` + configuration file, specifying the size and number of HugePages. +- **Pod specification**: HugePages are requested and allocated at the pod + level through resource requests and limits in the pod specification. Pods + can request specific sizes of HugePages (e.g., 2MB or 1GB). +- **Scheduler awareness**: The Kubernetes scheduler is aware of HugePages as a + resource and schedules pods onto nodes that have sufficient HugePages + available. This ensures that pods with HugePages requirements are placed + appropriately. Scheduler configurations and policies can be adjusted to + optimise HugePages allocation and utilisation. +- **Node Feature Discovery (NFD)**: Node Feature Discovery can be used to + label nodes with their HugePages capabilities. This enables scheduling + decisions to be based on the available HugePages resources. +- **Resource quotas and limits**: Kubernetes enables the definition of resource + quotas and limits to control the allocation of HugePages across namespaces. + This helps in managing and isolating resource usage effectively. +- **Monitoring and metrics**: Kubernetes provides tools and integrations + (e.g., Prometheus, Grafana) to monitor and visualise HugePages usage across + the cluster. This helps in tracking resource utilisation and performance. + Metrics can include HugePages allocation, usage and availability on each + node, aiding in capacity planning and optimisation. + +## Real-time kernel + +A real-time kernel ensures that high-priority tasks are run within a +predictable time frame, crucial for applications requiring low latency and high +determinism. Note that this can also impede applications which were not +designed with these considerations. + +### Key features + +- **Predictable task execution**: A real-time kernel ensures that + high-priority tasks are run within a predictable and bounded time frame, + reducing the variability in task execution time. +- **Low latency**: The kernel is optimised to minimise the time it takes to + respond to high-priority tasks, which is crucial for applications that + require immediate processing. +- **Priority-based scheduling**: Tasks are scheduled based on their priority + levels, with real-time tasks being given precedence over other types of + tasks to ensure they are processed promptly. +- **Deterministic behaviour**: The kernel guarantees deterministic behaviour, + meaning the same task will have the same response time every time it is + run, essential for time-sensitive applications. +- **Preemption:** The real-time kernel supports preemptive multitasking, + allowing high-priority tasks to interrupt lower-priority tasks to ensure + critical tasks are run without delay. +- **Resource reservation**: System resources (such as CPU and memory) can be + reserved by the kernel for real-time tasks, ensuring that these resources + are available when needed. +- **Enhanced interrupt handling**: Interrupt handling is optimised to ensure + minimal latency and jitter, which is critical for maintaining the + performance of real-time applications. +- **Real-time scheduling policies**: The kernel includes specific scheduling + policies (e.g., SCHED\_FIFO, SCHED\_RR) designed to manage real-time tasks + effectively and ensure they meet their deadlines. + +These features make a real-time kernel ideal for applications requiring precise +timing and high reliability. + +### Application to Kubernetes + +The architecture for integrating a real-time kernel into Kubernetes involves +several components and configurations to ensure that high-priority, low-latency +tasks can be managed effectively within a Kubernetes environment. Here are the +key architectural components and their roles: + +- **Real-time kernel installation**: Each Kubernetes node must run a real-time + kernel. This involves installing a real-time kernel package and configuring + the system to use it. +- **Kernel boot parameters**: The kernel boot parameters must be configured to + optimise for real-time performance. This includes isolating CPU cores and + configuring other kernel parameters for real-time behaviour. +- **Kubelet configuration**: The `kubelet` on each node must be configured to + recognise and manage real-time workloads. This can involve setting specific + `kubelet` flags and configurations. +- **Pod specification**: Real-time workloads are specified at the pod level + through resource requests and limits. Pods can request dedicated CPU cores + and other resources to ensure they meet real-time requirements. +- **CPU Manager**: Kubernetes’ CPU Manager is a critical component for + real-time workloads. It enables the static allocation of CPUs to + containers, ensuring that specific CPU cores are dedicated to particular + workloads. +- **Scheduler awareness**: The Kubernetes scheduler must be aware of real-time + requirements and prioritise scheduling pods onto nodes with available + real-time resources. +- **Priority and preemption**: Kubernetes supports priority and preemption to + ensure that critical real-time pods are scheduled and run as needed. This + involves defining pod priorities and enabling preemption to ensure + high-priority pods can displace lower-priority ones if necessary. +- **Resource quotas and limits**: Kubernetes can define resource quotas + and limits to control the allocation of resources for real-time workloads + across namespaces. This helps manage and isolate resource usage effectively. +- **Monitoring and metrics**: Monitoring tools such as Prometheus and Grafana + can be used to track the performance and resource utilisation of real-time + workloads. Metrics include CPU usage, latency and task scheduling times, + which help in optimising and troubleshooting real-time applications. +- **Security and isolation**: Security contexts and isolation mechanisms + ensure that real-time workloads are protected and run in a controlled + environment. This includes setting privileged containers and configuring + namespaces. + +## CPU pinning + +CPU pinning enables specific CPU cores to be dedicated to a particular process +or container, ensuring that the process runs on the same CPU core(s) every +time, which reduces context switching and cache invalidation. + +### Key features + +- **Dedicated CPU Cores**: CPU pinning allocates specific CPU cores to a + process or container, ensuring consistent and predictable CPU usage. +- **Reduced context switching**: By running a process or container on the same + CPU core(s), CPU pinning minimises the overhead associated with context + switching, leading to better performance. +- **Improved cache utilisation**: When a process runs on a dedicated CPU core, + it can take full advantage of the CPU cache, reducing the need to fetch data + from main memory and improving overall performance. +- **Enhanced application performance**: Applications that require low latency + and high performance benefit from CPU pinning as it ensures they have + dedicated processing power without interference from other processes. +- **Consistent performance**: CPU pinning ensures that a process or container + receives consistent CPU performance, which is crucial for real-time and + performance-sensitive applications. +- **Isolation of workloads**: CPU pinning isolates workloads on specific CPU + cores, preventing them from being affected by other workloads running on + different cores. This is especially useful in multi-tenant environments. +- **Improved predictability**: By eliminating the variability introduced by + sharing CPU cores, CPU pinning provides more predictable performance + characteristics for critical applications. +- **Integration with Kubernetes**: Kubernetes supports CPU pinning through the + CPU Manager (in GA since v1.26), which allows for the static allocation of + CPUs to containers. This ensures that containers with high CPU demands have + the necessary resources. + +### Application to Kubernetes + +The architecture for CPU pinning in Kubernetes involves several components and +configurations to ensure that specific CPU cores can be dedicated to particular +processes or containers, thereby enhancing performance and predictability. Here +are the key architectural components and their roles: + +- **Kubelet configuration**: The `kubelet` on each node must be configured to + enable CPU pinning. This involves setting specific `kubelet` flags to + activate the CPU Manager. +- **CPU manager**: Kubernetes’ CPU Manager is a critical component for CPU + pinning. It allows for the static allocation of CPUs to containers, ensuring + that specific CPU cores are dedicated to particular workloads. The CPU + Manager can be configured to either static or none. Static policy enables + exclusive CPU core allocation to Guaranteed QoS (Quality of Service) pods. +- **Pod specification**: Pods must be specified to request dedicated CPU + resources. This is done through resource requests and limits in the pod + specification. +- **Scheduler awareness**: The Kubernetes scheduler must be aware of the CPU + pinning requirements. It schedules pods onto nodes with available CPU + resources as requested by the pod specification. The scheduler ensures that + pods with specific CPU pinning requests are placed on nodes with sufficient + free dedicated CPUs. +- **NUMA Topology Awareness**: For optimal performance, CPU pinning should be + aligned with NUMA (Non-Uniform Memory Access) topology. This ensures that + memory accesses are local to the CPU, reducing latency. Kubernetes can be + configured to be NUMA-aware, using the Topology Manager to align CPU + and memory allocation with NUMA nodes. +- **Node Feature Discovery (NFD)**: Node Feature Discovery can be used to + label nodes with their CPU capabilities, including the availability of + isolated and reserved CPU cores. +- **Resource quotas and limits**: Kubernetes can define resource quotas + and limits to control the allocation of CPU resources across namespaces. + This helps in managing and isolating resource usage effectively. +- **Monitoring and metrics**: Monitoring tools such as Prometheus and Grafana + can be used to track the performance and resource utilisation of CPU-pinned + workloads. Metrics include CPU usage, core allocation and task scheduling + times, which help in optimising and troubleshooting performance-sensitive + applications. +- **Isolation and security**: Security contexts and isolation mechanisms + ensure that CPU-pinned workloads are protected and run in a controlled + environment. This includes setting privileged containers and configuring + namespaces to avoid resource contention. +- **Performance Tuning**: Additional performance tuning can be achieved by + isolating CPU cores at the OS level and configuring kernel parameters to + minimise interference from other processes. This includes setting CPU + isolation and `nohz_full` parameters (reduces the number of scheduling-clock + interrupts, improving energy efficiency and [reducing OS jitter][no_hz]). + +## NUMA topology awareness + +NUMA (Non-Uniform Memory Access) topology awareness ensures that the CPU and +memory allocation are aligned according to the NUMA architecture, which can +reduce memory latency and increase performance for memory-intensive +applications. + +The Kubernetes Memory Manager enables the feature of guaranteed memory (and +HugePages) allocation for pods in the Guaranteed QoS (Quality of Service) +class. + +The Memory Manager employs hint generation protocol to yield the most suitable +NUMA affinity for a pod. The Memory Manager feeds the central manager (Topology +Manager) with these affinity hints. Based on both the hints and Topology +Manager policy, the pod is rejected or admitted to the node. + +Moreover, the Memory Manager ensures that the memory which a pod requests is +allocated from a minimum number of NUMA nodes. + +### Key features + +- **Aligned CPU and memory allocation**: NUMA topology awareness ensures that + CPUs and memory are allocated in alignment with the NUMA architecture, + minimising cross-node memory access latency. +- **Reduced memory latency**: By ensuring that memory is accessed from the + same NUMA node as the CPU, NUMA topology awareness reduces memory latency, + leading to improved performance for memory-intensive applications. +- **Increased performance**: Applications benefit from increased performance + due to optimised memory access patterns, which is especially critical for + high-performance computing and data-intensive tasks. +- **Kubernetes Memory Manager**: The Kubernetes Memory Manager supports + guaranteed memory allocation for pods in the Guaranteed QoS (Quality of + Service) class, ensuring predictable performance. +- **Hint generation protocol**: The Memory Manager uses a hint generation + protocol to determine the most suitable NUMA affinity for a pod, helping to + optimise resource allocation based on NUMA topology. +- **Integration with Topology Manager**: The Memory Manager provides NUMA + affinity hints to the Topology Manager. The Topology Manager then decides + whether to admit or reject the pod based on these hints and the configured + policy. +- **Optimised resource allocation**: The Memory Manager ensures that the + memory requested by a pod is allocated from the minimum number of NUMA + nodes, thereby optimising resource usage and performance. +- **Enhanced scheduling decisions**: The Kubernetes scheduler, in conjunction + with the Topology Manager, makes informed decisions about pod placement to + ensure optimal NUMA alignment, improving overall cluster efficiency. +- **Support for HugePages**: The Memory Manager also supports the allocation + of HugePages, ensuring that large memory pages are allocated in a NUMA-aware + manner, further enhancing performance for applications that require large + memory pages. +- **Improved application predictability**: By aligning CPU and memory + allocation with NUMA topology, applications experience more predictable + performance characteristics, crucial for real-time and latency-sensitive + workloads. +- **Policy-Based Management**: NUMA topology awareness can be managed through + policies so that administrators can configure how resources should be + allocated based on the NUMA architecture, providing flexibility and control. + +### Application to Kubernetes + +The architecture for NUMA topology awareness in Kubernetes involves several +components and configurations to ensure that CPU and memory allocations are +optimised according to the NUMA architecture. This setup reduces memory latency +and enhances performance for memory intensive applications. Here are the key +architectural components and their roles: + +- **Node configuration**: Each Kubernetes node must have NUMA-aware hardware. + The system's NUMA topology can be inspected using tools such as `lscpu` or + `numactl`. +- **Kubelet configuration**: The `kubelet` on each node must be configured to + enable NUMA topology awareness. This involves setting specific `kubelet` + flags to activate the Topology Manager. +- **Topology Manager**: The Topology Manager is a critical component that + coordinates resource allocation based on NUMA topology. It receives NUMA + affinity hints from other managers (e.g., CPU Manager, Device Manager) and + makes informed scheduling decisions. +- **Memory Manager**: The Kubernetes Memory Manager is responsible for + managing memory allocation, including HugePages, in a NUMA-aware manner. It + ensures that memory is allocated from the minimum number of NUMA nodes + required. The Memory Manager uses a hint generation protocol to provide NUMA + affinity hints to the Topology Manager. +- **Pod specification**: Pods can be specified to request NUMA-aware resource + allocation through resource requests and limits, ensuring that they get + allocated in alignment with the NUMA topology. +- **Scheduler awareness**: The Kubernetes scheduler works in conjunction with + the Topology Manager to place pods on nodes that meet their NUMA affinity + requirements. The scheduler considers NUMA topology during the scheduling + process to optimise performance. +- **Node Feature Discovery (NFD)**: Node Feature Discovery can be used to + label nodes with their NUMA capabilities, providing the scheduler with + information to make more informed placement decisions. +- **Resource quotas and limits**: Kubernetes allows defining resource quotas + and limits to control the allocation of NUMA-aware resources across + namespaces. This helps in managing and isolating resource usage effectively. +- **Monitoring and metrics**: Monitoring tools such as Prometheus and Grafana + can be used to track the performance and resource utilisation of NUMA-aware + workloads. Metrics include CPU and memory usage per NUMA node, helping in + optimising and troubleshooting performance-sensitive applications. +- **Isolation and security**: Security contexts and isolation mechanisms + ensure that NUMA-aware workloads are protected and run in a controlled + environment. This includes setting privileged containers and configuring + namespaces to avoid resource contention. +- **Performance tuning**: Additional performance tuning can be achieved by + configuring kernel parameters and using tools like `numactl` to bind + processes to specific NUMA nodes. + +## SR-IOV (Single Root I/O Virtualisation) + +SR-IOV enables a single physical network device to appear as multiple separate +virtual devices. This can be beneficial for network-intensive applications that +require direct access to the network hardware. + +### Key features + +- **Multiple Virtual Functions (VFs)**: SR-IOV enables a single physical + network device to be partitioned into multiple virtual functions (VFs), each + of which can be assigned to a virtual machine or container as a separate + network interface. +- **Direct hardware access**: By providing direct access to the physical + network device, SR-IOV bypasses the software-based network stack, reducing + overhead and improving network performance and latency. +- **Improved network throughput**: Applications can achieve higher network + throughput as SR-IOV enables high-speed data transfer directly + between the network device and the application. +- **Reduced CPU utilisation**: Offloading network processing to the hardware + reduces the CPU load on the host system, freeing up CPU resources for other + tasks and improving overall system performance. +- **Isolation and security**: Each virtual function (VF) is isolated from + others, providing security and stability. This isolation ensures that issues + in one VF do not affect other VFs or the physical function (PF). +- **Dynamic resource allocation**: SR-IOV supports dynamic allocation of + virtual functions, enabling resources to be adjusted based on application + demands without requiring changes to the physical hardware setup. +- **Enhanced virtualisation support**: SR-IOV is particularly beneficial in + virtualised environments, enabling better network performance for virtual + machines and containers by providing them with dedicated network interfaces. +- **Kubernetes integration**: Kubernetes supports SR-IOV through the use of + network device plugins, enabling the automatic discovery, allocation, + and management of virtual functions. +- **Compatibility with Network Functions Virtualisation (NFV)**: SR-IOV is + widely used in NFV deployments to meet the high-performance networking + requirements of virtual network functions (VNFs), such as firewalls, + routers and load balancers. +- **Reduced network latency**: As network packets can bypass the + hypervisor's virtual switch, SR-IOV significantly reduces network latency, + making it ideal for latency-sensitive applications. + +### Application to Kubernetes + +The architecture for SR-IOV (Single Root I/O Virtualisation) in Kubernetes +involves several components and configurations to ensure that virtual functions +(VFs) from a single physical network device can be managed and allocated +efficiently. This setup enhances network performance and provides direct access +to network hardware for applications requiring high throughput and low latency. +Here are the key architectural components and their roles: + +- **Node configuration**: Each Kubernetes node with SR-IOV capable hardware + must have the SR-IOV drivers and tools installed. This includes the SR-IOV + network device plugin and associated drivers. +- **SR-IOV enabled network interface**: The physical network interface card + (NIC) must be configured to support SR-IOV. This involves enabling SR-IOV in + the system BIOS and configuring the NIC to create virtual functions (VFs). +- **SR-IOV network device plugin**: The SR-IOV network device plugin is + deployed as a DaemonSet in Kubernetes. It discovers SR-IOV capable network + interfaces and manages the allocation of virtual functions (VFs) to pods. +- **Device Plugin Configuration**: The SR-IOV device plugin requires a + configuration file that specifies the network devices and the number of + virtual functions (VFs) to be managed. +- **Pod specification**: Pods can request SR-IOV virtual functions by + specifying resource requests and limits in the pod specification. The SR-IOV + device plugin allocates the requested VFs to the pod. +- **Scheduler awareness**: The Kubernetes scheduler must be aware of the + SR-IOV resources available on each node. The device plugin advertises the + available VFs as extended resources, which the scheduler uses to place pods + accordingly. Scheduler configuration ensures pods with SR-IOV requests are + scheduled on nodes with available VFs. +- **Resource quotas and limits**: Kubernetes enables the definition of + resource quotas and limits to control the allocation of SR-IOV resources + across namespaces. This helps manage and isolate resource usage effectively. +- **Monitoring and metrics**: Monitoring tools such as Prometheus and Grafana + can be used to track the performance and resource utilisation of + SR-IOV-enabled workloads. Metrics include VF allocation, network throughput, + and latency, helping optimise and troubleshoot performance-sensitive + applications. +- **Isolation and security**: SR-IOV provides isolation between VFs, ensuring + that each VF operates independently and securely. This isolation is critical + for multi-tenant environments where different workloads share the same + physical network device. +- **Dynamic resource allocation**: SR-IOV supports dynamic allocation and + deallocation of VFs, enabling Kubernetes to adjust resources based on + application demands without requiring changes to the physical hardware + setup. + +## DPDK (Data Plane Development Kit) + +The Data Plane Development Kit (DPDK) is a set of libraries and drivers for +fast packet processing. It is designed to run in user space, so that +applications can achieve high-speed packet processing by bypassing the kernel. +DPDK is used to optimise network performance and reduce latency, making it +ideal for applications that require high-throughput and low-latency networking, +such as telecommunications, cloud data centres and network functions +virtualisation (NFV). + +### Key features + +- **High performance**: DPDK can process millions of packets per second per + core, using multi-core CPUs to scale performance. +- **User-space processing**: By running in user space, DPDK avoids the + overhead of kernel context switches and uses HugePages for better + memory performance. +- **Poll Mode Drivers (PMD)**: DPDK uses PMDs that poll for packets instead of + relying on interrupts, which reduces latency. + +### DPDK architecture + +The main goal of the DPDK is to provide a simple, complete framework for fast +packet processing in data plane applications. Anyone can use the code to +understand some of the techniques employed, to build upon for prototyping or to +add their own protocol stacks. + +The framework creates a set of libraries for specific environments through the +creation of an Environment Abstraction Layer (EAL), which may be specific to a +mode of the Intel® architecture (32-bit or 64-bit), user space +compilers or a specific platform. These environments are created through the +use of Meson files (needed by Meson, the software tool for automating the +building of software that DPDK uses) and configuration files. Once the EAL +library is created, the user may link with the library to create their own +applications. Other libraries, outside of EAL, including the Hash, Longest +Prefix Match (LPM) and rings libraries are also provided. Sample applications +are provided to help show the user how to use various features of the DPDK. + +The DPDK implements a run-to-completion model for packet processing, where all +resources must be allocated prior to calling data plane applications, running +as execution units on logical processing cores. The model does not support a +scheduler and all devices are accessed by polling. The primary reason for not +using interrupts is the performance overhead imposed by interrupt processing. + +In addition to the run-to-completion model, a pipeline model may also be used +by passing packets or messages between cores via the rings. This enables work +to be performed in stages and is a potentially more efficient use of code on +cores. This is suitable for scenarios where each pipeline must be mapped to a +specific application thread or when multiple pipelines must be mapped to the +same thread. + +### Application to Kubernetes + +The architecture for integrating the Data Plane Development Kit (DPDK) into +Kubernetes involves several components and configurations to ensure high-speed +packet processing and low-latency networking. DPDK enables applications to +bypass the kernel network stack, providing direct access to network hardware +and significantly enhancing network performance. Here are the key architectural +components and their roles: + +- **Node configuration**: Each Kubernetes node must have the DPDK libraries + and drivers installed. This includes setting up HugePages and binding + network interfaces to DPDK-compatible drivers. +- **HugePages configuration**: DPDK requires HugePages for efficient memory + management. Configure the system to reserve HugePages. +- **Network interface binding**: Network interfaces must be bound to + DPDK-compatible drivers (e.g., vfio-pci) to be used by DPDK applications. +- **DPDK application container**: Create a Docker container image with the + DPDK application and necessary libraries. Ensure that the container runs + with appropriate privileges and mounts HugePages. +- **Pod specification**: Deploy the DPDK application in Kubernetes by + specifying the necessary resources, including CPU pinning and HugePages, in + the pod specification. +- **CPU pinning**: For optimal performance, DPDK applications should use + dedicated CPU cores. Configure CPU pinning in the pod specification. +- **SR-IOV for network interfaces**: Combine DPDK with SR-IOV to provide + high-performance network interfaces. Allocate SR-IOV virtual functions (VFs) + to DPDK pods. +- **Scheduler awareness**: The Kubernetes scheduler must be aware of the + resources required by DPDK applications, including HugePages and CPU + pinning, to place pods appropriately on nodes with sufficient resources. +- **Monitoring and metrics**: Use monitoring tools like Prometheus and Grafana + to track the performance of DPDK applications, including network throughput, + latency and CPU usage. +- **Resource quotas and limits**: Define resource quotas and limits to control + the allocation of resources for DPDK applications across namespaces, + ensuring fair resource distribution and preventing resource contention. +- **Isolation and security**: Ensure that DPDK applications run in isolated + and secure environments. Use security contexts to provide the necessary + privileges while maintaining security best practices. + + + + +[no_hz]: https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt +[howto-epa]: ../howto/epa + diff --git a/docs/src/snap/explanation/index.md b/docs/src/snap/explanation/index.md index 3feb4fb83..5c95b7d39 100644 --- a/docs/src/snap/explanation/index.md +++ b/docs/src/snap/explanation/index.md @@ -16,7 +16,9 @@ certificates channels clustering ingress -/snap/explanation/security +epa +security +cis ``` --- diff --git a/docs/src/snap/explanation/ingress.md b/docs/src/snap/explanation/ingress.md index 6e7c73c5d..09ebf334b 100644 --- a/docs/src/snap/explanation/ingress.md +++ b/docs/src/snap/explanation/ingress.md @@ -19,7 +19,7 @@ CNI (Container Network Interface) called [Cilium][Cilium]. If you wish to use a different network plugin the implementation and configuration falls under your responsibility. -Learn how to use the {{product}} default network in the [networking HowTo guide][Network]. +Learn how to use the {{product}} default network in the [networking how-to guide][Network]. ## Kubernetes Pods and Services @@ -54,8 +54,7 @@ that routes traffic from outside of your cluster to services inside of your clus Please do not confuse this with the Kubernetes Service LoadBalancer type which operates at layer 4 and routes traffic directly to individual pods. -```{kroki} ../../assets/ingress.puml -``` +![cluster6][] With {{product}}, enabling Ingress is easy: See the [default Ingress guide][Ingress]. @@ -73,10 +72,14 @@ the responsibility of implementation falls upon you. You will need to create the Ingress resource, outlining rules that direct traffic to your application's Kubernetes service. + + +[cluster6]: https://assets.ubuntu.com/v1/e6d02e9c-cluster6.svg + -[Ingress]: /snap/howto/networking/default-ingress -[Network]: /snap/howto/networking/default-network +[Ingress]: ../howto/networking/default-ingress +[Network]: ../howto/networking/default-network [Cilium]: https://cilium.io/ [network plugin]: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/ [Service]: https://kubernetes.io/docs/concepts/services-networking/service/ diff --git a/docs/src/snap/explanation/security.md b/docs/src/snap/explanation/security.md index 53f5ea727..8daeb368f 100644 --- a/docs/src/snap/explanation/security.md +++ b/docs/src/snap/explanation/security.md @@ -44,11 +44,11 @@ have access to your cluster. Describing the security mechanisms of these clouds is out of the scope of this documentation, but you may find the following links useful. -- Amazon Web Services -- Google Cloud Platform -- Metal As A Service(MAAS) -- Microsoft Azure -- VMWare VSphere +- [Amazon Web Services security][] +- [Google Cloud Platform security][] +- [Metal As A Service(MAAS) hardening][] +- [Microsoft Azure security][] +- [VMware VSphere hardening guides][] ## Security Compliance @@ -62,4 +62,10 @@ check the [roadmap][] for current areas of work. [Kubernetes Security documentation]: https://kubernetes.io/docs/concepts/security/overview/ [snap documentation]: https://snapcraft.io/docs/security-sandboxing [rocks-security]: https://canonical-rockcraft.readthedocs-hosted.com/en/latest/explanation/rockcraft/ -[roadmap]: /snap/reference/roadmap +[roadmap]: ../reference/roadmap +[Amazon Web Services security]: https://aws.amazon.com/security/ +[Google Cloud Platform security]:https://cloud.google.com/security/ +[Metal As A Service(MAAS) hardening]:https://maas.io/docs/snap/3.0/ui/hardening-your-maas-installation +[Microsoft Azure security]:https://docs.microsoft.com/en-us/azure/security/azure-security +[VMware VSphere hardening guides]: https://www.vmware.com/security/hardening-guides.html + diff --git a/docs/src/snap/howto/backup-restore.md b/docs/src/snap/howto/backup-restore.md index cb5345ac4..dc54a9cab 100644 --- a/docs/src/snap/howto/backup-restore.md +++ b/docs/src/snap/howto/backup-restore.md @@ -64,7 +64,7 @@ sudo k8s kubectl expose deployment nginx -n workloads --port 80 ## Install Velero Download the Velero binary from the -[releases page on github][releases] and place it in our `PATH`. In this case we +[releases page on GitHub][releases] and place it in our `PATH`. In this case we install the v1.14.1 Linux binary for AMD64 under `/usr/local/bin`: ```bash @@ -100,7 +100,7 @@ EOF ``` We are now ready to install Velero into the cluster, with an aws plugin that -[matches][aws-plugin-matching] the velero release: +[matches][aws-plugin-matching] the Velero release: ```bash SERVICE_URL="http://${SERVICE}.velero.svc:9000" diff --git a/docs/src/snap/howto/cis-hardening.md b/docs/src/snap/howto/cis-hardening.md new file mode 100644 index 000000000..f44c65cb1 --- /dev/null +++ b/docs/src/snap/howto/cis-hardening.md @@ -0,0 +1,292 @@ +# CIS compliance + +CIS Hardening refers to the process of implementing security configurations that +align with the benchmarks set by the [Center for Internet Security (CIS)][]. The +open source tool [kube-bench][] is designed to automatically check whether +your Kubernetes clusters are configured according to the +[CIS Kubernetes Benchmark][]. This guide covers how to setup your {{product}} +cluster with kube-bench. + +## What you'll need + +This guide assumes the following: + +- You have a bootstrapped {{product}} cluster (see the [Getting Started] +[getting-started-guide] guide) +- You have root or sudo access to the machine + +## Install kube-bench + +Download the latest [kube-bench release][] on your Kubernetes nodes. Make sure +to select the appropriate binary version. + +For example, to download the Linux binary, use the following command. Replace +`KB` by the version listed in the releases page. + +``` +KB=8.0 +mkdir kube-bench +cd kube-bench +curl -L https://github.com/aquasecurity/kube-bench/releases/download/v0.$KB/kube-bench_0.$KB\_linux_amd64.tar.gz -o kube-bench_0.$KB\_linux_amd64.tar.gz +``` + +Extract the downloaded tarball and move the binary to a directory in your PATH: + +``` +tar -xvf kube-bench_0.$KB\_linux_amd64.tar.gz +sudo mv kube-bench /usr/local/bin/ +``` + +Verify kube-bench installation. + +``` +kube-bench version +``` + +The output should list the version installed. + +Install `kubectl` and configure it to interact with the cluster. + +```{warning} +This will override your ~/.kube/config if you already have kubectl installed in your cluster. +``` + +``` +sudo snap install kubectl --classic +mkdir ~/.kube/ +sudo k8s kubectl config view --raw > ~/.kube/config +export KUBECONFIG=~/.kube/config +``` + +Get CIS hardening checks applicable for {{product}}: + +``` +git clone -b ck8s https://github.com/canonical/kube-bench.git kube-bench-ck8s-cfg +``` + +Test-run kube-bench against {{product}}: + +``` +sudo -E kube-bench --version ck8s-dqlite-cis-1.24 --config-dir ./kube-bench-ck8s-cfg/cfg/ --config ./kube-bench-ck8s-cfg/cfg/config.yaml +``` + +## Harden your deployments + +Before running a CIS Kubernetes audit, it is essential to first harden your +{{product}} deployment to minimise vulnerabilities and ensure +compliance with industry best practices, as defined by the CIS Kubernetes +Benchmark. + +### Control plane nodes + +Run the following commands on your control plane nodes. + +#### Configure auditing + +Create an audit-policy.yaml file under `/var/snap/k8s/common/etc/` and specify +the level of auditing you desire based on the [upstream instructions][]. Here is +a minimal example of such a policy file. + +``` +sudo sh -c 'cat >/var/snap/k8s/common/etc/audit-policy.yaml <>/var/snap/k8s/common/args/kube-apiserver </var/snap/k8s/common/etc/eventconfig.yaml </var/snap/k8s/common/etc/admission-control-config-file.yaml <>/var/snap/k8s/common/args/kube-apiserver <>/var/snap/k8s/common/args/kubelet < +[Center for Internet Security (CIS)]:https://www.cisecurity.org/ +[kube-bench]:https://aquasecurity.github.io/kube-bench/v0.6.15/ +[CIS Kubernetes Benchmark]:https://www.cisecurity.org/benchmark/kubernetes +[getting-started-guide]: ../tutorial/getting-started +[kube-bench release]: https://github.com/aquasecurity/kube-bench/releases +[upstream instructions]:https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/ +[rate limits]:https://kubernetes.io/docs/reference/config-api/apiserver-eventratelimit.v1alpha1 diff --git a/docs/src/snap/howto/contribute.md b/docs/src/snap/howto/contribute.md index 05e08f1d2..67f1372b9 100644 --- a/docs/src/snap/howto/contribute.md +++ b/docs/src/snap/howto/contribute.md @@ -88,7 +88,7 @@ it on the [Diátaxis website]. In essence though, this guides the way we categorise and write our documentation. You can see there are four main categories of documentation: -- **Tutorials** for guided walkthroughs +- **Tutorials** for guided walk-throughs - **How to** pages for specific tasks and goals - **Explanation** pages which give background reasons and, well, explanations - **Reference**, where you will find the commands, the roadmap, etc. diff --git a/docs/src/snap/howto/epa.md b/docs/src/snap/howto/epa.md new file mode 100644 index 000000000..278cb3420 --- /dev/null +++ b/docs/src/snap/howto/epa.md @@ -0,0 +1,1148 @@ +# How to set up Enhanced Platform Awareness + +This section explains how to set up the Enhanced Platform Awareness (EPA) +features in a {{product}} cluster. Please see the [EPA explanation +page][explain-epa] for details about how EPA applies to {{product}}. + +The content starts with the setup of the environment (including steps for using +[MAAS][MAAS]). Then the setup of {{product}}, including the Multus & SR-IOV/DPDK +networking components. Finally, the steps needed to test every EPA feature: +HugePages, Real-time Kernel, CPU Pinning / NUMA Topology Awareness and +SR-IOV/DPDK. + +## What you'll need + +- An Ubuntu Pro subscription (required for real-time kernel) +- Ubuntu instances **or** a MAAS environment to run {{product}} on + + +## Prepare the Environment + + +`````{tabs} +````{group-tab} Ubuntu + +First, run the `numactl` command to get the number of CPUs available for NUMA: + +``` +numactl -s +``` + +This example output shows that there are 32 CPUs available for NUMA: + +``` +policy: default +preferred node: current +physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 +cpubind: 0 1 +nodebind: 0 1 +membind: 0 1 +``` + +```{dropdown} Detailed explanation of output + +- `policy: default`: indicates that the system is using the default NUMA policy. The default policy typically tries to allocate memory on the same node as the processor executing a task, but it can fall back to other nodes if necessary. +- `preferred node: current`: processes will prefer to allocate memory from the current node (the node where the process is running). However, if memory is not available on the current node, it can be allocated from other nodes. +- `physcpubind: 0 1 2 3 ... 31 `: shows the physical CPUs that processes are allowed to run on. In this case, the system has 32 physical CPUs enabled for NUMA, and processes can use any of them. +- `cpubind: 0 1 `: indicates the specific CPUs that the current process (meaning the process “numactl \-s”) is bound to. It's currently using CPUs 0 and 1. +- `nodebind: 0 1 `: shows the NUMA nodes that the current process (meaning the process “numactl \-s”) is allowed to use for memory allocation. It has access to both node 0 and node 1. +- `membind`: 0 1 `: confirms that the current process (meaning the process “numactl \-s”) can allocate memory from both node 0 and node 1. +``` + +### Enable the real-time kernel + +The real-time kernel enablement requires an ubuntu pro subscription and some additional tools to be available. + +``` +sudo pro attach +sudo apt update && sudo apt install ubuntu-advantage-tools +sudo pro enable realtime-kernel +``` + +This should produce output similar to: + +``` +One moment, checking your subscription first +Real-time kernel cannot be enabled with Livepatch. +Disable Livepatch and proceed to enable Real-time kernel? (y/N) y +Disabling incompatible service: Livepatch +The Real-time kernel is an Ubuntu kernel with PREEMPT_RT patches integrated. + +This will change your kernel. To revert to your original kernel, you will need +to make the change manually. + +Do you want to continue? [ default = Yes ]: (Y/n) Y +Updating Real-time kernel package lists +Updating standard Ubuntu package lists +Installing Real-time kernel packages +Real-time kernel enabled +A reboot is required to complete install. +``` + +First the Ubuntu system is attached to an Ubuntu Pro subscription +(needed to use the real-time kernel), requiring you to enter a token +associated with the subscription. After successful attachment, your +system gains access to the Ubuntu Pro repositories, including the one +containing the real-time kernel packages. Once the tools and +real-time kernel are installed, a reboot is required to start using +the new kernel. + +### Create a configuration file to enable HugePages and CPU isolation + +The bootloader will need a configuration file to enable the recommended +boot options (explained below) to enable HugePages and CPU isolation. + +In this example, the host has 128 CPUs, and 2M / 1G HugePages are enabled. +This is the command to update the boot options and reboot the system: + +``` +cat < /etc/default/grub.d/epa_kernel_options.cfg +GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} intel_iommu=on iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=0 crashkernel=auto softlockup_panic=0 audit=0 tsc=nowatchdog intel_pstate=disable mce=off hugepagesz=1G hugepages=1000 hugepagesz=2M hugepages=0 default_hugepagesz=1G kthread_cpus=0-31 irqaffinity=0-31 nohz=on nosoftlockup nohz_full=32-127 rcu_nocbs=32-127 rcu_nocb_poll skew_tick=1 isolcpus=managed_irq,32-127 console=tty0 console=ttyS0,115200n8" +EOF +sudo chmod 0644 /etc/netplan/99-sriov_vfs.yaml +update-grub +reboot +``` + +```{dropdown} Explanation of boot options + +- `intel_iommu=on`: Enables Intel's Input-Output Memory Management Unit (IOMMU), which is used for device virtualisation and Direct Memory Access (DMA) remapping. +- `iommu=pt`: Sets the IOMMU to passthrough mode, allowing devices to directly access physical memory without translation. +- `usbcore.autosuspend=-1`: Disables USB autosuspend, preventing USB devices from being automatically suspended to save power. +- `selinux=0`: Disables Security-Enhanced Linux (SELinux), a security module that provides mandatory access control. +- `enforcing=0`: If SELinux is enabled, this option sets it to permissive mode, where policies are not enforced but violations are logged. +- `nmi_watchdog=0`: Disables the Non-Maskable Interrupt (NMI) watchdog, which is used to detect and respond to system hangs. +- `crashkernel=auto`: Reserves a portion of memory for capturing a crash dump in the event of a kernel crash. +- `softlockup_panic=0`: Prevents the kernel from panicking (crashing) on detecting a soft lockup, where a CPU appears to be stuck. +- `audit=0`: Disables the kernel auditing system, which logs security-relevant events. +- `tsc=nowatchdog`: Disables the Time Stamp Counter (TSC) watchdog, which checks for issues with the TSC. +- `intel_pstate=disable`: Disables the Intel P-state driver, which controls CPU frequency scaling. +- `mce=off`: Disables Machine Check Exception (MCE) handling, which detects and reports hardware errors. +- `hugepagesz=1G hugepages=1000`: Allocates 1000 huge pages of 1GB each. +- `hugepagesz=2M hugepages=0`: Configures huge pages of 2MB size but sets their count to 0\. +- `default_hugepagesz=1G`: Sets the default size for huge pages to 1GB. +- `kthread_cpus=0-31`: Restricts kernel threads to run on CPUs 0-31. +- `irqaffinity=0-31`: Restricts interrupt handling to CPUs 0-31. +- `nohz=on`: Enables the nohz (no timer tick) mode, reducing timer interrupts on idle CPUs. +- `nosoftlockup`: Disables the detection of soft lockups. +- `nohz_full=32-127`: Enables nohz\_full (full tickless) mode on CPUs 32-127, reducing timer interrupts during application processing. +- `rcu_nocbs=32-127`: Offloads RCU (Read-Copy-Update) callbacks to CPUs 32-127, preventing them from running on these CPUs. +- `rcu_nocb_poll`: Enables polling for RCU callbacks instead of using interrupts. +- `skew_tick=1`: Skews the timer tick across CPUs to reduce contention. +- `isolcpus=managed_irq,32-127`: Isolates CPUs 32-127 and assigns managed IRQs to them, reducing their involvement in system processes and dedicating them to specific workloads. +- `console=tty0`: Sets the console output to the first virtual terminal. +- `console=ttyS0,115200n8`: Sets the console output to the serial port ttyS0 with a baud rate of 115200, 8 data bits, no parity, and 1 stop bit. +``` + +Once the reboot has taken place, ensure the HugePages configuration has been applied: + +``` +grep HugePages /proc/meminfo +``` + +This should generate output indicating the number of pages allocated + +``` +HugePages_Total: 1000 +HugePages_Free: 1000 +HugePages_Rsvd: 0 +HugePages_Surp: 0 +``` + + +Next, create a configuration file to configure the network interface +to use SR-IOV (so it can create virtual functions afterwards) using +Netplan. In the example below the file is created first, then the configuration is +applied, making 128 virtual functions available for use in the environment: + +``` +cat < /etc/netplan/99-sriov_vfs.yaml + network: + ethernets: + enp152s0f1: + virtual-function-count: 128 +EOF +sudo chmod 0600 /etc/netplan/99-sriov_vfs.yaml +sudo netplan apply +ip link show enp152s0f1 +``` + +The output of the last command should indicate the device is working and has generated the expected +virtual functions. + +``` +5: enp152s0f1: mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000 + link/ether 40:a6:b7:96:d8:89 brd ff:ff:ff:ff:ff:ff + vf 0 link/ether ae:31:7f:91:09:97 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off + vf 1 link/ether 32:09:8b:f7:07:4b brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off + vf 2 link/ether 12:b9:c6:08:fc:36 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off + .......... + vf 125 link/ether 92:10:ff:8a:e5:0c brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off + vf 126 link/ether 66:fe:ad:f2:d3:05 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off + vf 127 link/ether ca:20:00:c6:83:dd brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off +``` + +```{dropdown} Explanation of steps + * Breakdown of the content of the file `/etc/netplan/99-sriov\_vfs.yaml` : + * path: `/etc/netplan/99-sriov\_vfs.yaml`: This specifies the location of the configuration file. The "99" prefix in the filename usually indicates that it will be processed last, potentially overriding other configurations. + * enp152s0f1: This is the name of the physical network interface you want to create VFs on. This name may vary depending on your system. + * virtual-function-count: 128: This is the key line that instructs Netplan to create 128 virtual functions on the specified physical interface. Each of these VFs can be assigned to a different virtual machine or container, effectively allowing them to share the physical adapter's bandwidth. + * permissions: "0600": This is an optional line that sets the file permissions to 600 (read and write access only for the owner). + * Breakdown of the output of ip link show enp152s0f1 command: + * Main interface: + * 5: The index number of the network interface in the system. + * enp152s0f1: The name of the physical network interface. + * \: The interface's flags indicating its capabilities (e.g., broadcast, multicast) and current status (UP). + * mtu 9000: The maximum transmission unit (MTU) is set to 9000 bytes, larger than the typical 1500 bytes, likely for jumbo frames. + * qdisc mq: The queuing discipline (qdisc) is set to "mq" (multi-queue), designed for multi-core systems. + * state UP: The interface is currently active and operational. + * mode DEFAULT: The interface is in the default mode of operation. + * qlen 1000: The maximum number of packets allowed in the transmit queue. + * link/ether 40:a6:b7:96:d8:89: The interface's MAC address (a unique hardware identifier). + * Virtual functions: + * vf \: The index number of the virtual function. + * link/ether \: The MAC address assigned to the virtual function. + * spoof checking on: A security feature to prevent MAC address spoofing (pretending to be another device). + * link-state auto: The link state (up/down) is determined automatically based on the physical connection. + * trust off: The interface doesn't trust the incoming VLAN (Virtual LAN) tags. + * Results: + * Successful VF Creation: The output confirms a success creation of 128 VFs (numbered 0 through 127\) on the enp152s0f1 interface. + * VF Availability: Each VF is ready for use, and they can be assigned i.e. to {{product}} containers to give them direct access to the network through this physical network interface. + * MAC Addresses: Each VF has its own unique MAC address, which is essential for network communication. +``` + + +Now enable DPDK, first by cloning the DPDK repository, and then placing the script which +will bind the VFs to the VFIO-PCI driver in the location that will run +automatically each time the system boots up, so the VFIO +(Virtual Function I/O) bindings are applied consistently: + +``` +git clone https://github.com/DPDK/dpdk.git /home/ubuntu/dpdk +cat < /var/lib/cloud/scripts/per-boot/dpdk_bind.sh + #!/bin/bash + if [ -d /home/ubuntu/dpdk ]; then + modprobe vfio-pci + vfs=$(python3 /home/ubuntu/dpdk/usertools/dpdk-devbind.py -s | grep drv=iavf | awk '{print $1}' | tail -n +11) + python3 /home/ubuntu/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci $vfs + fi +sudo chmod 0755 /var/lib/cloud/scripts/per-boot/dpdk_bind.sh +``` + +```{dropdown} Explanation + * Load VFIO Module (modprobe vfio-pci): If the DPDK directory exists, the script loads the VFIO-PCI kernel module. This module is necessary for the VFIO driver to function. + * The script uses the `dpdk-devbind.py` tool (included with DPDK) to list the available network devices and their drivers. + * It filters this output using grep drv=iavf to find devices that are currently using the iavf driver (a common driver for Intel network adapters), excluding the physical network interface itself and just focusing on the virtual functions (VFs). + * Bind VFs to VFIO: The script uses `dpdk-devbind.py` again, this time with the \--bind=vfio-pci option, to bind the identified VFs to the VFIO-PCI driver. This step essentially tells the kernel to relinquish control of these devices to DPDK. +``` + +To test that the VFIO Kernel Module and DPDK are enabled: + +``` +lsmod | grep -E 'vfio' +``` + +...should indicate the kernel module is loaded + +``` +vfio_pci 16384 0 +vfio_pci_core 94208 1 vfio_pci +vfio_iommu_type1 53248 0 +vfio 73728 3 vfio_pci_core,vfio_iommu_type1,vfio_pci +iommufd 98304 1 vfio +irqbypass 12288 2 vfio_pci_core,kvm + +``` + +Running the helper script: + +``` +python3 /home/ubuntu/dpdk/usertools/dpdk-devbind.py -s +``` + +...should return a list of network devices using DPDK: + +``` +Network devices using DPDK-compatible driver +============================================ +0000:98:12.2 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf +0000:98:12.3 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf +0000:98:12.4 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf +.... +``` + +With these preparation steps we have enabled the features of EPA: + +- NUMA and CPU Pinning are available to the first 32 CPUs +- Real-Time Kernel is enabled +- HugePages are enabled and 1000 1G huge pages are available +- SR-IOV is enabled in the enp152s0f1 interface, with 128 virtual + function interfaces bound to the vfio-pci driver (that could also use the iavf driver) +- DPDK is enabled in all the 128 virtual function interfaces + +```` + +````{group-tab} MAAS + +To prepare a machine for CPU isolation, HugePages, real-time kernel, +SR-IOV and DPDK we leverage cloud-init through MAAS. + +``` +#cloud-config + +apt: + sources: + rtk.list: + source: "deb https://:@private-ppa.launchpadcontent.net/canonical-kernel-rt/ppa/ubuntu jammy main" + +write_files: + # set kernel option with hugepages and cpu isolation + - path: /etc/default/grub.d/100-telco_kernel_options.cfg + content: | + GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} intel_iommu=on iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=0 crashkernel=auto softlockup_panic=0 audit=0 tsc=nowatchdog intel_pstate=disable mce=off hugepagesz=1G hugepages=1000 hugepagesz=2M hugepages=0 default_hugepagesz=1G kthread_cpus=0-31 irqaffinity=0-31 nohz=on nosoftlockup nohz_full=32-127 rcu_nocbs=32-127 rcu_nocb_poll skew_tick=1 isolcpus=managed_irq,32-127 console=tty0 console=ttyS0,115200n8" + permissions: "0644" + + # create sriov VFs + - path: /etc/netplan/99-sriov_vfs.yaml + content: | + network: + ethernets: + enp152s0f1: + virtual-function-count: 128 + permissions: "0600" + + # ensure VFs are bound to vfio-pci driver (so they can be consumed by pods) + - path: /var/lib/cloud/scripts/per-boot/dpdk_bind.sh + content: | + #!/bin/bash + if [ -d /home/ubuntu/dpdk ]; then + modprobe vfio-pci + vfs=$(python3 /home/ubuntu/dpdk/usertools/dpdk-devbind.py -s | grep drv=iavf | awk '{print $1}' | tail -n +11) + python3 /home/ubuntu/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci $vfs + fi + permissions: "0755" + + # set proxy variables + - path: /etc/environment + content: | + HTTPS_PROXY=http://10.18.2.1:3128 + HTTP_PROXY=http://10.18.2.1:3128 + NO_PROXY=10.0.0.0/8,192.168.0.0/16,127.0.0.1,172.16.0.0/16,.svc,localhost + https_proxy=http://10.18.2.1:3128 + http_proxy=http://10.18.2.1:3128 + no_proxy=10.0.0.0/8,192.168.0.0/16,127.0.0.1,172.16.0.0/16,.svc,localhost + append: true + + # add rtk ppa key + - path: /etc/apt/trusted.gpg.d/rtk.asc + content: | + -----BEGIN PGP PUBLIC KEY BLOCK----- + Comment: Hostname: + Version: Hockeypuck 2.2 + + xsFNBGAervwBEADHCeEuR7WKRiEII+uFOu8J+W47MZOcVhfNpu4rdcveL4qe4gj4 + nsROMHaINeUPCmv7/4EXdXtTm1VksXeh4xTeqH6ZaQre8YZ9Hf4OYNRcnFOn0KR+ + aCk0OWe9xkoDbrSYd3wmx8NG/Eau2C7URzYzYWwdHgZv6elUKk6RDbDh6XzIaChm + kLsErCP1SiYhKQvD3Q0qfXdRG908lycCxgejcJIdYxgxOYFFPcyC+kJy2OynnvQr + 4Yw6LJ2LhwsA7bJ5hhQDCYZ4foKCXX9I59G71dO1fFit5O/0/oq0xe7yUYCejf7Z + OqD+TzEK4lxLr1u8j8lXoQyUXzkKIL0SWEFT4tzOFpWQ2IBs/sT4X2oVA18dPDoZ + H2SGxCUcABfne5zrEDgkUkbnQRihBtTyR7QRiE3GpU19RNVs6yAu+wA/hti8Pg9O + U/5hqifQrhJXiuEoSmmgNb9QfbR3tc0ZhKevz4y+J3vcnka6qlrP1lAirOVm2HA7 + STGRnaEJcTama85MSIzJ6aCx4omCgUIfDmsi9nAZRkmeomERVlIAvcUYxtqprLfu + 6plDs+aeff/MAmHbak7yF+Txj8+8F4k6FcfNBT51oVSZuqFwyLswjGVzWol6aEY7 + akVIrn3OdN2u6VWlU4ZO5+sjP4QYsf5K2oVnzFVIpYvqtO2fGbxq/8dRJQARAQAB + zSVMYXVuY2hwYWQgUFBBIGZvciBDYW5vbmljYWwgS2VybmVsIFJUwsGOBBMBCgA4 + FiEEc4Tsv+pcopCX6lNfLz1Vl/FsjCEFAmAervwCGwMFCwkIBwIGFQoJCAsCBBYC + AwECHgECF4AACgkQLz1Vl/FsjCF9WhAAnwfx9njs1M3rfsMMuhvPxx0WS65HDlq8 + SRgl9K2EHtZIcS7lHmcjiTR5RD1w+4rlKZuE5J3EuMnNX1PdCYLSyMQed+7UAtX6 + TNyuiuVZVxuzJ5iS7L2ZoX05ASgyoh/Loipc+an6HzHqQnNC16ZdrBL4AkkGhDgP + ZbYjM3FbBQkL2T/08NcwTrKuVz8DIxgH7yPAOpBzm91n/pV248eK0a46sKauR2DB + zPKjcc180qmaVWyv9C60roSslvnkZsqe/jYyDFuSsRWqGgE5jNyIb8EY7K7KraPv + 3AkusgCh4fqlBxOvF6FJkiYeZZs5YXvGQ296HTfVhPLOqctSFX2kuUKGIq2Z+H/9 + qfJFGS1iaUsoDEUOaU27lQg5wsYa8EsCm9otroH2P3g7435JYRbeiwlwfHMS9EfK + dwD38d8UzZj7TnxGG4T1aLb3Lj5tNG6DSko69+zqHhuknjkRuAxRAZfHeuRbACgE + nIa7Chit8EGhC2GB12pr5XFWzTvNFdxFhbG+ed7EiGn/v0pVQc0ZfE73FXltg7et + bkoC26o5Ksk1wK2SEs/f8aDZFtG01Ys0ASFICDGW2tusFvDs6LpPUUggMjf41s7j + 4tKotEE1Hzr38EdY+8faRaAS9teQdH5yob5a5Bp5F5wgmpqZom/gjle4JBVaV5dI + N5rcnHzcvXw= + =asqr + -----END PGP PUBLIC KEY BLOCK----- + permissions: "0644" + +# install the snap +snap: + commands: + 00: 'snap install k8s --classic --channel=1.31/beta' + +runcmd: +# fetch dpdk driver binding script +- su ubuntu -c "git config --global http.proxy http://10.18.2.1:3128" +- su ubuntu -c "git clone https://github.com/DPDK/dpdk.git /home/ubuntu/dpdk" +- apt update +- DEBIAN_FRONTEND=noninteractive apt install -y linux-headers-6.8.1-1004-realtime linux-image-6.8.1-1004-realtime linux-modules-6.8.1-1004-realtime linux-modules-extra-6.8.1-1004-realtime + +# enable kernel options +- update-grub + +# reboot to activate realtime-kernel and kernel options +power_state: + mode: reboot +``` + +```{note} + +In the above file, the `realtime kernel` 6.8 is installed from a private PPA. +It was recently backported from 24.04 to 22.04 and is still going through +some validation stages. Once it is officially released, it will be +installable via the Ubuntu Pro CLI. +``` + + + +```` +````` + +## {{product}} setup + +{{product}} is delivered as a [snap][]. + +This section explains how to set up a dual node {{product}} cluster for testing +EPA capabilities. + +### Control plane and worker node + +1. [Install the snap][install-link] from the relevant [channel][channel]. + + ```{note} + A pre-release channel is required currently until there is a stable release of {{product}}. + ``` + + For example: + + + ```{include} ../../_parts/install.md + ``` + +2. Create a file called *configuration.yaml*. In this configuration file we let + the snap start with its default CNI (calico), with CoreDNS deployed and we + also point k8s to the external etcd. + + ```yaml + cluster-config: + network: + enabled: true + dns: + enabled: true + local-storage: + enabled: true + extra-node-kubelet-args: + --reserved-cpus: "0-31" + --cpu-manager-policy: "static" + --topology-manager-policy: "best-effort" + ``` + +3. Bootstrap {{product}} using the above configuration file. + + ``` + sudo k8s bootstrap --file configuration.yaml + ``` + +#### Verify the control plane node is running + +After a few seconds you can query the API server with: + +``` +sudo k8s kubectl get all -A +``` + +### Add a second k8s node as a worker + +1. Install the k8s snap on the second node + + ```{include} ../../_parts/install.md + ``` + +2. On the control plane node generate a join token to be used for joining the + second node + + ``` + sudo k8s get-join-token --worker + ``` + +3. On the worker node create the configuration.yaml file + + ``` + extra-node-kubelet-args: + --reserved-cpus: "0-31" + --cpu-manager-policy: "static" + --topology-manager-policy: "best-effort" + ``` + +4. On the worker node use the token to join the cluster + + ``` + sudo k8s join-cluster --file configuration.yaml + ``` + + +#### Verify the two node cluster is ready + +After a few seconds the second worker node will register with the control +plane. You can query the available workers from the first node: + +``` +sudo k8s kubectl get nodes +``` + +The output should list the connected nodes: + +``` +NAME STATUS ROLES AGE VERSION +pc6b-rb4-n1 Ready control-plane,worker 22h v1.31.0 +pc6b-rb4-n3 Ready worker 22h v1.31.0 +``` + +### Multus and SR-IOV setup + +Apply the 'thick' Multus plugin (in case of resource scarcity we can consider +deploying the thin flavour) + +``` +sudo k8s kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml +``` + +```{note} +The memory limits for the Multus pod spec in the DaemonSet should be +increased (i.e. to 500Mi instead 50Mi) to avoid OOM issues when deploying +multiple workload pods in parallel. +``` + +#### SR-IOV Network Device Plugin + +Create `sriov-dp.yaml` configMap: + +``` +cat <TAbort- SERR- + Kernel driver in use: vfio-pci + Kernel modules: iavf +``` + +Now, create a test pod that will claim a network interface from the DPDK +network: + +``` +cat < + +[MAAS]: https://maas.io +[channel]: ../explanation/channels/ +[install-link]: install/snap +[snap]: https://snapcraft.io/docs +[cyclictest]: https://github.com/jlelli/rt-tests +[explain-epa]: ../explanation/epa \ No newline at end of file diff --git a/docs/src/snap/howto/external-datastore.md b/docs/src/snap/howto/external-datastore.md index 5c4204432..bd583a3ca 100644 --- a/docs/src/snap/howto/external-datastore.md +++ b/docs/src/snap/howto/external-datastore.md @@ -37,7 +37,7 @@ datastore-client-key: | ``` -* `datastore-url` expects a comma seperated list of addresses +* `datastore-url` expects a comma separated list of addresses (e.g. `https://10.42.254.192:2379,https://10.42.254.193:2379,https://10.42.254.194:2379`) * `datastore-ca-crt` expects a certificate for the CA in PEM format diff --git a/docs/src/snap/howto/index.md b/docs/src/snap/howto/index.md index 1445e2a32..0df54a503 100644 --- a/docs/src/snap/howto/index.md +++ b/docs/src/snap/howto/index.md @@ -17,13 +17,15 @@ Overview install/index networking/index storage/index -external-datastore -proxy +Use an external datastore backup-restore refresh-certs restore-quorum +two-node-ha +Set up Enhanced Platform Awareness +cis-hardening contribute -support +Get support ``` --- diff --git a/docs/src/snap/howto/install/index.md b/docs/src/snap/howto/install/index.md index 6c7403acc..76e1169c7 100644 --- a/docs/src/snap/howto/install/index.md +++ b/docs/src/snap/howto/install/index.md @@ -12,8 +12,8 @@ the current How-to guides below. :glob: :titlesonly: -snap +Install from a snap multipass -lxd -offline +Install in LXD +Install in air-gapped environments ``` diff --git a/docs/src/snap/howto/install/lxd.md b/docs/src/snap/howto/install/lxd.md index e81a1c11b..60f8df590 100644 --- a/docs/src/snap/howto/install/lxd.md +++ b/docs/src/snap/howto/install/lxd.md @@ -109,7 +109,7 @@ port assigned by Kubernetes. In this example, we will use [Microbot] as it provides a simple HTTP endpoint to expose. These steps can be applied to any other deployment. -First, initialize the k8s cluster with +First, initialise the k8s cluster with ``` lxc exec k8s -- sudo k8s bootstrap @@ -239,4 +239,4 @@ need to access for example storage devices (See comment in [^5]). [default-bridged-networking]: https://ubuntu.com/blog/lxd-networking-lxdbr0-explained [Microbot]: https://github.com/dontrebootme/docker-microbot [AppArmor]: https://apparmor.net/ -[channels]: /snap/explanation/channels +[channels]: ../../explanation/channels diff --git a/docs/src/snap/howto/install/multipass.md b/docs/src/snap/howto/install/multipass.md index 7c15847c5..d008b9815 100644 --- a/docs/src/snap/howto/install/multipass.md +++ b/docs/src/snap/howto/install/multipass.md @@ -1,6 +1,6 @@ # Install with Multipass (Ubuntu/Mac/Windows) -**Multipass** is a simple way to run Ubuntu in a +[Multipass][]is a simple way to run Ubuntu in a virtual machine, no matter what your underlying OS. It is the recommended way to run {{product}} on Windows and macOS systems, and is equally useful for running multiple instances of the `k8s` snap on Ubuntu too. @@ -26,7 +26,7 @@ Multipass is shipped as a snap for Ubuntu and other OSes which support the Windows users should download and install the Multipass installer from the website. -The latest version is available here , +The [latest Windows version][] is available to download, though you may wish to visit the [Multipass website][] for more details. @@ -37,7 +37,7 @@ though you may wish to visit the [Multipass website][] for more details. Users running macOS should download and install the Multipass installer from the website. -The latest version is available here , +The [latest macOS version] is available to download, though you may wish to visit the [Multipass website][] for more details, including an alternate install method using `brew`. @@ -60,14 +60,14 @@ multipass launch 22.04 --name k8s-node --memory 4G --disk 20G --cpus 2 This command specifies: -- **22.04**: The Ubuntu image used as the basis for the instance -- **--name**: The name by which you will refer to the instance -- **--memory**: The memory to allocate -- **--disk**: The disk space to allocate -- **--cpus**: The number of CPU cores to reserve for this instance +- `22.04`: The Ubuntu image used as the basis for the instance +- `--name`: The name by which you will refer to the instance +- `--memory`: The memory to allocate +- `--disk`: The disk space to allocate +- `--cpus`: The number of CPU cores to reserve for this instance For more details of creating instances with Multipass, please see the -[Multipass documentation][multipass-options] about instance creation. +[Multipass documentation][Multipass-options] about instance creation. ## Access the created instance @@ -111,8 +111,11 @@ multipass purge +[Multipass]:https://multipass.run/ [snap-support]: https://snapcraft.io/docs/installing-snapd -[multipass-options]: https://multipass.run/docs/get-started-with-multipass-linux#heading--create-a-customised-instance +[Multipass-options]: https://multipass.run/docs/get-started-with-multipass-linux#heading--create-a-customised-instance [install instructions]: ./snap [Getting started]: ../../tutorial/getting-started [Multipass website]: https://multipass.run/docs +[latest Window version]:https://multipass.run/download/windows +[latest macOS version]:https://multipass.run/download/macos diff --git a/docs/src/snap/howto/install/offline.md b/docs/src/snap/howto/install/offline.md index dc251347e..28776effc 100644 --- a/docs/src/snap/howto/install/offline.md +++ b/docs/src/snap/howto/install/offline.md @@ -29,7 +29,7 @@ are necessary to verify the integrity of the packages. ```{note} Update the version of k8s by adjusting the channel parameter. For more information on channels visit the -[channels explanation](/snap/explanation/channels.md). +[channels explanation](../../explanation/channels.md). ``` ```{note} @@ -61,13 +61,13 @@ add a dummy default route on the `eth0` interface using the following command: ip route add default dev eth0 ``` -```{note} +```{note} Ensure that `eth0` is the name of the default network interface used for pod-to-pod communication. ``` -The dummy gateway will only be used by the Kubernetes services to -know which interface to use, actual connectivity to the internet is not +The dummy gateway will only be used by the Kubernetes services to +know which interface to use, actual connectivity to the internet is not required. Ensure that the dummy gateway rule survives a node reboot. #### Ensure proxy access @@ -91,10 +91,10 @@ All workloads in a Kubernetes cluster are run as an OCI image. Kubernetes needs to be able to fetch these images and load them into the container runtime. For {{product}}, it is also necessary to fetch the images used -by its features (network, dns, etc.) as well as any images that are +by its features (network, DNS, etc.) as well as any images that are needed to run specific workloads. -```{note} +```{note} The image options are presented in the order of increasing complexity of implementation. It may be helpful to combine these options for different scenarios. @@ -120,12 +120,12 @@ ghcr.io/canonical/k8s-snap/sig-storage/csi-node-driver-registrar:v2.10.1 ghcr.io/canonical/k8s-snap/sig-storage/csi-provisioner:v5.0.1 ghcr.io/canonical/k8s-snap/sig-storage/csi-resizer:v1.11.1 ghcr.io/canonical/k8s-snap/sig-storage/csi-snapshotter:v8.0.1 -ghcr.io/canonical/metrics-server:0.7.0-ck1 +ghcr.io/canonical/metrics-server:0.7.0-ck2 ghcr.io/canonical/rawfile-localpv:0.8.0-ck4 ``` -A list of images can also be found in the `images.txt` file when unsquashing the -downloaded k8s snap. +A list of images can also be found in the `images.txt` file when the +downloaded k8s snap is unsquashed. Please ensure that the images used by workloads are tracked as well. @@ -167,12 +167,18 @@ any upstream registries (e.g. `docker.io`) and the private mirror. ##### Load images with regsync We recommend using [regsync][regsync] to copy images from the upstream registry -to your private registry. Refer to the [sync-images.yaml][sync-images-yaml] -file that contains the configuration for syncing images from the upstream -registry to the private registry. Using the output from `k8s list-images` -update the images in the [sync-images.yaml][sync-images-yaml] file if -necessary. Update the file with the appropriate mirror, and specify a mirror -for ghcr.io that points to the registry. +to your private registry. +For that, create a `sync-images.yaml` file that maps the output from +`k8s list-images` to the private registry mirror and specify a mirror for +ghcr.io that points to the registry. + +``` +sync: + - source: ghcr.io/canonical/k8s-snap/pause:3.10 + target: '{{ env "MIRROR" }}/canonical/k8s-snap/pause:3.10' + type: image + ... +``` After creating the `sync-images.yaml` file, use [regsync][regsync] to sync the images. Assuming your registry mirror is at http://10.10.10.10:5050, run: @@ -264,7 +270,7 @@ capabilities = ["pull", "resolve"] HTTPS requires the additionally specification of the registry CA certificate. Copy the certificate to `/var/snap/k8s/common/etc/containerd/hosts.d/ghcr.io/ca.crt`. -Then add the configuration in +Then add the configuration in `/var/snap/k8s/common/etc/containerd/hosts.d/ghcr.io/hosts.toml`: ``` @@ -299,11 +305,9 @@ After a while, confirm that all the cluster nodes show up in the output of the [Core20]: https://canonical.com/blog/ubuntu-core-20-secures-linux-for-iot -[svc-ports]: /snap/explanation/services-and-ports.md -[proxy]: /snap/howto/proxy.md -[sync-images-yaml]: https://github.com/canonical/k8s-snap/blob/main/build-scripts/hack/sync-images.yaml +[proxy]: ../networking/proxy.md [regsync]: https://github.com/regclient/regclient/blob/main/docs/regsync.md [regctl]: https://github.com/regclient/regclient/blob/main/docs/regctl.md [regctl.sh]: https://github.com/canonical/k8s-snap/blob/main/src/k8s/tools/regctl.sh -[nodes]: /snap/tutorial/add-remove-nodes.md +[nodes]: ../../tutorial/add-remove-nodes.md [squid]: https://www.squid-cache.org/ diff --git a/docs/src/snap/howto/install/snap.md b/docs/src/snap/howto/install/snap.md index cc16e7cd6..b18afdf1c 100644 --- a/docs/src/snap/howto/install/snap.md +++ b/docs/src/snap/howto/install/snap.md @@ -80,4 +80,4 @@ ready state. [channels page]: ../../explanation/channels [snap]: https://snapcraft.io/docs [snapd support]: https://snapcraft.io/docs/installing-snapd -[bootstrap]: /snap/reference/bootstrap-config-reference \ No newline at end of file +[bootstrap]: ../../reference/bootstrap-config-reference \ No newline at end of file diff --git a/docs/src/snap/howto/networking/default-dns.md b/docs/src/snap/howto/networking/default-dns.md index 34722d3a1..6cc6e4aac 100644 --- a/docs/src/snap/howto/networking/default-dns.md +++ b/docs/src/snap/howto/networking/default-dns.md @@ -94,4 +94,4 @@ sudo k8s help disable -[getting-started-guide]: /snap/tutorial/getting-started +[getting-started-guide]: ../../tutorial/getting-started diff --git a/docs/src/snap/howto/networking/default-ingress.md b/docs/src/snap/howto/networking/default-ingress.md index 90498d910..e70d66157 100644 --- a/docs/src/snap/howto/networking/default-ingress.md +++ b/docs/src/snap/howto/networking/default-ingress.md @@ -55,7 +55,7 @@ You should see three options: ### TLS Secret You can create a TLS secret by following the official -[Kubernetes documentation][kubectl-create-secret-tls/]. +[Kubernetes documentation][kubectl-create-secret-TLS/]. Please remember to use `sudo k8s kubectl` (See the [kubectl-guide]). Tell Ingress to use your new Ingress certificate: @@ -105,7 +105,7 @@ sudo k8s help disable -[kubectl-create-secret-tls/]: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_secret_tls/ +[kubectl-create-secret-TLS/]: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_secret_tls/ [proxy-protocol]: https://kubernetes.io/docs/reference/networking/service-protocols/#protocol-proxy-special -[getting-started-guide]: /snap/tutorial/getting-started -[kubectl-guide]: /snap/tutorial/kubectl +[getting-started-guide]: ../../tutorial/getting-started +[kubectl-guide]: ../../tutorial/kubectl diff --git a/docs/src/snap/howto/networking/default-loadbalancer.md b/docs/src/snap/howto/networking/default-loadbalancer.md index 88a2a20fb..6552b87a0 100644 --- a/docs/src/snap/howto/networking/default-loadbalancer.md +++ b/docs/src/snap/howto/networking/default-loadbalancer.md @@ -9,7 +9,7 @@ explains how to configure and enable the load-balancer. This guide assumes the following: - You have root or sudo access to the machine. -- You have a bootstraped {{product}} cluster (see the [Getting +- You have a bootstrapped {{product}} cluster (see the [Getting Started][getting-started-guide] guide). ## Check the status and configuration @@ -28,13 +28,15 @@ To check the current configuration of the load-balancer, run the following: ``` sudo k8s get load-balancer ``` + This should output a list of values like this: -- `cidrs` - a list containing [cidr] or IP address range definitions of the +- `cidrs` - a list containing [CIDR] or IP address range definitions of the pool of IP addresses to use - `l2-mode` - whether L2 mode (failover) is turned on -- `l2-interfaces` - optional list of interfaces to announce services over (defaults to all) +- `l2-interfaces` - optional list of interfaces to announce services over + (defaults to all) - `bgp-mode` - whether BGP mode is active. - `bgp-local-asn` - the local Autonomous System Number (ASN) - `bgp-peer-address` - the peer address @@ -47,7 +49,8 @@ These values are configured using the `k8s set`command, e.g.: sudo k8s set load-balancer.l2-mode=true ``` -Note that for the BGP mode, it is necessary to set ***all*** the values simultaneously. E.g. +Note that for the BGP mode, it is necessary to set ***all*** the values +simultaneously. E.g. ``` sudo k8s set load-balancer.bgp-mode=true load-balancer.bgp-local-asn=64512 load-balancer.bgp-peer-address=10.0.10.55/32 load-balancer.bgp-peer-asn=64512 load-balancer.bgp-peer-port=7012 @@ -77,6 +80,5 @@ sudo k8s disable load-balancer - -[cidr]: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing -[getting-started-guide]: /snap/tutorial/getting-started +[CIDR]: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing +[getting-started-guide]: ../../tutorial/getting-started diff --git a/docs/src/snap/howto/networking/default-network.md b/docs/src/snap/howto/networking/default-network.md index bde0aad9e..d39537bc2 100644 --- a/docs/src/snap/howto/networking/default-network.md +++ b/docs/src/snap/howto/networking/default-network.md @@ -91,4 +91,4 @@ sudo k8s disable network --help -[getting-started-guide]: /snap/tutorial/getting-started +[getting-started-guide]: ../../tutorial/getting-started diff --git a/docs/src/snap/howto/networking/dualstack.md b/docs/src/snap/howto/networking/dualstack.md index a7114ce6f..efa1b0c19 100644 --- a/docs/src/snap/howto/networking/dualstack.md +++ b/docs/src/snap/howto/networking/dualstack.md @@ -6,13 +6,13 @@ both IPv4 and IPv6 addresses, allowing them to communicate over either protocol. This document will guide you through enabling dual-stack, including necessary configurations, known limitations, and common issues. -### Prerequisites +## Prerequisites Before enabling dual-stack, ensure that your environment supports IPv6, and that your network configuration (including any underlying infrastructure) is compatible with dual-stack operation. -### Enabling Dual-Stack +## Enabling Dual-Stack Dual-stack can be enabled by specifying both IPv4 and IPv6 CIDRs during the cluster bootstrap process. The key configuration parameters are: @@ -133,11 +133,18 @@ cluster bootstrap process. The key configuration parameters are: working. -### CIDR Size Limitations +## CIDR Size Limitations When setting up dual-stack networking, it is important to consider the limitations regarding CIDR size: -- **/64 is too large for the Service CIDR**: Using a `/64` CIDR for services -may cause issues like failure to initialize the IPv6 allocator. This is due +- **/108 is the maximum size for the Service CIDR** +Using a smaller value than `/108` for service CIDRs +may cause issues like failure to initialise the IPv6 allocator. This is due to the CIDR size being too large for Kubernetes to handle efficiently. + +See upstream reference: [kube-apiserver validation][kube-apiserver-test] + + + +[kube-apiserver-test]: https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/options/validation_test.go#L435 diff --git a/docs/src/snap/howto/networking/index.md b/docs/src/snap/howto/networking/index.md index 98d42bd55..d015577a2 100644 --- a/docs/src/snap/howto/networking/index.md +++ b/docs/src/snap/howto/networking/index.md @@ -11,9 +11,11 @@ how to configure and use key capabilities of {{product}}. ```{toctree} :titlesonly: -/snap/howto/networking/default-dns.md -/snap/howto/networking/default-network.md -/snap/howto/networking/default-ingress.md -/snap/howto/networking/default-loadbalancer.md -/snap/howto/networking/dualstack.md +Use default DNS +Use default network +Use default Ingress +Use default load-balancer +Enable Dual-Stack networking +Set up an IPv6-only cluster +Configure proxy settings ``` diff --git a/docs/src/snap/howto/networking/ipv6.md b/docs/src/snap/howto/networking/ipv6.md new file mode 100644 index 000000000..65bd5cc99 --- /dev/null +++ b/docs/src/snap/howto/networking/ipv6.md @@ -0,0 +1,142 @@ +# How to set up an IPv6-Only Cluster + +An IPv6-only Kubernetes cluster operates exclusively using IPv6 addresses, +without support for IPv4. This configuration is ideal for environments that +are transitioning away from IPv4 or want to take full advantage of IPv6's +expanded address space. This document, explains how to set up +an IPv6-only cluster, including key configurations and necessary checks +to ensure proper setup. + +## Prerequisites + +Before setting up an IPv6-only cluster, ensure that: + +- Your environment supports IPv6. +- Network infrastructure, such as routers, firewalls, and DNS, are configured +to handle IPv6 traffic. +- Any underlying infrastructure (e.g. cloud providers, bare metal setups) +must be IPv6-compatible. + +## Setting Up an IPv6-Only Cluster + +The process of creating an IPv6-only cluster involves specifying only IPv6 +CIDRs for pods and services during the bootstrap process. Unlike dual-stack, +only IPv6 CIDRs are used. + +1. **Bootstrap Kubernetes with IPv6 CIDRs** + +Start by bootstrapping the Kubernetes cluster and providing only IPv6 +CIDRs for pods and services: + +```bash +sudo k8s bootstrap --timeout 10m --interactive +``` + +When prompted, set the pod and service CIDRs to IPv6 ranges. For example: + +``` +Please set the Pod CIDR: [fd01::/108] +Please set the Service CIDR: [fd98::/108] +``` + +Alternatively, these values can be configured in a bootstrap configuration file +named `bootstrap-config.yaml` in this example: + +```yaml +pod-cidr: fd01::/108 +service-cidr: fd98::/108 +``` + +Specify the configuration file during the bootstrapping process: + +```bash +sudo k8s bootstrap --file bootstrap-config.yaml +``` + +2. **Verify Pod and Service Creation** + +Once the cluster is up, verify that all pods are running: + +```sh +sudo k8s kubectl get pods -A +``` + +Deploy a pod with an nginx web-server and expose it via a service to verify +connectivity of the IPv6-only cluster. Create a manifest file +`nginx-ipv6.yaml` with the following content: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-ipv6 +spec: + selector: + matchLabels: + run: nginx-ipv6 + replicas: 1 + template: + metadata: + labels: + run: nginx-ipv6 + spec: + containers: + - name: nginx-ipv6 + image: rocks.canonical.com/cdk/diverdane/nginxipv6:1.0.0 + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Service +metadata: + name: nginx-ipv6 + labels: + run: nginx-ipv6 +spec: + type: NodePort + ipFamilies: + - IPv6 + ports: + - port: 80 + protocol: TCP + selector: + run: nginx-ipv6 +``` + +Deploy the web-server and its service by running: + +```sh +sudo k8s kubectl apply -f nginx-ipv6.yaml +``` + +3. **Verify IPv6 Connectivity** + +Retrieve the service details to confirm an IPv6 address is assigned: + +```sh +sudo k8s kubectl get service nginx-ipv6 -n default +``` + +Obtain the service’s IPv6 address from the output: + +``` +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +nginx-ipv6 NodePort fd98::7534 80:32248/TCP 2m +``` + +Use the assigned IPv6 address to test connectivity: + +```bash +curl http://[fd98::7534]/ +``` + +A welcome message from the nginx web-server is displayed when IPv6 +connectivity is set up correctly. + +## IPv6-Only Cluster Considerations + +**Service and Pod CIDR Sizing** + +Use `/108` as the maximum size for Service CIDRs. Larger ranges (e.g., `/64`) +may lead to allocation errors or Kubernetes failing to initialise the IPv6 +address allocator. diff --git a/docs/src/snap/howto/proxy.md b/docs/src/snap/howto/networking/proxy.md similarity index 71% rename from docs/src/snap/howto/proxy.md rename to docs/src/snap/howto/networking/proxy.md index ad4186625..d0f64414d 100644 --- a/docs/src/snap/howto/proxy.md +++ b/docs/src/snap/howto/networking/proxy.md @@ -1,11 +1,11 @@ # Configure proxy settings for K8s -{{product}} packages a number of utilities (eg curl, helm) which need +{{product}} packages a number of utilities (for example curl, helm) which need to fetch resources they expect to find on the internet. In a constrained network environment, such access is usually controlled through proxies. To set up a proxy using squid follow the -[how-to-install-a-squid-server][squid] tutorial. +[How to install a Squid server][squid] tutorial. ## Adding proxy configuration for the k8s snap @@ -37,19 +37,21 @@ Environment="no_proxy=10.0.0.0/8,10.152.183.1,192.168.0.0/16,127.0.0.1,172.16.0. Note that you may need to restart for these settings to take effect. -```{note} The **10.152.183.0/24** CIDR needs to be covered in the juju-no-proxy - list as it is the Kubernetes service CIDR. Without this any pods will not be - able to reach the cluster's kubernetes-api. You should also exclude the range - used by pods (which defaults to **10.1.0.0/16**) and any required - local networks. + +```{note} Include the CIDR **10.152.183.0/24** in both the +`no_proxy` and `NO_PROXY` environment variables, as it's the default Kubernetes +service CIDR. If you are using a different service CIDR, update this setting +accordingly. This ensures pods can access the cluster's Kubernetes API Server. +Also, include the default pod range (**10.1.0.0/16**) and any local networks +needed. ``` ## Adding proxy configuration for the k8s charms Proxy configuration is handled by Juju when deploying the `k8s` charms. Please -see the [documentation for adding proxy configuration via Juju]. +see the [documentation for adding proxy configuration via Juju][juju-proxy]. -[documentation for adding proxy configuration via Juju]: /charm/howto/proxy +[juju-proxy]: ../../../charm/howto/proxy [squid]: https://ubuntu.com/server/docs/how-to-install-a-squid-server diff --git a/docs/src/snap/howto/restore-quorum.md b/docs/src/snap/howto/restore-quorum.md index aeb15b721..9050797c7 100755 --- a/docs/src/snap/howto/restore-quorum.md +++ b/docs/src/snap/howto/restore-quorum.md @@ -1,9 +1,9 @@ # Recovering a Cluster After Quorum Loss Highly available {{product}} clusters can survive losing one or more -nodes. [Dqlite], the default datastore, implements a [Raft] based protocol where -an elected leader holds the definitive copy of the database, which is then -replicated on two or more secondary nodes. +nodes. [Dqlite], the default datastore, implements a [Raft] based protocol +where an elected leader holds the definitive copy of the database, which is +then replicated on two or more secondary nodes. When the a majority of the nodes are lost, the cluster becomes unavailable. If at least one database node survived, the cluster can be recovered using the @@ -64,8 +64,8 @@ sudo snap stop k8s ## Recover the Database -Choose one of the remaining alive cluster nodes that has the most recent version -of the Raft log. +Choose one of the remaining alive cluster nodes that has the most recent +version of the Raft log. Update the ``cluster.yaml`` files, changing the role of the lost nodes to "spare" (2). Additionally, double check the addresses and IDs specified in @@ -73,7 +73,8 @@ Update the ``cluster.yaml`` files, changing the role of the lost nodes to files were moved across nodes. The following command guides us through the recovery process, prompting a text -editor with informative inline comments for each of the dqlite configuration files. +editor with informative inline comments for each of the dqlite configuration +files. ``` sudo /snap/k8s/current/bin/k8sd cluster-recover \ @@ -82,29 +83,40 @@ sudo /snap/k8s/current/bin/k8sd cluster-recover \ --log-level 0 ``` -Please adjust the log level for additional debug messages by increasing its value. -The command creates database backups before making any changes. +Please adjust the log level for additional debug messages by increasing its +value. The command creates database backups before making any changes. -The above command will reconfigure the Raft members and create recovery tarballs -that are used to restore the lost nodes, once the Dqlite configuration is updated. +The above command will reconfigure the Raft members and create recovery +tarballs that are used to restore the lost nodes, once the Dqlite +configuration is updated. ```{note} -By default, the command will recover both Dqlite databases. If one of the databases -needs to be skipped, use the ``--skip-k8sd`` or ``--skip-k8s-dqlite`` flags. -This can be useful when using an external Etcd database. +By default, the command will recover both Dqlite databases. If one of the +databases needs to be skipped, use the ``--skip-k8sd`` or ``--skip-k8s-dqlite`` +flags. This can be useful when using an external Etcd database. ``` -Once the "cluster-recover" command completes, restart the k8s services on the node: +```{note} +Non-interactive mode can be requested using the ``--non-interactive`` flag. +In this case, no interactive prompts or text editors will be displayed and +the command will assume that the configuration files have already been updated. + +This allows automating the recovery procedure. +``` + +Once the "cluster-recover" command completes, restart the k8s services on the +node: ``` sudo snap start k8s ``` -Ensure that the services started successfully by using ``sudo snap services k8s``. -Use ``k8s status --wait-ready`` to wait for the cluster to become ready. +Ensure that the services started successfully by using +``sudo snap services k8s``. Use ``k8s status --wait-ready`` to wait for the +cluster to become ready. -You may notice that we have not returned to an HA cluster yet: ``high availability: no``. -This is expected as we need to recover +You may notice that we have not returned to an HA cluster yet: +``high availability: no``. This is expected as we need to recover ## Recover the remaining nodes @@ -113,28 +125,34 @@ nodes. For k8sd, copy ``recovery_db.tar.gz`` to ``/var/snap/k8s/common/var/lib/k8sd/state/recovery_db.tar.gz``. When the k8sd -service starts, it will load the archive and perform the necessary recovery steps. +service starts, it will load the archive and perform the necessary recovery +steps. The k8s-dqlite archive needs to be extracted manually. First, create a backup of the current k8s-dqlite state directory: ``` -sudo mv /var/snap/k8s/common/var/lib/k8s-dqlite /var/snap/k8s/common/var/lib/k8s-dqlite.bkp +sudo mv /var/snap/k8s/common/var/lib/k8s-dqlite \ + /var/snap/k8s/common/var/lib/k8s-dqlite.bkp ``` Then, extract the backup archive: ``` sudo mkdir /var/snap/k8s/common/var/lib/k8s-dqlite -sudo tar xf recovery-k8s-dqlite-$timestamp-post-recovery.tar.gz -C /var/snap/k8s/common/var/lib/k8s-dqlite +sudo tar xf recovery-k8s-dqlite-$timestamp-post-recovery.tar.gz \ + -C /var/snap/k8s/common/var/lib/k8s-dqlite ``` -Node specific files need to be copied back to the k8s-dqlite state dir: +Node specific files need to be copied back to the k8s-dqlite state directory: ``` -sudo cp /var/snap/k8s/common/var/lib/k8s-dqlite.bkp/cluster.crt /var/snap/k8s/common/var/lib/k8s-dqlite -sudo cp /var/snap/k8s/common/var/lib/k8s-dqlite.bkp/cluster.key /var/snap/k8s/common/var/lib/k8s-dqlite -sudo cp /var/snap/k8s/common/var/lib/k8s-dqlite.bkp/info.yaml /var/snap/k8s/common/var/lib/k8s-dqlite +sudo cp /var/snap/k8s/common/var/lib/k8s-dqlite.bkp/cluster.crt \ + /var/snap/k8s/common/var/lib/k8s-dqlite +sudo cp /var/snap/k8s/common/var/lib/k8s-dqlite.bkp/cluster.key \ + /var/snap/k8s/common/var/lib/k8s-dqlite +sudo cp /var/snap/k8s/common/var/lib/k8s-dqlite.bkp/info.yaml \ + /var/snap/k8s/common/var/lib/k8s-dqlite ``` Once these steps are completed, restart the k8s services: @@ -143,13 +161,15 @@ Once these steps are completed, restart the k8s services: sudo snap start k8s ``` -Repeat these steps for all remaining nodes. Once a quorum is achieved, the cluster -will be reported as "highly available": +Repeat these steps for all remaining nodes. Once a quorum is achieved, +the cluster will be reported as "highly available": ``` $ sudo k8s status cluster status: ready -control plane nodes: 10.80.130.168:6400 (voter), 10.80.130.167:6400 (voter), 10.80.130.164:6400 (voter) +control plane nodes: 10.80.130.168:6400 (voter), + 10.80.130.167:6400 (voter), + 10.80.130.164:6400 (voter) high availability: yes datastore: k8s-dqlite network: enabled diff --git a/docs/src/snap/howto/storage/ceph.md b/docs/src/snap/howto/storage/ceph.md index b448cab9f..b379d2e5f 100644 --- a/docs/src/snap/howto/storage/ceph.md +++ b/docs/src/snap/howto/storage/ceph.md @@ -29,7 +29,7 @@ this demonstration will have less than 5 OSDs. (See [placement groups]) ceph osd pool create kubernetes 128 ``` -Initialize the pool as a Ceph block device pool. +Initialise the pool as a Ceph block device pool. ``` rbd pool init kubernetes @@ -48,8 +48,7 @@ capabilities to administer your Ceph cluster: ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes' ``` -For more information on user capabilities in Ceph, see -[https://docs.ceph.com/en/latest/rados/operations/user-management/#authorization-capabilities] +For more information on user capabilities in Ceph, see the [authorisation capabilities page][] ``` [client.kubernetes] @@ -60,7 +59,7 @@ Note the generated key, you will need it at a later step. ## Generate csi-config-map.yaml -First, get the fsid and the monitor addresses of your cluster. +First, get the `fsid` and the monitor addresses of your cluster. ``` sudo ceph mon dump @@ -79,7 +78,7 @@ election_strategy: 1 dumped monmap epoch 2 ``` -Keep note of the v1 IP (`10.0.0.136:6789`) and the fsid +Keep note of the v1 IP (`10.0.0.136:6789`) and the `fsid` (`6d5c12c9-6dfb-445a-940f-301aa7de0f29`) as you will need to refer to them soon. ``` @@ -131,11 +130,10 @@ Then apply: kubectl apply -f csi-kms-config-map.yaml ``` -If you do need to configure a KMS provider, an example ConfigMap is available in -the Ceph repository: -[https://github.com/ceph/ceph-csi/blob/devel/examples/kms/vault/kms-config.yaml] +If you do need to configure a KMS provider, an [example ConfigMap][] is available +in the Ceph repository. -Create the `ceph-config-map.yaml` which will be stored inside a ceph.conf file +Create the `ceph-config-map.yaml` which will be stored inside a `ceph.conf` file in the CSI containers. This `ceph.conf` file will be used by Ceph daemons on each container to authenticate with the Ceph cluster. @@ -188,7 +186,7 @@ Then apply: kubectl apply -f csi-rbd-secret.yaml ``` -## Create ceph-csi's custom Kubernetes objects +## Create ceph-csi custom Kubernetes objects Create the ServiceAccount and RBAC ClusterRole/ClusterRoleBinding objects: @@ -251,7 +249,7 @@ Then apply: kubectl apply -f csi-rbd-sc.yaml ``` -## Create a Persistant Volume Claim (PVC) for a RBD-backed file-system +## Create a Persistent Volume Claim (PVC) for a RBD-backed file-system This PVC will allow users to request RBD-backed storage. @@ -279,7 +277,7 @@ Then apply: kubectl apply -f pvc.yaml ``` -## Create a pod that binds to the Rados Block Device PVC +## Create a pod that binds to the RADOS Block Device PVC Finally, create a pod configuration that uses the RBD-backed PVC. @@ -313,7 +311,7 @@ kubectl apply -f pod.yaml ## Verify that the pod is using the RBD PV -To verify that the csi-rbd-demo-pod is indeed using a RBD Persistant Volume, run +To verify that the `csi-rbd-demo-pod` is indeed using a RBD Persistent Volume, run the following commands, you should see information related to attached volumes in both of their outputs: @@ -331,7 +329,9 @@ Ceph documentation: [Intro to Ceph]. [Ceph]: https://ceph.com/ -[getting-started-guide]: ../tutorial/getting-started.md +[getting-started-guide]: ../../tutorial/getting-started.md [block-devices-and-kubernetes]: https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/ [placement groups]: https://docs.ceph.com/en/mimic/rados/operations/placement-groups/ [Intro to Ceph]: https://docs.ceph.com/en/latest/start/intro/ +[authorisation capabilities page]:[https://docs.ceph.com/en/latest/rados/operations/user-management/#authorization-capabilities] +[example ConfigMap]:https://github.com/ceph/ceph-csi/blob/devel/examples/kms/vault/kms-config.yaml diff --git a/docs/src/snap/howto/storage/cloud.md b/docs/src/snap/howto/storage/cloud.md new file mode 100644 index 000000000..920b297dc --- /dev/null +++ b/docs/src/snap/howto/storage/cloud.md @@ -0,0 +1,496 @@ +# How to use cloud storage + +{{product}} simplifies the process of integrating and managing cloud storage +solutions like Amazon EBS. This guide provides steps to configure IAM policies, +deploy the cloud controller manager, and set up the necessary drivers for you +to take advantage of cloud storage solutions in the context of Kubernetes. + +## What you'll need + +This guide is for AWS and assumes the following: + +- You have root or sudo access to an Amazon EC2 instance +- You can create roles and policies in AWS + + +## Set IAM Policies + +Your instance will need a few IAM policies to be able to communciate with the +AWS APIs. The policies provided here are quite open and should be scoped down +based on your security requirements. + +You will most likely want to create a role for your instance. You can call this +role "k8s-control-plane" or "k8s-worker". Then, define and attach the following +policies to the role. Once the role is created with the required policies, +attach the role to the instance. + +For a control plane node: + +```{dropdown} Control Plane Policies +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "autoscaling:DescribeAutoScalingGroups", + "autoscaling:DescribeLaunchConfigurations", + "autoscaling:DescribeTags", + "ec2:DescribeInstances", + "ec2:DescribeRegions", + "ec2:DescribeRouteTables", + "ec2:DescribeSecurityGroups", + "ec2:DescribeSubnets", + "ec2:DescribeVolumes", + "ec2:DescribeAvailabilityZones", + "ec2:CreateSecurityGroup", + "ec2:CreateTags", + "ec2:CreateVolume", + "ec2:ModifyInstanceAttribute", + "ec2:ModifyVolume", + "ec2:AttachVolume", + "ec2:AuthorizeSecurityGroupIngress", + "ec2:CreateRoute", + "ec2:DeleteRoute", + "ec2:DeleteSecurityGroup", + "ec2:DeleteVolume", + "ec2:DetachVolume", + "ec2:RevokeSecurityGroupIngress", + "ec2:DescribeVpcs", + "ec2:DescribeInstanceTopology", + "elasticloadbalancing:AddTags", + "elasticloadbalancing:AttachLoadBalancerToSubnets", + "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer", + "elasticloadbalancing:CreateLoadBalancer", + "elasticloadbalancing:CreateLoadBalancerPolicy", + "elasticloadbalancing:CreateLoadBalancerListeners", + "elasticloadbalancing:ConfigureHealthCheck", + "elasticloadbalancing:DeleteLoadBalancer", + "elasticloadbalancing:DeleteLoadBalancerListeners", + "elasticloadbalancing:DescribeLoadBalancers", + "elasticloadbalancing:DescribeLoadBalancerAttributes", + "elasticloadbalancing:DetachLoadBalancerFromSubnets", + "elasticloadbalancing:DeregisterInstancesFromLoadBalancer", + "elasticloadbalancing:ModifyLoadBalancerAttributes", + "elasticloadbalancing:RegisterInstancesWithLoadBalancer", + "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer", + "elasticloadbalancing:AddTags", + "elasticloadbalancing:CreateListener", + "elasticloadbalancing:CreateTargetGroup", + "elasticloadbalancing:DeleteListener", + "elasticloadbalancing:DeleteTargetGroup", + "elasticloadbalancing:DescribeListeners", + "elasticloadbalancing:DescribeLoadBalancerPolicies", + "elasticloadbalancing:DescribeTargetGroups", + "elasticloadbalancing:DescribeTargetHealth", + "elasticloadbalancing:ModifyListener", + "elasticloadbalancing:ModifyTargetGroup", + "elasticloadbalancing:RegisterTargets", + "elasticloadbalancing:DeregisterTargets", + "elasticloadbalancing:SetLoadBalancerPoliciesOfListener", + "iam:CreateServiceLinkedRole", + "kms:DescribeKey" + ], + "Resource": [ + "*" + ] + } + ] +} +``` + +For a worker node: + +```{dropdown} Worker Policies +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:DescribeInstances", + "ec2:DescribeRegions", + "ecr:GetAuthorizationToken", + "ecr:BatchCheckLayerAvailability", + "ecr:GetDownloadUrlForLayer", + "ecr:GetRepositoryPolicy", + "ecr:DescribeRepositories", + "ecr:ListImages", + "ecr:BatchGetImage" + ], + "Resource": "*" + } + ] +} +``` + +## Add a tag to your EC2 Instance + +A cluster using the AWS cloud provider needs to label existing nodes and +resources with a ClusterID or the kube-controller-manager will not start. Add +the following tag to your instance, making sure to replace the placeholder id +with your own (this can simply be "k8s" or "my-k8s-cluster"). + +``` +kubernetes.io/cluster/=owned +``` + +## Set your host name + +The cloud controller manager uses the node name to correctly associate the node +with an EC2 instance. In {{product}}, the node name is derived from the +hostname of the machine. Therefore, before bootstrapping the cluster, you must +first set an appropriate host name. + +```bash +echo "$(sudo cloud-init query ds.meta_data.local-hostname)" | sudo tee /etc/hostname +``` + +Then, reboot the machine. + +```bash +sudo reboot +``` + +When the machine is up, use `hostname -f` to check the host name. It should +look like: + +```bash +ip-172-31-11-86.us-east-2.compute.internal +``` + +This host name format is called IP-based naming and is specific to AWS. + + +## Bootstrap {{product}} + +Now that your machine has an appropriate host name, you are ready to bootstrap +{{product}}. + +First, create a bootstrap configuration file that sets the cloud-provider +configuration to "external". + +```bash +echo "cluster-config: + cloud-provider: external" > bootstrap-config.yaml +``` + +Then, bootstrap the cluster: + +```bash +sudo k8s bootstrap --file ./bootstrap-config.yaml +sudo k8s status --wait-ready +``` + +## Deploy the cloud controller manager + +Now that you have an appropriate host name, policies, and a {{product}} +cluster, you have everything you need to deploy the cloud controller manager. + +Here is a YAML definition file that sets appropriate defaults for you, it +configures the necessary service accounts, roles, and daemonsets: + +```{dropdown} CCM deployment manifest +```yaml +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: aws-cloud-controller-manager + namespace: kube-system + labels: + k8s-app: aws-cloud-controller-manager +spec: + selector: + matchLabels: + k8s-app: aws-cloud-controller-manager + updateStrategy: + type: RollingUpdate + template: + metadata: + labels: + k8s-app: aws-cloud-controller-manager + spec: + nodeSelector: + node-role.kubernetes.io/control-plane: "" + tolerations: + - key: node.cloudprovider.kubernetes.io/uninitialized + value: "true" + effect: NoSchedule + - effect: NoSchedule + key: node-role.kubernetes.io/control-plane + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/control-plane + operator: Exists + serviceAccountName: cloud-controller-manager + containers: + - name: aws-cloud-controller-manager + image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.28.3 + args: + - --v=2 + - --cloud-provider=aws + - --use-service-account-credentials=true + - --configure-cloud-routes=false + resources: + requests: + cpu: 200m + hostNetwork: true +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: cloud-controller-manager + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: cloud-controller-manager:apiserver-authentication-reader + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: extension-apiserver-authentication-reader +subjects: + - apiGroup: "" + kind: ServiceAccount + name: cloud-controller-manager + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: system:cloud-controller-manager +rules: +- apiGroups: + - "" + resources: + - events + verbs: + - create + - patch + - update +- apiGroups: + - "" + resources: + - nodes + verbs: + - '*' +- apiGroups: + - "" + resources: + - nodes/status + verbs: + - patch +- apiGroups: + - "" + resources: + - services + verbs: + - list + - patch + - update + - watch +- apiGroups: + - "" + resources: + - services/status + verbs: + - list + - patch + - update + - watch +- apiGroups: + - "" + resources: + - serviceaccounts + verbs: + - create + - get + - list + - watch +- apiGroups: + - "" + resources: + - persistentvolumes + verbs: + - get + - list + - update + - watch +- apiGroups: + - "" + resources: + - endpoints + verbs: + - create + - get + - list + - watch + - update +- apiGroups: + - coordination.k8s.io + resources: + - leases + verbs: + - create + - get + - list + - watch + - update +- apiGroups: + - "" + resources: + - serviceaccounts/token + verbs: + - create +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: system:cloud-controller-manager +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:cloud-controller-manager +subjects: + - apiGroup: "" + kind: ServiceAccount + name: cloud-controller-manager + namespace: kube-system +``` + +You can apply the CCM manifest easily by running the following command: + +```bash +sudo k8s kubectl apply -f https://raw.githubusercontent.com/canonical/k8s-snap/main/docs/src/assets/how-to-cloud-storage-aws-ccm.yaml +``` + +After a moment, you should see the cloud controller manager pod was +successfully deployed. + +```bash +NAME READY STATUS RESTARTS AGE +aws-cloud-controller-manager-ndbtq 1/1 Running 1 (3h51m ago) 9h +``` + +## Deploy the EBS CSI Driver + +Now that the cloud controller manager is deployed and can communicate with AWS, +you are ready to deploy the EBS CSI driver. The easiest way to deploy the +driver is with the Helm chart. Luckily, {{product}} has a built-in Helm +command. + +If you want to create encrypted drives, you need to add the statement to the +policy you are using for the instance. + +```json +{ + "Effect": "Allow", + "Action": [ + "kms:Decrypt", + "kms:GenerateDataKeyWithoutPlaintext", + "kms:CreateGrant" + ], + "Resource": "*" +} +``` + +Then, add the Helm repo for the EBS CSI Driver. + +```bash +sudo k8s helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver +sudo k8s helm repo update +``` + +Finally, install the Helm chart, making sure to set the correct region as an +argument. + +```bash +sudo k8s helm upgrade --install aws-ebs-csi-driver \ + --namespace kube-system \ + aws-ebs-csi-driver/aws-ebs-csi-driver \ + --set controller.region= +``` + +Once the command completes, you can verify the pods are successfully deployed: + +```bash +kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver +``` + +```bash +NAME READY STATUS RESTARTS AGE +ebs-csi-controller-78bcd46cf8-5zk8q 5/5 Running 2 (3h48m ago) 8h +ebs-csi-controller-78bcd46cf8-g7l5h 5/5 Running 1 (3h48m ago) 8h +ebs-csi-node-nx6rg 3/3 Running 0 9h +``` + +The status of all pods should be "Running". + +## Deploy a workload + +Everything is in place for you to deploy a workload that dynamically creates +and uses an EBS volume. + +First, create a StorageClass and a PersistentVolumeClaim: + +``` +sudo k8s kubectl apply -f - < Volumes` page in AWS, you should see a 10Gi gp3 volume. diff --git a/docs/src/snap/howto/storage/index.md b/docs/src/snap/howto/storage/index.md index 4310beac7..f79732d4d 100644 --- a/docs/src/snap/howto/storage/index.md +++ b/docs/src/snap/howto/storage/index.md @@ -12,6 +12,7 @@ default storage built-in to {{product}}. ```{toctree} :titlesonly: -storage -ceph -``` \ No newline at end of file +Use default storage +Use Ceph storage +Use cloud storage +``` diff --git a/docs/src/snap/howto/storage/storage.md b/docs/src/snap/howto/storage/storage.md index dbba33631..3b7d0a261 100644 --- a/docs/src/snap/howto/storage/storage.md +++ b/docs/src/snap/howto/storage/storage.md @@ -62,4 +62,4 @@ Disabling storage only removes the CSI driver. The persistent volume claims will still be available and your data will remain on disk. -[getting-started-guide]: ../tutorial/getting-started.md +[getting-started-guide]: ../../tutorial/getting-started.md diff --git a/docs/src/snap/howto/two-node-ha.md b/docs/src/snap/howto/two-node-ha.md new file mode 100644 index 000000000..e7b1cdc99 --- /dev/null +++ b/docs/src/snap/howto/two-node-ha.md @@ -0,0 +1,430 @@ +# Two-Node High-Availability with Dqlite + +High availability (HA) is a mandatory requirement for most production-grade +Kubernetes deployments, usually implying three or more nodes. + +Two-node HA clusters are sometimes preferred for cost savings and operational +efficiency considerations. Follow this guide to learn how Canonical Kubernetes +can achieve high availability with just two nodes while using the default +datastore, [Dqlite]. Both nodes will be active members of the cluster, sharing +the Kubernetes load. + +Dqlite cannot achieve a [Raft] quorum with fewer than three nodes. This means +that Dqlite will not be able to replicate data and the secondaries will simply +forward the queries to the primary node. + +In the event of a node failure, database recovery will require following the +steps in the [Dqlite recovery guide]. + +## Proposed solution + +Since Dqlite data replication is not available in this situation, we propose +using synchronous block level replication through +[Distributed Replicated Block Device] (DRBD). + +The cluster monitoring and failover process will be handled by [Pacemaker] and +[Corosync]. After a node failure, the DRBD volume will be mounted on the +standby node, allowing access to the latest Dqlite database version. + +Additional recovery steps are automated and invoked through Pacemaker. + +### Prerequisites: + +* Please ensure that both nodes are part of the Kubernetes cluster. + See the [getting started] and [add/remove nodes] guides. +* The user associated with the HA service has SSH access to the peer node and + passwordless sudo configured. For simplicity, the default "ubuntu" user can + be used. +* We recommend using static IP configuration. + +The [two-node-ha.sh script] automates most operations related to the two-node +HA scenario and is included in the snap. + +The first step is to install the required packages: + +``` +/snap/k8s/current/k8s/hack/two-node-ha.sh install_packages +``` + +### Distributed Replicated Block Device (DRBD) + +This example uses a loopback device as DRBD backing storage: + +``` +sudo dd if=/dev/zero of=/opt/drbd0-backstore bs=1M count=2000 +``` + +Ensure that the loopback device is attached at boot time, before Pacemaker +starts. + +``` +cat < +HATWO_ADDR= + +cat < +HATWO_ADDR= + +sudo mv /etc/corosync/corosync.conf /etc/corosync/corosync.conf.orig + +cat < +HATWO_ADDR= +DRBD_MOUNT_DIR=${DRBD_MOUNT_DIR:-"/mnt/drbd0"} + +sudo crm configure < + +# remove the node constraint. +sudo crm resource clear fs_res +``` + +### Managing Kubernetes Snap Services + +For the two-node HA setup, k8s snap services should no longer start +automatically. Instead, they will be managed by a wrapper service. + +``` +for f in `sudo snap services k8s | awk 'NR>1 {print $1}'`; do + echo "disabling snap.$f" + sudo systemctl disable "snap.$f"; +done +``` + +### Preparing the wrapper service + +The next step is to define the wrapper service. Add the following to +``/etc/systemd/system/two-node-ha-k8s.service``. + +```{note} +the sample uses the ``ubuntu`` user, feel free to use a different one as long as the prerequisites +are met. +``` + +``` +[Unit] +Description=K8s service wrapper handling Dqlite recovery for two-node HA setups. +After=network.target pacemaker.service + +[Service] +User=ubuntu +Group=ubuntu +Type=oneshot +ExecStart=/bin/bash /snap/k8s/current/k8s/hack/two-node-ha.sh start_service +ExecStop=/bin/bash sudo snap stop k8s +RemainAfterExit=true + +[Install] +WantedBy=multi-user.target +``` + +```{note} +The ``two-node-ha.sh start_service`` command used by the service wrapper +automatically detects the expected Dqlite role based on the DRBD state. +It then takes the necessary steps to bootstrap the Dqlite state directories, +synchronise with the peer node (if available) and recover the database. +``` + +When a DRBD failover occurs, the ``two-node-ha-k8s`` service needs to be +restarted. To accomplish this, we are going to define a separate service that +will be invoked by Pacemaker. Create a file called +``/etc/systemd/system/two-node-ha-k8s-failover.service`` containing the +following: + +``` +[Unit] +Description=Managed by Pacemaker, restarts two-node-ha-k8s on failover. +After=network.target home-ubuntu-workspace.mount + +[Service] +Type=oneshot +ExecStart=systemctl restart two-node-ha-k8s +RemainAfterExit=true +``` + +Reload the systemd configuration and set ``two-node-ha-k8s`` to start +automatically. Notice that ``two-node-ha-k8s-failover`` must not be configured +to start automatically, but instead is going to be managed through Pacemaker. + +``` +sudo systemctl enable two-node-ha-k8s +sudo systemctl daemon-reload +``` + +Make sure that both nodes have been configured using the above steps before +moving forward. + +### Automating the failover procedure + +Define a new Pacemaker resource that will invoke the +``two-node-ha-k8s-failover`` service when a DRBD failover occurs. + +``` +sudo crm configure < +[Dqlite]: https://dqlite.io/ +[Raft]: https://raft.github.io/ +[Distributed Replicated Block Device]: https://ubuntu.com/server/docs/distributed-replicated-block-device-drbd +[Dqlite recovery guide]: restore-quorum +[external datastore guide]: external-datastore +[two-node-ha.sh script]: https://github.com/canonical/k8s-snap/blob/main/k8s/hack/two-node-ha.sh +[getting started]: ../tutorial/getting-started +[add/remove nodes]: ../tutorial/add-remove-nodes +[Pacemaker]: https://clusterlabs.org/pacemaker/ +[Corosync]: https://clusterlabs.org/corosync.html +[Pacemaker fencing]: https://clusterlabs.org/pacemaker/doc/2.1/Pacemaker_Explained/html/fencing.html +[split brain]: https://en.wikipedia.org/wiki/Split-brain_(computing) diff --git a/docs/src/snap/index.md b/docs/src/snap/index.md index 7022a799f..d4b1e4f92 100644 --- a/docs/src/snap/index.md +++ b/docs/src/snap/index.md @@ -1,5 +1,20 @@ # {{product}} snap documentation +```{toctree} +:hidden: +Overview +``` + +```{toctree} +:hidden: +:titlesonly: +:maxdepth: 6 +tutorial/index.md +howto/index.md +explanation/index.md +reference/index.md +``` + The {{product}} snap is a performant, lightweight, secure and opinionated distribution of **Kubernetes** which includes everything needed to create and manage a scalable cluster suitable for all use cases. @@ -70,4 +85,4 @@ and constructive feedback. [roadmap]: ./reference/roadmap [overview page]: ./explanation/about [architecture documentation]: ./reference/architecture -[Juju charm]: /charm/index +[Juju charm]: ../charm/index diff --git a/docs/src/snap/reference/annotations.md b/docs/src/snap/reference/annotations.md index b5e4404d8..010c3143f 100644 --- a/docs/src/snap/reference/annotations.md +++ b/docs/src/snap/reference/annotations.md @@ -1,13 +1,191 @@ # Annotations This page outlines the annotations that can be configured during cluster -[bootstrap]. To do this, set the cluster-config/annotations parameter in +[bootstrap]. To do this, set the `cluster-config.annotations` parameter in the bootstrap configuration. -| Name | Description | Values | -|---------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------| -| `k8sd/v1alpha/lifecycle/skip-cleanup-kubernetes-node-on-remove` | If set, only microcluster and file cleanup are performed. This is helpful when an external controller (e.g., CAPI) manages the Kubernetes node lifecycle. By default, k8sd will remove the Kubernetes node when it is removed from the cluster. | "true"\|"false" | +For example: + +```yaml +cluster-config: +... + annotations: + k8sd/v1alpha/lifecycle/skip-cleanup-kubernetes-node-on-remove: true + k8sd/v1alpha/lifecycle/skip-stop-services-on-remove: true +``` + +Please refer to the [Kubernetes website] for more information on annnotations. + +## `k8sd/v1alpha/lifecycle/skip-cleanup-kubernetes-node-on-remove` + +| | | +|---|---| +| **Values**| "true"\|"false"| +| **Description**| If set, only MicroCluster and file cleanup are performed. This is helpful when an external controller (e.g., CAPI) manages the Kubernetes node lifecycle. By default, k8sd will remove the Kubernetes node when it is removed from the cluster. | + +## `k8sd/v1alpha/lifecycle/skip-cleanup-kubernetes-node-on-remove` + +| | | +|---|---| +|**Values**| "true"\|"false"| +|**Description**|If set, the k8s services will not be stopped on the leaving node when removing the node. This is helpful when an external controller (e.g., CAPI) manages the Kubernetes node lifecycle. By default, all services are stopped on leaving nodes.| + +## `k8sd/v1alpha1/csrsigning/auto-approve` + +| | | +|---|---| +|**Values**| "true"\|"false"| +|**Description**|If set, certificate signing requests created by worker nodes are auto approved.| + +## `k8sd/v1alpha1/calico/apiserver-enabled` + +| | | +|---|---| +|**Values**| "true"\|"false"| +|**Description**|Enable the installation of the Calico API server to enable management of Calico APIs using kubectl.| + +## `k8sd/v1alpha1/calico/encapsulation-v4` + +| | | +|---|---| +|**Values**| “IPIP”\|”VXLAN”\|”IPIPCrossSubnet”\|”VXLANCrossSubnet”\|”None”| +|**Description**|The type of encapsulation to use on the IPv4 pool.| + +## `k8sd/v1alpha1/calico/encapsulation-v6` + +| | | +|---|---| +|**Values**| “IPIP”\|”VXLAN”\|”IPIPCrossSubnet”\|”VXLANCrossSubnet”\|”None”| +|**Description**|The type of encapsulation to use on the IPv6 pool.| + +## `k8sd/v1alpha1/calico/autodetection-v4/firstFound` + +| | | +|---|---| +|**Values**| "true"\|"false"| +|**Description**|Use default interface matching parameters to select an interface, performing best-effort filtering based on well-known interface names.| + +## `k8sd/v1alpha1/calico/autodetection-v4/kubernetes` + +| | | +|---|---| +|**Values**| “NodeInternalIP”| +|**Description**|Configure Calico to detect node addresses based on the Kubernetes API.| + +## `k8sd/v1alpha1/calico/autodetection-v4/interface` + +| | | +|---|---| +|**Values**| string | +|**Description**|Enable IP auto-detection based on interfaces that match the given regex.| + +## `k8sd/v1alpha1/calico/autodetection-v4/skipInterface` + +| | | +|---|---| +|**Values**| string | +|**Description**|Enable IP auto-detection based on interfaces that do not match the given regex.| + +## `k8sd/v1alpha1/calico/autodetection-v4/canReach` + +| | | +|---|---| +|**Values**| string | +|**Description**|Enable IP auto-detection based on which source address on the node is used to reach the specified IP or domain.| + +## `k8sd/v1alpha1/calico/autodetection-v4/cidrs` + +| | | +|---|---| +|**Values**| \[] (string values comma separated) | +|**Description**|Enable IP auto-detection based on which addresses on the nodes are within one of the provided CIDRs.| + +## `k8sd/v1alpha1/calico/autodetection-v6/firstFound` + +| | | +|---|---| +|**Values**| "true"\|"false" | +|**Description**|Use default interface matching parameters to select an interface, performing best-effort filtering based on well-known interface names.| + +## `k8sd/v1alpha1/calico/autodetection-v6/kubernetes` + +| | | +|---|---| +|**Values**| “NodeInternalIP” | +|**Description**|Configure Calico to detect node addresses based on the Kubernetes API.| + +## `k8sd/v1alpha1/calico/autodetection-v6/interface` + +| | | +|---|---| +|**Values**| string | +|**Description**|Enable IP auto-detection based on interfaces that match the given regex.| + +## `k8sd/v1alpha1/calico/autodetection-v6/skipInterface` + +| | | +|---|---| +|**Values**| string | +|**Description**|Enable IP auto-detection based on interfaces that do not match the given regex.| + +## `k8sd/v1alpha1/calico/autodetection-v6/canReach` + +| | | +|---|---| +|**Values**| string | +|**Description**|Enable IP auto-detection based on which source address on the node is used to reach the specified IP or domain.| + +## `k8sd/v1alpha1/calico/autodetection-v6/cidrs` + +| | | +|---|---| +|**Values**| \[] (string values comma separated) | +|**Description**|Enable IP auto-detection based on which addresses on the nodes are within one of the provided CIDRs.| + +## `k8sd/v1alpha1/cilium/devices` + +| | | +|---|---| +|**Values**| string| +|**Description**|List of devices facing cluster/external network (used for BPF NodePort, BPF masquerading and host firewall); supports `+` as wildcard in device name, e.g. `eth+,ens+` | + +## `k8sd/v1alpha1/cilium/direct-routing-device` + +| | | +|---|---| +|**Values**| string| +|**Description**|Device name used to connect nodes in direct routing mode (used by BPF NodePort, BPF host routing); if empty, automatically set to a device with k8s InternalIP/ExternalIP or with a default route. Bridge type devices are ignored in automatic selection| + +## `k8sd/v1alpha1/cilium/vlan-bpf-bypass` + +| | | +|---|---| +|**Values**| \[] (string values comma separated)| +|**Description**|Comma separated list of VLAN tags to bypass eBPF filtering on native devices. Cilium enables a firewall on native devices and filters all unknown traffic, including VLAN 802.1q packets, which pass through the main device with the associated tag (e.g., VLAN device eth0.4000 and its main interface eth0). Supports `0` as wildcard for bypassing all VLANs. e.g. `4001,4002`| + +## `k8sd/v1alpha1/metrics-server/image-repo` + +| | | +|---|---| +|**Values**| string| +|**Description**|Override the default image repository for the metrics-server.| + +## `k8sd/v1alpha1/metrics-server/image-tag` + +| | | +|---|---| +|**Values**| string| +|**Description**|Override the default image tag for the metrics-server.| + + -[bootstrap]: /snap/reference/bootstrap-config-reference +[Kubernetes website]:https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/ +[bootstrap]: bootstrap-config-reference diff --git a/docs/src/snap/reference/architecture.md b/docs/src/snap/reference/architecture.md index b3adad13e..02170835e 100644 --- a/docs/src/snap/reference/architecture.md +++ b/docs/src/snap/reference/architecture.md @@ -10,8 +10,7 @@ current design of {{product}}, following the [C4 model]. This overview of {{product}} demonstrates the interactions of Kubernetes with users and with other systems. -```{kroki} ../../assets/overview.puml -``` +![cluster5][] Two actors interact with the Kubernetes snap: @@ -20,7 +19,8 @@ Two actors interact with the Kubernetes snap: access to the cluster. That initial user is able to configure the cluster to match their needs and of course create other users that may or may not have admin privileges. The K8s admin is also able to maintain workloads running - in the cluster. + in the cluster. If you deploy {{product}} from a snap, this is how the cluster + is manually orchestrated. - **K8s user**: A user consuming the workloads hosted in the cluster. Users do not have access to the Kubernetes API server. They need to access the cluster @@ -52,8 +52,7 @@ distribution. We have identified the following: Looking more closely at what is contained within the K8s snap itself: -```{kroki} ../../assets/k8s-container.puml -``` +![cluster1][] The `k8s` snap distribution includes the following: @@ -74,8 +73,7 @@ The `k8s` snap distribution includes the following: K8sd is the component that implements and exposes the operations functionality needed for managing the Kubernetes cluster. -```{kroki} ../../assets/k8sd-component.puml -``` +![cluster2][] At the core of the `k8sd` functionality we have the cluster manager that is responsible for configuring the services, workload and features we deem @@ -107,8 +105,7 @@ This functionality is exposed via the following interfaces: Canonical `k8s` Charms encompass two primary components: the [`k8s` charm][K8s charm] and the [`k8s-worker` charm][K8s-worker charm]. -```{kroki} ../../assets/charms-architecture.puml -``` +![cluster4][] Charms are instantiated on a machine as a Juju unit, and a collection of units constitutes an application. Both `k8s` and `k8s-worker` units are responsible @@ -119,7 +116,7 @@ determines the node's role in the Kubernetes cluster. The `k8s` charm manages directing the `juju` controller to reach the model's eventually consistent state. For more detail on Juju's concepts, see the [Juju docs][]. -The administrator may choose any supported cloud-types (Openstack, MAAS, AWS, +The administrator may choose any supported cloud-types (OpenStack, MAAS, AWS, GCP, Azure...) on which to manage the machines making up the Kubernetes cluster. Juju selects a single leader unit per application to act as a centralised figure with the model. The `k8s` leader oversees Kubernetes @@ -140,6 +137,12 @@ and the sharing of observability data with the [`Canonical Observability Stack (COS)`][COS docs]. This modular and integrated approach facilitates a robust and flexible {{product}} deployment managed through Juju. + + +[cluster1]: https://assets.ubuntu.com/v1/dfc43753-cluster1.svg +[cluster2]: https://assets.ubuntu.com/v1/f634743e-k8sd.svg +[cluster4]: https://assets.ubuntu.com/v1/24fd1773-cluster4.svg +[cluster5]: https://assets.ubuntu.com/v1/bcfe150f-overview.svg [C4 model]: https://c4model.com/ diff --git a/docs/src/snap/reference/bootstrap-config-reference.md b/docs/src/snap/reference/bootstrap-config-reference.md index 047622c5d..758828e04 100644 --- a/docs/src/snap/reference/bootstrap-config-reference.md +++ b/docs/src/snap/reference/bootstrap-config-reference.md @@ -1,520 +1,14 @@ # Bootstrap configuration file reference -A YAML file can be supplied to the `k8s bootstrap` command to configure and +A YAML file can be supplied to the `k8s join-cluster` command to configure and customise the cluster. This reference section provides the format of this file by listing all available options and their details. See below for an example. -## Format Specification +## Configuration options -### cluster-config.network - -**Type:** `object`
-**Required:** `No` - -Configuration options for the network feature - -#### cluster-config.network.enabled - -**Type:** `bool`
-**Required:** `No`
- -Determines if the feature should be enabled. -If omitted defaults to `true` - -### cluster-config.dns - -**Type:** `object`
-**Required:** `No` - -Configuration options for the dns feature - -#### cluster-config.dns.enabled - -**Type:** `bool`
-**Required:** `No`
- -Determines if the feature should be enabled. -If omitted defaults to `true` - -#### cluster-config.dns.cluster-domain - -**Type:** `string`
-**Required:** `No`
- -Sets the local domain of the cluster. -If omitted defaults to `cluster.local` - -#### cluster-config.dns.service-ip - -**Type:** `string`
-**Required:** `No`
- -Sets the IP address of the dns service. If omitted defaults to the IP address -of the Kubernetes service created by the feature. - -Can be used to point to an external dns server when feature is disabled. - - -#### cluster-config.dns.upstream-nameservers - -**Type:** `list[string]`
-**Required:** `No`
- -Sets the upstream nameservers used to forward queries for out-of-cluster -endpoints. -If omitted defaults to `/etc/resolv.conf` and uses the nameservers of the node. - - -### cluster-config.ingress - -**Type:** `object`
-**Required:** `No` - -Configuration options for the ingress feature - -#### cluster-config.ingress.enabled - -**Type:** `bool`
-**Required:** `No`
- -Determines if the feature should be enabled. -If omitted defaults to `false` - -#### cluster-config.ingress.default-tls-secret - -**Type:** `string`
-**Required:** `No`
- -Sets the name of the secret to be used for providing default encryption to -ingresses. - -Ingresses can specify another TLS secret in their resource definitions, -in which case the default secret won't be used. - -#### cluster-config.ingress.enable-proxy-protocol - -**Type:** `bool`
-**Required:** `No`
- -Determines if the proxy protocol should be enabled for ingresses. -If omitted defaults to `false` - - -### cluster-config.load-balancer - -**Type:** `object`
-**Required:** `No` - -Configuration options for the load-balancer feature - -#### cluster-config.load-balancer.enabled - -**Type:** `bool`
-**Required:** `No`
- -Determines if the feature should be enabled. -If omitted defaults to `false` - -#### cluster-config.load-balancer.cidrs - -**Type:** `list[string]`
-**Required:** `No`
- -Sets the CIDRs used for assigning IP addresses to Kubernetes services with type -`LoadBalancer`. - -#### cluster-config.load-balancer.l2-mode - -**Type:** `bool`
-**Required:** `No`
- -Determines if L2 mode should be enabled. -If omitted defaults to `false` - -#### cluster-config.load-balancer.l2-interfaces - -**Type:** `list[string]`
-**Required:** `No`
- -Sets the interfaces to be used for announcing IP addresses through ARP. -If omitted all interfaces will be used. - -#### cluster-config.load-balancer.bgp-mode - -**Type:** `bool`
-**Required:** `No`
- -Determines if BGP mode should be enabled. -If omitted defaults to `false` - -#### cluster-config.load-balancer.bgp-local-asn - -**Type:** `int`
-**Required:** `Yes if bgp-mode is true`
- -Sets the ASN to be used for the local virtual BGP router. - -#### cluster-config.load-balancer.bgp-peer-address - -**Type:** `string`
-**Required:** `Yes if bgp-mode is true`
- -Sets the IP address of the BGP peer. - -#### cluster-config.load-balancer.bgp-peer-asn - -**Type:** `int`
-**Required:** `Yes if bgp-mode is true`
- -Sets the ASN of the BGP peer. - -#### cluster-config.load-balancer.bgp-peer-port - -**Type:** `int`
-**Required:** `Yes if bgp-mode is true`
- -Sets the port of the BGP peer. - - -### cluster-config.local-storage - -**Type:** `object`
-**Required:** `No` - -Configuration options for the local-storage feature - -#### cluster-config.local-storage.enabled - -**Type:** `bool`
-**Required:** `No`
- -Determines if the feature should be enabled. -If omitted defaults to `false` - -#### cluster-config.local-storage.local-path - -**Type:** `string`
-**Required:** `No`
- -Sets the path to be used for storing volume data. -If omitted defaults to `/var/snap/k8s/common/rawfile-storage` - -#### cluster-config.local-storage.reclaim-policy - -**Type:** `string`
-**Required:** `No`
-**Possible Values:** `Retain | Recycle | Delete` - -Sets the reclaim policy of the storage class. -If omitted defaults to `Delete` - -#### cluster-config.local-storage.default - -**Type:** `bool`
-**Required:** `No`
- -Determines if the storage class should be set as default. -If omitted defaults to `true` - - -### cluster-config.gateway - -**Type:** `object`
-**Required:** `No` - -Configuration options for the gateway feature - -#### cluster-config.gateway.enabled - -**Type:** `bool`
-**Required:** `No`
- -Determines if the feature should be enabled. -If omitted defaults to `true` - -### cluster-config.cloud-provider - -**Type:** `string`
-**Required:** `No`
-**Possible Values:** `external` - -Sets the cloud provider to be used by the cluster. - -When this is set as `external`, node will wait for an external cloud provider to -do cloud specific setup and finish node initialization. - -### control-plane-taints - -**Type:** `list[string]`
-**Required:** `No` - -List of taints to be applied to control plane nodes. - -### pod-cidr - -**Type:** `string`
-**Required:** `No` - -The CIDR to be used for assigning pod addresses. -If omitted defaults to `10.1.0.0/16` - -### service-cidr - -**Type:** `string`
-**Required:** `No` - -The CIDR to be used for assigning service addresses. -If omitted defaults to `10.152.183.0/24` - -### disable-rbac - -**Type:** `bool`
-**Required:** `No` - -Determines if RBAC should be disabled. -If omitted defaults to `false` - -### secure-port - -**Type:** `int`
-**Required:** `No` - -The port number for kube-apiserver to use. -If omitted defaults to `6443` - -### k8s-dqlite-port - -**Type:** `int`
-**Required:** `No` - -The port number for k8s-dqlite to use. -If omitted defaults to `9000` - -### datastore-type - -**Type:** `string`
-**Required:** `No`
-**Possible Values:** `k8s-dqlite | external` - -The type of datastore to be used. -If omitted defaults to `k8s-dqlite` - -Can be used to point to an external datastore like etcd. - -### datastore-servers - -**Type:** `list[string]`
-**Required:** `No`
- -The server addresses to be used when `datastore-type` is set to `external`. - -### datastore-ca-crt - -**Type:** `string`
-**Required:** `No`
- -The CA certificate to be used when communicating with the external datastore. - -### datastore-client-crt - -**Type:** `string`
-**Required:** `No`
- -The client certificate to be used when communicating with the external -datastore. - -### datastore-client-key - -**Type:** `string`
-**Required:** `No`
- -The client key to be used when communicating with the external datastore. - -### extra-sans - -**Type:** `list[string]`
-**Required:** `No`
- -List of extra SANs to be added to certificates. - -### ca-crt - -**Type:** `string`
-**Required:** `No`
- -The CA certificate to be used for Kubernetes services. -If omitted defaults to an auto generated certificate. - -### ca-key - -**Type:** `string`
-**Required:** `No`
- -The CA key to be used for Kubernetes services. -If omitted defaults to an auto generated key. - -### front-proxy-ca-crt - -**Type:** `string`
-**Required:** `No`
- -The CA certificate to be used for the front proxy. -If omitted defaults to an auto generated certificate. - -### front-proxy-ca-key - -**Type:** `string`
-**Required:** `No`
- -The CA key to be used for the front proxy. -If omitted defaults to an auto generated key. - -### front-proxy-client-crt - -**Type:** `string`
-**Required:** `No`
- -The client certificate to be used for the front proxy. -If omitted defaults to an auto generated certificate. - -### front-proxy-client-key - -**Type:** `string`
-**Required:** `No`
- -The client key to be used for the front proxy. -If omitted defaults to an auto generated key. - - -### apiserver-kubelet-client-crt - -**Type:** `string`
-**Required:** `No`
- -The client certificate to be used by kubelet for communicating with the -kube-apiserver. -If omitted defaults to an auto generated certificate. - -### apiserver-kubelet-client-key - -**Type:** `string`
-**Required:** `No`
- -The client key to be used by kubelet for communicating with the kube-apiserver. -If omitted defaults to an auto generated key. - -### service-account-key - -**Type:** `string`
-**Required:** `No`
- -The key to be used by the default service account. -If omitted defaults to an auto generated key. - -### apiserver-crt - -**Type:** `string`
-**Required:** `No`
- -The certificate to be used for the kube-apiserver. -If omitted defaults to an auto generated certificate. - -### apiserver-key - -**Type:** `string`
-**Required:** `No`
- -The key to be used for the kube-apiserver. -If omitted defaults to an auto generated key. - -### kubelet-crt - -**Type:** `string`
-**Required:** `No`
- -The certificate to be used for the kubelet. -If omitted defaults to an auto generated certificate. - -### kubelet-key - -**Type:** `string`
-**Required:** `No`
- -The key to be used for the kubelet. -If omitted defaults to an auto generated key. - -### extra-node-config-files - -**Type:** `map[string]string`
-**Required:** `No`
- -Additional files that are uploaded `/var/snap/k8s/common/args/conf.d/` -to a node on bootstrap. These files can them be references by Kubernetes -service arguments. -The format is `map[]`. - -### extra-node-kube-apiserver-args - -**Type:** `map[string]string`
-**Required:** `No`
- -Additional arguments that are passed to the `kube-apiserver` only for that -specific node. Overwrites default configuration. A parameter that is explicitly -set to `null` is deleted. The format is `map[<--flag-name>]`. - -### extra-node-kube-controller-manager-args - -**Type:** `map[string]string`
-**Required:** `No`
- -Additional arguments that are passed to the `kube-controller-manager` only for -that specific node. Overwrites default configuration. A parameter that is -explicitly set to `null` is deleted. The format is `map[<--flag-name>]`. - -### extra-node-kube-scheduler-args - -**Type:** `map[string]string`
-**Required:** `No`
- -Additional arguments that are passed to the `kube-scheduler` only for that -specific node. Overwrites default configuration. A parameter that is explicitly -set to `null` is deleted. The format is `map[<--flag-name>]`. - -### extra-node-kube-proxy-args - -**Type:** `map[string]string`
-**Required:** `No`
- -Additional arguments that are passed to the `kube-proxy` only for that -specific node. Overwrites default configuration. A parameter that is explicitly -set to `null` is deleted. The format is `map[<--flag-name>]`. - -### extra-node-kubelet-args - -**Type:** `map[string]string`
-**Required:** `No`
- -Additional arguments that are passed to the `kubelet` only for that -specific node. Overwrites default configuration. A parameter that is explicitly -set to `null` is deleted. The format is `map[<--flag-name>]`. - -### extra-node-containerd-args - -**Type:** `map[string]string`
-**Required:** `No`
- -Additional arguments that are passed to `containerd` only for that -specific node. Overwrites default configuration. A parameter that is explicitly -set to `null` is deleted. The format is `map[<--flag-name>]`. - -### extra-node-k8s-dqlite-args - -**Type:** `map[string]string`
-**Required:** `No`
+```{include} ../../_parts/bootstrap_config.md +``` -Additional arguments that are passed to `k8s-dqlite` only for that -specific node. Overwrites default configuration. A parameter that is explicitly -set to `null` is deleted. The format is `map[<--flag-name>]`. ## Example diff --git a/docs/src/snap/reference/certificates.md b/docs/src/snap/reference/certificates.md index 29df8bdb0..996d80f6c 100644 --- a/docs/src/snap/reference/certificates.md +++ b/docs/src/snap/reference/certificates.md @@ -26,13 +26,13 @@ their issuance. | **Common Name** | **Purpose** | **File Location** | **Primary Function** | **Signed By** | |--------------------------------------------|-----------|------------------------------------------------------|------------------------------------------------------------------|-----------------------------| | `kube-apiserver` | Server | `/etc/kubernetes/pki/apiserver.crt` | Securing the API server endpoint | `kubernetes-ca` | -| `apiserver-kubelet-client` | Client | `/etc/kubernetes/pki/apiserver-kubelet-client.crt` | API server communication with kubelets | `kubernetes-ca-client` | +| `apiserver-kubelet-client` | Client | `/etc/kubernetes/pki/apiserver-kubelet-client.crt` | API server communication with kubelet | `kubernetes-ca-client` | | `kube-apiserver-etcd-client` | Client | `/etc/kubernetes/pki/apiserver-etcd-client.crt` | API server communication with etcd | `kubernetes-ca-client` | | `front-proxy-client` | Client | `/etc/kubernetes/pki/front-proxy-client.crt` | API server communication with the front-proxy | `kubernetes-front-proxy-ca` | | `system:kube-controller-manager` | Client | `/etc/kubernetes/pki/controller-manager.crt` | Communication between the controller manager and the API server | `kubernetes-ca-client` | | `system:kube-scheduler` | Client | `/etc/kubernetes/pki/scheduler.crt` | Communication between the scheduler and the API server | `kubernetes-ca-client` | | `system:kube-proxy` | Client | `/etc/kubernetes/pki/proxy.crt` | Communication between kube-proxy and the API server | `kubernetes-ca-client` | -| `system:node:$hostname` | Client | `/etc/kubernetes/pki/kubelet-client.crt` | Authentication of kubelets to the API server | `kubernetes-ca-client` | +| `system:node:$hostname` | Client | `/etc/kubernetes/pki/kubelet-client.crt` | Authentication of kubelet to the API server | `kubernetes-ca-client` | | `k8s-dqlite` | Client | `/var/snap/k8s/common/var/lib/k8s-dqlite/cluster.crt`| Communication between k8s-dqlite nodes and API server | `self-signed` | | `root@$hostname` | Client | `/var/snap/k8s/common/var/lib/k8s-dqlite/cluster.crt` | Communication between k8sd nodes | `self-signed` | diff --git a/docs/src/snap/reference/control-plane-join-config-reference.md b/docs/src/snap/reference/control-plane-join-config-reference.md new file mode 100755 index 000000000..06875521a --- /dev/null +++ b/docs/src/snap/reference/control-plane-join-config-reference.md @@ -0,0 +1,11 @@ +# Control plane node join configuration file reference + +A YAML file can be supplied to the `k8s join-cluster ` command to configure and +customise new nodes. + +This reference section provides all available options for control plane nodes. + +## Configuration options + +```{include} ../../_parts/control_plane_join_config.md +``` diff --git a/docs/src/snap/reference/index.md b/docs/src/snap/reference/index.md index f1720e760..bd7a9217f 100644 --- a/docs/src/snap/reference/index.md +++ b/docs/src/snap/reference/index.md @@ -16,10 +16,12 @@ commands annotations certificates bootstrap-config-reference +control-plane-join-config-reference +worker-join-config-reference proxy troubleshooting architecture -community +Community roadmap ``` diff --git a/docs/src/snap/reference/proxy.md b/docs/src/snap/reference/proxy.md index 1fa29f765..31acde759 100644 --- a/docs/src/snap/reference/proxy.md +++ b/docs/src/snap/reference/proxy.md @@ -37,5 +37,6 @@ how to set these. -[How to guide for configuring proxies for the k8s snap]: /snap/howto/proxy -[How to guide for configuring proxies for k8s charms]: /charm/howto/proxy +[How to guide for configuring proxies for the k8s snap]: ../howto/networking/proxy +[How to guide for configuring proxies for k8s charms]: ../../charm/howto/proxy + diff --git a/docs/src/snap/reference/releases.md b/docs/src/snap/reference/releases.md index 319f16737..b6035eb58 100644 --- a/docs/src/snap/reference/releases.md +++ b/docs/src/snap/reference/releases.md @@ -18,7 +18,7 @@ Currently {{product}} is working towards general availability, but you can install it now to try: - **Clustering** - need high availability or just an army of worker nodes? - {{product}} is emminently scaleable, see the [tutorial on adding + {{product}} is eminently scalable, see the [tutorial on adding more nodes][nodes]. - **Networking** - Our built-in network component allows cluster administrators to automatically scale and secure network policies across the cluster. Find @@ -32,8 +32,8 @@ Follow along with the [tutorial] to get started! -[tutorial]: /snap/tutorial/getting-started -[nodes]: /snap/tutorial/add-remove-nodes +[tutorial]: ../tutorial/getting-started +[nodes]: ../tutorial/add-remove-nodes [COS Lite]: https://charmhub.io/cos-lite -[networking]: /snap/howto/networking/index -[observability documentation]: /charm/howto/cos-lite \ No newline at end of file +[networking]: ../howto/networking/index +[observability documentation]: ../../charm/howto/cos-lite \ No newline at end of file diff --git a/docs/src/snap/reference/roadmap.md b/docs/src/snap/reference/roadmap.md index 63550f85c..a97ae5657 100644 --- a/docs/src/snap/reference/roadmap.md +++ b/docs/src/snap/reference/roadmap.md @@ -7,7 +7,7 @@ future direction and priorities of the project. Our roadmap matches the cadence of the Ubuntu release cycle, so `24.10` is the same as the release date for Ubuntu 24.10. This does not precisely map to the release cycle of Kubernetes versions, so please consult the [release notes] for -specifics of whatfeatures have been delivered. +specifics of what features have been delivered. ``` {csv-table} {{product}} public roadmap diff --git a/docs/src/snap/reference/troubleshooting.md b/docs/src/snap/reference/troubleshooting.md index f6b44a3f1..a4edf0d94 100644 --- a/docs/src/snap/reference/troubleshooting.md +++ b/docs/src/snap/reference/troubleshooting.md @@ -44,7 +44,7 @@ the kubelet. kubelet needs a feature from cgroup and the kernel may not be set up appropriately to provide the cpuset feature. ``` -E0125 00:20:56.003890 2172 kubelet.go:1466] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: root container [kubepods] doesn't exist" +E0125 00:20:56.003890 2172 kubelet.go:1466] "Failed to start ContainerManager" err="failed to initialise top level QOS containers: root container [kubepods] doesn't exist" ``` ### Explanation @@ -54,7 +54,7 @@ An excellent deep-dive of the issue exists at Commenter [@haircommander][] [states][kubernetes-122955-2020403422] > basically: we've figured out that this issue happens because libcontainer -> doesn't initialize the cpuset cgroup for the kubepods slice when the kubelet +> doesn't initialise the cpuset cgroup for the kubepods slice when the kubelet > initially calls into it to do so. This happens because there isn't a cpuset > defined on the top level of the cgroup. however, we fail to validate all of > the cgroup controllers we need are present. It's possible this is a @@ -68,7 +68,7 @@ Commenter [@haircommander][] [states][kubernetes-122955-2020403422] ### Solution This is in the process of being fixed upstream via -[kubernetes/kuberetes #125923][kubernetes-125923]. +[kubernetes/kubernetes #125923][kubernetes-125923]. In the meantime, the best solution is to create a `Delegate=yes` configuration in systemd. diff --git a/docs/src/snap/reference/worker-join-config-reference.md b/docs/src/snap/reference/worker-join-config-reference.md new file mode 100755 index 000000000..d10ea5ba2 --- /dev/null +++ b/docs/src/snap/reference/worker-join-config-reference.md @@ -0,0 +1,11 @@ +# Worker node join configuration file reference + +A YAML file can be supplied to the `k8s join-cluster ` command to configure and +customise new worker nodes. + +This reference section provides all available options for worker nodes. + +## Configuration options + +```{include} ../../_parts/worker_join_config.md +``` diff --git a/docs/src/snap/tutorial/add-remove-nodes.md b/docs/src/snap/tutorial/add-remove-nodes.md index 736474b46..9065c1a6c 100644 --- a/docs/src/snap/tutorial/add-remove-nodes.md +++ b/docs/src/snap/tutorial/add-remove-nodes.md @@ -1,4 +1,4 @@ -# Adding and Removing Nodes +# Adding and removing nodes Typical production clusters are hosted across multiple data centres and cloud environments, enabling them to leverage geographical distribution for improved @@ -8,7 +8,7 @@ This tutorial simplifies the concept by creating a cluster within a controlled environment using two Multipass VMs. The approach here allows us to focus on the foundational aspects of clustering using {{product}} without the complexities of a full-scale, production setup. If your nodes are already -installed, you can skip the multipass setup and go to [step 2](step2). +installed, you can skip the Multipass setup and go to [step 2](step2). ## Before starting @@ -56,14 +56,15 @@ sudo snap install --classic --edge k8s ### 2. Bootstrap your control plane node -Bootstrap the control plane node: +Bootstrap the control plane node with default configuration: ``` sudo k8s bootstrap ``` {{product}} allows you to create two types of nodes: control plane and -worker nodes. In this example, we're creating a worker node. +worker nodes. In this example, we just initialised a control plane node, now +let's create a worker node. Generate the token required for the worker node to join the cluster by executing the following command on the control-plane node: @@ -72,45 +73,48 @@ the following command on the control-plane node: sudo k8s get-join-token worker --worker ``` +`worker` refers to the name of the node we want to join. `--worker` is the type +of node we want to join. + A base64 token will be printed to your terminal. Keep it handy as you will need it for the next step. ```{note} It's advisable to name the new node after the hostname of the - worker node (in this case, the VM's hostname is worker). + worker node (in this case, the VM hostname is worker). ``` ### 3. Join the cluster on the worker node -To join the worker node to the cluster, run: +To join the worker node to the cluster, run on worker node: ``` sudo k8s join-cluster ``` -After a few seconds, you should see: `Joined the cluster.` +After a few seconds, you should see: `Joined the cluster.` ### 4. View the status of your cluster -To see what we've accomplished in this tutorial: +Let's review what we've accomplished in this tutorial. -If you created a control plane node, check that it joined successfully: +To see the control plane node created: ``` sudo k8s status ``` -If you created a worker node, verify with this command: +Verify the worker node joined successfully with this command +on control-plane node: ``` sudo k8s kubectl get nodes ``` -You should see that you've successfully added a worker or control plane node to -your cluster. +You should see that you've successfully added a worker to your cluster. Congratulations! -### 4. Remove Nodes and delete the VMs (Optional) +### 4. Remove nodes and delete the VMs (Optional) It is important to clean-up your nodes before tearing down the VMs. @@ -139,10 +143,10 @@ multipass delete worker multipass purge ``` -## Next Steps +## Next steps - Discover how to enable and configure Ingress resources [Ingress][Ingress] -- Keep mastering {{product}} with kubectl [How to use +- Learn more about {{product}} with kubectl [How to use kubectl][Kubectl] - Explore Kubernetes commands with our [Command Reference Guide][Command Reference] @@ -153,8 +157,8 @@ multipass purge [Getting started]: getting-started [Multipass Installation]: https://multipass.run/install -[Ingress]: /snap/howto/networking/default-ingress +[Ingress]: ../howto/networking/default-ingress [Kubectl]: kubectl -[Command Reference]: /snap/reference/commands -[Storage]: /snap/howto/storage -[Networking]: /snap/howto/networking/index.md +[Command Reference]: ../reference/commands +[Storage]: ../howto/storage/index +[Networking]: ../howto/networking/index.md diff --git a/docs/src/snap/tutorial/getting-started.md b/docs/src/snap/tutorial/getting-started.md index d613e202b..90b66d594 100644 --- a/docs/src/snap/tutorial/getting-started.md +++ b/docs/src/snap/tutorial/getting-started.md @@ -19,22 +19,24 @@ Install the {{product}} snap with: sudo snap install k8s --edge --classic ``` -### 2. Bootstrap a Kubernetes Cluster +### 2. Bootstrap a Kubernetes cluster -Bootstrap a Kubernetes cluster with default configuration using: +The bootstrap command initialises your cluster and configures your host system +as a Kubernetes node. If you would like to bootstrap a Kubernetes cluster with +default configuration run: ``` sudo k8s bootstrap ``` -This command initialises your cluster and configures your host system -as a Kubernetes node. For custom configurations, you can explore additional options using: ``` sudo k8s bootstrap --help ``` +Bootstrapping the cluster can only be done once. + ### 3. Check cluster status To confirm the installation was successful and your node is ready you @@ -44,6 +46,13 @@ should run: sudo k8s status ``` +It may take a few moments for the cluster to be ready. Confirm that {{product}} +has transitioned to the `cluster status ready` state by running: + +``` +sudo k8s status --wait-ready +``` + Run the following command to list all the pods in the `kube-system` namespace: @@ -51,19 +60,13 @@ namespace: sudo k8s kubectl get pods -n kube-system ``` -You will observe at least three pods running: +You will observe at least three pods running. The functions of these three pods +are: - **CoreDNS**: Provides DNS resolution services. - **Network operator**: Manages the life-cycle of the networking solution. - **Network agent**: Facilitates network management. -Confirm that {{product}} has transitioned to the `k8s is ready` state -by running: - -``` -sudo k8s status --wait-ready -``` - ### 5. Access Kubernetes The standard tool for deploying and managing workloads on Kubernetes @@ -94,9 +97,10 @@ Let's deploy a demo NGINX server: sudo k8s kubectl create deployment nginx --image=nginx ``` -This command launches a [pod](https://kubernetes.io/docs/concepts/workloads/pods/), -the smallest deployable unit in Kubernetes, -running the NGINX application within a container. +This command launches a +[pod](https://kubernetes.io/docs/concepts/workloads/pods/), the smallest +deployable unit in Kubernetes, running the NGINX application within a +container. You can check the status of your pods by running: @@ -123,7 +127,7 @@ running: sudo k8s kubectl get pods ``` -### 8. Enable Local Storage +### 8. Enable local storage In scenarios where you need to preserve application data beyond the life-cycle of the pod, Kubernetes provides persistent volumes. @@ -165,7 +169,7 @@ You can inspect the storage-writer-pod with: sudo k8s kubectl describe pod storage-writer-pod ``` -### 9. Disable Local Storage +### 9. Disable local storage Begin by removing the pod along with the persistent volume claim: @@ -200,20 +204,20 @@ sudo snap remove k8s --purge This option ensures complete removal of the snap and its associated data. -## Next Steps +## Next steps -- Keep mastering {{product}} with kubectl: [How to use kubectl] +- Learn more about {{product}} with kubectl: [How to use kubectl] - Explore Kubernetes commands with our [Command Reference Guide] -- Learn how to set up a multi-node environment [Setting up a K8s cluster] -- Configure storage options [Storage] +- Learn how to set up a multi-node environment by [Adding and Removing Nodes] +- Configure storage options: [Storage] - Master Kubernetes networking concepts: [Networking] -- Discover how to enable and configure Ingress resources [Ingress] +- Discover how to enable and configure Ingress resources: [Ingress] [How to use kubectl]: kubectl -[Command Reference Guide]: /snap/reference/commands -[Setting up a K8s cluster]: add-remove-nodes -[Storage]: /snap/howto/storage -[Networking]: /snap/howto/networking/index.md -[Ingress]: /snap/howto/networking/default-ingress.md \ No newline at end of file +[Command Reference Guide]: ../reference/commands +[Adding and Removing Nodes]: add-remove-nodes +[Storage]: ../howto/storage/index +[Networking]: ../howto/networking/index.md +[Ingress]: ../howto/networking/default-ingress.md \ No newline at end of file diff --git a/docs/src/snap/tutorial/index.md b/docs/src/snap/tutorial/index.md index 237c89aed..b124db24b 100644 --- a/docs/src/snap/tutorial/index.md +++ b/docs/src/snap/tutorial/index.md @@ -33,6 +33,6 @@ Finally, our [Reference section] is for when you need to check specific details or information such as the command reference or release notes. -[How-to guides]: /snap/howto/index -[Explanation section]: /snap/explanation/index -[Reference section]: /snap/reference/index +[How-to guides]: ../howto/index +[Explanation section]: ../explanation/index +[Reference section]: ../reference/index diff --git a/docs/src/snap/tutorial/kubectl.md b/docs/src/snap/tutorial/kubectl.md index 2f9fed807..5bdf04ef9 100644 --- a/docs/src/snap/tutorial/kubectl.md +++ b/docs/src/snap/tutorial/kubectl.md @@ -13,7 +13,7 @@ Before you begin, make sure you have the following: [Getting Started]) - You are using the built-in `kubectl` command from the snap. -### 1. The Kubectl Command +### 1. The kubectl command The `kubectl` command communicates with the [Kubernetes API server][kubernetes-api-server]. @@ -21,17 +21,25 @@ The `kubectl` command communicates with the The `kubectl` command included with {{product}} is built from the original upstream source into the `k8s` snap you have installed. -### 2. How To Use Kubectl +### 2. How to use kubectl -To access `kubectl`, run the following command: +To access `kubectl`, run the following: ``` -sudo k8s kubectl +sudo k8s kubectl ``` +This will display a list of commands possible with `kubectl`. + > **Note**: Only control plane nodes can use the `kubectl` command. Worker > nodes do not have access to this command. +The format of `kubectl` commands are: + +``` +sudo k8s kubectl +``` + ### 3. Configuration In {{product}}, the `kubeconfig` file that is being read to display @@ -49,7 +57,7 @@ Let's review what was created in the [Getting Started] guide. To see what pods were created when we enabled the `network` and `dns` -components: +components during the cluster bootstrap: ``` sudo k8s kubectl get pods -o wide -n kube-system @@ -68,7 +76,7 @@ The `kubernetes` service in the `default` namespace is where the Kubernetes API server resides, and it's the endpoint with which other nodes in your cluster will communicate. -### 5. Creating and Managing Objects +### 5. Creating and managing objects Let's deploy an NGINX server using this command: @@ -123,7 +131,7 @@ pods will have a status of `ContainerCreating`. -[Command Reference Guide]: /snap/reference/commands +[Command Reference Guide]: ../reference/commands [Getting Started]: getting-started [kubernetes-api-server]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ [kubeconfig-doc]: https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/ diff --git a/docs/tools/.custom_wordlist.txt b/docs/tools/.custom_wordlist.txt index e69de29bb..40495c3f4 100644 --- a/docs/tools/.custom_wordlist.txt +++ b/docs/tools/.custom_wordlist.txt @@ -0,0 +1,266 @@ +adapter's +adapters +allocatable +allocator +AlwaysPullImages +api +apiserver +apparmor +AppArmor +args +ARP +asn +ASN +autostart +autosuspend +aws +backend +backported +balancers +benoitblanchon +bgp +BGP +bootloader +CABPCK +CACPCK +capi +CAPI +CAs +Center +ceph +Ceph +cephcsi +cephx +cgroup +cgroups +cidr +CIDR +cidrs +CIDRs +CK8sControlPlane +CLI +CLIs +CloudFormation +ClusterAPI +clusterctl +ClusterRole +ClusterRoleBinding +CMK +CNI +Commenter +config +configMap +ConfigMap +containerd +CoreDNS +Corosync +CPUs +cpuset +crt +csi +CSI +CSRs +cyclictest +daemonset +DaemonSet +datastore +datastores +dbus +de +deallocation +deployable +discoverable +DMA +dns +DNS +DPDK +DRBD +drv +dqlite +EAL +EasyRSA +enp +enum +etcd +EventRateLimit +failover +gapped +GCP +ghcr +Gi +github +GPLv +Graber +Graber's +grafana +haircommander +Harbor +hostname +hostpath +HPC +html +http +https +HugePage +HugePages +iavf +init +initialise +integrations +io +IOMMU +IOV +ip +IPv +IPv4 +IPv6 +IRQs +Jinja +jitter +juju +Juju's +KMS +kube +kube-apiserver +kube-controller-manager +kube-proxy +kube-scheduler +kube-system +kubeconfig +kubectl +kubelet +kubepods +kubernetes +latencies +Latencies +libcontainer +lifecycle +linux +Lite's +LoadBalancer +localhost +Lookaside +lookups +loopback +LPM +lxc +LxcSecurity +LXD +MAAS +macOS +Maskable +MCE +MetalLB +Microbot +MicroCluster +MicroK +MicroK8s +MinIO +modprobe +Moonray +mq +mtu +MTU +multicast +MULTICAST +Multipass +Multus +nameservers +Netplan +NetworkAttachmentDefinition +NFD +NFV +nginx +NGINX +NIC +NMI +nodeport +nohz +NUMA +numactl +OCI +OOM +OpenStack +OSDs +ParseDuration +passthrough +passwordless +pci +PEM +performant +PID +PMD +PMDs +PPA +proc +programmatically +provisioner +PRs +PV +qdisc +qlen +QoS +RADOS +rbac +RBAC +RBD +rc +RCU +README +regctl +regsync +roadmap +Rockcraft +rollout +runtimes +rw +sandboxed +SANs +scalable +SCHED +sControlPlane +sd +SELinux +ServiceAccount +Snapcraft +snapd +SR-IOV +stackexchange +stgraber +STONITH +StorageClass +sudo +sys +systemd +taskset +Telco +throughs +tickless +TLB +tls +TLS +toml +TSC +TTL +ttyS +ubuntu +unix +unschedulable +unsquashed +Velero +vf +VF +vfio +VFIO +VFs +virtualised +VLAN +VMs +VMware +VNFs +VPCs +VSphere +WIP +www +yaml +YAMLs diff --git a/docs/tools/.sphinx/_static/404.svg b/docs/tools/.sphinx/_static/404.svg new file mode 100644 index 000000000..b353cd339 --- /dev/null +++ b/docs/tools/.sphinx/_static/404.svg @@ -0,0 +1,13 @@ + + + + + + + + + diff --git a/docs/tools/.sphinx/_static/custom.css b/docs/tools/.sphinx/_static/custom.css index 443412c5c..2b9e81fb1 100644 --- a/docs/tools/.sphinx/_static/custom.css +++ b/docs/tools/.sphinx/_static/custom.css @@ -84,8 +84,8 @@ Based on: https://github.com/canonical/vanilla-framework/blob/main/scss/_base_typography-definitions.scss regular text: 400, - bold: 550, - thin: 300, + bold: 550, + thin: 300, h1: bold, h2: 180; @@ -168,34 +168,63 @@ a.headerlink { border-left: 2px solid var(--color-brand-primary); } -/** Some tweaks for issue #16 **/ +/** Some tweaks for Sphinx tabs **/ [role="tablist"] { border-bottom: 1px solid var(--color-sidebar-item-background--hover); } -.sphinx-tabs-tab[aria-selected="true"] { +.sphinx-tabs-tab[aria-selected="true"], .sd-tab-set>input:checked+label{ border: 0; border-bottom: 2px solid var(--color-brand-primary); - background-color: var(--color-sidebar-item-background--current); - font-weight:300; + font-weight: 400; + font-size: 1rem; + color: var(--color-brand-primary); +} + +body[data-theme="dark"] .sphinx-tabs-tab[aria-selected="true"] { + background: var(--color-background-primary); + border-bottom: 2px solid var(--color-brand-primary); +} + +button.sphinx-tabs-tab[aria-selected="false"]:hover, .sd-tab-set>input:not(:checked)+label:hover { + border-bottom: 2px solid var(--color-foreground-border); +} + +button.sphinx-tabs-tab[aria-selected="false"]{ + border-bottom: 2px solid var(--color-background-primary); } -.sphinx-tabs-tab{ +body[data-theme="dark"] .sphinx-tabs-tab { + background: var(--color-background-primary); +} + +.sphinx-tabs-tab, .sd-tab-set>label{ color: var(--color-brand-primary); - font-weight:300; + font-family: var(--font-stack); + font-weight: 400; + font-size: 1rem; + padding: 1em 1.25em .5em } .sphinx-tabs-panel { border: 0; border-bottom: 1px solid var(--color-sidebar-item-background--hover); background: var(--color-background-primary); + padding: 0.75rem 0 0.75rem 0; } -button.sphinx-tabs-tab:hover { - background-color: var(--color-sidebar-item-background--hover); +body[data-theme="dark"] .sphinx-tabs-panel { + background: var(--color-background-primary); +} + +/** A tweak for issue #190 **/ + +.highlight .hll { + background-color: var(--color-highlighted-background); } + /** Custom classes to fix scrolling in tables by decreasing the font size or breaking certain columns. Specify the classes in the Markdown file with, for example: @@ -268,6 +297,15 @@ button.version_select { .available_versions a:hover {background-color: var(--color-sidebar-item-background--current)} +/** Suppress link underlines outside on-hover **/ +a { + text-decoration: none; +} + +a:hover, a:visited:hover { + text-decoration: underline; +} + .show {display:block;} /** Fix for nested numbered list - the nested list is lettered **/ @@ -319,3 +357,36 @@ details summary { .sidebar-search-container input[type=submit]:hover { text-decoration: underline; } + +/* Make inline code the same size as code blocks */ +p code.literal { + border: 0; + font-size: var(--code-font-size); +} + +/* Use the general admonition font size for inline code */ +.admonition p code.literal { + font-size: var(--admonition-font-size); +} + +.highlight .s, .highlight .s1, .highlight .s2 { + color: #3F8100; +} + +.highlight .o { + color: #BB5400; +} + +.rubric > .hclass2 { + display: block; + font-size: 2em; + border-radius: .5rem; + font-weight: 300; + line-height: 1.25; + margin-top: 1.75rem; + margin-right: -0.5rem; + margin-bottom: 0.5rem; + margin-left: -0.5rem; + padding-left: .5rem; + padding-right: .5rem; +} \ No newline at end of file diff --git a/docs/tools/.sphinx/_static/footer.css b/docs/tools/.sphinx/_static/footer.css new file mode 100644 index 000000000..a0a1db454 --- /dev/null +++ b/docs/tools/.sphinx/_static/footer.css @@ -0,0 +1,47 @@ +.display-contributors { + color: var(--color-sidebar-link-text); + cursor: pointer; +} +.all-contributors { + display: none; + z-index: 55; + list-style: none; + position: fixed; + top: 0; + bottom: 0; + left: 0; + right: 0; + width: 200px; + height: 200px; + overflow-y: scroll; + margin: auto; + padding: 0; + background: var(--color-background-primary); + scrollbar-color: var(--color-foreground-border) transparent; + scrollbar-width: thin; +} + +.all-contributors li:hover { + background: var(--color-sidebar-item-background--hover); + width: 100%; +} + +.all-contributors li a{ + color: var(--color-sidebar-link-text); + padding: 1rem; + display: inline-block; +} + +#overlay { + position: fixed; + display: none; + width: 100%; + height: 100%; + top: 0; + left: 0; + right: 0; + bottom: 0; + background-color: rgba(0,0,0,0.5); + z-index: 2; + cursor: pointer; +} diff --git a/docs/tools/.sphinx/_static/footer.js b/docs/tools/.sphinx/_static/footer.js new file mode 100644 index 000000000..9a08b1e99 --- /dev/null +++ b/docs/tools/.sphinx/_static/footer.js @@ -0,0 +1,12 @@ +$(document).ready(function() { + $(document).on("click", function () { + $(".all-contributors").hide(); + $("#overlay").hide(); + }); + + $('.display-contributors').click(function(event) { + $('.all-contributors').toggle(); + $("#overlay").toggle(); + event.stopPropagation(); + }); +}) diff --git a/docs/tools/.sphinx/_static/furo_colors.css b/docs/tools/.sphinx/_static/furo_colors.css index 422c777d9..4cfdbe7bf 100644 --- a/docs/tools/.sphinx/_static/furo_colors.css +++ b/docs/tools/.sphinx/_static/furo_colors.css @@ -11,7 +11,7 @@ body { --color-background-hover: #f2f2f2; --color-brand-primary: #111; --color-brand-content: #06C; - --color-api-background: #cdcdcd; + --color-api-background: #E3E3E3; --color-inline-code-background: rgba(0,0,0,.03); --color-sidebar-link-text: #111; --color-sidebar-item-background--current: #ebebeb; @@ -25,9 +25,11 @@ body { --color-admonition-title--tip: #24598F; --color-admonition-title--important: #C7162B; --color-admonition-title--caution: #F99B11; - --color-highlighted-background: #EbEbEb; - --color-link-underline: var(--color-background-primary); - --color-link-underline--hover: var(--color-background-primary); + --color-highlighted-background: #EBEBEB; + --color-link-underline: var(--color-link); + --color-link-underline--hover: var(--color-link); + --color-link-underline--visited: var(--color-link--visited); + --color-link-underline--visited--hover: var(--color-link--visited); --color-version-popup: #772953; } @@ -40,7 +42,7 @@ body { --color-background-secondary: var(--color-background-primary); --color-background-hover: #666; --color-brand-primary: #fff; - --color-brand-content: #06C; + --color-brand-content: #69C; --color-sidebar-link-text: #f7f7f7; --color-sidebar-item-background--current: #666; --color-sidebar-item-background--hover: #333; @@ -54,12 +56,11 @@ body { --color-admonition-title--important: #C7162B; --color-admonition-title--caution: #F99B11; --color-highlighted-background: #666; - --color-link-underline: var(--color-background-primary); - --color-link-underline--hover: var(--color-background-primary); --color-version-popup: #F29879; } @media (prefers-color-scheme: dark) { body:not([data-theme="light"]) { + --color-api-background: #A4A4A4; --color-code-background: #202020; --color-code-foreground: #d0d0d0; --color-foreground-secondary: var(--color-foreground-primary); @@ -67,7 +68,7 @@ body { --color-background-secondary: var(--color-background-primary); --color-background-hover: #666; --color-brand-primary: #fff; - --color-brand-content: #06C; + --color-brand-content: #69C; --color-sidebar-link-text: #f7f7f7; --color-sidebar-item-background--current: #666; --color-sidebar-item-background--hover: #333; @@ -81,8 +82,7 @@ body { --color-admonition-title--important: #C7162B; --color-admonition-title--caution: #F99B11; --color-highlighted-background: #666; - --color-link-underline: var(--color-background-primary); - --color-link-underline--hover: var(--color-background-primary); + --color-link: #F9FCFF; --color-version-popup: #F29879; } } diff --git a/docs/tools/.sphinx/_static/github_issue_links.js b/docs/tools/.sphinx/_static/github_issue_links.js index 7963d9554..f0706038b 100644 --- a/docs/tools/.sphinx/_static/github_issue_links.js +++ b/docs/tools/.sphinx/_static/github_issue_links.js @@ -14,8 +14,7 @@ window.onload = function() { link.href = ( github_url + "/issues/new?" - + "title=Docs:+ADD+YOUR+TITLE" - + "&labels=documentation" + + "title=docs%3A+TYPE+YOUR+QUESTION+HERE" + "&body=*Please describe the question or issue you're facing with " + `"${document.title}"` + ".*" diff --git a/docs/tools/.sphinx/_templates/404.html b/docs/tools/.sphinx/_templates/404.html new file mode 100644 index 000000000..4cb2d50d3 --- /dev/null +++ b/docs/tools/.sphinx/_templates/404.html @@ -0,0 +1,17 @@ +{% extends "page.html" %} + +{% block content -%} +
+

Page not found

+
+
+
+ {{ body }} +
+
+ Penguin with a question mark +
+
+
+
+{%- endblock content %} diff --git a/docs/tools/.sphinx/_templates/footer.html b/docs/tools/.sphinx/_templates/footer.html index 403c482ea..6839f0154 100644 --- a/docs/tools/.sphinx/_templates/footer.html +++ b/docs/tools/.sphinx/_templates/footer.html @@ -70,6 +70,26 @@ {%- endif %} +
{# mod: replaced RTD icons with our links #} @@ -80,6 +100,12 @@
{% endif %} + {% if mattermost %} +
+ Ask a question on Mattermost +
+ {% endif %} + {% if matrix %}
Ask a question on Matrix @@ -95,7 +121,7 @@ {% endif %}
- Edit this page on GitHub + Edit this page on GitHub
{% endif %} diff --git a/docs/tools/.sphinx/build_requirements.py b/docs/tools/.sphinx/build_requirements.py new file mode 100644 index 000000000..df6f149b4 --- /dev/null +++ b/docs/tools/.sphinx/build_requirements.py @@ -0,0 +1,127 @@ +import sys + +sys.path.append('./') +from custom_conf import * + +# The file contains helper functions and the mechanism to build the +# .sphinx/requirements.txt file that is needed to set up the virtual +# environment. + +# You should not do any modifications to this file. Put your custom +# requirements into the custom_required_modules array in the custom_conf.py +# file. If you need to change this file, contribute the changes upstream. + +legacyCanonicalSphinxExtensionNames = [ + "youtube-links", + "related-links", + "custom-rst-roles", + "terminal-output" + ] + +def IsAnyCanonicalSphinxExtensionUsed(): + for extension in custom_extensions: + if (extension.startswith("canonical.") or + extension in legacyCanonicalSphinxExtensionNames): + return True + + return False + +def IsNotFoundExtensionUsed(): + return "notfound.extension" in custom_extensions + +def IsSphinxTabsUsed(): + for extension in custom_extensions: + if extension.startswith("sphinx_tabs."): + return True + + return False + +def AreRedirectsDefined(): + return ("sphinx_reredirects" in custom_extensions) or ( + ("redirects" in globals()) and \ + (redirects is not None) and \ + (len(redirects) > 0)) + +def IsOpenGraphConfigured(): + if "sphinxext.opengraph" in custom_extensions: + return True + + for global_variable_name in list(globals()): + if global_variable_name.startswith("ogp_"): + return True + + return False + +def IsMyStParserUsed(): + return ("myst_parser" in custom_extensions) or \ + ("custom_myst_extensions" in globals()) + +def DeduplicateExtensions(extensionNames: [str]): + extensionNames = dict.fromkeys(extensionNames) + resultList = [] + encounteredCanonicalExtensions = [] + + for extensionName in extensionNames: + if extensionName in legacyCanonicalSphinxExtensionNames: + extensionName = "canonical." + extensionName + + if extensionName.startswith("canonical."): + if extensionName not in encounteredCanonicalExtensions: + encounteredCanonicalExtensions.append(extensionName) + resultList.append(extensionName) + else: + resultList.append(extensionName) + + return resultList + +if __name__ == "__main__": + requirements = [ + "furo", + "pyspelling", + "sphinx", + "sphinx-autobuild", + "sphinx-copybutton", + "sphinx-design", + "sphinxcontrib-jquery", + "watchfiles", + "GitPython" + + ] + + requirements.extend(custom_required_modules) + + if IsAnyCanonicalSphinxExtensionUsed(): + requirements.append("canonical-sphinx-extensions") + + if IsNotFoundExtensionUsed(): + requirements.append("sphinx-notfound-page") + + if IsSphinxTabsUsed(): + requirements.append("sphinx-tabs") + + if AreRedirectsDefined(): + requirements.append("sphinx-reredirects") + + if IsOpenGraphConfigured(): + requirements.append("sphinxext-opengraph") + + if IsMyStParserUsed(): + requirements.append("myst-parser") + requirements.append("linkify-it-py") + + # removes duplicate entries + requirements = list(dict.fromkeys(requirements)) + requirements.sort() + + with open(".sphinx/requirements.txt", 'w') as requirements_file: + requirements_file.write( + "# DO NOT MODIFY THIS FILE DIRECTLY!\n" + "#\n" + "# This file is generated automatically.\n" + "# Add custom requirements to the custom_required_modules\n" + "# array in the custom_conf.py file and run:\n" + "# make clean && make install\n") + + for requirement in requirements: + requirements_file.write(requirement) + requirements_file.write('\n') diff --git a/docs/tools/.sphinx/fonts/Ubuntu-B.ttf b/docs/tools/.sphinx/fonts/Ubuntu-B.ttf new file mode 100644 index 000000000..b173da274 Binary files /dev/null and b/docs/tools/.sphinx/fonts/Ubuntu-B.ttf differ diff --git a/docs/tools/.sphinx/fonts/Ubuntu-R.ttf b/docs/tools/.sphinx/fonts/Ubuntu-R.ttf new file mode 100644 index 000000000..d748728a2 Binary files /dev/null and b/docs/tools/.sphinx/fonts/Ubuntu-R.ttf differ diff --git a/docs/tools/.sphinx/fonts/Ubuntu-RI.ttf b/docs/tools/.sphinx/fonts/Ubuntu-RI.ttf new file mode 100644 index 000000000..4f2d2bc7c Binary files /dev/null and b/docs/tools/.sphinx/fonts/Ubuntu-RI.ttf differ diff --git a/docs/tools/.sphinx/fonts/UbuntuMono-B.ttf b/docs/tools/.sphinx/fonts/UbuntuMono-B.ttf new file mode 100644 index 000000000..7bd666576 Binary files /dev/null and b/docs/tools/.sphinx/fonts/UbuntuMono-B.ttf differ diff --git a/docs/tools/.sphinx/fonts/UbuntuMono-R.ttf b/docs/tools/.sphinx/fonts/UbuntuMono-R.ttf new file mode 100644 index 000000000..fdd309d71 Binary files /dev/null and b/docs/tools/.sphinx/fonts/UbuntuMono-R.ttf differ diff --git a/docs/tools/.sphinx/fonts/UbuntuMono-RI.ttf b/docs/tools/.sphinx/fonts/UbuntuMono-RI.ttf new file mode 100644 index 000000000..18f81a292 Binary files /dev/null and b/docs/tools/.sphinx/fonts/UbuntuMono-RI.ttf differ diff --git a/docs/tools/.sphinx/fonts/ubuntu-font-licence-1.0.txt b/docs/tools/.sphinx/fonts/ubuntu-font-licence-1.0.txt new file mode 100644 index 000000000..ae78a8f94 --- /dev/null +++ b/docs/tools/.sphinx/fonts/ubuntu-font-licence-1.0.txt @@ -0,0 +1,96 @@ +------------------------------- +UBUNTU FONT LICENCE Version 1.0 +------------------------------- + +PREAMBLE +This licence allows the licensed fonts to be used, studied, modified and +redistributed freely. The fonts, including any derivative works, can be +bundled, embedded, and redistributed provided the terms of this licence +are met. The fonts and derivatives, however, cannot be released under +any other licence. The requirement for fonts to remain under this +licence does not require any document created using the fonts or their +derivatives to be published under this licence, as long as the primary +purpose of the document is not to be a vehicle for the distribution of +the fonts. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this licence and clearly marked as such. This may +include source files, build scripts and documentation. + +"Original Version" refers to the collection of Font Software components +as received under this licence. + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to +a new environment. + +"Copyright Holder(s)" refers to all individuals and companies who have a +copyright ownership of the Font Software. + +"Substantially Changed" refers to Modified Versions which can be easily +identified as dissimilar to the Font Software by users of the Font +Software comparing the Original Version with the Modified Version. + +To "Propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification and with or without charging +a redistribution fee), making available to the public, and in some +countries other activities as well. + +PERMISSION & CONDITIONS +This licence does not grant any rights under trademark law and all such +rights are reserved. + +Permission is hereby granted, free of charge, to any person obtaining a +copy of the Font Software, to propagate the Font Software, subject to +the below conditions: + +1) Each copy of the Font Software must contain the above copyright +notice and this licence. These can be included either as stand-alone +text files, human-readable headers or in the appropriate machine- +readable metadata fields within text or binary files as long as those +fields can be easily viewed by the user. + +2) The font name complies with the following: +(a) The Original Version must retain its name, unmodified. +(b) Modified Versions which are Substantially Changed must be renamed to +avoid use of the name of the Original Version or similar names entirely. +(c) Modified Versions which are not Substantially Changed must be +renamed to both (i) retain the name of the Original Version and (ii) add +additional naming elements to distinguish the Modified Version from the +Original Version. The name of such Modified Versions must be the name of +the Original Version, with "derivative X" where X represents the name of +the new work, appended to that name. + +3) The name(s) of the Copyright Holder(s) and any contributor to the +Font Software shall not be used to promote, endorse or advertise any +Modified Version, except (i) as required by this licence, (ii) to +acknowledge the contribution(s) of the Copyright Holder(s) or (iii) with +their explicit written permission. + +4) The Font Software, modified or unmodified, in part or in whole, must +be distributed entirely under this licence, and must not be distributed +under any other licence. The requirement for fonts to remain under this +licence does not affect any document created using the Font Software, +except any version of the Font Software extracted from a document +created using the Font Software may only be distributed under this +licence. + +TERMINATION +This licence becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF +COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER +DEALINGS IN THE FONT SOFTWARE. diff --git a/docs/tools/.sphinx/get_vale_conf.py b/docs/tools/.sphinx/get_vale_conf.py new file mode 100644 index 000000000..23d890153 --- /dev/null +++ b/docs/tools/.sphinx/get_vale_conf.py @@ -0,0 +1,41 @@ +#! /usr/bin/env python + +import requests +import os + +DIR=os.getcwd() + +def main(): + + if os.path.exists(f"{DIR}/.sphinx/styles/Canonical"): + print("Vale directory exists") + else: + os.makedirs(f"{DIR}/.sphinx/styles/Canonical") + + url = "https://api.github.com/repos/canonical/praecepta/contents/styles/Canonical" + r = requests.get(url) + for item in r.json(): + download = requests.get(item["download_url"]) + file = open(".sphinx/styles/Canonical/" + item["name"], "w") + file.write(download.text) + file.close() + + if os.path.exists(f"{DIR}/.sphinx/styles/config/vocabularies/Canonical"): + print("Vocab directory exists") + else: + os.makedirs(f"{DIR}/.sphinx/styles/config/vocabularies/Canonical") + + url = "https://api.github.com/repos/canonical/praecepta/contents/styles/config/vocabularies/Canonical" + r = requests.get(url) + for item in r.json(): + download = requests.get(item["download_url"]) + file = open(".sphinx/styles/config/vocabularies/Canonical/" + item["name"], "w") + file.write(download.text) + file.close() + config = requests.get("https://raw.githubusercontent.com/canonical/praecepta/main/vale.ini") + file = open(".sphinx/vale.ini", "w") + file.write(config.text) + file.close() + +if __name__ == "__main__": + main() diff --git a/docs/tools/.sphinx/images/Canonical-logo-4x.png b/docs/tools/.sphinx/images/Canonical-logo-4x.png new file mode 100644 index 000000000..fd75696eb Binary files /dev/null and b/docs/tools/.sphinx/images/Canonical-logo-4x.png differ diff --git a/docs/tools/.sphinx/images/front-page-light.pdf b/docs/tools/.sphinx/images/front-page-light.pdf new file mode 100644 index 000000000..bb68cdf8f Binary files /dev/null and b/docs/tools/.sphinx/images/front-page-light.pdf differ diff --git a/docs/tools/.sphinx/images/front-page.png b/docs/tools/.sphinx/images/front-page.png new file mode 100644 index 000000000..c80e84303 Binary files /dev/null and b/docs/tools/.sphinx/images/front-page.png differ diff --git a/docs/tools/.sphinx/images/normal-page-footer.pdf b/docs/tools/.sphinx/images/normal-page-footer.pdf new file mode 100644 index 000000000..dfd73cbc7 Binary files /dev/null and b/docs/tools/.sphinx/images/normal-page-footer.pdf differ diff --git a/docs/tools/.sphinx/latex_elements_template.txt b/docs/tools/.sphinx/latex_elements_template.txt new file mode 100644 index 000000000..2b13b514a --- /dev/null +++ b/docs/tools/.sphinx/latex_elements_template.txt @@ -0,0 +1,119 @@ +{ + 'papersize': 'a4paper', + 'pointsize': '11pt', + 'fncychap': '', + 'preamble': r''' +%\usepackage{charter} +%\usepackage[defaultsans]{lato} +%\usepackage{inconsolata} +\setmainfont[UprightFont = *-R, BoldFont = *-B, ItalicFont=*-RI, Extension = .ttf]{Ubuntu} +\setmonofont[UprightFont = *-R, BoldFont = *-B, ItalicFont=*-RI, Extension = .ttf]{UbuntuMono} +\usepackage[most]{tcolorbox} +\tcbuselibrary{breakable} +\usepackage{lastpage} +\usepackage{tabto} +\usepackage{ifthen} +\usepackage{etoolbox} +\usepackage{fancyhdr} +\usepackage{graphicx} +\usepackage{titlesec} +\usepackage{fontspec} +\usepackage{tikz} +\usepackage{changepage} +\usepackage{array} +\usepackage{tabularx} +\definecolor{yellowgreen}{RGB}{154, 205, 50} +\definecolor{title}{RGB}{76, 17, 48} +\definecolor{subtitle}{RGB}{116, 27, 71} +\definecolor{label}{RGB}{119, 41, 100} +\definecolor{copyright}{RGB}{174, 167, 159} +\makeatletter +\def\tcb@finalize@environment{% + \color{.}% hack for xelatex + \tcb@layer@dec% +} +\makeatother +\newenvironment{sphinxclassprompt}{\color{yellowgreen}\setmonofont[Color = 9ACD32, UprightFont = *-R, Extension = .ttf]{UbuntuMono}}{} +\tcbset{enhanced jigsaw, colback=black, fontupper=\color{white}} +\newtcolorbox{termbox}{use color stack, breakable, colupper=white, halign=flush left} +\newenvironment{sphinxclassterminal}{\setmonofont[Color = white, UprightFont = *-R, Extension = .ttf]{UbuntuMono}\sphinxsetup{VerbatimColor={black}}\begin{termbox}}{\end{termbox}} +\newcommand{\dimtorightedge}{% + \dimexpr\paperwidth-1in-\hoffset-\oddsidemargin\relax} +\newcommand{\dimtotop}{% + \dimexpr\height-1in-\voffset-\topmargin-\headheight-\headsep\relax} +\newtoggle{tpage} +\AtBeginEnvironment{titlepage}{\global\toggletrue{tpage}} +\fancypagestyle{plain}{ + \fancyhf{} + \fancyfoot[R]{\thepage\ of \pageref*{LastPage}} + \renewcommand{\headrulewidth}{0pt} + \renewcommand{\footrulewidth}{0pt} +} +\fancypagestyle{normal}{ + \fancyhf{} + \fancyfoot[R]{\thepage\ of \pageref*{LastPage}} + \renewcommand{\headrulewidth}{0pt} + \renewcommand{\footrulewidth}{0pt} +} +\fancypagestyle{titlepage}{% + \fancyhf{} + \fancyfoot[L]{\footnotesize \textcolor{copyright}{© 2024 Canonical Ltd. All rights reserved.}} +} +\newcommand\sphinxbackoftitlepage{\thispagestyle{titlepage}} +\titleformat{\chapter}[block]{\Huge \color{title} \bfseries\filright}{\thechapter .}{1.5ex}{} +\titlespacing{\chapter}{0pt}{0pt}{0pt} +\titleformat{\section}[block]{\huge \bfseries\filright}{\thesection .}{1.5ex}{} +\titlespacing{\section}{0pt}{0pt}{0pt} +\titleformat{\subsection}[block]{\Large \bfseries\filright}{\thesubsection .}{1.5ex}{} +\titlespacing{\subsection}{0pt}{0pt}{0pt} +\setcounter{tocdepth}{1} +\renewcommand\pagenumbering[1]{} +''', + 'sphinxsetup': 'verbatimwithframe=false, pre_border-radius=0pt, verbatimvisiblespace=\\phantom{}, verbatimcontinued=\\phantom{}', + 'extraclassoptions': 'openany,oneside', + 'maketitle': r''' +\begin{titlepage} +\begin{flushleft} + \begin{tikzpicture}[remember picture,overlay] + \node[anchor=south east, inner sep=0] at (current page.south east) { + \includegraphics[width=\paperwidth, height=\paperheight]{front-page-light} + }; + \end{tikzpicture} +\end{flushleft} + +\vspace*{3cm} + +\begin{adjustwidth}{8cm}{0pt} +\begin{flushleft} + \huge \textcolor{black}{\textbf{}{\raggedright{$PROJECT}}} +\end{flushleft} +\end{adjustwidth} + +\vfill + +\begin{adjustwidth}{8cm}{0pt} +\begin{tabularx}{0.5\textwidth}{ l l } + \textcolor{lightgray}{© 2024 Canonical Ltd.} & \hspace{3cm} \\ + \textcolor{lightgray}{All rights reserved.} & \hspace{3cm} \\ + & \hspace{3cm} \\ + & \hspace{3cm} \\ + +\end{tabularx} +\end{adjustwidth} + +\end{titlepage} +\RemoveFromHook{shipout/background} +\AddToHook{shipout/background}{ + \begin{tikzpicture}[remember picture,overlay] + \node[anchor=south west, align=left, inner sep=0] at (current page.south west) { + \includegraphics[width=\paperwidth]{normal-page-footer} + }; + \end{tikzpicture} + \begin{tikzpicture}[remember picture,overlay] + \node[anchor=north east, opacity=0.5, inner sep=35] at (current page.north east) { + \includegraphics[width=4cm]{Canonical-logo-4x} + }; + \end{tikzpicture} + } +''', +} \ No newline at end of file diff --git a/docs/tools/.sphinx/pa11y.json b/docs/tools/.sphinx/pa11y.json new file mode 100644 index 000000000..8df0cb9cb --- /dev/null +++ b/docs/tools/.sphinx/pa11y.json @@ -0,0 +1,9 @@ +{ + "chromeLaunchConfig": { + "args": [ + "--no-sandbox" + ] + }, + "reporter": "cli", + "standard": "WCAG2AA" +} \ No newline at end of file diff --git a/docs/tools/.sphinx/requirements.txt b/docs/tools/.sphinx/requirements.txt index 977f93134..4ebefde3c 100644 --- a/docs/tools/.sphinx/requirements.txt +++ b/docs/tools/.sphinx/requirements.txt @@ -1,7 +1,13 @@ +# DO NOT MODIFY THIS FILE DIRECTLY! Unless you want to +# +# This file is generated automatically. +# Add custom requirements to the custom_required_modules +# array in the custom_conf.py file and run: +# make clean && make install +GitPython canonical-sphinx-extensions furo linkify-it-py -matplotlib myst-parser pyspelling sphinx @@ -9,9 +15,8 @@ sphinx-autobuild sphinx-copybutton sphinx-design sphinx-notfound-page -sphinx-reredirects sphinx-tabs sphinxcontrib-jquery +sphinxcontrib-svg2pdfconverter[CairoSVG] sphinxext-opengraph -sphinxcontrib-plantuml -sphinxcontrib-kroki +watchfiles \ No newline at end of file diff --git a/docs/tools/.sphinx/spellingcheck.yaml b/docs/tools/.sphinx/spellingcheck.yaml index fc9d3c503..b907c5d67 100644 --- a/docs/tools/.sphinx/spellingcheck.yaml +++ b/docs/tools/.sphinx/spellingcheck.yaml @@ -9,7 +9,7 @@ matrix: - .custom_wordlist.txt output: .sphinx/.wordlist.dic sources: - - _build/**/*.html + - ../_build/**/*.html pipeline: - pyspelling.filters.html: comments: false @@ -21,7 +21,6 @@ matrix: - pre - spellexception - link - - title - div.relatedlinks - strong.command - div.visually-hidden diff --git a/docs/tools/Makefile b/docs/tools/Makefile index dbbf1057b..72fe7c602 100644 --- a/docs/tools/Makefile +++ b/docs/tools/Makefile @@ -1,101 +1,31 @@ -# Minimal makefile for Sphinx documentation +# This Makefile stub allows you to customize starter pack (SP) targets. +# Consider this file as a bridge between your project +# and the starter pack's predefined targets that reside in Makefile.sp. # - -# You can set these variables from the command line, and also -# from the environment for the first two. -SPHINXOPTS ?= -c . -d .sphinx/.doctrees -SPHINXBUILD ?= sphinx-build -SPHINXDIR = .sphinx -SOURCEDIR = ../src/ -BUILDDIR = ../_build -VENVDIR = $(SPHINXDIR)/venv -PA11Y = $(SPHINXDIR)/node_modules/pa11y/bin/pa11y.js -VENV = $(VENVDIR)/bin/activate - -.PHONY: help woke-install pa11y-install install run html epub serve clean \ - clean-doc spelling linkcheck woke pa11y Makefile +# You can add your own, non-SP targets here or override SP targets +# to fit your project's needs. For example, you can define and use targets +# named "install" or "run", but continue to use SP targets like "sp-install" +# or "sp-run" when working on the documentation. # Put it first so that "make" without argument is like "make help". -help: $(VENVDIR) - @. $(VENV); $(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) - -# Explicit target avoids fall-through to the "Makefile" target. -$(SPHINXDIR)/requirements.txt: - test -f $(SPHINXDIR)/requirements.txt - -# If requirements are updated, venv should be rebuilt and timestamped. -$(VENVDIR): $(SPHINXDIR)/requirements.txt - @echo "... setting up virtualenv" - python3 -m venv $(VENVDIR) - . $(VENV); pip install --require-virtualenv \ - --upgrade -r $(SPHINXDIR)/requirements.txt \ - --log $(VENVDIR)/pip_install.log - @test ! -f $(VENVDIR)/pip_list.txt || \ - mv $(VENVDIR)/pip_list.txt $(VENVDIR)/pip_list.txt.bak - @. $(VENV); pip list --local --format=freeze > $(VENVDIR)/pip_list.txt +help: @echo "\n" \ - "--------------------------------------------------------------- \n" \ - "* watch, build and serve the documentation: make run \n" \ - "* only build: make html \n" \ - "* only serve: make serve \n" \ - "* clean built doc files: make clean-doc \n" \ - "* clean full environment: make clean \n" \ - "* check links: make linkcheck \n" \ - "* check spelling: make spelling \n" \ - "* check inclusive language: make woke \n" \ - "* check accessibility: make pa11y \n" \ - "* other possible targets: make \n" \ - "--------------------------------------------------------------- \n" - @touch $(VENVDIR) - -woke-install: - @type woke >/dev/null 2>&1 || \ - { echo "Installing \"woke\" snap... \n"; sudo snap install woke; } - -pa11y-install: - @type $(PA11Y) >/dev/null 2>&1 || { \ - echo "Installing \"pa11y\" from npm... \n"; \ - mkdir -p $(SPHINXDIR)/node_modules/ ; \ - npm install --prefix $(SPHINXDIR) pa11y; \ - } - -install: $(VENVDIR) woke-install - -run: install - . $(VENV); sphinx-autobuild -b dirhtml "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) - -# Doesn't depend on $(BUILDDIR) to rebuild properly at every run. -html: install - . $(VENV); $(SPHINXBUILD) -b dirhtml "$(SOURCEDIR)" "$(BUILDDIR)" -w .sphinx/warnings.txt $(SPHINXOPTS) - -epub: install - . $(VENV); $(SPHINXBUILD) -b epub "$(SOURCEDIR)" "$(BUILDDIR)" -w .sphinx/warnings.txt $(SPHINXOPTS) - -serve: html - cd "$(BUILDDIR)"; python3 -m http.server 8000 - -clean: clean-doc - @test ! -e "$(VENVDIR)" -o -d "$(VENVDIR)" -a "$(abspath $(VENVDIR))" != "$(VENVDIR)" - rm -rf $(VENVDIR) - -clean-doc: - git clean -fx "$(BUILDDIR)" - rm -rf .sphinx/.doctrees - -spelling: html - . $(VENV) ; python3 -m pyspelling -c .sphinx/spellingcheck.yaml - -linkcheck: install - . $(VENV) ; $(SPHINXBUILD) -b linkcheck "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) - -woke: woke-install - woke *.rst **/*.rst --exit-1-on-failure \ - -c https://github.com/canonical/Inclusive-naming/raw/main/config.yml - -pa11y: pa11y-install html - find $(BUILDDIR) -name *.html -print0 | xargs -n 1 -0 $(PA11Y) - -# Catch-all target: route all unknown targets to Sphinx using the new -# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). -%: Makefile - . $(VENV); $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + "------------------------------------------------------------- \n" \ + "* watch, build and serve the documentation: make run \n" \ + "* only build: make html \n" \ + "* only serve: make serve \n" \ + "* clean built doc files: make clean-doc \n" \ + "* clean full environment: make clean \n" \ + "* check links: make linkcheck \n" \ + "* check spelling: make spelling \n" \ + "* check spelling (without building again): make spellcheck \n" \ + "* check inclusive language: make woke \n" \ + "* check accessibility: make pa11y \n" \ + "* check style guide compliance: make vale \n" \ + "* check style guide compliance on target: make vale TARGET=* \n" \ + "* check metrics for documentation: make allmetrics \n" \ + "* other possible targets: make \n" \ + "------------------------------------------------------------- \n" + +%: + $(MAKE) -f Makefile.sp sp-$@ diff --git a/docs/tools/Makefile.sp b/docs/tools/Makefile.sp new file mode 100644 index 000000000..0ad2c8b62 --- /dev/null +++ b/docs/tools/Makefile.sp @@ -0,0 +1,156 @@ +# Minimal makefile for Sphinx documentation +# +# `Makefile.sp` is from the Sphinx starter pack and should not be +# modified. +# Add your customisation to `Makefile` instead. + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXDIR = .sphinx +SPHINXOPTS ?= -c . -d $(SPHINXDIR)/.doctrees -j auto +SPHINXBUILD ?= sphinx-build +SOURCEDIR = ../src +METRICSDIR = $(SOURCEDIR)/metrics +BUILDDIR = ../_build +VENVDIR = $(SPHINXDIR)/venv +PA11Y = $(SPHINXDIR)/node_modules/pa11y/bin/pa11y.js --config $(SPHINXDIR)/pa11y.json +VENV = $(VENVDIR)/bin/activate +TARGET = * +ALLFILES = *.rst **/*.rst +ADDPREREQS ?= +REQPDFPACKS = latexmk fonts-freefont-otf texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended texlive-font-utils texlive-lang-cjk texlive-xetex plantuml xindy tex-gyre dvipng + +.PHONY: sp-full-help sp-woke-install sp-pa11y-install sp-install sp-run sp-html \ + sp-epub sp-serve sp-clean sp-clean-doc sp-spelling sp-spellcheck sp-linkcheck sp-woke \ + sp-allmetrics sp-pa11y sp-pdf-prep-force sp-pdf-prep sp-pdf Makefile.sp sp-vale sp-bash + +sp-full-help: $(VENVDIR) + @. $(VENV); $(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + @echo "\n\033[1;31mNOTE: This help texts shows unsupported targets!\033[0m" + @echo "Run 'make help' to see supported targets." + +# Shouldn't assume that venv is available on Ubuntu by default; discussion here: +# https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1290847 +$(SPHINXDIR)/requirements.txt: + @python3 -c "import venv" || \ + (echo "You must install python3-venv before you can build the documentation."; exit 1) + python3 -m venv $(VENVDIR) + @if [ ! -z "$(ADDPREREQS)" ]; then \ + . $(VENV); pip install \ + $(PIPOPTS) --require-virtualenv $(ADDPREREQS); \ + fi + . $(VENV); python3 $(SPHINXDIR)/build_requirements.py + +# If requirements are updated, venv should be rebuilt and timestamped. +$(VENVDIR): $(SPHINXDIR)/requirements.txt + @echo "... setting up virtualenv" + python3 -m venv $(VENVDIR) + . $(VENV); pip install $(PIPOPTS) --require-virtualenv \ + --upgrade -r $(SPHINXDIR)/requirements.txt \ + --log $(VENVDIR)/pip_install.log + @test ! -f $(VENVDIR)/pip_list.txt || \ + mv $(VENVDIR)/pip_list.txt $(VENVDIR)/pip_list.txt.bak + @. $(VENV); pip list --local --format=freeze > $(VENVDIR)/pip_list.txt + @touch $(VENVDIR) + +sp-woke-install: + @type woke >/dev/null 2>&1 || \ + { echo "Installing \"woke\" snap... \n"; sudo snap install woke; } + +sp-pa11y-install: + @type $(PA11Y) >/dev/null 2>&1 || { \ + echo "Installing \"pa11y\" from npm... \n"; \ + mkdir -p $(SPHINXDIR)/node_modules/ ; \ + npm install --prefix $(SPHINXDIR) pa11y; \ + } + +sp-install: $(VENVDIR) + +sp-run: sp-install + . $(VENV); sphinx-autobuild -b dirhtml "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) + +# Doesn't depend on $(BUILDDIR) to rebuild properly at every run. +sp-html: sp-install + . $(VENV); $(SPHINXBUILD) -W --keep-going -b dirhtml "$(SOURCEDIR)" "$(BUILDDIR)" -w $(SPHINXDIR)/warnings.txt $(SPHINXOPTS) + +sp-epub: sp-install + . $(VENV); $(SPHINXBUILD) -b epub "$(SOURCEDIR)" "$(BUILDDIR)" -w $(SPHINXDIR)/warnings.txt $(SPHINXOPTS) + +sp-serve: sp-html + cd "$(BUILDDIR)"; python3 -m http.server --bind 127.0.0.1 8000 + +sp-clean: sp-clean-doc + @test ! -e "$(VENVDIR)" -o -d "$(VENVDIR)" -a "$(abspath $(VENVDIR))" != "$(VENVDIR)" + rm -rf $(VENVDIR) + rm -f $(SPHINXDIR)/requirements.txt + rm -rf $(SPHINXDIR)/node_modules/ + rm -rf $(SPHINXDIR)/styles + rm -rf $(SPHINXDIR)/vale.ini + +sp-clean-doc: + git clean -fx "$(BUILDDIR)" + rm -rf $(SPHINXDIR)/.doctrees + +sp-spellcheck: + . $(VENV) ; python3 -m pyspelling -c $(SPHINXDIR)/spellingcheck.yaml -j $(shell nproc) + +sp-spelling: sp-html sp-spellcheck + +sp-linkcheck: sp-install + . $(VENV) ; $(SPHINXBUILD) -b linkcheck "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) || { grep --color -F "[broken]" "$(BUILDDIR)/output.txt"; exit 1; } + exit 0 + +sp-woke: sp-woke-install + woke $(ALLFILES) --exit-1-on-failure \ + -c https://github.com/canonical/Inclusive-naming/raw/main/config.yml + +sp-pa11y: sp-pa11y-install sp-html + find $(BUILDDIR) -name *.html -print0 | xargs -n 1 -0 $(PA11Y) + +sp-vale: sp-install + @. $(VENV); test -d $(SPHINXDIR)/venv/lib/python*/site-packages/vale || pip install vale + @. $(VENV); test -f $(SPHINXDIR)/vale.ini || python3 $(SPHINXDIR)/get_vale_conf.py + @. $(VENV); find $(SPHINXDIR)/venv/lib/python*/site-packages/vale/vale_bin -size 195c -exec vale --config "$(SPHINXDIR)/vale.ini" $(TARGET) > /dev/null \; + @cat $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept.txt > $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept_backup.txt + @cat $(SOURCEDIR)/.wordlist.txt $(SOURCEDIR)/.custom_wordlist.txt >> $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept.txt + @echo "" + @echo "Running Vale against $(TARGET). To change target set TARGET= with make command" + @echo "" + @. $(VENV); vale --config "$(SPHINXDIR)/vale.ini" --glob='*.{md,txt,rst}' $(TARGET) || true + @cat $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept_backup.txt > $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept.txt && rm $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept_backup.txt + +sp-pdf-prep: sp-install + @for packageName in $(REQPDFPACKS); do (dpkg-query -W -f='$${Status}' $$packageName 2>/dev/null | \ + grep -c "ok installed" >/dev/null && echo "Package $$packageName is installed") && continue || \ + (echo "\nPDF generation requires the installation of the following packages: $(REQPDFPACKS)" && \ + echo "" && echo "Run sudo make pdf-prep-force to install these packages" && echo "" && echo \ + "Please be aware these packages will be installed to your system") && exit 1 ; done + +sp-pdf-prep-force: + apt-get update + apt-get upgrade -y + apt-get install --no-install-recommends -y $(REQPDFPACKS) \ + +sp-pdf: sp-pdf-prep + @. $(VENV); sphinx-build -M latexpdf "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) + @rm ./$(BUILDDIR)/latex/front-page-light.pdf || true + @rm ./$(BUILDDIR)/latex/normal-page-footer.pdf || true + @find ./$(BUILDDIR)/latex -name "*.pdf" -exec mv -t ./$(BUILDDIR) {} + + @rm -r $(BUILDDIR)/latex + @echo "\nOutput can be found in ./$(BUILDDIR)\n" + +sp-allmetrics: sp-html + @echo "Recording documentation metrics..." + @echo "Checking for existence of vale..." + . $(VENV) + @. $(VENV); test -d $(SPHINXDIR)/venv/lib/python*/site-packages/vale || pip install vale + @. $(VENV); test -f $(SPHINXDIR)/vale.ini || python3 $(SPHINXDIR)/get_vale_conf.py + @. $(VENV); find $(SPHINXDIR)/venv/lib/python*/site-packages/vale/vale_bin -size 195c -exec vale --config "$(SPHINXDIR)/vale.ini" $(TARGET) > /dev/null \; + @eval '$(METRICSDIR)/scripts/source_metrics.sh $(PWD)' + @eval '$(METRICSDIR)/scripts/build_metrics.sh $(PWD) $(METRICSDIR)' + + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile.sp + . $(VENV); $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/tools/conf.py b/docs/tools/conf.py index f4346485e..bd99aa2d7 100644 --- a/docs/tools/conf.py +++ b/docs/tools/conf.py @@ -1,9 +1,16 @@ import sys import os +import requests +from urllib.parse import urlparse +from git import Repo, InvalidGitRepositoryError +import time +import ast import yaml sys.path.append('./') from custom_conf import * +sys.path.append('.sphinx/') +from build_requirements import * # Configuration file for the Sphinx documentation builder. # You should not do any modifications to this file. Put your custom @@ -19,42 +26,65 @@ extensions = [ 'sphinx_design', - 'sphinx_tabs.tabs', - 'sphinx_reredirects', - 'canonical.youtube-links', - 'canonical.related-links', - 'canonical.custom-rst-roles', - 'canonical.terminal-output', 'sphinx_copybutton', - 'sphinxext.opengraph', - 'myst_parser', 'sphinxcontrib.jquery', - 'notfound.extension' ] + +# Only add redirects extension if any redirects are specified. +if AreRedirectsDefined(): + extensions.append('sphinx_reredirects') + +# Only add myst extensions if any configuration is present. +if IsMyStParserUsed(): + extensions.append('myst_parser') + + # Additional MyST syntax + myst_enable_extensions = [ + 'substitution', + 'deflist', + 'linkify' + ] + myst_enable_extensions.extend(custom_myst_extensions) + +# Only add Open Graph extension if any configuration is present. +if IsOpenGraphConfigured(): + extensions.append('sphinxext.opengraph') + extensions.extend(custom_extensions) +extensions = DeduplicateExtensions(extensions) ### Configuration for extensions -# Additional MyST syntax -myst_enable_extensions = [ - 'substitution', - 'deflist', - 'linkify' -] -myst_enable_extensions.extend(custom_myst_extensions) - # Used for related links if not 'discourse_prefix' in html_context and 'discourse' in html_context: html_context['discourse_prefix'] = html_context['discourse'] + '/t/' -# The default for notfound_urls_prefix usually works, but not for -# documentation on documentation.ubuntu.com +# The URL prefix for the notfound extension depends on whether the documentation uses versions. +# For documentation on documentation.ubuntu.com, we also must add the slug. +url_version = '' +url_lang = '' + +# Determine if the URL uses versions and language +if 'READTHEDOCS_CANONICAL_URL' in os.environ and os.environ['READTHEDOCS_CANONICAL_URL']: + url_parts = os.environ['READTHEDOCS_CANONICAL_URL'].split('/') + + if len(url_parts) >= 2 and 'READTHEDOCS_VERSION' in os.environ and os.environ['READTHEDOCS_VERSION'] == url_parts[-2]: + url_version = url_parts[-2] + '/' + + if len(url_parts) >= 3 and 'READTHEDOCS_LANGUAGE' in os.environ and os.environ['READTHEDOCS_LANGUAGE'] == url_parts[-3]: + url_lang = url_parts[-3] + '/' + +# Set notfound_urls_prefix to the slug (if defined) and the version/language affix if slug: - notfound_urls_prefix = '/' + slug + '/en/latest/' + notfound_urls_prefix = '/' + slug + '/' + url_lang + url_version +elif len(url_lang + url_version) > 0: + notfound_urls_prefix = '/' + url_lang + url_version +else: + notfound_urls_prefix = '' notfound_context = { 'title': 'Page not found', - 'body': '

Page not found

\n\n

Sorry, but the documentation page that you are looking for was not found.

\n

Documentation changes over time, and pages are moved around. We try to redirect you to the updated content where possible, but unfortunately, that didn\'t work this time (maybe because the content you were looking for does not exist in this version of the documentation).

\n

You can try to use the navigation to locate the content you\'re looking for, or search for a similar page.

\n', + 'body': '

Sorry, but the documentation page that you are looking for was not found.

\n\n

Documentation changes over time, and pages are moved around. We try to redirect you to the updated content where possible, but unfortunately, that didn\'t work this time (maybe because the content you were looking for does not exist in this version of the documentation).

\n

You can try to use the navigation to locate the content you\'re looking for, or search for a similar page.

\n', } # Default image for OGP (to prevent font errors, see @@ -99,6 +129,10 @@ for tag in custom_tags: tags.add(tag) +# html_context['get_contribs'] is a function and cannot be +# cached (see https://github.com/sphinx-doc/sphinx/issues/12300) +suppress_warnings = ["config.cache"] + ############################################################ ### Styling ############################################################ @@ -111,6 +145,7 @@ # Setting templates_path for epub makes the build fail if builder == 'dirhtml' or builder == 'html': templates_path = ['.sphinx/_templates'] + notfound_template = '404.html' # Theme configuration html_theme = 'furo' @@ -132,15 +167,52 @@ 'custom.css', 'header.css', 'github_issue_links.css', - 'furo_colors.css' + 'furo_colors.css', + 'footer.css' ] html_css_files.extend(custom_html_css_files) -html_js_files = ['header-nav.js'] +html_js_files = ['header-nav.js', 'footer.js'] if 'github_issues' in html_context and html_context['github_issues'] and not disable_feedback_button: html_js_files.append('github_issue_links.js') html_js_files.extend(custom_html_js_files) +############################################################# +# Display the contributors + +def get_contributors_for_file(github_url, github_folder, pagename, page_source_suffix, display_contributors_since=None): + filename = f"{pagename}{page_source_suffix}" + paths=html_context['github_folder'][1:] + filename + + try: + repo = Repo(".") + except InvalidGitRepositoryError: + cwd = os.getcwd() + ghfolder = html_context['github_folder'][:-1] + if ghfolder and cwd.endswith(ghfolder): + repo = Repo(cwd.rpartition(ghfolder)[0]) + else: + print("The local Git repository could not be found.") + return + + since = display_contributors_since if display_contributors_since and display_contributors_since.strip() else None + + commits = repo.iter_commits(paths=paths, since=since) + + contributors_dict = {} + for commit in commits: + contributor = commit.author.name + if contributor not in contributors_dict or commit.committed_date > contributors_dict[contributor]['date']: + contributors_dict[contributor] = { + 'date': commit.committed_date, + 'sha': commit.hexsha + } + # The github_page contains the link to the contributor's latest commit. + contributors_list = [{'name': name, 'github_page': f"{github_url}/commit/{data['sha']}"} for name, data in contributors_dict.items()] + sorted_contributors_list = sorted(contributors_list, key=lambda x: x['name']) + return sorted_contributors_list + +html_context['get_contribs'] = get_contributors_for_file ############################################################ ### Myst configuration @@ -148,3 +220,30 @@ if os.path.exists('./reuse/substitutions.yaml'): with open('./reuse/substitutions.yaml', 'r') as fd: myst_substitutions = yaml.safe_load(fd.read()) + + + +############################################################ +### PDF configuration +############################################################ + +latex_additional_files = [ + "./.sphinx/fonts/Ubuntu-B.ttf", + "./.sphinx/fonts/Ubuntu-R.ttf", + "./.sphinx/fonts/Ubuntu-RI.ttf", + "./.sphinx/fonts/UbuntuMono-R.ttf", + "./.sphinx/fonts/UbuntuMono-RI.ttf", + "./.sphinx/fonts/UbuntuMono-B.ttf", + "./.sphinx/images/Canonical-logo-4x.png", + "./.sphinx/images/front-page-light.pdf", + "./.sphinx/images/normal-page-footer.pdf", +] + +latex_engine = 'xelatex' +latex_show_pagerefs = True +latex_show_urls = 'footnote' + +with open(".sphinx/latex_elements_template.txt", "rt") as file: + latex_config = file.read() + +latex_elements = ast.literal_eval(latex_config.replace("$PROJECT", project)) \ No newline at end of file diff --git a/docs/tools/conf.py-old b/docs/tools/conf.py-old new file mode 100644 index 000000000..f4346485e --- /dev/null +++ b/docs/tools/conf.py-old @@ -0,0 +1,150 @@ +import sys +import os +import yaml + +sys.path.append('./') +from custom_conf import * + +# Configuration file for the Sphinx documentation builder. +# You should not do any modifications to this file. Put your custom +# configuration into the custom_conf.py file. +# If you need to change this file, contribute the changes upstream. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +############################################################ +### Extensions +############################################################ + +extensions = [ + 'sphinx_design', + 'sphinx_tabs.tabs', + 'sphinx_reredirects', + 'canonical.youtube-links', + 'canonical.related-links', + 'canonical.custom-rst-roles', + 'canonical.terminal-output', + 'sphinx_copybutton', + 'sphinxext.opengraph', + 'myst_parser', + 'sphinxcontrib.jquery', + 'notfound.extension' +] +extensions.extend(custom_extensions) + +### Configuration for extensions + +# Additional MyST syntax +myst_enable_extensions = [ + 'substitution', + 'deflist', + 'linkify' +] +myst_enable_extensions.extend(custom_myst_extensions) + +# Used for related links +if not 'discourse_prefix' in html_context and 'discourse' in html_context: + html_context['discourse_prefix'] = html_context['discourse'] + '/t/' + +# The default for notfound_urls_prefix usually works, but not for +# documentation on documentation.ubuntu.com +if slug: + notfound_urls_prefix = '/' + slug + '/en/latest/' + +notfound_context = { + 'title': 'Page not found', + 'body': '

Page not found

\n\n

Sorry, but the documentation page that you are looking for was not found.

\n

Documentation changes over time, and pages are moved around. We try to redirect you to the updated content where possible, but unfortunately, that didn\'t work this time (maybe because the content you were looking for does not exist in this version of the documentation).

\n

You can try to use the navigation to locate the content you\'re looking for, or search for a similar page.

\n', +} + +# Default image for OGP (to prevent font errors, see +# https://github.com/canonical/sphinx-docs-starter-pack/pull/54 ) +if not 'ogp_image' in locals(): + ogp_image = 'https://assets.ubuntu.com/v1/253da317-image-document-ubuntudocs.svg' + +############################################################ +### General configuration +############################################################ + +exclude_patterns = [ + '_build', + 'Thumbs.db', + '.DS_Store', + '.sphinx', + '_parts' +] +exclude_patterns.extend(custom_excludes) + +rst_epilog = ''' +.. include:: /reuse/links.txt +''' +if 'custom_rst_epilog' in locals(): + rst_epilog = custom_rst_epilog + +source_suffix = { + '.rst': 'restructuredtext', + '.md': 'markdown', +} + +if not 'conf_py_path' in html_context and 'github_folder' in html_context: + html_context['conf_py_path'] = html_context['github_folder'] + +# For ignoring specific links +linkcheck_anchors_ignore_for_url = [ + r'https://github\.com/.*' +] +linkcheck_anchors_ignore_for_url.extend(custom_linkcheck_anchors_ignore_for_url) + +# Tags cannot be added directly in custom_conf.py, so add them here +for tag in custom_tags: + tags.add(tag) + +############################################################ +### Styling +############################################################ + +# Find the current builder +builder = 'dirhtml' +if '-b' in sys.argv: + builder = sys.argv[sys.argv.index('-b')+1] + +# Setting templates_path for epub makes the build fail +if builder == 'dirhtml' or builder == 'html': + templates_path = ['.sphinx/_templates'] + +# Theme configuration +html_theme = 'furo' +html_last_updated_fmt = '' +html_permalinks_icon = '¶' + +if html_title == '': + html_theme_options = { + 'sidebar_hide_name': True + } + +############################################################ +### Additional files +############################################################ + +html_static_path = ['.sphinx/_static'] + +html_css_files = [ + 'custom.css', + 'header.css', + 'github_issue_links.css', + 'furo_colors.css' +] +html_css_files.extend(custom_html_css_files) + +html_js_files = ['header-nav.js'] +if 'github_issues' in html_context and html_context['github_issues'] and not disable_feedback_button: + html_js_files.append('github_issue_links.js') +html_js_files.extend(custom_html_js_files) + + +############################################################ +### Myst configuration +############################################################ +if os.path.exists('./reuse/substitutions.yaml'): + with open('./reuse/substitutions.yaml', 'r') as fd: + myst_substitutions = yaml.safe_load(fd.read()) diff --git a/docs/tools/custom_conf.py b/docs/tools/custom_conf.py index 53dafffbe..aa876861c 100644 --- a/docs/tools/custom_conf.py +++ b/docs/tools/custom_conf.py @@ -41,7 +41,7 @@ # -H 'Accept: application/vnd.github.v3.raw' \ # https://api.github.com/repos/canonical/ | jq '.created_at' -copyright = '%s, %s' % (datetime.date.today().year, author) +copyright = '%s CC-BY-SA, %s' % (datetime.date.today().year, author) ## Open Graph configuration - defines what is displayed as a link preview ## when linking to the documentation from another website (see https://ogp.me/) @@ -89,7 +89,7 @@ # Change to the folder that contains the documentation # (usually "/" or "/docs/") - 'github_folder': '/docs/', + 'github_folder': '/docs/src/', # Change to an empty value if your GitHub repo doesn't have issues enabled. # This will disable the feedback button and the issue link in the footer. @@ -100,7 +100,12 @@ # You can override the default setting on a page-by-page basis by specifying # it as file-wide metadata at the top of the file, see # https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html - 'sequential_nav': "none" + 'sequential_nav': "none", + # Controls if to display the contributors of a file or not + "display_contributors": True, + + # Controls time frame for showing the contributors + "display_contributors_since": "" } # If your project is on documentation.ubuntu.com, specify the project @@ -140,11 +145,35 @@ ## Use them to extend the default features. # Add extensions -custom_extensions = ['sphinxcontrib.kroki', ] +custom_extensions = [ ] # Add MyST extensions custom_myst_extensions = [] +# Add custom Sphinx extensions as needed. +# This array contains recommended extensions that should be used. +# NOTE: The following extensions are handled automatically and do +# not need to be added here: myst_parser, sphinx_copybutton, sphinx_design, +# sphinx_reredirects, sphinxcontrib.jquery, sphinxext.opengraph +custom_extensions = [ + 'sphinx_tabs.tabs', + 'canonical.youtube-links', + 'canonical.related-links', + 'canonical.custom-rst-roles', + 'canonical.terminal-output', + 'notfound.extension', + 'sphinxcontrib.cairosvgconverter', + ] +# Add custom required Python modules that must be added to the +# .sphinx/requirements.txt file. +# NOTE: The following modules are handled automatically and do not need to be +# added here: canonical-sphinx-extensions, furo, linkify-it-py, myst-parser, +# pyspelling, sphinx, sphinx-autobuild, sphinx-copybutton, sphinx-design, +# sphinx-notfound-page, sphinx-reredirects, sphinx-tabs, sphinxcontrib-jquery, +# sphinxext-opengraph +custom_required_modules = [ + 'sphinxcontrib-svg2pdfconverter[CairoSVG]' +] # Add files or directories that should be excluded from processing. custom_excludes = [ 'doc-cheat-sheet*', @@ -186,4 +215,6 @@ rst_prolog = ''' .. role:: center :class: align-center +.. role:: h2 + :class: hclass2 ''' diff --git a/docs/tools/make.bat b/docs/tools/make.bat new file mode 100644 index 000000000..32bb24529 --- /dev/null +++ b/docs/tools/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=_build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/docs/tools/metrics/scripts/build_metrics.sh b/docs/tools/metrics/scripts/build_metrics.sh new file mode 100755 index 000000000..b7140a1a9 --- /dev/null +++ b/docs/tools/metrics/scripts/build_metrics.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +links=0 +images=0 + +# count number of links +links=$(find . -type d -path './.sphinx' -prune -o -name '*.html' -exec cat {} + | grep -o "&2 echo -e "$msg" +} + +function get_dqlite_node_id() { + local infoYamlPath=$1 + sudo cat $infoYamlPath | yq -r '.ID' +} + +function get_dqlite_node_addr() { + local infoYamlPath=$1 + sudo cat $infoYamlPath | yq -r '.Address' +} + +function get_dqlite_node_role() { + local infoYamlPath=$1 + sudo cat $infoYamlPath | yq -r '.Role' +} + +function get_dqlite_role_from_cluster_yaml() { + # Note that the cluster.yaml role may not match the info.yaml role. + # In case of a freshly joined node, info.yaml will show it as a "voter" + # while cluster.yaml lists it as a "spare" node. + local clusterYamlPath=$1 + local nodeId=$2 + + # Update the specified node. + sudo cat $clusterYamlPath | \ + yq -r "(.[] | select(.ID == \"$nodeId\") | .Role )" +} + +function set_dqlite_node_role() { + # The yq snap installs in confined mode, so it's unable to access the + # Dqlite config files. + # In order to modify files in-place, we're using sponge. It reads all + # the stdin data before opening the output file. + local infoYamlPath=$1 + local role=$2 + sudo cat $infoYamlPath | \ + yq ".Role = $role" | + sudo sponge $infoYamlPath +} + +# Update cluster.yaml, setting the specified node as voter (role = 0). +# The other nodes will become spares, having the role set to 2. +function set_dqlite_node_as_sole_voter() { + local clusterYamlPath=$1 + local nodeId=$2 + + # Update the specified node. + sudo cat $clusterYamlPath | \ + yq "(.[] | select(.ID == \"$nodeId\") | .Role ) = 0" | \ + sudo sponge $clusterYamlPath + + # Update the other nodes. + sudo cat $clusterYamlPath | \ + yq "(.[] | select(.ID != \"$nodeId\") | .Role ) = 2" | \ + sudo sponge $clusterYamlPath +} + +function get_dql_peer_ip() { + local clusterYamlPath=$1 + local nodeId=$2 + + local addresses=( $(sudo cat $clusterYamlPath | \ + yq "(.[] | select(.ID != \"$nodeId\") | .Address )") ) + + if [[ ${#addresses[@]} -gt 1 ]]; then + log_message "More than one dql peers found: ${addresses[@]}" + exit 1 + fi + + if [[ ${#addresses[@]} -lt 1 ]]; then + log_message "No dql peers found." + exit 1 + fi + + echo ${addresses[0]} | cut -d ":" -f 1 +} + +# This function moves the Dqlite state directories to the DRBD mount, +# replacing them with symlinks. This ensures that the primary will always use +# the latest DRBD data. +# +# The existing contents are moved to a backup folder, which can be used as +# part of the recovery process. +function move_statedirs() { + sudo mkdir -p $DRBD_MOUNT_DIR/k8s-dqlite + sudo mkdir -p $DRBD_MOUNT_DIR/k8sd + + log_message "Validating Dqlite state directories." + check_statedir $K8S_DQLITE_STATE_DIR $DRBD_MOUNT_DIR/k8s-dqlite + check_statedir $K8SD_STATE_DIR $DRBD_MOUNT_DIR/k8sd + + if [[ ! -L $K8S_DQLITE_STATE_DIR ]] || [[ ! -L $K8SD_STATE_DIR ]]; then + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + + local expRole=`get_expected_dqlite_role` + # For fresh k8s clusters, the info.yaml role may not match the cluster.yaml role. + local k8sDqliteRole=`get_dqlite_role_from_cluster_yaml \ + $K8S_DQLITE_CLUSTER_YAML $k8sDqliteNodeId` + + if [[ $expRole -ne $k8sDqliteRole ]]; then + # TODO: consider automating this. We may move the pacemaker resource + # ourselves and maybe even copy the remote files through scp or ssh. + # However, there's a risk of race conditions. + log_message "DRBD volume mounted on replica, refusing to transfer Dqlite files." + log_message "Move the DRBD volume to the primary node (through the fs_res Pacemaker resource) and try again." + log_message "Example: sudo crm resource move fs_res && sudo crm resource clear fs_res" + exit 1 + fi + fi + + # Ensure that the k8s services are stopped. + log_message "Stopping k8s services." + sudo snap stop k8s + + if [[ ! -L $K8S_DQLITE_STATE_DIR ]]; then + log_message "Not a symlink: $K8S_DQLITE_STATE_DIR, " \ + "transferring to $DRBD_MOUNT_DIR/k8s-dqlite" + sudo cp -r $K8S_DQLITE_STATE_DIR/. $DRBD_MOUNT_DIR/k8s-dqlite + + log_message "Creating k8s-dqlite state dir backup: $K8S_DQLITE_STATE_BKP_DIR" + sudo rm -rf $K8S_DQLITE_STATE_BKP_DIR + sudo mv $K8S_DQLITE_STATE_DIR/ $K8S_DQLITE_STATE_BKP_DIR + + log_message "Creating symlink $K8S_DQLITE_STATE_DIR -> $DRBD_MOUNT_DIR/k8s-dqlite" + sudo ln -sf $DRBD_MOUNT_DIR/k8s-dqlite $K8S_DQLITE_STATE_DIR + else + log_message "Symlink $K8S_DQLITE_STATE_DIR points to $DRBD_MOUNT_DIR/k8s-dqlite" + fi + + if [[ ! -L $K8SD_STATE_DIR ]]; then + log_message "Not a symlink: $K8SD_STATE_DIR, " \ + "transferring to $DRBD_MOUNT_DIR/k8sd" + sudo cp -r $K8SD_STATE_DIR/. $DRBD_MOUNT_DIR/k8sd + + log_message "Creating k8sd state dir backup: $K8SD_STATE_BKP_DIR" + sudo rm -rf $K8SD_STATE_BKP_DIR + sudo mv $K8SD_STATE_DIR/ $K8SD_STATE_BKP_DIR + + log_message "Creating symlink $K8SD_STATE_DIR -> $DRBD_MOUNT_DIR/k8sd" + sudo ln -sf $DRBD_MOUNT_DIR/k8sd $K8SD_STATE_DIR + else + log_message "Symlink $K8SD_STATE_DIR points to $DRBD_MOUNT_DIR/k8sd" + fi +} + +function ensure_mount_rw() { + if ! mount | grep "on $DRBD_MOUNT_DIR type" &> /dev/null; then + log_message "Missing DRBD mount: $DRBD_MOUNT_DIR" + return 1 + fi + + if ! mount | grep "on $DRBD_MOUNT_DIR type" | grep "rw" &> /dev/null; then + log_message "DRBD mount read-only: $DRBD_MOUNT_DIR" + return 1 + fi +} + +function wait_drbd_promoted() { + log_message "Waiting for one of the DRBD nodes to be promoted." + + local pollInterval=2 + # Special parameter, no need to increase it ourselves. + SECONDS=0 + + while [[ $SECONDS -lt $DRBD_READY_TIMEOUT ]]; do + if sudo crm resource status drbd_master_slave | grep Promoted ; then + log_message "DRBD node promoted." + return 0 + else + log_message "No DRBD node promoted yet, retrying in ${pollInterval}s" + sleep $pollInterval + fi + done + + log_message "Timed out waiting for primary DRBD node." \ + "Waited: ${SECONDS}. Timeout: ${DRBD_READY_TIMEOUT}s." + return 1 +} + +function ensure_drbd_unmounted() { + if mount | grep "on $DRBD_MOUNT_DIR type" &> /dev/null ; then + log_message "DRBD device mounted: $DRBD_MOUNT_DIR" + return 1 + fi +} + +function ensure_drbd_ready() { + ensure_mount_rw + + diskStatus=`sudo drbdadm status r0 | grep disk | head -1 | cut -d ":" -f 2` + if [[ $diskStatus != "UpToDate" ]]; then + log_message "DRBD disk status not ready. Current status: $diskStatus" + return 1 + else + log_message "DRBD disk up to date." + fi +} + +function wait_drbd_primary () { + log_message "Waiting for primary DRBD node to be ready." + + local pollInterval=2 + # Special parameter, no need to increase it ourselves. + SECONDS=0 + + while [[ $SECONDS -lt $DRBD_READY_TIMEOUT ]]; do + if ensure_drbd_ready; then + log_message "Primary DRBD node ready." + return 0 + else + log_message "Primary DRBD node not ready yet, retrying in ${pollInterval}s" + sleep $pollInterval + fi + done + + log_message "Timed out waiting for primary DRBD node." \ + "Waited: ${SECONDS}. Timeout: ${DRBD_READY_TIMEOUT}s." + return 1 +} + +function wait_for_peer_k8s() { + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + local peerIp=`get_dql_peer_ip $K8S_DQLITE_CLUSTER_BKP_YAML $k8sDqliteNodeId` + if [[ -z $peerIp ]]; then + log_message "Couldn't retrieve Dqlite peer ip." + exit 1 + fi + + log_message "Waiting for k8s to start on peer: $peerIp. Timeout: ${PEER_READY_TIMEOUT}s." + + local pollInterval=2 + # Special parameter, no need to increase it ourselves. + SECONDS=0 + + while [[ $SECONDS -lt $PEER_READY_TIMEOUT ]]; do + if ssh $SSH_OPTS $SSH_USERNAME@$peerIp sudo k8s status &> /dev/null; then + log_message "Peer ready." + return 0 + else + log_message "Peer not ready yet, retrying in ${pollInterval}s." + sleep $pollInterval + fi + done + + log_message "Timed out waiting for k8s services to start on peer." \ + "Waited: ${SECONDS}. Timeout: ${PEER_READY_TIMEOUT}s." + return 1 + +} + +# "drbdadm status" throws the following if our service starts before +# Pacemaker initialized DRBD (even on the secondary). +# +# r0: No such resource +# Command 'drbdsetup-84 status r0' terminated with exit code 10 +function wait_drbd_resource () { + log_message "Waiting for DRBD resource." + + local pollInterval=2 + # Special parameter, no need to increase it ourselves. + SECONDS=0 + + while [[ $SECONDS -lt $DRBD_READY_TIMEOUT ]]; do + if sudo drbdadm status &> /dev/null; then + log_message "DRBD ready." + return 0 + else + log_message "DRBD not ready yet, retrying in ${pollInterval}s" + sleep $pollInterval + fi + done + + log_message "Timed out waiting for DRBD resource." \ + "Waited: ${SECONDS}. Timeout: ${DRBD_READY_TIMEOUT}s." + return 1 +} + +# Based on the DRBD volume state, we decide if this node should be a +# Dqlite voter or a spare. +function get_expected_dqlite_role() { + drbdResRole=`sudo drbdadm status $DRBD_RES_NAME | head -1 | grep role | cut -d ":" -f 2` + + case $drbdResRole in + "Primary") + echo $DQLITE_ROLE_VOTER + ;; + "Secondary") + echo $DQLITE_ROLE_SPARE + ;; + *) + log_message "Unexpected DRBD role: $drbdResRole" + exit 1 + ;; + esac +} + +function validate_drbd_state() { + wait_drbd_promoted + + drbdResRole=`sudo drbdadm status $DRBD_RES_NAME | head -1 | grep role | cut -d ":" -f 2` + + case $drbdResRole in + "Primary") + wait_drbd_primary + ;; + "Secondary") + ensure_drbd_unmounted + ;; + *) + log_message "Unexpected DRBD role: $drbdResRole" + exit 1 + ;; + esac +} + +# After a failover, the state dir points to the shared DRBD volume. +# We need to restore the node certificate and config files. +function restore_dqlite_confs_and_certs() { + log_message "Restoring Dqlite configs and certificates." + + sudo cp $K8S_DQLITE_STATE_BKP_DIR/info.yaml $K8S_DQLITE_STATE_DIR + + sudo cp $K8SD_STATE_BKP_DIR/database/info.yaml $K8SD_STATE_DIR/database/ + sudo cp $K8SD_STATE_BKP_DIR/daemon.yaml $K8SD_STATE_DIR/ + + # restore k8s-dqlite certificates + sudo cp $K8S_DQLITE_STATE_BKP_DIR/cluster.crt $K8S_DQLITE_STATE_DIR + sudo cp $K8S_DQLITE_STATE_BKP_DIR/cluster.key $K8S_DQLITE_STATE_DIR + + # restore k8sd certificates + sudo cp $K8SD_STATE_BKP_DIR/cluster.crt $K8SD_STATE_DIR + sudo cp $K8SD_STATE_BKP_DIR/cluster.key $K8SD_STATE_DIR + sudo cp $K8SD_STATE_BKP_DIR/server.crt $K8SD_STATE_DIR + sudo cp $K8SD_STATE_BKP_DIR/server.key $K8SD_STATE_DIR +} + +# Promote the current node as primary and prepare the recovery archives. +function promote_as_primary() { + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + local k8sdNodeId=`get_dqlite_node_id $K8SD_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + local peerIp=`get_dql_peer_ip $K8S_DQLITE_CLUSTER_YAML $k8sDqliteNodeId` + if [[ -z $peerIp ]]; then + log_message "Couldn't retrieve Dqlite peer ip." + exit 1 + fi + + log_message "Stopping local k8s services." + sudo snap stop k8s + + # After a node crash, there may be a leaked control socket file and + # k8sd will refuse to perform the recovery. We've just stopped the k8s snap, + # it should be safe to remove such stale unix sockets. + log_message "Removing stale control sockets." + sudo rm -f $K8SD_STATE_DIR/control.socket + + local stoppedPeer=0 + log_message "Checking peer k8s services: $peerIp" + if ssh $SSH_OPTS $SSH_USERNAME@$peerIp sudo snap services k8s | grep -v inactive | grep "active"; then + log_message "Attempting to stop peer k8s services." + # Stop the k8s snap directly instead of the wrapper service so that + # we won't cause failures if both nodes start at the same time. + # The secondary will wait for the k8s services to start on the primary. + if ssh $SSH_OPTS $SSH_USERNAME@$peerIp sudo snap stop k8s; then + stoppedPeer=1 + log_message "Successfully stopped peer k8s services." + log_message "The stopped services are going to be restarted after the recovery finishes." + else + log_message "Couldn't stop k8s services on the peer node." \ + "Assuming that the peer node is stopped and proceeding with the recovery." + fi + fi + + log_message "Ensuring rw access to DRBD mount." + # Having RW access to the DRBD mount implies that this is the primary node. + ensure_mount_rw + + restore_dqlite_confs_and_certs + + log_message "Updating Dqlite roles." + # Update info.yaml + set_dqlite_node_role $K8S_DQLITE_INFO_YAML $DQLITE_ROLE_VOTER + set_dqlite_node_role $K8SD_INFO_YAML $DQLITE_ROLE_VOTER + + # Update cluster.yaml + set_dqlite_node_as_sole_voter $K8S_DQLITE_CLUSTER_YAML $k8sDqliteNodeId + set_dqlite_node_as_sole_voter $K8SD_CLUSTER_YAML $k8sdNodeId + + log_message "Restoring Dqlite." + sudo $K8SD_PATH cluster-recover \ + --state-dir=$K8SD_STATE_DIR \ + --k8s-dqlite-state-dir=$K8S_DQLITE_STATE_DIR \ + --log-level $K8SD_LOG_LEVEL \ + --non-interactive + + # TODO: consider removing offending segments if the last snapshot is behind + # and then try again. + + log_message "Copying k8sd recovery tarball to $K8SD_RECOVERY_TARBALL_BKP" + sudo cp $K8SD_RECOVERY_TARBALL $K8SD_RECOVERY_TARBALL_BKP + + log_message "Restarting k8s services." + sudo snap start k8s + + # TODO: validate k8s status + + if [[ $stoppedPeer -ne 0 ]]; then + log_message "Restarting peer k8s services: $peerIp" + # It's importand to issue a restart here since we stopped the k8s snap + # directly and the wrapper service doesn't currently monitor it. + ssh $SSH_OPTS $SSH_USERNAME@$peerIp sudo systemctl restart $SYSTEMD_SERVICE_NAME || + log_message "Couldn't start peer k8s services." + fi +} + +function process_recovery_files_on_secondary() { + local peerIp="$1" + + log_message "Ensuring that the DRBD volume is unmounted." + ensure_drbd_unmounted + + log_message "Restoring local Dqlite backup files." + sudo cp -r $K8S_DQLITE_STATE_BKP_DIR/. $DRBD_MOUNT_DIR/k8s-dqlite/ + sudo cp -r $K8SD_STATE_BKP_DIR/. $DRBD_MOUNT_DIR/k8sd/ + + sudo rm -f $DRBD_MOUNT_DIR/k8s-dqlite/00*-* + sudo rm -f $DRBD_MOUNT_DIR/k8s-dqlite/snapshot-* + sudo rm -f $DRBD_MOUNT_DIR/k8s-dqlite/metadata* + + sudo rm -f $DRBD_MOUNT_DIR/k8sd/database/00*-* + sudo rm -f $DRBD_MOUNT_DIR/k8sd/database/snapshot-* + sudo rm -f $DRBD_MOUNT_DIR/k8sd/database/metadata* + + log_message "Retrieving k8sd recovery tarball." + scp $SSH_OPTS $SSH_USERNAME@$peerIp:$K8SD_RECOVERY_TARBALL_BKP /tmp/ + sudo mv /tmp/`basename $K8SD_RECOVERY_TARBALL_BKP` \ + $K8SD_RECOVERY_TARBALL + + # TODO: do we really need to transfer recovery tarballs in this situation? + # the spare is simply forwarding the requests to the primary, it doesn't really + # hold any data. + lastK8sDqliteRecoveryTarball=`ssh $SSH_USERNAME@$peerIp \ + sudo ls /var/snap/k8s/common/ | \ + grep -P "recovery-k8s-dqlite-.*post-recovery" | \ + tail -1` + if [ -z "$lastK8sDqliteRecoveryTarball" ]; then + log_message "couldn't retrieve latest k8s-dqlite recovery tarball from $peerIp" + exit 1 + fi + + log_message "Retrieving k8s-dqlite recovery tarball." + scp $SSH_USERNAME@$peerIp:/var/snap/k8s/common/$lastK8sDqliteRecoveryTarball /tmp/ + sudo tar -xf /tmp/$lastK8sDqliteRecoveryTarball -C $K8S_DQLITE_STATE_DIR + + log_message "Updating Dqlite roles." + # Update info.yaml + set_dqlite_node_role $K8S_DQLITE_INFO_YAML $DQLITE_ROLE_SPARE + set_dqlite_node_role $K8SD_INFO_YAML $DQLITE_ROLE_SPARE + # We're skipping cluster.yaml, we expect the recovery archives to contain + # updated cluster.yaml files. +} + +# Recover a former primary, now secondary Dqlite node. +# Run "promote_as_primary" on the ther node first. +function rejoin_secondary() { + log_message "Recovering secondary node." + + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + local peerIp=`get_dql_peer_ip $K8S_DQLITE_CLUSTER_BKP_YAML $k8sDqliteNodeId` + if [[ -z $peerIp ]]; then + log_message "Couldn't retrieve Dqlite peer ip." + exit 1 + fi + + log_message "Stopping k8s services." + sudo snap stop k8s + + log_message "Adding temporary Pacemaker constraint." + # We need to prevent failovers from happening while restoring secondary + # Dqlite data, otherwise we may end up overriding or deleting the primary + # node data. + # + # TODO: consider reducing the constraint scope (e.g. resource level constraint + # instead of putting the entire node in standby). + sudo crm node standby + if ! process_recovery_files_on_secondary $peerIp; then + log_message "Dqlite recovery filed, removing temporary Pacemaker constraints." + sudo crm node online + exit 1 + fi + + log_message "Restoring Pacemaker state." + sudo crm node online + + log_message "Restarting k8s services" + sudo snap start k8s +} + +function install_packages() { + sudo apt-get update + + sudo DEBIAN_FRONTEND=noninteractive apt-get install \ + python3 python3-netaddr \ + pacemaker resource-agents-extra \ + drbd-utils ntp linux-image-generic snap moreutils -y + sudo modprobe drbd || sudo apt-get install -y linux-modules-extra-$(uname -r) + + sudo snap install jq + sudo snap install yq + sudo snap install install k8s --classic $K8S_SNAP_CHANNEL +} + +function check_statedir() { + local stateDir="$1" + local expLink="$2" + + if [[ ! -e $stateDir ]]; then + log_message "State directory missing: $stateDir" + exit 1 + fi + + target=`readlink -f $stateDir` + if [[ -L "$stateDir" ]] && [[ "$target" != "$expLink" ]]; then + log_message "Unexpected symlink target. " \ + "State directory: $stateDir. " \ + "Expected symlink target: $expLink. " \ + "Actual symlink target: $target." + exit 1 + fi + + if [[ ! -L $stateDir ]] && [[ ! -z "$( ls -A $expLink )" ]]; then + log_message "State directory is not a symlink, however the " \ + "expected link target exists and is not empty. " \ + "We can't know which files to keep, erroring out. " \ + "State directory: $stateDir. " \ + "Expected symlink target: $expLink." + exit 1 + fi +} + +function check_peer_recovery_tarballs() { + log_message "Retrieving k8s-dqlite node id." + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + log_message "Retrieving Dqlite peer ip." + local peerIp=`get_dql_peer_ip $K8S_DQLITE_CLUSTER_BKP_YAML $k8sDqliteNodeId` + if [[ -z $peerIp ]]; then + log_message "Couldn't retrieve Dqlite peer ip." + exit 1 + fi + + log_message "Checking for recovery taballs on $peerIp." + + k8sdRecoveryTarball=`ssh $SSH_OPTS $SSH_USERNAME@$peerIp \ + sudo ls -A "$K8SD_RECOVERY_TARBALL_BKP"` + if [[ -z $k8sdRecoveryTarball ]]; then + log_message "Peer $peerIp doesn't have k8sd recovery tarball." + return 1 + fi + + lastK8sDqliteRecoveryTarball=`ssh $SSH_OPTS $SSH_USERNAME@$peerIp \ + sudo ls /var/snap/k8s/common/ | \ + grep -P "recovery-k8s-dqlite-.*post-recovery"` + if [[ -z $k8sdRecoveryTarball ]]; then + log_message "Peer $peerIp doesn't have k8s-dqlite recovery tarball." + return 1 + fi +} + +function start_service() { + log_message "Initializing node." + + # DRBD is the primary source of truth for the Dqlite role. + # We need to wait for it to become available. + wait_drbd_resource + + # dump the DRBD and pacemaker status for debugging purposes. + sudo drbdadm status + sudo crm status + + validate_drbd_state + + move_statedirs + + local expRole=`get_expected_dqlite_role` + case $expRole in + $DQLITE_ROLE_VOTER) + log_message "Assuming the Dqlite voter role (primary)." + + # We'll assume that if the primary stopped, it needs to go through + # the recovery process. + promote_as_primary + ;; + $DQLITE_ROLE_SPARE) + log_message "Assuming the Dqlite spare role (secondary)." + + wait_for_peer_k8s + + if check_peer_recovery_tarballs; then + log_message "Recovery tarballs found, initiating recovery." + rejoin_secondary + else + # Maybe the primary didn't change and we don't need to go + # through the recovery process. + # TODO: consider comparing the cluster.yaml files from the + # two nodes. + log_message "Recovery tarballs missing, skipping recovery." + log_message "Starting k8s services." + sudo snap k8s start + fi + ;; + *) + log_message "Unexpected Dqlite role: $expRole" + exit 1 + ;; + esac +} + +function clean_recovery_data() { + log_message "Cleaning up Dqlite recovery data." + rm -f $K8SD_RECOVERY_TARBALL + rm -f $K8SD_RECOVERY_TARBALL_BKP + rm -f $K8S_DQLITE_STATE_DIR/recovery-k8s-dqlite* +} + +function purge() { + log_message "Removing the k8s snap and all the associated files." + + sudo snap remove --purge k8s + + if [[ -d $DRBD_MOUNT_DIR ]]; then + log_message "Cleaning up $DRBD_MOUNT_DIR." + sudo rm -rf $DRBD_MOUNT_DIR/k8sd + sudo rm -rf $DRBD_MOUNT_DIR/k8s-dqlite + + if ! ensure_drbd_unmounted; then + log_message "Cleaning up $DRBD_MOUNT_DIR mount point." + + # The replicas use the mount dir directly, without a block device + # attachment. We need to clean up the mount point as well. + # + # We're using another mount with "--bind" to bypass the DRBD mount. + tempdir=`mktemp -d` + # We need to mount the parent dir. + sudo mount --bind `dirname $DRBD_MOUNT_DIR` $tempdir + sudo rm -rf $tempdir/`basename $DRBD_MOUNT_DIR`/k8sd + sudo rm -rf $tempdir/`basename $DRBD_MOUNT_DIR`/k8s-dqlite + sudo umount $tempdir + sudo rm -rf $tempdir + fi + fi +} + +function clear_taints() { + log_message "Clearing tainted Pacemaker resources." + sudo crm resource clear ha_k8s_failover_service + sudo crm resource clear fs_res + sudo crm resource clear drbd_master_slave + + sudo crm resource cleanup ha_k8s_failover_service + sudo crm resource cleanup fs_res + sudo crm resource cleanup drbd_master_slave +} + +function main() { + local command=$1 + + case $command in + "move_statedirs") + move_statedirs + ;; + "install_packages") + install_packages + ;; + "start_service") + start_service + ;; + "clean_recovery_data") + clean_recovery_data + ;; + "purge") + purge + ;; + "clear_taints") + clear_taints + ;; + *) + cat << EOF +Unknown command: $1 + +usage: $0 + +Commands: + move_statedirs Move the Dqlite state directories to the DRBD mount, + replacing them with symlinks. + The existing contents are moved to a backup folder, + which can be used as part of the recovery process. + install_packages Install the packages required by the two-node HA + cluster. + start_service Initialize the k8s services, taking the following + steps: + 1. Based on the DRBD state, decide if this node + should assume the primary (dqlite voter) or + secondary (spare) role. + 2. If this is the first start, transfer the Dqlite + state directories and create backups. + 3. If this node is a primary, promote it and initiate + the Dqlite recovery, creating recovery tarballs. + Otherwise, copy over the recovery files and + join the existing cluster as a spare. + 4. Start the k8s services. + IMPORTANT: ensure that the DRBD volume is attached + to the primary node when running the command for + the first time. + clean_recovery_data Remove database recovery files. Should be called + after the cluster has been fully recovered. + purge Remove the k8s snap and all its associated files. + clear_taints Clear tainted Pacemaker resources. + +EOF + ;; + esac +} + +if [[ $sourced -ne 1 ]]; then + main $@ +fi diff --git a/k8s/lib.sh b/k8s/lib.sh index e3b871cfb..e81b7aefd 100755 --- a/k8s/lib.sh +++ b/k8s/lib.sh @@ -47,6 +47,8 @@ k8s::common::is_strict() { k8s::remove::network() { k8s::common::setup_env + "${SNAP}/bin/kube-proxy" --cleanup || true + k8s::cmd::k8s x-cleanup network || true } @@ -60,17 +62,48 @@ k8s::remove::containers() { # delete cni network namespaces ip netns list | cut -f1 -d' ' | grep -- "^cni-" | xargs -n1 -r -t ip netns delete || true - # unmount NFS volumes forcefully, as unmounting them normally may hang otherwise. + # The PVC loopback devices use container paths, making them tricky to identify. + # We'll rely on the volume mount paths (/var/lib/kubelet/*). + local LOOP_DEVICES=`cat /proc/mounts | grep /var/lib/kubelet/pods | grep /dev/loop | cut -d " " -f 1` + + # unmount Pod NFS volumes forcefully, as unmounting them normally may hang otherwise. cat /proc/mounts | grep /run/containerd/io.containerd. | grep "nfs[34]" | cut -f2 -d' ' | xargs -r -t umount -f || true cat /proc/mounts | grep /var/lib/kubelet/pods | grep "nfs[34]" | cut -f2 -d' ' | xargs -r -t umount -f || true - # unmount volumes + # unmount Pod volumes gracefully. cat /proc/mounts | grep /run/containerd/io.containerd. | cut -f2 -d' ' | xargs -r -t umount || true cat /proc/mounts | grep /var/lib/kubelet/pods | cut -f2 -d' ' | xargs -r -t umount || true - # umount lingering volumes by force, to prevent potential volume leaks. + # unmount lingering Pod volumes by force, to prevent potential volume leaks. cat /proc/mounts | grep /run/containerd/io.containerd. | cut -f2 -d' ' | xargs -r -t umount -f || true cat /proc/mounts | grep /var/lib/kubelet/pods | cut -f2 -d' ' | xargs -r -t umount -f || true + + # unmount various volumes exposed by CSI plugin drivers. + cat /proc/mounts | grep /var/lib/kubelet/plugins | cut -f2 -d' ' | xargs -r -t umount -f || true + + # remove kubelet plugin sockets, as we don't have the containers associated with them anymore, + # so kubelet won't try to access inexistent plugins on reinstallation. + find /var/lib/kubelet/plugins/ -name "*.sock" | xargs rm -f || true + rm /var/lib/kubelet/plugins_registry/*.sock || true + + cat /proc/mounts | grep /var/snap/k8s/common/var/lib/containerd/ | cut -f2 -d' ' | xargs -r -t umount || true + + # cleanup loopback devices + for dev in $LOOP_DEVICES; do + losetup -d $dev + done +} + +k8s::remove::containerd() { + k8s::common::setup_env + + # only remove containerd if the snap was already bootstrapped. + # this is to prevent removing containerd when it is not installed by the snap. + for file in "containerd-socket-path" "containerd-config-dir" "containerd-root-dir" "containerd-cni-bin-dir"; do + if [ -f "$SNAP_COMMON/lock/$file" ]; then + rm -rf $(cat "$SNAP_COMMON/lock/$file") + fi + done } # Run a ctr command against the local containerd socket @@ -169,3 +202,11 @@ k8s::kubelet::ensure_shared_root_dir() { mount -o remount --make-rshared "$SNAP_COMMON/var/lib/kubelet" /var/lib/kubelet fi } + +# Loads the kernel module names given as arguments +# Example: 'k8s::util::load_kernel_modules mod1 mod2 mod3' +k8s::util::load_kernel_modules() { + k8s::common::setup_env + + modprobe $@ +} diff --git a/k8s/manifests/charts/cilium-1.15.2.tgz b/k8s/manifests/charts/cilium-1.15.2.tgz deleted file mode 100644 index 6bf08bd03..000000000 Binary files a/k8s/manifests/charts/cilium-1.15.2.tgz and /dev/null differ diff --git a/k8s/manifests/charts/cilium-1.16.3.tgz b/k8s/manifests/charts/cilium-1.16.3.tgz new file mode 100644 index 000000000..eaca333a4 Binary files /dev/null and b/k8s/manifests/charts/cilium-1.16.3.tgz differ diff --git a/k8s/manifests/charts/coredns-1.29.0.tgz b/k8s/manifests/charts/coredns-1.29.0.tgz deleted file mode 100644 index 47b44f442..000000000 Binary files a/k8s/manifests/charts/coredns-1.29.0.tgz and /dev/null differ diff --git a/k8s/manifests/charts/coredns-1.36.0.tgz b/k8s/manifests/charts/coredns-1.36.0.tgz new file mode 100644 index 000000000..7e44977a5 Binary files /dev/null and b/k8s/manifests/charts/coredns-1.36.0.tgz differ diff --git a/k8s/manifests/charts/gateway-api-1.0.0.tgz b/k8s/manifests/charts/gateway-api-1.0.0.tgz deleted file mode 100644 index 7b84d44a5..000000000 Binary files a/k8s/manifests/charts/gateway-api-1.0.0.tgz and /dev/null differ diff --git a/k8s/manifests/charts/gateway-api-1.1.0.tgz b/k8s/manifests/charts/gateway-api-1.1.0.tgz new file mode 100644 index 000000000..57ade850a Binary files /dev/null and b/k8s/manifests/charts/gateway-api-1.1.0.tgz differ diff --git a/k8s/manifests/charts/metallb-0.14.5.tgz b/k8s/manifests/charts/metallb-0.14.5.tgz deleted file mode 100644 index c37846191..000000000 Binary files a/k8s/manifests/charts/metallb-0.14.5.tgz and /dev/null differ diff --git a/k8s/manifests/charts/metallb-0.14.8.tgz b/k8s/manifests/charts/metallb-0.14.8.tgz new file mode 100644 index 000000000..a78eb231c Binary files /dev/null and b/k8s/manifests/charts/metallb-0.14.8.tgz differ diff --git a/k8s/manifests/charts/metrics-server-3.12.0.tgz b/k8s/manifests/charts/metrics-server-3.12.0.tgz deleted file mode 100644 index 22f9f8dc2..000000000 Binary files a/k8s/manifests/charts/metrics-server-3.12.0.tgz and /dev/null differ diff --git a/k8s/manifests/charts/metrics-server-3.12.2.tgz b/k8s/manifests/charts/metrics-server-3.12.2.tgz new file mode 100644 index 000000000..4538e8a10 Binary files /dev/null and b/k8s/manifests/charts/metrics-server-3.12.2.tgz differ diff --git a/k8s/scripts/inspect.sh b/k8s/scripts/inspect.sh index a4ae51cc5..9f559451c 100755 --- a/k8s/scripts/inspect.sh +++ b/k8s/scripts/inspect.sh @@ -111,6 +111,17 @@ function collect_service_diagnostics { journalctl -n 100000 -u "snap.$service" &>"$INSPECT_DUMP/$service/journal.log" } +function collect_registry_mirror_logs { + local mirror_units=`systemctl list-unit-files --state=enabled | grep "registry-" | awk '{print $1}'` + if [ -n "$mirror_units" ]; then + mkdir -p "$INSPECT_DUMP/mirrors" + + for mirror_unit in $mirror_units; do + journalctl -n 100000 -u "$mirror_unit" &>"$INSPECT_DUMP/mirrors/$mirror_unit.log" + done + fi +} + function collect_network_diagnostics { log_info "Copy network diagnostics to the final report tarball" ip a &>"$INSPECT_DUMP/ip-a.log" || true @@ -182,6 +193,9 @@ else check_expected_services "${worker_services[@]}" fi +printf -- 'Collecting registry mirror logs\n' +collect_registry_mirror_logs + printf -- 'Collecting service arguments\n' collect_args diff --git a/k8s/wrappers/services/kube-proxy b/k8s/wrappers/services/kube-proxy index 6cd55a999..ef368665a 100755 --- a/k8s/wrappers/services/kube-proxy +++ b/k8s/wrappers/services/kube-proxy @@ -3,4 +3,16 @@ . "$SNAP/k8s/lib.sh" k8s::util::wait_kube_apiserver + +# NOTE: kube-proxy reads some values related to the `nf_conntrack` +# module from procfs on startup, so we must ensure it's loaded: +# https://github.com/canonical/k8s-snap/issues/626 +if [ -f "/proc/sys/net/netfilter/nf_conntrack_max" ]; then + echo "Kernel module nf_conntrack was already loaded before kube-proxy startup." +else + k8s::util::load_kernel_modules nf_conntrack \ + && echo "Successfully modprobed nf_conntrack before kube-proxy startup." \ + || echo "WARN: Failed to 'modprobe nf_conntrack' before kube-proxy startup." +fi + k8s::common::execute_service kube-proxy diff --git a/snap/hooks/remove b/snap/hooks/remove index 29cf572c3..e84c36dd9 100755 --- a/snap/hooks/remove +++ b/snap/hooks/remove @@ -7,3 +7,5 @@ k8s::common::setup_env k8s::remove::containers k8s::remove::network + +k8s::remove::containerd diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml index 9d21e55f1..435f40fb2 100644 --- a/snap/snapcraft.yaml +++ b/snap/snapcraft.yaml @@ -164,6 +164,7 @@ parts: - ethtool - hostname - iproute2 + - ipset - kmod - libatm1 - libnss-resolve diff --git a/src/k8s/Makefile b/src/k8s/Makefile index 4e7d33154..8911da0d8 100644 --- a/src/k8s/Makefile +++ b/src/k8s/Makefile @@ -7,6 +7,13 @@ go.fmt: go mod tidy go fmt ./... +go.lint: +ifeq (, $(shell which golangci-lint)) + echo "golangci-lint not found, installing it" + go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.61.0 +endif + golangci-lint run + go.vet: $(DQLITE_BUILD_SCRIPTS_DIR)/static-go-vet.sh ./... @@ -14,7 +21,7 @@ go.unit: $(DQLITE_BUILD_SCRIPTS_DIR)/static-go-test.sh -v ./pkg/... ./cmd/... -coverprofile=coverage.txt --cover go.doc: bin/static/k8s - bin/static/k8s generate-docs --output-dir ../../docs/src/_parts/commands/ + bin/static/k8s generate-docs --output-dir ../../docs/src/_parts/ --project-dir . ## Static Builds static: bin/static/k8s bin/static/k8sd bin/static/k8s-apiserver-proxy diff --git a/src/k8s/cmd/k8s/hooks.go b/src/k8s/cmd/k8s/hooks.go index 481d63df6..77a945580 100644 --- a/src/k8s/cmd/k8s/hooks.go +++ b/src/k8s/cmd/k8s/hooks.go @@ -34,3 +34,36 @@ func hookInitializeFormatter(env cmdutil.ExecutionEnvironment, format *string) f } } } + +// hookCheckLXD verifies the ownership of directories needed for Kubernetes to function. +// If a potential issue is detected, it displays a warning to the user. +func hookCheckLXD() func(*cobra.Command, []string) { + return func(cmd *cobra.Command, args []string) { + // pathsOwnershipCheck paths to validate root is the owner + pathsOwnershipCheck := []string{"/sys", "/proc", "/dev/kmsg"} + inLXD, err := cmdutil.InLXDContainer() + if err != nil { + cmd.PrintErrf("Failed to check if running inside LXD container: %s", err.Error()) + return + } + if inLXD { + var errMsgs []string + for _, pathToCheck := range pathsOwnershipCheck { + if err = cmdutil.ValidateRootOwnership(pathToCheck); err != nil { + errMsgs = append(errMsgs, err.Error()) + } + } + if len(errMsgs) > 0 { + if debug, _ := cmd.Flags().GetBool("debug"); debug { + cmd.PrintErrln("Warning: When validating required resources potential issues found:") + for _, errMsg := range errMsgs { + cmd.PrintErrln("\t", errMsg) + } + } + cmd.PrintErrln("The lxc profile for Canonical Kubernetes might be missing.") + cmd.PrintErrln("For running k8s inside LXD container refer to " + + "https://documentation.ubuntu.com/canonical-kubernetes/latest/snap/howto/install/lxd/") + } + } + } +} diff --git a/src/k8s/cmd/k8s/k8s.go b/src/k8s/cmd/k8s/k8s.go index 45e26d0c8..815faff7e 100644 --- a/src/k8s/cmd/k8s/k8s.go +++ b/src/k8s/cmd/k8s/k8s.go @@ -35,13 +35,11 @@ func addCommands(root *cobra.Command, group *cobra.Group, commands ...*cobra.Com } func NewRootCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { - var ( - opts struct { - logDebug bool - logVerbose bool - stateDir string - } - ) + var opts struct { + logDebug bool + logVerbose bool + stateDir string + } cmd := &cobra.Command{ Use: "k8s", Short: "Canonical Kubernetes CLI", diff --git a/src/k8s/cmd/k8s/k8s_bootstrap.go b/src/k8s/cmd/k8s/k8s_bootstrap.go index b4243d824..9ba6525d3 100644 --- a/src/k8s/cmd/k8s/k8s_bootstrap.go +++ b/src/k8s/cmd/k8s/k8s_bootstrap.go @@ -13,6 +13,7 @@ import ( apiv1 "github.com/canonical/k8s-snap-api/api/v1" cmdutil "github.com/canonical/k8s/cmd/util" + "github.com/canonical/k8s/pkg/client/snapd" "github.com/canonical/k8s/pkg/config" "github.com/canonical/k8s/pkg/k8sd/features" "github.com/canonical/k8s/pkg/utils" @@ -45,8 +46,26 @@ func newBootstrapCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { Use: "bootstrap", Short: "Bootstrap a new Kubernetes cluster", Long: "Generate certificates, configure service arguments and start the Kubernetes services.", - PreRun: chainPreRunHooks(hookRequireRoot(env), hookInitializeFormatter(env, &opts.outputFormat)), + PreRun: chainPreRunHooks(hookRequireRoot(env), hookInitializeFormatter(env, &opts.outputFormat), hookCheckLXD()), Run: func(cmd *cobra.Command, args []string) { + snapdClient, err := snapd.NewClient() + if err != nil { + cmd.PrintErrln("Error: failed to create snapd client: %w", err) + env.Exit(1) + return + } + microk8sInfo, err := snapdClient.GetSnapInfo("microk8s") + if err != nil { + cmd.PrintErrln("Error: failed to check if microk8s is installed: %w", err) + env.Exit(1) + return + } + if microk8sInfo.StatusCode == 200 && microk8sInfo.HasInstallDate() { + cmd.PrintErrln("Error: microk8s snap is installed. Please remove it using the following command and try again:\n\n sudo snap remove microk8s") + env.Exit(1) + return + } + if opts.interactive && opts.configFile != "" { cmd.PrintErrln("Error: --interactive and --file flags cannot be set at the same time.") env.Exit(1) diff --git a/src/k8s/cmd/k8s/k8s_bootstrap_test.go b/src/k8s/cmd/k8s/k8s_bootstrap_test.go index 39fa074d8..ca24ef624 100644 --- a/src/k8s/cmd/k8s/k8s_bootstrap_test.go +++ b/src/k8s/cmd/k8s/k8s_bootstrap_test.go @@ -3,11 +3,11 @@ package k8s import ( "bytes" _ "embed" - "os" "path/filepath" "testing" apiv1 "github.com/canonical/k8s-snap-api/api/v1" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations" cmdutil "github.com/canonical/k8s/cmd/util" "github.com/canonical/k8s/pkg/utils" . "github.com/onsi/gomega" @@ -62,7 +62,10 @@ var testCases = []testCase{ Enabled: utils.Pointer(true), }, CloudProvider: utils.Pointer("external"), - Annotations: map[string]string{apiv1.AnnotationSkipCleanupKubernetesNodeOnRemove: "true"}, + Annotations: map[string]string{ + apiv1_annotations.AnnotationSkipCleanupKubernetesNodeOnRemove: "true", + apiv1_annotations.AnnotationSkipStopServicesOnRemove: "true", + }, }, ControlPlaneTaints: []string{"node-role.kubernetes.io/control-plane:NoSchedule"}, PodCIDR: utils.Pointer("10.100.0.0/16"), @@ -105,7 +108,7 @@ var testCases = []testCase{ func mustAddConfigToTestDir(t *testing.T, configPath string, data string) { t.Helper() // Create the cluster bootstrap config file - err := os.WriteFile(configPath, []byte(data), 0644) + err := utils.WriteFile(configPath, []byte(data), 0o644) if err != nil { t.Fatal(err) } diff --git a/src/k8s/cmd/k8s/k8s_generate_docs.go b/src/k8s/cmd/k8s/k8s_generate_docs.go index 10a8ea715..b95bb8b46 100644 --- a/src/k8s/cmd/k8s/k8s_generate_docs.go +++ b/src/k8s/cmd/k8s/k8s_generate_docs.go @@ -1,29 +1,64 @@ package k8s import ( + "path" + + apiv1 "github.com/canonical/k8s-snap-api/api/v1" cmdutil "github.com/canonical/k8s/cmd/util" + "github.com/canonical/k8s/pkg/docgen" "github.com/spf13/cobra" "github.com/spf13/cobra/doc" ) func newGenerateDocsCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { var opts struct { - outputDir string + outputDir string + projectDir string } cmd := &cobra.Command{ Use: "generate-docs", Hidden: true, Short: "Generate markdown documentation", Run: func(cmd *cobra.Command, args []string) { - if err := doc.GenMarkdownTree(cmd.Parent(), opts.outputDir); err != nil { + outPath := path.Join(opts.outputDir, "commands") + if err := doc.GenMarkdownTree(cmd.Parent(), outPath); err != nil { cmd.PrintErrf("Error: Failed to generate markdown documentation for k8s command.\n\nThe error was: %v\n", err) env.Exit(1) return } + + outPath = path.Join(opts.outputDir, "bootstrap_config.md") + err := docgen.MarkdownFromJsonStructToFile(apiv1.BootstrapConfig{}, outPath, opts.projectDir) + if err != nil { + cmd.PrintErrf("Error: Failed to generate markdown documentation for bootstrap configuration\n\n") + cmd.PrintErrf("Error: %v", err) + env.Exit(1) + return + } + + outPath = path.Join(opts.outputDir, "control_plane_join_config.md") + err = docgen.MarkdownFromJsonStructToFile(apiv1.ControlPlaneJoinConfig{}, outPath, opts.projectDir) + if err != nil { + cmd.PrintErrf("Error: Failed to generate markdown documentation for ctrl plane join configuration\n\n") + cmd.PrintErrf("Error: %v", err) + env.Exit(1) + return + } + + outPath = path.Join(opts.outputDir, "worker_join_config.md") + err = docgen.MarkdownFromJsonStructToFile(apiv1.WorkerJoinConfig{}, outPath, opts.projectDir) + if err != nil { + cmd.PrintErrf("Error: Failed to generate markdown documentation for worker join configuration\n\n") + cmd.PrintErrf("Error: %v", err) + env.Exit(1) + return + } + cmd.Printf("Generated documentation in %s\n", opts.outputDir) }, } cmd.Flags().StringVar(&opts.outputDir, "output-dir", ".", "directory where the markdown docs will be written") + cmd.Flags().StringVar(&opts.projectDir, "project-dir", "../../", "the path to k8s-snap/src/k8s") return cmd } diff --git a/src/k8s/cmd/k8s/k8s_join_cluster.go b/src/k8s/cmd/k8s/k8s_join_cluster.go index 7507fedcb..4cd5bfe6d 100644 --- a/src/k8s/cmd/k8s/k8s_join_cluster.go +++ b/src/k8s/cmd/k8s/k8s_join_cluster.go @@ -32,7 +32,7 @@ func newJoinClusterCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { cmd := &cobra.Command{ Use: "join-cluster ", Short: "Join a cluster using the provided token", - PreRun: chainPreRunHooks(hookRequireRoot(env), hookInitializeFormatter(env, &opts.outputFormat)), + PreRun: chainPreRunHooks(hookRequireRoot(env), hookInitializeFormatter(env, &opts.outputFormat), hookCheckLXD()), Args: cmdutil.ExactArgs(env, 1), Run: func(cmd *cobra.Command, args []string) { token := args[0] diff --git a/src/k8s/cmd/k8s/k8s_set_test.go b/src/k8s/cmd/k8s/k8s_set_test.go index 8c6bc23d1..7431f1b04 100644 --- a/src/k8s/cmd/k8s/k8s_set_test.go +++ b/src/k8s/cmd/k8s/k8s_set_test.go @@ -192,7 +192,7 @@ func Test_updateConfigMapstructure(t *testing.T) { if tc.expectErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cfg).To(SatisfyAll(tc.assertions...)) } }) diff --git a/src/k8s/cmd/k8s/k8s_x_capi.go b/src/k8s/cmd/k8s/k8s_x_capi.go index 2da816bcb..2c4658fd8 100644 --- a/src/k8s/cmd/k8s/k8s_x_capi.go +++ b/src/k8s/cmd/k8s/k8s_x_capi.go @@ -1,10 +1,9 @@ package k8s import ( - "os" - apiv1 "github.com/canonical/k8s-snap-api/api/v1" cmdutil "github.com/canonical/k8s/cmd/util" + "github.com/canonical/k8s/pkg/utils" "github.com/spf13/cobra" ) @@ -48,7 +47,7 @@ func newXCAPICmd(env cmdutil.ExecutionEnvironment) *cobra.Command { return } - if err := os.WriteFile(env.Snap.NodeTokenFile(), []byte(token), 0600); err != nil { + if err := utils.WriteFile(env.Snap.NodeTokenFile(), []byte(token), 0o600); err != nil { cmd.PrintErrf("Error: Failed to write the node token to file.\n\nThe error was: %v\n", err) env.Exit(1) return diff --git a/src/k8s/cmd/k8s/k8s_x_snapd_config.go b/src/k8s/cmd/k8s/k8s_x_snapd_config.go index f950cec57..b0f07f9a1 100644 --- a/src/k8s/cmd/k8s/k8s_x_snapd_config.go +++ b/src/k8s/cmd/k8s/k8s_x_snapd_config.go @@ -64,11 +64,7 @@ func newXSnapdConfigCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { ctx, cancel := context.WithTimeout(cmd.Context(), opts.timeout) defer cancel() if err := control.WaitUntilReady(ctx, func() (bool, error) { - _, partOfCluster, err := client.NodeStatus(cmd.Context()) - if !partOfCluster { - cmd.PrintErrf("Node is not part of a cluster: %v\n", err) - env.Exit(1) - } + _, _, err := client.NodeStatus(cmd.Context()) return err == nil, nil }); err != nil { cmd.PrintErrf("Error: k8sd did not come up in time: %v\n", err) diff --git a/src/k8s/cmd/k8s/testdata/bootstrap-config-full.yaml b/src/k8s/cmd/k8s/testdata/bootstrap-config-full.yaml index 79def822f..2fa172ca6 100644 --- a/src/k8s/cmd/k8s/testdata/bootstrap-config-full.yaml +++ b/src/k8s/cmd/k8s/testdata/bootstrap-config-full.yaml @@ -23,6 +23,7 @@ cluster-config: cloud-provider: external annotations: k8sd/v1alpha/lifecycle/skip-cleanup-kubernetes-node-on-remove: true + k8sd/v1alpha/lifecycle/skip-stop-services-on-remove: true control-plane-taints: - node-role.kubernetes.io/control-plane:NoSchedule pod-cidr: 10.100.0.0/16 diff --git a/src/k8s/cmd/k8sd/k8sd.go b/src/k8s/cmd/k8sd/k8sd.go index dedfafdf7..3fd28264a 100644 --- a/src/k8s/cmd/k8sd/k8sd.go +++ b/src/k8s/cmd/k8sd/k8sd.go @@ -91,7 +91,7 @@ func NewRootCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { addCommands( cmd, &cobra.Group{ID: "cluster", Title: "K8sd clustering commands:"}, - newClusterRecoverCmd(), + newClusterRecoverCmd(env), ) return cmd diff --git a/src/k8s/cmd/k8sd/k8sd_cluster_recover.go b/src/k8s/cmd/k8sd/k8sd_cluster_recover.go old mode 100755 new mode 100644 index 766acc49f..983aa52b2 --- a/src/k8s/cmd/k8sd/k8sd_cluster_recover.go +++ b/src/k8s/cmd/k8sd/k8sd_cluster_recover.go @@ -16,6 +16,9 @@ import ( "github.com/canonical/go-dqlite" "github.com/canonical/go-dqlite/app" "github.com/canonical/go-dqlite/client" + cmdutil "github.com/canonical/k8s/cmd/util" + "github.com/canonical/k8s/pkg/log" + "github.com/canonical/k8s/pkg/utils" "github.com/canonical/lxd/shared" "github.com/canonical/lxd/shared/termios" "github.com/canonical/microcluster/v3/cluster" @@ -23,12 +26,9 @@ import ( "github.com/spf13/cobra" "golang.org/x/sys/unix" "gopkg.in/yaml.v2" - - "github.com/canonical/k8s/pkg/log" - "github.com/canonical/k8s/pkg/utils" ) -const recoveryConfirmation = `You should only run this command if: +const preRecoveryMessage = `You should only run this command if: - A quorum of cluster members is permanently lost - You are *absolutely* sure all k8s daemons are stopped (sudo snap stop k8s) - This instance has the most up to date database @@ -36,8 +36,17 @@ const recoveryConfirmation = `You should only run this command if: Note that before applying any changes, a database backup is created at: * k8sd (microcluster): /var/snap/k8s/common/var/lib/k8sd/state/db_backup..tar.gz * k8s-dqlite: /var/snap/k8s/common/recovery-k8s-dqlite--pre-recovery.tar.gz +` -Do you want to proceed? (yes/no): ` +const recoveryConfirmation = "Do you want to proceed? (yes/no): " + +const nonInteractiveMessage = `Non-interactive mode requested. + +The command will assume that the dqlite configuration files have already been +modified with the updated cluster member roles and addresses. + +Initiating the dqlite database recovery. +` const clusterK8sdYamlRecoveryComment = `# Member roles can be modified. Unrecoverable nodes should be given the role "spare". # @@ -75,6 +84,7 @@ const yamlHelperCommentFooter = "# ------- everything below will be written ---- var clusterRecoverOpts struct { K8sDqliteStateDir string + NonInteractive bool SkipK8sd bool SkipK8sDqlite bool } @@ -87,29 +97,30 @@ func logDebugf(format string, args ...interface{}) { msg := fmt.Sprintf(format, args...) log.L().Info(msg) } - } -func newClusterRecoverCmd() *cobra.Command { +func newClusterRecoverCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { cmd := &cobra.Command{ Use: "cluster-recover", Short: "Recover the cluster from this member if quorum is lost", - RunE: func(cmd *cobra.Command, args []string) error { + Run: func(cmd *cobra.Command, args []string) { log.Configure(log.Options{ LogLevel: rootCmdOpts.logLevel, AddDirHeader: true, }) - if err := recoveryCmdPrechecks(cmd.Context()); err != nil { - return err + if err := recoveryCmdPrechecks(cmd); err != nil { + cmd.PrintErrf("Recovery precheck failed: %v\n", err) + env.Exit(1) } if clusterRecoverOpts.SkipK8sd { - cmd.Printf("Skipping k8sd recovery.") + cmd.Printf("Skipping k8sd recovery.\n") } else { k8sdTarballPath, err := recoverK8sd() if err != nil { - return fmt.Errorf("failed to recover k8sd, error: %w", err) + cmd.PrintErrf("Failed to recover k8sd, error: %v\n", err) + env.Exit(1) } cmd.Printf("K8sd cluster changes applied.\n") cmd.Printf("New database state saved to %s\n", k8sdTarballPath) @@ -120,15 +131,15 @@ func newClusterRecoverCmd() *cobra.Command { } if clusterRecoverOpts.SkipK8sDqlite { - cmd.Printf("Skipping k8s-dqlite recovery.") + cmd.Printf("Skipping k8s-dqlite recovery.\n") } else { k8sDqlitePreRecoveryTarball, k8sDqlitePostRecoveryTarball, err := recoverK8sDqlite() if err != nil { - return fmt.Errorf( - "failed to recover k8s-dqlite, error: %w, "+ - "pre-recovery backup: %s", - err, k8sDqlitePreRecoveryTarball, - ) + cmd.PrintErrf( + "Failed to recover k8s-dqlite, error: %v, "+ + "pre-recovery backup: %s\n", + err, k8sDqlitePreRecoveryTarball) + env.Exit(1) } cmd.Printf("K8s-dqlite cluster changes applied.\n") cmd.Printf("New database state saved to %s\n", @@ -138,13 +149,13 @@ func newClusterRecoverCmd() *cobra.Command { k8sDqlitePostRecoveryTarball, clusterRecoverOpts.K8sDqliteStateDir) cmd.Printf("Pre-recovery database backup: %s\n\n", k8sDqlitePreRecoveryTarball) } - - return nil }, } cmd.Flags().StringVar(&clusterRecoverOpts.K8sDqliteStateDir, "k8s-dqlite-state-dir", "", "k8s-dqlite datastore location") + cmd.Flags().BoolVar(&clusterRecoverOpts.NonInteractive, "non-interactive", + false, "disable interactive prompts, assume that the configs have been updated") cmd.Flags().BoolVar(&clusterRecoverOpts.SkipK8sd, "skip-k8sd", false, "skip k8sd recovery") cmd.Flags().BoolVar(&clusterRecoverOpts.SkipK8sDqlite, "skip-k8s-dqlite", @@ -166,13 +177,13 @@ func removeEmptyLines(content []byte) []byte { return out } -func recoveryCmdPrechecks(ctx context.Context) error { - log := log.FromContext(ctx) +func recoveryCmdPrechecks(cmd *cobra.Command) error { + log := log.FromContext(cmd.Context()) log.V(1).Info("Running prechecks.") - if !termios.IsTerminal(unix.Stdin) { - return fmt.Errorf("this command is meant to be run in an interactive terminal") + if !termios.IsTerminal(unix.Stdin) && !clusterRecoverOpts.NonInteractive { + return fmt.Errorf("interactive mode requested in a non-interactive terminal") } if clusterRecoverOpts.K8sDqliteStateDir == "" { @@ -182,21 +193,31 @@ func recoveryCmdPrechecks(ctx context.Context) error { return fmt.Errorf("k8sd state dir not specified") } - reader := bufio.NewReader(os.Stdin) - fmt.Print(recoveryConfirmation) + cmd.Print(preRecoveryMessage) + cmd.Print("\n") - input, err := reader.ReadString('\n') - if err != nil { - return fmt.Errorf("couldn't read user input, error: %w", err) - } - input = strings.TrimSuffix(input, "\n") + if clusterRecoverOpts.NonInteractive { + cmd.Print(nonInteractiveMessage) + cmd.Print("\n") + } else { + reader := bufio.NewReader(os.Stdin) + cmd.Print(recoveryConfirmation) + + input, err := reader.ReadString('\n') + if err != nil { + return fmt.Errorf("couldn't read user input, error: %w", err) + } + input = strings.TrimSuffix(input, "\n") - if strings.ToLower(input) != "yes" { - return fmt.Errorf("cluster edit aborted; no changes made") + if strings.ToLower(input) != "yes" { + return fmt.Errorf("cluster edit aborted; no changes made") + } + + cmd.Print("\n") } if !clusterRecoverOpts.SkipK8sDqlite { - if err = ensureK8sDqliteMembersStopped(ctx); err != nil { + if err := ensureK8sDqliteMembersStopped(cmd.Context()); err != nil { return err } } @@ -272,7 +293,7 @@ func ensureK8sDqliteMembersStopped(ctx context.Context) error { }(ctx, dial, member.Address) } - for _, _ = range members { + for range members { addr, ok := <-c if !ok { return fmt.Errorf("channel closed unexpectedly") @@ -325,7 +346,7 @@ func yamlEditorGuide( newContent = removeEmptyLines(newContent) if applyChanges { - err = os.WriteFile(path, newContent, os.FileMode(0o644)) + err = utils.WriteFile(path, newContent, os.FileMode(0o644)) if err != nil { return nil, fmt.Errorf("could not write file: %s, error: %w", path, err) } @@ -376,59 +397,64 @@ func recoverK8sd() (string, error) { clusterYamlPath := path.Join(m.FileSystem.DatabaseDir, "cluster.yaml") clusterYamlCommentHeader := fmt.Sprintf("# K8sd cluster configuration\n# (based on the trust store and %s)\n", clusterYamlPath) - clusterYamlContent, err := yamlEditorGuide( - "", - false, - slices.Concat( - []byte(clusterYamlCommentHeader), - []byte("#\n"), - []byte(clusterK8sdYamlRecoveryComment), - []byte(yamlHelperCommentFooter), - []byte("\n"), - oldMembersYaml, - ), - false, - ) - if err != nil { - return "", fmt.Errorf("interactive text editor failed, error: %w", err) - } + clusterYamlContent := oldMembersYaml + if !clusterRecoverOpts.NonInteractive { + // Interactive mode requested (default). + // Assist the user in configuring dqlite. + clusterYamlContent, err = yamlEditorGuide( + "", + false, + slices.Concat( + []byte(clusterYamlCommentHeader), + []byte("#\n"), + []byte(clusterK8sdYamlRecoveryComment), + []byte(yamlHelperCommentFooter), + []byte("\n"), + oldMembersYaml, + ), + false, + ) + if err != nil { + return "", fmt.Errorf("interactive text editor failed, error: %w", err) + } - infoYamlPath := path.Join(m.FileSystem.DatabaseDir, "info.yaml") - infoYamlCommentHeader := fmt.Sprintf("# K8sd info.yaml\n# (%s)\n", infoYamlPath) - _, err = yamlEditorGuide( - infoYamlPath, - true, - slices.Concat( - []byte(infoYamlCommentHeader), - []byte("#\n"), - []byte(infoYamlRecoveryComment), - utils.YamlCommentLines(clusterYamlContent), - []byte("\n"), - []byte(yamlHelperCommentFooter), - ), - true, - ) - if err != nil { - return "", fmt.Errorf("interactive text editor failed, error: %w", err) - } + infoYamlPath := path.Join(m.FileSystem.DatabaseDir, "info.yaml") + infoYamlCommentHeader := fmt.Sprintf("# K8sd info.yaml\n# (%s)\n", infoYamlPath) + _, err = yamlEditorGuide( + infoYamlPath, + true, + slices.Concat( + []byte(infoYamlCommentHeader), + []byte("#\n"), + []byte(infoYamlRecoveryComment), + utils.YamlCommentLines(clusterYamlContent), + []byte("\n"), + []byte(yamlHelperCommentFooter), + ), + true, + ) + if err != nil { + return "", fmt.Errorf("interactive text editor failed, error: %w", err) + } - daemonYamlPath := path.Join(m.FileSystem.StateDir, "daemon.yaml") - daemonYamlCommentHeader := fmt.Sprintf("# K8sd daemon.yaml\n# (%s)\n", daemonYamlPath) - _, err = yamlEditorGuide( - daemonYamlPath, - true, - slices.Concat( - []byte(daemonYamlCommentHeader), - []byte("#\n"), - []byte(daemonYamlRecoveryComment), - utils.YamlCommentLines(clusterYamlContent), - []byte("\n"), - []byte(yamlHelperCommentFooter), - ), - true, - ) - if err != nil { - return "", fmt.Errorf("interactive text editor failed, error: %w", err) + daemonYamlPath := path.Join(m.FileSystem.StateDir, "daemon.yaml") + daemonYamlCommentHeader := fmt.Sprintf("# K8sd daemon.yaml\n# (%s)\n", daemonYamlPath) + _, err = yamlEditorGuide( + daemonYamlPath, + true, + slices.Concat( + []byte(daemonYamlCommentHeader), + []byte("#\n"), + []byte(daemonYamlRecoveryComment), + utils.YamlCommentLines(clusterYamlContent), + []byte("\n"), + []byte(yamlHelperCommentFooter), + ), + true, + ) + if err != nil { + return "", fmt.Errorf("interactive text editor failed, error: %w", err) + } } newMembers := []cluster.DqliteMember{} @@ -465,40 +491,53 @@ func recoverK8sd() (string, error) { func recoverK8sDqlite() (string, string, error) { k8sDqliteStateDir := clusterRecoverOpts.K8sDqliteStateDir + var err error + clusterYamlContent := []byte{} clusterYamlPath := path.Join(k8sDqliteStateDir, "cluster.yaml") clusterYamlCommentHeader := fmt.Sprintf("# k8s-dqlite cluster configuration\n# (%s)\n", clusterYamlPath) - clusterYamlContent, err := yamlEditorGuide( - clusterYamlPath, - true, - slices.Concat( - []byte(clusterYamlCommentHeader), - []byte("#\n"), - []byte(clusterK8sDqliteRecoveryComment), - []byte(yamlHelperCommentFooter), - ), - true, - ) - if err != nil { - return "", "", fmt.Errorf("interactive text editor failed, error: %w", err) - } - infoYamlPath := path.Join(k8sDqliteStateDir, "info.yaml") - infoYamlCommentHeader := fmt.Sprintf("# k8s-dqlite info.yaml\n# (%s)\n", infoYamlPath) - _, err = yamlEditorGuide( - infoYamlPath, - true, - slices.Concat( - []byte(infoYamlCommentHeader), - []byte("#\n"), - []byte(infoYamlRecoveryComment), - utils.YamlCommentLines(clusterYamlContent), - []byte("\n"), - []byte(yamlHelperCommentFooter), - ), - true, - ) - if err != nil { - return "", "", fmt.Errorf("interactive text editor failed, error: %w", err) + if clusterRecoverOpts.NonInteractive { + clusterYamlContent, err = os.ReadFile(clusterYamlPath) + if err != nil { + return "", "", fmt.Errorf( + "could not read k8s-dqlite cluster.yaml, error: %w", err) + } + } else { + // Interactive mode requested (default). + // Assist the user in configuring dqlite. + clusterYamlContent, err = yamlEditorGuide( + clusterYamlPath, + true, + slices.Concat( + []byte(clusterYamlCommentHeader), + []byte("#\n"), + []byte(clusterK8sDqliteRecoveryComment), + []byte(yamlHelperCommentFooter), + ), + true, + ) + if err != nil { + return "", "", fmt.Errorf("interactive text editor failed, error: %w", err) + } + + infoYamlPath := path.Join(k8sDqliteStateDir, "info.yaml") + infoYamlCommentHeader := fmt.Sprintf("# k8s-dqlite info.yaml\n# (%s)\n", infoYamlPath) + _, err = yamlEditorGuide( + infoYamlPath, + true, + slices.Concat( + []byte(infoYamlCommentHeader), + []byte("#\n"), + []byte(infoYamlRecoveryComment), + utils.YamlCommentLines(clusterYamlContent), + []byte("\n"), + []byte(yamlHelperCommentFooter), + ), + true, + ) + if err != nil { + return "", "", fmt.Errorf("interactive text editor failed, error: %w", err) + } } newMembers := []dqlite.NodeInfo{} diff --git a/src/k8s/cmd/main.go b/src/k8s/cmd/main.go index 578449308..5e4b38bf1 100644 --- a/src/k8s/cmd/main.go +++ b/src/k8s/cmd/main.go @@ -30,14 +30,26 @@ func main() { // choose command based on the binary name base := filepath.Base(os.Args[0]) + var err error switch base { case "k8s-apiserver-proxy": - k8s_apiserver_proxy.NewRootCmd(env).ExecuteContext(ctx) + err = k8s_apiserver_proxy.NewRootCmd(env).ExecuteContext(ctx) case "k8sd": - k8sd.NewRootCmd(env).ExecuteContext(ctx) + err = k8sd.NewRootCmd(env).ExecuteContext(ctx) case "k8s": - k8s.NewRootCmd(env).ExecuteContext(ctx) + err = k8s.NewRootCmd(env).ExecuteContext(ctx) default: panic(fmt.Errorf("invalid entrypoint name %q", base)) } + + // Although k8s commands typically use Run instead of RunE and handle + // errors directly within the command execution, this acts as a safeguard in + // case any are overlooked. + // + // Furthermore, the Cobra framework may not invoke the "Run*" entry points + // at all in case of argument parsing errors, in which case we *need* to + // handle the errors here. + if err != nil { + env.Exit(1) + } } diff --git a/src/k8s/cmd/util/hooks.go b/src/k8s/cmd/util/hooks.go new file mode 100644 index 000000000..6069a8f12 --- /dev/null +++ b/src/k8s/cmd/util/hooks.go @@ -0,0 +1,56 @@ +package cmdutil + +import ( + "fmt" + "os" + "strings" + "syscall" +) + +// getFileOwnerAndGroup retrieves the UID and GID of a file. +func getFileOwnerAndGroup(filePath string) (uid, gid uint32, err error) { + // Get file info using os.Stat + fileInfo, err := os.Stat(filePath) + if err != nil { + return 0, 0, fmt.Errorf("error getting file info: %w", err) + } + // Convert the fileInfo.Sys() to syscall.Stat_t to access UID and GID + stat, ok := fileInfo.Sys().(*syscall.Stat_t) + if !ok { + return 0, 0, fmt.Errorf("failed to cast to syscall.Stat_t") + } + // Return the UID and GID + return stat.Uid, stat.Gid, nil +} + +// ValidateRootOwnership checks if the specified path is owned by the root user and root group. +func ValidateRootOwnership(path string) (err error) { + uid, gid, err := getFileOwnerAndGroup(path) + if err != nil { + return err + } + if uid != 0 { + return fmt.Errorf("owner of %s is user with UID %d expected 0", path, uid) + } + if gid != 0 { + return fmt.Errorf("owner of %s is group with GID %d expected 0", path, gid) + } + return nil +} + +// InLXDContainer checks if k8s runs in a lxd container. +func InLXDContainer() (isLXD bool, err error) { + initialProcessEnvironmentVariables := "/proc/1/environ" + content, err := os.ReadFile(initialProcessEnvironmentVariables) + if err != nil { + // if the permission to file is missing we still want to display info about lxd + if os.IsPermission(err) { + return true, fmt.Errorf("cannnot access %s to check if runing in LXD container: %w", initialProcessEnvironmentVariables, err) + } + return false, fmt.Errorf("cannnot read %s to check if runing in LXD container: %w", initialProcessEnvironmentVariables, err) + } + if strings.Contains(string(content), "container=lxc") { + return true, nil + } + return false, nil +} diff --git a/src/k8s/go.mod b/src/k8s/go.mod index 57164d98b..68d3f1cde 100644 --- a/src/k8s/go.mod +++ b/src/k8s/go.mod @@ -5,7 +5,7 @@ go 1.22.6 require ( dario.cat/mergo v1.0.0 github.com/canonical/go-dqlite v1.22.0 - github.com/canonical/k8s-snap-api v1.0.5 + github.com/canonical/k8s-snap-api v1.0.13 github.com/canonical/lxd v0.0.0-20240822122218-e7b2a7a83230 github.com/canonical/microcluster/v3 v3.0.0-20240827143335-f7a4d3984970 github.com/go-logr/logr v1.4.2 @@ -14,6 +14,7 @@ require ( github.com/onsi/gomega v1.32.0 github.com/pelletier/go-toml v1.9.5 github.com/spf13/cobra v1.8.1 + golang.org/x/mod v0.20.0 golang.org/x/net v0.28.0 golang.org/x/sync v0.8.0 golang.org/x/sys v0.24.0 diff --git a/src/k8s/go.sum b/src/k8s/go.sum index 0a695e62d..20f296a62 100644 --- a/src/k8s/go.sum +++ b/src/k8s/go.sum @@ -99,8 +99,8 @@ github.com/bugsnag/panicwrap v0.0.0-20151223152923-e2c28503fcd0 h1:nvj0OLI3YqYXe github.com/bugsnag/panicwrap v0.0.0-20151223152923-e2c28503fcd0/go.mod h1:D/8v3kj0zr8ZAKg1AQ6crr+5VwKN5eIywRkfhyM/+dE= github.com/canonical/go-dqlite v1.22.0 h1:DuJmfcREl4gkQJyvZzjl2GHFZROhbPyfdjDRQXpkOyw= github.com/canonical/go-dqlite v1.22.0/go.mod h1:Uvy943N8R4CFUAs59A1NVaziWY9nJ686lScY7ywurfg= -github.com/canonical/k8s-snap-api v1.0.5 h1:49bgi6CGtFjCPweeTz55Sv/waKgCl6ftx4BqXt3RI9k= -github.com/canonical/k8s-snap-api v1.0.5/go.mod h1:LDPoIYCeYnfgOFrwVPJ/4edGU264w7BB7g0GsVi36AY= +github.com/canonical/k8s-snap-api v1.0.13 h1:Z+IW6Knvycu+DrkmH+9qB1UNyYiHfL+rFvT9DtSO2+g= +github.com/canonical/k8s-snap-api v1.0.13/go.mod h1:LDPoIYCeYnfgOFrwVPJ/4edGU264w7BB7g0GsVi36AY= github.com/canonical/lxd v0.0.0-20240822122218-e7b2a7a83230 h1:YOqZ+/14OPZ+/TOXpRHIX3KLT0C+wZVpewKIwlGUmW0= github.com/canonical/lxd v0.0.0-20240822122218-e7b2a7a83230/go.mod h1:YVGI7HStOKsV+cMyXWnJ7RaMPaeWtrkxyIPvGWbgACc= github.com/canonical/microcluster/v3 v3.0.0-20240827143335-f7a4d3984970 h1:UrnpglbXELlxtufdk6DGDytu2JzyzuS3WTsOwPrkQLI= diff --git a/src/k8s/hack/env.sh b/src/k8s/hack/env.sh index 66af208c7..60d0d9865 100755 --- a/src/k8s/hack/env.sh +++ b/src/k8s/hack/env.sh @@ -2,7 +2,7 @@ ## Component repositories REPO_MUSL="https://git.launchpad.net/musl" -REPO_LIBTIRPC="https://salsa.debian.org/debian/libtirpc.git" +REPO_LIBTIRPC="https://git.launchpad.net/libtirpc" REPO_LIBNSL="https://github.com/thkukuk/libnsl.git" REPO_LIBUV="https://github.com/libuv/libuv.git" REPO_LIBLZ4="https://github.com/lz4/lz4.git" diff --git a/src/k8s/pkg/client/dqlite/remove_test.go b/src/k8s/pkg/client/dqlite/remove_test.go index a2f316a65..8252add83 100644 --- a/src/k8s/pkg/client/dqlite/remove_test.go +++ b/src/k8s/pkg/client/dqlite/remove_test.go @@ -16,21 +16,21 @@ func TestRemoveNodeByAddress(t *testing.T) { client, err := dqlite.NewClient(ctx, dqlite.ClientOpts{ ClusterYAML: filepath.Join(dirs[0], "cluster.yaml"), }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(client).NotTo(BeNil()) members, err := client.ListMembers(ctx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(members).To(HaveLen(2)) memberToRemove := members[0].Address if members[0].Role == dqlite.Voter { memberToRemove = members[1].Address } - g.Expect(client.RemoveNodeByAddress(ctx, memberToRemove)).To(BeNil()) + g.Expect(client.RemoveNodeByAddress(ctx, memberToRemove)).To(Succeed()) members, err = client.ListMembers(ctx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(members).To(HaveLen(1)) }) }) @@ -41,11 +41,11 @@ func TestRemoveNodeByAddress(t *testing.T) { client, err := dqlite.NewClient(ctx, dqlite.ClientOpts{ ClusterYAML: filepath.Join(dirs[0], "cluster.yaml"), }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(client).NotTo(BeNil()) members, err := client.ListMembers(ctx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(members).To(HaveLen(2)) memberToRemove := members[0] @@ -61,7 +61,7 @@ func TestRemoveNodeByAddress(t *testing.T) { g.Expect(client.RemoveNodeByAddress(ctx, memberToRemove.Address)).To(Succeed()) members, err = client.ListMembers(ctx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(members).To(HaveLen(1)) g.Expect(members[0].Role).To(Equal(dqlite.Voter)) g.Expect(members[0].Address).ToNot(Equal(memberToRemove.Address)) diff --git a/src/k8s/pkg/client/dqlite/util_test.go b/src/k8s/pkg/client/dqlite/util_test.go index b1d8084f1..83cea9d63 100644 --- a/src/k8s/pkg/client/dqlite/util_test.go +++ b/src/k8s/pkg/client/dqlite/util_test.go @@ -26,7 +26,7 @@ var nextDqlitePort = 37312 // }) // } // -// ``` +// ```. func withDqliteCluster(t *testing.T, size int, f func(ctx context.Context, dirs []string)) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() diff --git a/src/k8s/pkg/client/helm/client.go b/src/k8s/pkg/client/helm/client.go index abe6c5da6..faf2ad031 100644 --- a/src/k8s/pkg/client/helm/client.go +++ b/src/k8s/pkg/client/helm/client.go @@ -4,6 +4,7 @@ import ( "bytes" "context" "encoding/json" + "errors" "fmt" "path/filepath" @@ -57,7 +58,7 @@ func (h *client) Apply(ctx context.Context, c InstallableChart, desired State, v get := action.NewGet(cfg) release, err := get.Run(c.Name) if err != nil { - if err != driver.ErrReleaseNotFound { + if !errors.Is(err, driver.ErrReleaseNotFound) { return false, fmt.Errorf("failed to get status of release %s: %w", c.Name, err) } isInstalled = false @@ -93,7 +94,7 @@ func (h *client) Apply(ctx context.Context, c InstallableChart, desired State, v // there is already a release installed, so we must run an upgrade action upgrade := action.NewUpgrade(cfg) upgrade.Namespace = c.Namespace - upgrade.ReuseValues = true + upgrade.ResetThenReuseValues = true chart, err := loader.Load(filepath.Join(h.manifestsBaseDir, c.ManifestPath)) if err != nil { diff --git a/src/k8s/pkg/client/helm/mock/mock.go b/src/k8s/pkg/client/helm/mock/mock.go index 6eea2d797..52a0f2f45 100644 --- a/src/k8s/pkg/client/helm/mock/mock.go +++ b/src/k8s/pkg/client/helm/mock/mock.go @@ -13,14 +13,14 @@ type MockApplyArguments struct { Values map[string]any } -// Mock is a mock implementation of helm.Client +// Mock is a mock implementation of helm.Client. type Mock struct { ApplyCalledWith []MockApplyArguments ApplyChanged bool ApplyErr error } -// Apply implements helm.Client +// Apply implements helm.Client. func (m *Mock) Apply(ctx context.Context, c helm.InstallableChart, desired helm.State, values map[string]any) (bool, error) { m.ApplyCalledWith = append(m.ApplyCalledWith, MockApplyArguments{Context: ctx, Chart: c, State: desired, Values: values}) return m.ApplyChanged, m.ApplyErr diff --git a/src/k8s/pkg/client/k8sd/mock/mock.go b/src/k8s/pkg/client/k8sd/mock/mock.go index d306e1a95..62915eb5b 100644 --- a/src/k8s/pkg/client/k8sd/mock/mock.go +++ b/src/k8s/pkg/client/k8sd/mock/mock.go @@ -56,14 +56,17 @@ func (m *Mock) BootstrapCluster(_ context.Context, request apiv1.BootstrapCluste m.BootstrapClusterCalledWith = request return m.BootstrapClusterResponse, m.BootstrapClusterErr } + func (m *Mock) GetJoinToken(_ context.Context, request apiv1.GetJoinTokenRequest) (apiv1.GetJoinTokenResponse, error) { m.GetJoinTokenCalledWith = request return m.GetJoinTokenResponse, m.GetJoinTokenErr } + func (m *Mock) JoinCluster(_ context.Context, request apiv1.JoinClusterRequest) error { m.JoinClusterCalledWith = request return m.JoinClusterErr } + func (m *Mock) RemoveNode(_ context.Context, request apiv1.RemoveNodeRequest) error { m.RemoveNodeCalledWith = request return m.RemoveNodeErr @@ -72,6 +75,7 @@ func (m *Mock) RemoveNode(_ context.Context, request apiv1.RemoveNodeRequest) er func (m *Mock) NodeStatus(_ context.Context) (apiv1.NodeStatusResponse, bool, error) { return m.NodeStatusResponse, m.NodeStatusInitialized, m.NodeStatusErr } + func (m *Mock) ClusterStatus(_ context.Context, waitReady bool) (apiv1.ClusterStatusResponse, error) { return m.ClusterStatusResponse, m.ClusterStatusErr } @@ -87,6 +91,7 @@ func (m *Mock) RefreshCertificatesRun(_ context.Context, request apiv1.RefreshCe func (m *Mock) GetClusterConfig(_ context.Context) (apiv1.GetClusterConfigResponse, error) { return m.GetClusterConfigResponse, m.GetClusterConfigErr } + func (m *Mock) SetClusterConfig(_ context.Context, request apiv1.SetClusterConfigRequest) error { m.SetClusterConfigCalledWith = request return m.SetClusterConfigErr diff --git a/src/k8s/pkg/client/kubernetes/configmap_test.go b/src/k8s/pkg/client/kubernetes/configmap_test.go index ea11d6aba..55b8ce713 100644 --- a/src/k8s/pkg/client/kubernetes/configmap_test.go +++ b/src/k8s/pkg/client/kubernetes/configmap_test.go @@ -79,7 +79,6 @@ func TestWatchConfigMap(t *testing.T) { case <-time.After(time.Second): t.Fatal("Timed out waiting for watch to complete") } - }) } } diff --git a/src/k8s/pkg/client/kubernetes/endpoints.go b/src/k8s/pkg/client/kubernetes/endpoints.go index 3d6709800..0840effb9 100644 --- a/src/k8s/pkg/client/kubernetes/endpoints.go +++ b/src/k8s/pkg/client/kubernetes/endpoints.go @@ -5,6 +5,7 @@ import ( "fmt" "sort" + "github.com/canonical/k8s/pkg/utils" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/util/retry" @@ -40,7 +41,13 @@ func (c *Client) GetKubeAPIServerEndpoints(ctx context.Context) ([]string, error } for _, addr := range subset.Addresses { if addr.IP != "" { - addresses = append(addresses, fmt.Sprintf("%s:%d", addr.IP, portNumber)) + var address string + if utils.IsIPv4(addr.IP) { + address = addr.IP + } else { + address = fmt.Sprintf("[%s]", addr.IP) + } + addresses = append(addresses, fmt.Sprintf("%s:%d", address, portNumber)) } } } diff --git a/src/k8s/pkg/client/kubernetes/endpoints_test.go b/src/k8s/pkg/client/kubernetes/endpoints_test.go index 238886aa6..d750829a5 100644 --- a/src/k8s/pkg/client/kubernetes/endpoints_test.go +++ b/src/k8s/pkg/client/kubernetes/endpoints_test.go @@ -109,7 +109,7 @@ func TestGetKubeAPIServerEndpoints(t *testing.T) { g.Expect(err).To(HaveOccurred()) g.Expect(servers).To(BeEmpty()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(servers).To(Equal(tc.expectedAddresses)) } }) diff --git a/src/k8s/pkg/client/kubernetes/node_test.go b/src/k8s/pkg/client/kubernetes/node_test.go index 55165e3c4..22ab79a1b 100644 --- a/src/k8s/pkg/client/kubernetes/node_test.go +++ b/src/k8s/pkg/client/kubernetes/node_test.go @@ -28,7 +28,7 @@ func TestDeleteNode(t *testing.T) { }, metav1.CreateOptions{}) err := client.DeleteNode(context.Background(), nodeName) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("node does not exist is successful", func(t *testing.T) { @@ -37,7 +37,7 @@ func TestDeleteNode(t *testing.T) { nodeName := "test-node" err := client.DeleteNode(context.Background(), nodeName) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("node deletion fails", func(t *testing.T) { diff --git a/src/k8s/pkg/client/kubernetes/pods_test.go b/src/k8s/pkg/client/kubernetes/pods_test.go index 5f3922e1f..b10fe76c7 100644 --- a/src/k8s/pkg/client/kubernetes/pods_test.go +++ b/src/k8s/pkg/client/kubernetes/pods_test.go @@ -98,7 +98,7 @@ func TestCheckForReadyPods(t *testing.T) { err := client.CheckForReadyPods(context.Background(), tc.namespace, tc.listOptions) if tc.expectedError == "" { - g.Expect(err).Should(BeNil()) + g.Expect(err).ShouldNot(HaveOccurred()) } else { g.Expect(err).Should(MatchError(tc.expectedError)) } diff --git a/src/k8s/pkg/client/kubernetes/restart_daemonset_test.go b/src/k8s/pkg/client/kubernetes/restart_daemonset_test.go index 55ecc8061..3f88b6b21 100644 --- a/src/k8s/pkg/client/kubernetes/restart_daemonset_test.go +++ b/src/k8s/pkg/client/kubernetes/restart_daemonset_test.go @@ -66,9 +66,9 @@ func TestRestartDaemonset(t *testing.T) { if tc.expectError { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) ds, err := client.AppsV1().DaemonSets("namespace").Get(context.Background(), "test", metav1.GetOptions{}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ds.Spec.Template.Annotations["kubectl.kubernetes.io/restartedAt"]).NotTo(BeEmpty()) } }) diff --git a/src/k8s/pkg/client/kubernetes/restart_deployment_test.go b/src/k8s/pkg/client/kubernetes/restart_deployment_test.go index 2bc7aa9d7..cdb59608d 100644 --- a/src/k8s/pkg/client/kubernetes/restart_deployment_test.go +++ b/src/k8s/pkg/client/kubernetes/restart_deployment_test.go @@ -66,9 +66,9 @@ func TestRestartDeployment(t *testing.T) { if tc.expectError { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) deploy, err := client.AppsV1().Deployments("namespace").Get(context.Background(), "test", metav1.GetOptions{}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(deploy.Spec.Template.Annotations["kubectl.kubernetes.io/restartedAt"]).NotTo(BeEmpty()) } }) diff --git a/src/k8s/pkg/client/kubernetes/server_groups.go b/src/k8s/pkg/client/kubernetes/server_groups.go index cb58b9d0c..a3581d7a8 100644 --- a/src/k8s/pkg/client/kubernetes/server_groups.go +++ b/src/k8s/pkg/client/kubernetes/server_groups.go @@ -6,7 +6,7 @@ import ( v1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) -// ListResourcesForGroupVersion lists the resources for a given group version (e.g. "cilium.io/v2alpha1") +// ListResourcesForGroupVersion lists the resources for a given group version (e.g. "cilium.io/v2alpha1"). func (c *Client) ListResourcesForGroupVersion(groupVersion string) (*v1.APIResourceList, error) { resources, err := c.Discovery().ServerResourcesForGroupVersion(groupVersion) if err != nil { diff --git a/src/k8s/pkg/client/kubernetes/server_groups_test.go b/src/k8s/pkg/client/kubernetes/server_groups_test.go index 3126442a5..028f311e6 100644 --- a/src/k8s/pkg/client/kubernetes/server_groups_test.go +++ b/src/k8s/pkg/client/kubernetes/server_groups_test.go @@ -3,12 +3,11 @@ package kubernetes_test import ( "testing" - fakediscovery "k8s.io/client-go/discovery/fake" - fakeclientset "k8s.io/client-go/kubernetes/fake" - "github.com/canonical/k8s/pkg/client/kubernetes" . "github.com/onsi/gomega" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + fakediscovery "k8s.io/client-go/discovery/fake" + fakeclientset "k8s.io/client-go/kubernetes/fake" ) func TestListResourcesForGroupVersion(t *testing.T) { @@ -59,7 +58,7 @@ func TestListResourcesForGroupVersion(t *testing.T) { if tt.expectedError { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(resources).To(Equal(tt.expectedList)) } }) diff --git a/src/k8s/pkg/client/kubernetes/status.go b/src/k8s/pkg/client/kubernetes/status.go index e98f87778..c89646c43 100644 --- a/src/k8s/pkg/client/kubernetes/status.go +++ b/src/k8s/pkg/client/kubernetes/status.go @@ -34,9 +34,8 @@ func (c *Client) CheckKubernetesEndpoint(ctx context.Context) error { // HasReadyNodes returns true if there is at least one Ready node in the cluster, false otherwise. func (c *Client) HasReadyNodes(ctx context.Context) (bool, error) { nodes, err := c.CoreV1().Nodes().List(ctx, metav1.ListOptions{}) - if err != nil { - return false, fmt.Errorf("failed to list nodes: %v", err) + return false, fmt.Errorf("failed to list nodes: %w", err) } for _, node := range nodes.Items { diff --git a/src/k8s/pkg/client/kubernetes/status_test.go b/src/k8s/pkg/client/kubernetes/status_test.go index 7dae3e115..64b4bc063 100644 --- a/src/k8s/pkg/client/kubernetes/status_test.go +++ b/src/k8s/pkg/client/kubernetes/status_test.go @@ -93,7 +93,7 @@ func TestClusterHasReadyNodes(t *testing.T) { ready, err := client.HasReadyNodes(context.Background()) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ready).To(Equal(tt.expectedReady)) }) } diff --git a/src/k8s/pkg/client/snapd/refresh_status.go b/src/k8s/pkg/client/snapd/refresh_status.go index e2feb37a7..c6fa55581 100644 --- a/src/k8s/pkg/client/snapd/refresh_status.go +++ b/src/k8s/pkg/client/snapd/refresh_status.go @@ -17,15 +17,16 @@ func (c *Client) GetRefreshStatus(changeID string) (*types.RefreshStatus, error) if err != nil { return nil, fmt.Errorf("failed to get snapd change status: %w", err) } + defer resp.Body.Close() resBody, err := io.ReadAll(resp.Body) if err != nil { - return nil, fmt.Errorf("client: could not read response body: %s", err) + return nil, fmt.Errorf("client: could not read response body: %w", err) } var changeResponse snapdChangeResponse if err := json.Unmarshal(resBody, &changeResponse); err != nil { - return nil, fmt.Errorf("client: could not unmarshal response body: %s", err) + return nil, fmt.Errorf("client: could not unmarshal response body: %w", err) } return &changeResponse.Result, nil diff --git a/src/k8s/pkg/client/snapd/snap_info.go b/src/k8s/pkg/client/snapd/snap_info.go new file mode 100644 index 000000000..b09f61ec4 --- /dev/null +++ b/src/k8s/pkg/client/snapd/snap_info.go @@ -0,0 +1,41 @@ +package snapd + +import ( + "encoding/json" + "fmt" + "io" + "time" +) + +type SnapInfoResult struct { + InstallDate time.Time `json:"install-date"` +} + +type SnapInfoResponse struct { + StatusCode int `json:"status-code"` + Result SnapInfoResult `json:"result"` +} + +func (c *Client) GetSnapInfo(snap string) (*SnapInfoResponse, error) { + resp, err := c.client.Get(fmt.Sprintf("http://localhost/v2/snaps/%s", snap)) + if err != nil { + return nil, fmt.Errorf("failed to get snapd snap info: %w", err) + } + defer resp.Body.Close() + + resBody, err := io.ReadAll(resp.Body) + if err != nil { + return nil, fmt.Errorf("client: could not read response body: %w", err) + } + + var snapInfoResponse SnapInfoResponse + if err := json.Unmarshal(resBody, &snapInfoResponse); err != nil { + return nil, fmt.Errorf("client: could not unmarshal response body: %w", err) + } + + return &snapInfoResponse, nil +} + +func (s SnapInfoResponse) HasInstallDate() bool { + return !s.Result.InstallDate.IsZero() +} diff --git a/src/k8s/pkg/docgen/godoc.go b/src/k8s/pkg/docgen/godoc.go new file mode 100755 index 000000000..eaa50c572 --- /dev/null +++ b/src/k8s/pkg/docgen/godoc.go @@ -0,0 +1,130 @@ +package docgen + +import ( + "fmt" + "go/ast" + "go/doc" + "go/parser" + "go/token" + "reflect" +) + +var packageDocCache = make(map[string]*doc.Package) + +func findTypeSpec(decl *ast.GenDecl, symbol string) (*ast.TypeSpec, error) { + for _, spec := range decl.Specs { + typeSpec, ok := spec.(*ast.TypeSpec) + if !ok { + return nil, fmt.Errorf("spec is not *ast.TypeSpec") + } + if symbol == typeSpec.Name.Name { + return typeSpec, nil + } + } + return nil, nil +} + +func getStructTypeFromDoc(packageDoc *doc.Package, structName string) (*ast.StructType, error) { + for _, docType := range packageDoc.Types { + if structName != docType.Name { + continue + } + typeSpec, err := findTypeSpec(docType.Decl, docType.Name) + if err != nil { + return nil, fmt.Errorf("failed to find type spec: %w", err) + } + if typeSpec == nil { + continue + } + structType, ok := typeSpec.Type.(*ast.StructType) + if !ok { + // Not a structure. + continue + } + return structType, nil + } + return nil, nil +} + +func parsePackageDir(packageDir string) (*ast.Package, error) { + fset := token.NewFileSet() + packages, err := parser.ParseDir(fset, packageDir, nil, parser.ParseComments) + if err != nil { + return nil, fmt.Errorf("couldn't parse go package: %s", packageDir) + } + + if len(packages) == 0 { + return nil, fmt.Errorf("no go package found: %s", packageDir) + } + if len(packages) > 1 { + return nil, fmt.Errorf("multiple go package found: %s", packageDir) + } + + // We have a map containing a single entry and we need to return it. + for _, pkg := range packages { + return pkg, nil + } + + // shouldn't really get here. + return nil, fmt.Errorf("failed to parse go package") +} + +func getAstStructField(structType *ast.StructType, fieldName string) (*ast.Field, error) { + for _, field := range structType.Fields.List { + for _, fieldIdent := range field.Names { + if fieldIdent.Name == fieldName { + return field, nil + } + } + } + return nil, nil +} + +func getPackageDoc(packagePath string, projectDir string) (*doc.Package, error) { + packageDoc, found := packageDocCache[packagePath] + if found { + return packageDoc, nil + } + + packageDir, err := getGoPackageDir(packagePath, projectDir) + if err != nil { + return nil, fmt.Errorf("failed to retrieve package dir: %w", err) + } + + pkg, err := parsePackageDir(packageDir) + if err != nil { + return nil, fmt.Errorf("failed to parse package dir: %w", err) + } + + packageDoc = doc.New(pkg, packageDir, doc.AllDecls|doc.PreserveAST) + packageDocCache[packagePath] = packageDoc + + return packageDoc, nil +} + +func getFieldDocstring(i any, field reflect.StructField, projectDir string) (string, error) { + inType := reflect.TypeOf(i) + + packageDoc, err := getPackageDoc(inType.PkgPath(), projectDir) + if err != nil { + return "", fmt.Errorf("failed to retrieve package doc: %w", err) + } + + structType, err := getStructTypeFromDoc(packageDoc, inType.Name()) + if err != nil { + return "", fmt.Errorf("failed to retrieve struct type: %w", err) + } + if structType == nil { + return "", fmt.Errorf("could not find %s structure definition", inType.Name()) + } + + astField, err := getAstStructField(structType, field.Name) + if err != nil { + return "", fmt.Errorf("failed to retrieve struct field: %w", err) + } + if astField == nil { + return "", fmt.Errorf("could not find %s.%s field definition", inType.Name(), field.Name) + } + + return astField.Doc.Text(), nil +} diff --git a/src/k8s/pkg/docgen/gomod.go b/src/k8s/pkg/docgen/gomod.go new file mode 100755 index 000000000..f51ec2e52 --- /dev/null +++ b/src/k8s/pkg/docgen/gomod.go @@ -0,0 +1,100 @@ +package docgen + +import ( + "fmt" + "os" + "path" + "strings" + + "golang.org/x/mod/modfile" + "golang.org/x/mod/module" +) + +func getGoDepModulePath(name string, version string) (string, error) { + cachePath := os.Getenv("GOMODCACHE") + if cachePath == "" { + goPath := os.Getenv("GOPATH") + if goPath == "" { + goPath = path.Join(os.Getenv("HOME"), "/go") + } + cachePath = path.Join(goPath, "pkg", "mod") + } + + escapedPath, err := module.EscapePath(name) + if err != nil { + return "", fmt.Errorf( + "couldn't escape module path %s: %w", name, err) + } + + escapedVersion, err := module.EscapeVersion(version) + if err != nil { + return "", fmt.Errorf( + "couldn't escape module version %s: %w", version, err) + } + + path := path.Join(cachePath, escapedPath+"@"+escapedVersion) + + // Validate the path. + if _, err := os.Stat(path); err != nil { + return "", fmt.Errorf( + "go module path not accessible %s %s %s: %w", + name, version, path, err) + } + + return path, nil +} + +func getDependencyVersionFromGoMod(goModPath string, packageName string, directOnly bool) (string, string, error) { + goModContents, err := os.ReadFile(goModPath) + if err != nil { + return "", "", fmt.Errorf("could not read go.mod file %s: %w", goModPath, err) + } + goModFile, err := modfile.ParseLax(goModPath, goModContents, nil) + if err != nil { + return "", "", fmt.Errorf("could not parse go.mod file %s: %w", goModPath, err) + } + + for _, dep := range goModFile.Require { + if directOnly && dep.Indirect { + continue + } + if strings.HasPrefix(packageName, dep.Mod.Path) { + return dep.Mod.Path, dep.Mod.Version, nil + } + } + + return "", "", fmt.Errorf("could not find dependency %s in %s", packageName, goModPath) +} + +func getGoModPath(projectDir string) (string, error) { + return path.Join(projectDir, "go.mod"), nil +} + +func getGoPackageDir(packageName string, projectDir string) (string, error) { + if packageName == "" { + return "", fmt.Errorf("could not retrieve package dir, no package name specified.") + } + + if strings.HasPrefix(packageName, "github.com/canonical/k8s/") { + return strings.Replace(packageName, "github.com/canonical/k8s", projectDir, 1), nil + } + + // Dependency, need to retrieve its version from go.mod. + goModPath, err := getGoModPath(projectDir) + if err != nil { + return "", err + } + + basePackageName, version, err := getDependencyVersionFromGoMod(goModPath, packageName, false) + if err != nil { + return "", err + } + + basePath, err := getGoDepModulePath(basePackageName, version) + if err != nil { + return "", err + } + + subPath := strings.TrimPrefix(packageName, basePackageName) + return path.Join(basePath, subPath), nil +} diff --git a/src/k8s/pkg/docgen/json_struct.go b/src/k8s/pkg/docgen/json_struct.go new file mode 100644 index 000000000..5dc5e5a67 --- /dev/null +++ b/src/k8s/pkg/docgen/json_struct.go @@ -0,0 +1,139 @@ +package docgen + +import ( + "fmt" + "os" + "reflect" + "strings" + + "github.com/canonical/k8s/pkg/utils" +) + +type JsonTag struct { + Name string + Options []string +} + +type Field struct { + Name string + TypeName string + JsonTag JsonTag + FullJsonPath string + Docstring string +} + +// Generate Markdown documentation for a JSON or YAML based on +// the Go structure definition, parsing field annotations. +func MarkdownFromJsonStruct(i any, projectDir string) (string, error) { + fields, err := ParseStruct(i, projectDir) + if err != nil { + return "", err + } + + entryTemplate := `### %s +**Type:** ` + "`%s`" + `
+ +%s +` + + var out strings.Builder + for _, field := range fields { + outFieldType := strings.ReplaceAll(field.TypeName, "*", "") + entry := fmt.Sprintf(entryTemplate, field.FullJsonPath, outFieldType, field.Docstring) + out.WriteString(entry) + } + + return out.String(), nil +} + +// Generate Markdown documentation for a JSON or YAML based on +// the Go structure definition, parsing field annotations. +// Write the output to the specified file path. +// The project dir is used to parse the source code and identify dependencies +// based on the go.mod file. +func MarkdownFromJsonStructToFile(i any, outFilePath string, projectDir string) error { + content, err := MarkdownFromJsonStruct(i, projectDir) + if err != nil { + return err + } + + err = utils.WriteFile(outFilePath, []byte(content), 0o644) + if err != nil { + return fmt.Errorf("failed to write markdown documentation to %s: %w", outFilePath, err) + } + return nil +} + +func getJsonTag(field reflect.StructField) JsonTag { + jsonTag := JsonTag{} + + jsonTagStr := field.Tag.Get("json") + if jsonTagStr == "" { + // Use yaml tags as fallback, which have the same format. + jsonTagStr = field.Tag.Get("yaml") + } + if jsonTagStr != "" { + jsonTagSlice := strings.Split(jsonTagStr, ",") + if len(jsonTagSlice) > 0 { + jsonTag.Name = jsonTagSlice[0] + } + if len(jsonTagSlice) > 1 { + jsonTag.Options = jsonTagSlice[1:] + } + } + + return jsonTag +} + +func ParseStruct(i any, projectDir string) ([]Field, error) { + inType := reflect.TypeOf(i) + + if inType.Kind() != reflect.Struct { + return nil, fmt.Errorf("structure parsing failed, not a structure: %s", inType.Name()) + } + + outFields := []Field{} + fields := reflect.VisibleFields(inType) + for _, field := range fields { + jsonTag := getJsonTag(field) + docstring, err := getFieldDocstring(i, field, projectDir) + if err != nil { + fmt.Fprintf(os.Stderr, "WARNING: could not retrieve field docstring: %s.%s, error: %v", + inType.Name(), field.Name, err) + } + + if field.Type.Kind() == reflect.Struct { + fieldIface := reflect.ValueOf(i).FieldByName(field.Name).Interface() + nestedFields, err := ParseStruct(fieldIface, projectDir) + if err != nil { + return nil, fmt.Errorf("couldn't parse %s.%s: %w", inType, field.Name, err) + } + + outField := Field{ + Name: field.Name, + TypeName: "object", + JsonTag: jsonTag, + FullJsonPath: jsonTag.Name, + Docstring: docstring, + } + outFields = append(outFields, outField) + + for _, nestedField := range nestedFields { + // Update the json paths of the nested fields based on the field name. + nestedField.FullJsonPath = jsonTag.Name + "." + nestedField.FullJsonPath + outFields = append(outFields, nestedField) + } + } else { + outField := Field{ + Name: field.Name, + TypeName: field.Type.String(), + JsonTag: jsonTag, + FullJsonPath: jsonTag.Name, + Docstring: docstring, + } + outFields = append(outFields, outField) + } + } + + return outFields, nil +} diff --git a/src/k8s/pkg/k8sd/api/capi_certificate_refresh.go b/src/k8s/pkg/k8sd/api/capi_certificate_refresh.go new file mode 100644 index 000000000..51ed19c75 --- /dev/null +++ b/src/k8s/pkg/k8sd/api/capi_certificate_refresh.go @@ -0,0 +1,87 @@ +package api + +import ( + "fmt" + "net/http" + + apiv1 "github.com/canonical/k8s-snap-api/api/v1" + "github.com/canonical/k8s/pkg/utils" + "github.com/canonical/lxd/lxd/response" + "github.com/canonical/microcluster/v3/state" + "golang.org/x/sync/errgroup" + certv1 "k8s.io/api/certificates/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// postApproveWorkerCSR approves the worker node CSR for the specified seed. +// The certificate approval process follows these steps: +// 1. The CAPI provider calls the /x/capi/refresh-certs/plan endpoint from a +// worker node, which generates a CSR and creates a CertificateSigningRequest +// object in the cluster. +// 2. The CAPI provider then calls the /k8sd/refresh-certs/run endpoint with +// the seed. This endpoint waits until the CSR is approved and the certificate +// is signed. Note that this is a blocking call. +// 3. The CAPI provider calls the /x/capi/refresh-certs/approve endpoint from +// any control plane node to approve the CSR. +// 4. The /x/capi/refresh-certs/run endpoint completes and returns once the +// certificate is approved and signed. +func (e *Endpoints) postApproveWorkerCSR(s state.State, r *http.Request) response.Response { + snap := e.provider.Snap() + + req := apiv1.ClusterAPIApproveWorkerCSRRequest{} + + if err := utils.NewStrictJSONDecoder(r.Body).Decode(&req); err != nil { + return response.BadRequest(fmt.Errorf("failed to parse request: %w", err)) + } + + if err := r.Body.Close(); err != nil { + return response.InternalError(fmt.Errorf("failed to close request body: %w", err)) + } + + client, err := snap.KubernetesClient("") + if err != nil { + return response.InternalError(fmt.Errorf("failed to get Kubernetes client: %w", err)) + } + + g, ctx := errgroup.WithContext(r.Context()) + + // CSR names + csrNames := []string{ + fmt.Sprintf("k8sd-%d-worker-kubelet-serving", req.Seed), + fmt.Sprintf("k8sd-%d-worker-kubelet-client", req.Seed), + fmt.Sprintf("k8sd-%d-worker-kube-proxy-client", req.Seed), + } + + for _, csrName := range csrNames { + g.Go(func() error { + if err := client.WatchCertificateSigningRequest( + ctx, + csrName, + func(request *certv1.CertificateSigningRequest) (bool, error) { + request.Status.Conditions = append(request.Status.Conditions, certv1.CertificateSigningRequestCondition{ + Type: certv1.CertificateApproved, + Status: corev1.ConditionTrue, + Reason: "ApprovedByCK8sCAPI", + Message: "This CSR was approved by the Canonical Kubernetes CAPI Provider", + LastUpdateTime: metav1.Now(), + }) + _, err := client.CertificatesV1().CertificateSigningRequests().UpdateApproval(ctx, csrName, request, metav1.UpdateOptions{}) + if err != nil { + return false, fmt.Errorf("failed to update CSR %s: %w", csrName, err) + } + return true, nil + }, + ); err != nil { + return fmt.Errorf("certificate signing request failed: %w", err) + } + return nil + }) + } + + if err := g.Wait(); err != nil { + return response.InternalError(fmt.Errorf("failed to approve worker node CSR: %w", err)) + } + + return response.SyncResponse(true, apiv1.ClusterAPIApproveWorkerCSRResponse{}) +} diff --git a/src/k8s/pkg/k8sd/api/capi_certificates_expiry.go b/src/k8s/pkg/k8sd/api/capi_certificates_expiry.go new file mode 100644 index 000000000..c6bc96ac8 --- /dev/null +++ b/src/k8s/pkg/k8sd/api/capi_certificates_expiry.go @@ -0,0 +1,50 @@ +package api + +import ( + "fmt" + "net/http" + "time" + + apiv1 "github.com/canonical/k8s-snap-api/api/v1" + databaseutil "github.com/canonical/k8s/pkg/k8sd/database/util" + pkiutil "github.com/canonical/k8s/pkg/utils/pki" + "github.com/canonical/lxd/lxd/response" + "github.com/canonical/microcluster/v3/state" +) + +func (e *Endpoints) postCertificatesExpiry(s state.State, r *http.Request) response.Response { + config, err := databaseutil.GetClusterConfig(r.Context(), s) + if err != nil { + return response.InternalError(fmt.Errorf("failed to get cluster config: %w", err)) + } + + certificates := []string{ + config.Certificates.GetCACert(), + config.Certificates.GetClientCACert(), + config.Certificates.GetAdminClientCert(), + config.Certificates.GetAPIServerKubeletClientCert(), + config.Certificates.GetFrontProxyCACert(), + } + + var earliestExpiry time.Time + // Find the earliest expiry certificate + // They should all be about the same but better double-check this. + for _, cert := range certificates { + if cert == "" { + continue + } + + cert, _, err := pkiutil.LoadCertificate(cert, "") + if err != nil { + return response.InternalError(fmt.Errorf("failed to load certificate: %w", err)) + } + + if earliestExpiry.IsZero() || cert.NotAfter.Before(earliestExpiry) { + earliestExpiry = cert.NotAfter + } + } + + return response.SyncResponse(true, &apiv1.CertificatesExpiryResponse{ + ExpiryDate: earliestExpiry.Format(time.RFC3339), + }) +} diff --git a/src/k8s/pkg/k8sd/api/certificates_refresh.go b/src/k8s/pkg/k8sd/api/certificates_refresh.go index 2d8153dc1..c94f33433 100644 --- a/src/k8s/pkg/k8sd/api/certificates_refresh.go +++ b/src/k8s/pkg/k8sd/api/certificates_refresh.go @@ -1,10 +1,15 @@ package api import ( + "context" + "crypto/rand" + "crypto/rsa" + "crypto/sha256" "crypto/x509/pkix" + "encoding/base64" "fmt" "math" - "math/rand" + "math/big" "net" "net/http" "path/filepath" @@ -28,7 +33,11 @@ import ( ) func (e *Endpoints) postRefreshCertsPlan(s state.State, r *http.Request) response.Response { - seed := rand.Intn(math.MaxInt) + seedBigInt, err := rand.Int(rand.Reader, big.NewInt(math.MaxInt)) + if err != nil { + return response.InternalError(fmt.Errorf("failed to generate seed: %w", err)) + } + seed := int(seedBigInt.Int64()) snap := e.provider.Snap() isWorker, err := snaputil.IsWorker(snap) @@ -49,7 +58,6 @@ func (e *Endpoints) postRefreshCertsPlan(s state.State, r *http.Request) respons return response.SyncResponse(true, apiv1.RefreshCertificatesPlanResponse{ Seed: seed, }) - } func (e *Endpoints) postRefreshCertsRun(s state.State, r *http.Request) response.Response { @@ -66,6 +74,8 @@ func (e *Endpoints) postRefreshCertsRun(s state.State, r *http.Request) response // refreshCertsRunControlPlane refreshes the certificates for a control plane node. func refreshCertsRunControlPlane(s state.State, r *http.Request, snap snap.Snap) response.Response { + log := log.FromContext(r.Context()) + req := apiv1.RefreshCertificatesRunRequest{} if err := utils.NewStrictJSONDecoder(r.Body).Decode(&req); err != nil { return response.BadRequest(fmt.Errorf("failed to parse request: %w", err)) @@ -81,6 +91,13 @@ func refreshCertsRunControlPlane(s state.State, r *http.Request, snap snap.Snap) return response.InternalError(fmt.Errorf("failed to parse node IP address %q", s.Address().Hostname())) } + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + serviceIPs, err := utils.GetKubernetesServiceIPsFromServiceCIDRs(clusterConfig.Network.GetServiceCIDR()) if err != nil { return response.InternalError(fmt.Errorf("failed to get IP address(es) from ServiceCIDR %q: %w", clusterConfig.Network.GetServiceCIDR(), err)) @@ -119,27 +136,67 @@ func refreshCertsRunControlPlane(s state.State, r *http.Request, snap snap.Snap) return response.InternalError(fmt.Errorf("failed to write control plane certificates: %w", err)) } - if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), clusterConfig.APIServer.GetSecurePort(), *certificates); err != nil { + if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), localhostAddress, clusterConfig.APIServer.GetSecurePort(), *certificates); err != nil { return response.InternalError(fmt.Errorf("failed to generate control plane kubeconfigs: %w", err)) } - if err := snaputil.RestartControlPlaneServices(r.Context(), snap); err != nil { - return response.InternalError(fmt.Errorf("failed to restart control plane services: %w", err)) - } + // NOTE: Restart the control plane services in a separate goroutine to avoid + // restarting the API server, which would break the k8sd proxy connection + // and cause missed responses in the proxy side. + readyCh := make(chan error) + go func() { + // NOTE: Create a new context independent of the request context to ensure + // the restart process is not cancelled by the client. + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + select { + case err := <-readyCh: + if err != nil { + log.Error(err, "Failed to refresh certificates") + return + } + case <-ctx.Done(): + log.Error(ctx.Err(), "Timeout waiting for certificates to be refreshed") + return + } - kubeletCert, _, err := pkiutil.LoadCertificate(certificates.KubeletCert, "") + if err := snaputil.RestartControlPlaneServices(ctx, snap); err != nil { + log.Error(err, "Failed to restart control plane services") + } + }() + + apiServerCert, _, err := pkiutil.LoadCertificate(certificates.APIServerCert, "") if err != nil { return response.InternalError(fmt.Errorf("failed to read kubelet certificate: %w", err)) } - expirationTimeUNIX := kubeletCert.NotAfter.Unix() + expirationTimeUNIX := apiServerCert.NotAfter.Unix() + + return response.ManualResponse(func(w http.ResponseWriter) (rerr error) { + defer func() { + readyCh <- rerr + close(readyCh) + }() - return response.SyncResponse(true, apiv1.RefreshCertificatesRunResponse{ - ExpirationSeconds: int(expirationTimeUNIX), + err := response.SyncResponse(true, apiv1.RefreshCertificatesRunResponse{ + ExpirationSeconds: int(expirationTimeUNIX), + }).Render(w) + if err != nil { + return fmt.Errorf("failed to render response: %w", err) + } + + f, ok := w.(http.Flusher) + if !ok { + return fmt.Errorf("ResponseWriter is not type http.Flusher") + } + + f.Flush() + return nil }) } -// refreshCertsRunWorker refreshes the certificates for a worker node +// refreshCertsRunWorker refreshes the certificates for a worker node. func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) response.Response { log := log.FromContext(r.Context()) @@ -167,6 +224,18 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo certificates.CACert = clusterConfig.Certificates.GetCACert() certificates.ClientCACert = clusterConfig.Certificates.GetClientCACert() + k8sdPublicKey, err := pkiutil.LoadRSAPublicKey(clusterConfig.Certificates.GetK8sdPublicKey()) + if err != nil { + return response.InternalError(fmt.Errorf("failed to load k8sd public key, error: %w", err)) + } + + hostnames := []string{snap.Hostname()} + ips := []net.IP{net.ParseIP(s.Address().Hostname())} + + extraIPs, extraNames := utils.SplitIPAndDNSSANs(req.ExtraSANs) + hostnames = append(hostnames, extraNames...) + ips = append(ips, extraIPs...) + g, ctx := errgroup.WithContext(r.Context()) for _, csr := range []struct { @@ -185,8 +254,8 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo commonName: fmt.Sprintf("system:node:%s", snap.Hostname()), organization: []string{"system:nodes"}, usages: []certv1.KeyUsage{certv1.UsageDigitalSignature, certv1.UsageKeyEncipherment, certv1.UsageServerAuth}, - hostnames: []string{snap.Hostname()}, - ips: []net.IP{net.ParseIP(s.Address().Hostname())}, + hostnames: hostnames, + ips: ips, signerName: "k8sd.io/kubelet-serving", certificate: &certificates.KubeletCert, key: &certificates.KubeletKey, @@ -209,7 +278,6 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo key: &certificates.KubeProxyClientKey, }, } { - csr := csr g.Go(func() error { csrPEM, keyPEM, err := pkiutil.GenerateCSR( pkix.Name{ @@ -224,14 +292,34 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo return fmt.Errorf("failed to generate CSR for %s: %w", csr.name, err) } + // Obtain the SHA256 sum of the CSR request. + hash := sha256.New() + _, err = hash.Write([]byte(csrPEM)) + if err != nil { + return fmt.Errorf("failed to checksum CSR %s, err: %w", csr.name, err) + } + + signature, err := rsa.EncryptPKCS1v15(rand.Reader, k8sdPublicKey, hash.Sum(nil)) + if err != nil { + return fmt.Errorf("failed to sign CSR %s, err: %w", csr.name, err) + } + signatureB64 := base64.StdEncoding.EncodeToString(signature) + + expirationSeconds := int32(req.ExpirationSeconds) + if _, err = client.CertificatesV1().CertificateSigningRequests().Create(ctx, &certv1.CertificateSigningRequest{ ObjectMeta: metav1.ObjectMeta{ Name: csr.name, + Annotations: map[string]string{ + "k8sd.io/signature": signatureB64, + "k8sd.io/node": snap.Hostname(), + }, }, Spec: certv1.CertificateSigningRequestSpec{ - Request: []byte(csrPEM), - Usages: csr.usages, - SignerName: csr.signerName, + Request: []byte(csrPEM), + ExpirationSeconds: &expirationSeconds, + Usages: csr.usages, + SignerName: csr.signerName, }, }, metav1.CreateOptions{}); err != nil { return fmt.Errorf("failed to create CSR for %s: %w", csr.name, err) @@ -249,9 +337,7 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo } return nil - }) - } if err := g.Wait(); err != nil { @@ -262,21 +348,54 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo return response.InternalError(fmt.Errorf("failed to write worker PKI: %w", err)) } + nodeIP := net.ParseIP(s.Address().Hostname()) + if nodeIP == nil { + return response.InternalError(fmt.Errorf("failed to parse node IP address %q", s.Address().Hostname())) + } + + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + // Kubeconfigs - if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), "127.0.0.1:6443", certificates.CACert, certificates.KubeletClientCert, certificates.KubeletClientKey); err != nil { + if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), fmt.Sprintf("%s:%d", localhostAddress, clusterConfig.APIServer.GetSecurePort()), certificates.CACert, certificates.KubeletClientCert, certificates.KubeletClientKey); err != nil { return response.InternalError(fmt.Errorf("failed to generate kubelet kubeconfig: %w", err)) } - if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), "127.0.0.1:6443", certificates.CACert, certificates.KubeProxyClientCert, certificates.KubeProxyClientKey); err != nil { + if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), fmt.Sprintf("%s:%d", localhostAddress, clusterConfig.APIServer.GetSecurePort()), certificates.CACert, certificates.KubeProxyClientCert, certificates.KubeProxyClientKey); err != nil { return response.InternalError(fmt.Errorf("failed to generate kube-proxy kubeconfig: %w", err)) } - // Restart the services - if err := snap.RestartService(r.Context(), "kubelet"); err != nil { - return response.InternalError(fmt.Errorf("failed to restart kubelet: %w", err)) - } - if err := snap.RestartService(r.Context(), "kube-proxy"); err != nil { - return response.InternalError(fmt.Errorf("failed to restart kube-proxy: %w", err)) - } + // NOTE: Restart the worker services in a separate goroutine to avoid + // restarting the kube-proxy and kubelet, which would break the + // proxy connection and cause missed responses in the proxy side. + readyCh := make(chan error, 1) + go func() { + // NOTE: Create a new context independent of the request context to ensure + // the restart process is not cancelled by the client. + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + select { + case err := <-readyCh: + if err != nil { + log.Error(err, "Failed to refresh certificates") + return + } + case <-ctx.Done(): + log.Error(ctx.Err(), "Timeout waiting for certificates to be refreshed") + return + } + + if err := snap.RestartService(ctx, "kubelet"); err != nil { + log.Error(err, "Failed to restart kubelet") + } + if err := snap.RestartService(ctx, "kube-proxy"); err != nil { + log.Error(err, "Failed to restart kube-proxy") + } + }() cert, _, err := pkiutil.LoadCertificate(certificates.KubeletCert, "") if err != nil { @@ -284,10 +403,27 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo } expirationTimeUNIX := cert.NotAfter.Unix() - return response.SyncResponse(true, apiv1.RefreshCertificatesRunResponse{ - ExpirationSeconds: int(expirationTimeUNIX), - }) + return response.ManualResponse(func(w http.ResponseWriter) (rerr error) { + defer func() { + readyCh <- rerr + close(readyCh) + }() + err := response.SyncResponse(true, apiv1.RefreshCertificatesRunResponse{ + ExpirationSeconds: int(expirationTimeUNIX), + }).Render(w) + if err != nil { + return fmt.Errorf("failed to render response: %w", err) + } + + f, ok := w.(http.Flusher) + if !ok { + return fmt.Errorf("ResponseWriter is not type http.Flusher") + } + + f.Flush() + return nil + }) } // isCertificateSigningRequestApprovedAndIssued checks if the certificate @@ -298,7 +434,6 @@ func isCertificateSigningRequestApprovedAndIssued(csr *certv1.CertificateSigning for _, condition := range csr.Status.Conditions { if condition.Type == certv1.CertificateApproved && condition.Status == corev1.ConditionTrue { return len(csr.Status.Certificate) > 0, nil - } if condition.Type == certv1.CertificateDenied && condition.Status == corev1.ConditionTrue { return false, fmt.Errorf("CSR %s was denied: %s", csr.Name, condition.Reason) diff --git a/src/k8s/pkg/k8sd/api/cluster_remove.go b/src/k8s/pkg/k8sd/api/cluster_remove.go index 6cbc22e62..731389683 100644 --- a/src/k8s/pkg/k8sd/api/cluster_remove.go +++ b/src/k8s/pkg/k8sd/api/cluster_remove.go @@ -8,6 +8,7 @@ import ( "time" apiv1 "github.com/canonical/k8s-snap-api/api/v1" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations" databaseutil "github.com/canonical/k8s/pkg/k8sd/database/util" "github.com/canonical/k8s/pkg/log" "github.com/canonical/k8s/pkg/utils" @@ -87,7 +88,7 @@ func (e *Endpoints) postClusterRemove(s state.State, r *http.Request) response.R return response.InternalError(fmt.Errorf("failed to get cluster config: %w", err)) } - if _, ok := cfg.Annotations[apiv1.AnnotationSkipCleanupKubernetesNodeOnRemove]; ok { + if _, ok := cfg.Annotations[apiv1_annotations.AnnotationSkipCleanupKubernetesNodeOnRemove]; ok { // Explicitly skip removing the node from Kubernetes. log.Info("Skipping Kubernetes worker node removal") return response.SyncResponse(true, nil) diff --git a/src/k8s/pkg/k8sd/api/cluster_tokens.go b/src/k8s/pkg/k8sd/api/cluster_tokens.go index d685ce1b9..1562f4b23 100644 --- a/src/k8s/pkg/k8sd/api/cluster_tokens.go +++ b/src/k8s/pkg/k8sd/api/cluster_tokens.go @@ -10,6 +10,7 @@ import ( apiv1 "github.com/canonical/k8s-snap-api/api/v1" "github.com/canonical/k8s/pkg/k8sd/database" "github.com/canonical/k8s/pkg/k8sd/types" + "github.com/canonical/k8s/pkg/log" "github.com/canonical/k8s/pkg/utils" "github.com/canonical/lxd/lxd/response" "github.com/canonical/microcluster/v3/microcluster" @@ -48,17 +49,19 @@ func (e *Endpoints) postClusterJoinTokens(s state.State, r *http.Request) respon } func getOrCreateJoinToken(ctx context.Context, m *microcluster.MicroCluster, tokenName string, ttl time.Duration) (string, error) { + log := log.FromContext(ctx) + // grab token if it exists and return it records, err := m.ListJoinTokens(ctx) if err != nil { - fmt.Println("Failed to get existing tokens. Trying to create a new token.") + log.V(1).Info("Failed to get existing tokens. Trying to create a new token.") } else { for _, record := range records { if record.Name == tokenName { return record.Token, nil } } - fmt.Println("No token exists yet. Creating a new token.") + log.V(1).Info("No token exists yet. Creating a new token.") } token, err := m.NewJoinToken(ctx, tokenName, ttl) diff --git a/src/k8s/pkg/k8sd/api/endpoints.go b/src/k8s/pkg/k8sd/api/endpoints.go index e7e7af5d0..4ae9946e3 100644 --- a/src/k8s/pkg/k8sd/api/endpoints.go +++ b/src/k8s/pkg/k8sd/api/endpoints.go @@ -85,18 +85,18 @@ func (e *Endpoints) Endpoints() []rest.Endpoint { Post: rest.EndpointAction{ Handler: e.postWorkerInfo, AllowUntrusted: true, - AccessHandler: ValidateWorkerInfoAccessHandler("worker-name", "worker-token"), + AccessHandler: ValidateWorkerInfoAccessHandler("Worker-Name", "Worker-Token"), }, }, // Certificates { Name: "RefreshCerts/Plan", - Path: "k8sd/refresh-certs/plan", + Path: apiv1.RefreshCertificatesPlanRPC, Post: rest.EndpointAction{Handler: e.postRefreshCertsPlan}, }, { Name: "RefreshCerts/Run", - Path: "k8sd/refresh-certs/run", + Path: apiv1.RefreshCertificatesRunRPC, Post: rest.EndpointAction{Handler: e.postRefreshCertsRun}, }, // Kubeconfig @@ -140,6 +140,26 @@ func (e *Endpoints) Endpoints() []rest.Endpoint { Path: apiv1.ClusterAPIRemoveNodeRPC, Post: rest.EndpointAction{Handler: e.postClusterRemove, AccessHandler: ValidateCAPIAuthTokenAccessHandler("capi-auth-token"), AllowUntrusted: true}, }, + { + Name: "ClusterAPI/CertificatesExpiry", + Path: apiv1.ClusterAPICertificatesExpiryRPC, + Post: rest.EndpointAction{Handler: e.postCertificatesExpiry, AccessHandler: e.ValidateNodeTokenAccessHandler("node-token"), AllowUntrusted: true}, + }, + { + Name: "ClusterAPI/RefreshCerts/Plan", + Path: apiv1.ClusterAPICertificatesPlanRPC, + Post: rest.EndpointAction{Handler: e.postRefreshCertsPlan, AccessHandler: e.ValidateNodeTokenAccessHandler("node-token"), AllowUntrusted: true}, + }, + { + Name: "ClusterAPI/RefreshCerts/Run", + Path: apiv1.ClusterAPICertificatesRunRPC, + Post: rest.EndpointAction{Handler: e.postRefreshCertsRun, AccessHandler: e.ValidateNodeTokenAccessHandler("node-token"), AllowUntrusted: true}, + }, + { + Name: "ClusterAPI/RefreshCerts/Approve", + Path: apiv1.ClusterAPIApproveWorkerCSRRPC, + Post: rest.EndpointAction{Handler: e.postApproveWorkerCSR, AccessHandler: ValidateCAPIAuthTokenAccessHandler("capi-auth-token"), AllowUntrusted: true}, + }, // Snap refreshes { Name: "Snap/Refresh", diff --git a/src/k8s/pkg/k8sd/api/impl/k8sd.go b/src/k8s/pkg/k8sd/api/impl/k8sd.go index bc8eabdb5..e062a12f5 100644 --- a/src/k8s/pkg/k8sd/api/impl/k8sd.go +++ b/src/k8s/pkg/k8sd/api/impl/k8sd.go @@ -59,5 +59,4 @@ func GetLocalNodeStatus(ctx context.Context, s state.State, snap snap.Snap) (api Address: s.Address().Hostname(), ClusterRole: clusterRole, }, nil - } diff --git a/src/k8s/pkg/k8sd/api/response.go b/src/k8s/pkg/k8sd/api/response.go index 626aadfa8..01c94e373 100644 --- a/src/k8s/pkg/k8sd/api/response.go +++ b/src/k8s/pkg/k8sd/api/response.go @@ -5,9 +5,9 @@ import ( ) const ( - // StatusNodeUnavailable is the Http status code that the API returns if the node isn't in the cluster + // StatusNodeUnavailable is the Http status code that the API returns if the node isn't in the cluster. StatusNodeUnavailable = 520 - // StatusNodeInUse is the Http status code that the API returns if the node is already in the cluster + // StatusNodeInUse is the Http status code that the API returns if the node is already in the cluster. StatusNodeInUse = 521 ) diff --git a/src/k8s/pkg/k8sd/api/worker.go b/src/k8s/pkg/k8sd/api/worker.go index efd750598..d030ba31b 100644 --- a/src/k8s/pkg/k8sd/api/worker.go +++ b/src/k8s/pkg/k8sd/api/worker.go @@ -26,7 +26,7 @@ func (e *Endpoints) postWorkerInfo(s state.State, r *http.Request) response.Resp } // Existence of this header is already checked in the access handler. - workerName := r.Header.Get("worker-name") + workerName := r.Header.Get("Worker-Name") nodeIP := net.ParseIP(req.Address) if nodeIP == nil { return response.BadRequest(fmt.Errorf("failed to parse node IP address %s", req.Address)) @@ -63,7 +63,7 @@ func (e *Endpoints) postWorkerInfo(s state.State, r *http.Request) response.Resp return response.InternalError(fmt.Errorf("failed to retrieve list of known kube-apiserver endpoints: %w", err)) } - workerToken := r.Header.Get("worker-token") + workerToken := r.Header.Get("Worker-Token") if err := s.Database().Transaction(r.Context(), func(ctx context.Context, tx *sql.Tx) error { return database.DeleteWorkerNodeToken(ctx, tx, workerToken) }); err != nil { @@ -86,5 +86,6 @@ func (e *Endpoints) postWorkerInfo(s state.State, r *http.Request) response.Resp KubeProxyClientCert: workerCertificates.KubeProxyClientCert, KubeProxyClientKey: workerCertificates.KubeProxyClientKey, K8sdPublicKey: cfg.Certificates.GetK8sdPublicKey(), + Annotations: cfg.Annotations, }) } diff --git a/src/k8s/pkg/k8sd/app/app.go b/src/k8s/pkg/k8sd/app/app.go index d6172d124..720401e3f 100644 --- a/src/k8s/pkg/k8sd/app/app.go +++ b/src/k8s/pkg/k8sd/app/app.go @@ -236,7 +236,7 @@ func (a *App) Run(ctx context.Context, customHooks *state.Hooks) error { // markNodeReady will decrement the readyWg counter to signal that the node is ready. // The node is ready if: // - the microcluster database is accessible -// - the kubernetes endpoint is reachable +// - the kubernetes endpoint is reachable. func (a *App) markNodeReady(ctx context.Context, s state.State) error { log := log.FromContext(ctx).WithValues("startup", "waitForReady") diff --git a/src/k8s/pkg/k8sd/app/cluster_util.go b/src/k8s/pkg/k8sd/app/cluster_util.go index 9255be3e4..46b23d366 100644 --- a/src/k8s/pkg/k8sd/app/cluster_util.go +++ b/src/k8s/pkg/k8sd/app/cluster_util.go @@ -3,10 +3,12 @@ package app import ( "context" "fmt" + "net" "github.com/canonical/k8s/pkg/k8sd/setup" "github.com/canonical/k8s/pkg/snap" snaputil "github.com/canonical/k8s/pkg/snap/util" + mctypes "github.com/canonical/microcluster/v3/rest/types" ) func startControlPlaneServices(ctx context.Context, snap snap.Snap, datastore string) error { @@ -58,3 +60,22 @@ func waitApiServerReady(ctx context.Context, snap snap.Snap) error { return nil } + +func DetermineLocalhostAddress(clusterMembers []mctypes.ClusterMember) (string, error) { + // Check if any of the cluster members have an IPv6 address, if so return "::1" + // if one member has an IPv6 address, other members should also have IPv6 interfaces + for _, clusterMember := range clusterMembers { + memberAddress := clusterMember.Address.Addr().String() + nodeIP := net.ParseIP(memberAddress) + if nodeIP == nil { + return "", fmt.Errorf("failed to parse node IP address %q", memberAddress) + } + + if nodeIP.To4() == nil { + return "[::1]", nil + } + } + + // If no IPv6 addresses are found this means the cluster is IPv4 only + return "127.0.0.1", nil +} diff --git a/src/k8s/pkg/k8sd/app/cluster_util_test.go b/src/k8s/pkg/k8sd/app/cluster_util_test.go new file mode 100644 index 000000000..acd6b5e1e --- /dev/null +++ b/src/k8s/pkg/k8sd/app/cluster_util_test.go @@ -0,0 +1,120 @@ +package app_test + +import ( + "net/netip" + "testing" + + "github.com/canonical/k8s/pkg/k8sd/app" + mctypes "github.com/canonical/microcluster/v3/rest/types" + . "github.com/onsi/gomega" +) + +func TestDetermineLocalhostAddress(t *testing.T) { + t.Run("IPv4Only", func(t *testing.T) { + g := NewWithT(t) + + mockMembers := []mctypes.ClusterMember{ + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node1", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.1:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node2", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.2:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node3", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.3:1234"), + }, + }, + }, + } + + localhostAddress, err := app.DetermineLocalhostAddress(mockMembers) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(localhostAddress).To(Equal("127.0.0.1")) + }) + + t.Run("IPv6Only", func(t *testing.T) { + g := NewWithT(t) + + mockMembers := []mctypes.ClusterMember{ + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node1", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("[fda1:8e75:b6ef::]:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node2", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("[fd51:d664:aca3::]:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node3", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("[fda3:c11d:3cda::]:1234"), + }, + }, + }, + } + + localhostAddress, err := app.DetermineLocalhostAddress(mockMembers) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(localhostAddress).To(Equal("[::1]")) + }) + + t.Run("IPv4_IPv6_Mixed", func(t *testing.T) { + g := NewWithT(t) + + mockMembers := []mctypes.ClusterMember{ + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node1", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.1:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node2", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("[fd51:d664:aca3::]:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node3", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.3:1234"), + }, + }, + }, + } + + localhostAddress, err := app.DetermineLocalhostAddress(mockMembers) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(localhostAddress).To(Equal("[::1]")) + }) +} diff --git a/src/k8s/pkg/k8sd/app/hooks_bootstrap.go b/src/k8s/pkg/k8sd/app/hooks_bootstrap.go index 4144a0f84..1386edca9 100644 --- a/src/k8s/pkg/k8sd/app/hooks_bootstrap.go +++ b/src/k8s/pkg/k8sd/app/hooks_bootstrap.go @@ -10,6 +10,8 @@ import ( "net" "net/http" "path/filepath" + "strconv" + "strings" "time" apiv1 "github.com/canonical/k8s-snap-api/api/v1" @@ -28,7 +30,6 @@ import ( // onBootstrap is called after we bootstrap the first cluster node. // onBootstrap configures local services then writes the cluster config on the database. func (a *App) onBootstrap(ctx context.Context, s state.State, initConfig map[string]string) error { - // NOTE(neoaggelos): context timeout is passed over configuration, so that hook failures are propagated to the client ctx, cancel := context.WithCancel(ctx) defer cancel() @@ -119,8 +120,8 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT if err != nil { return fmt.Errorf("failed to prepare HTTP request: %w", err) } - httpRequest.Header.Add("worker-name", s.Name()) - httpRequest.Header.Add("worker-token", token.Secret) + httpRequest.Header.Add("Worker-Name", s.Name()) + httpRequest.Header.Add("Worker-Token", token.Secret) httpResponse, err := httpClient.Do(httpRequest) if err != nil { @@ -178,11 +179,29 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT return fmt.Errorf("failed to write worker node certificates: %w", err) } + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + + port := "6443" + if len(response.APIServers) == 0 { + return fmt.Errorf("no APIServers found in worker node info") + } + // Get the secure port from the first APIServer since they should all be the same. + port = response.APIServers[0][strings.LastIndex(response.APIServers[0], ":")+1:] + securePort, err := strconv.Atoi(port) + if err != nil { + return fmt.Errorf("failed to parse apiserver secure port: %w", err) + } + // Kubeconfigs - if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), "127.0.0.1:6443", certificates.CACert, certificates.KubeletClientCert, certificates.KubeletClientKey); err != nil { + if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), fmt.Sprintf("%s:%d", localhostAddress, securePort), certificates.CACert, certificates.KubeletClientCert, certificates.KubeletClientKey); err != nil { return fmt.Errorf("failed to generate kubelet kubeconfig: %w", err) } - if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), "127.0.0.1:6443", certificates.CACert, certificates.KubeProxyClientCert, certificates.KubeProxyClientKey); err != nil { + if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), fmt.Sprintf("%s:%d", localhostAddress, securePort), certificates.CACert, certificates.KubeProxyClientCert, certificates.KubeProxyClientKey); err != nil { return fmt.Errorf("failed to generate kube-proxy kubeconfig: %w", err) } @@ -197,6 +216,9 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT // TODO(neoaggelos): We should be explicit here and try to avoid having worker nodes use // or set other cluster configuration keys by accident. cfg := types.ClusterConfig{ + APIServer: types.APIServer{ + SecurePort: utils.Pointer(securePort), + }, Network: types.Network{ PodCIDR: utils.Pointer(response.PodCIDR), ServiceCIDR: utils.Pointer(response.ServiceCIDR), @@ -206,6 +228,7 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT CACert: utils.Pointer(response.CACert), ClientCACert: utils.Pointer(response.ClientCACert), }, + Annotations: response.Annotations, } // Pre-init checks @@ -229,10 +252,10 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT if err := setup.KubeletWorker(snap, s.Name(), nodeIP, response.ClusterDNS, response.ClusterDomain, response.CloudProvider, joinConfig.ExtraNodeKubeletArgs); err != nil { return fmt.Errorf("failed to configure kubelet: %w", err) } - if err := setup.KubeProxy(ctx, snap, s.Name(), response.PodCIDR, joinConfig.ExtraNodeKubeProxyArgs); err != nil { + if err := setup.KubeProxy(ctx, snap, s.Name(), response.PodCIDR, localhostAddress, joinConfig.ExtraNodeKubeProxyArgs); err != nil { return fmt.Errorf("failed to configure kube-proxy: %w", err) } - if err := setup.K8sAPIServerProxy(snap, response.APIServers, joinConfig.ExtraNodeK8sAPIServerProxyArgs); err != nil { + if err := setup.K8sAPIServerProxy(snap, response.APIServers, securePort, joinConfig.ExtraNodeK8sAPIServerProxyArgs); err != nil { return fmt.Errorf("failed to configure k8s-apiserver-proxy: %w", err) } if err := setup.ExtraNodeConfigFiles(snap, joinConfig.ExtraNodeConfigFiles); err != nil { @@ -277,6 +300,13 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst return fmt.Errorf("failed to parse node IP address %q", s.Address().Hostname()) } + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + // Create directories if err := setup.EnsureAllDirectories(snap); err != nil { return fmt.Errorf("failed to create directories: %w", err) @@ -296,7 +326,7 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst // NOTE: Default certificate expiration is set to 20 years. certificates := pki.NewK8sDqlitePKI(pki.K8sDqlitePKIOpts{ Hostname: s.Name(), - IPSANs: []net.IP{{127, 0, 0, 1}}, + IPSANs: []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")}, NotBefore: notBefore, NotAfter: notBefore.AddDate(20, 0, 0), AllowSelfSignedCA: true, @@ -395,14 +425,15 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst } // Generate kubeconfigs - if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), cfg.APIServer.GetSecurePort(), *certificates); err != nil { + if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), localhostAddress, cfg.APIServer.GetSecurePort(), *certificates); err != nil { return fmt.Errorf("failed to generate kubeconfigs: %w", err) } // Configure datastore switch cfg.Datastore.GetType() { case "k8s-dqlite": - if err := setup.K8sDqlite(snap, fmt.Sprintf("%s:%d", nodeIP.String(), cfg.Datastore.GetK8sDqlitePort()), nil, bootstrapConfig.ExtraNodeK8sDqliteArgs); err != nil { + address := fmt.Sprintf("%s:%d", utils.ToIPString(nodeIP), cfg.Datastore.GetK8sDqlitePort()) + if err := setup.K8sDqlite(snap, address, nil, bootstrapConfig.ExtraNodeK8sDqliteArgs); err != nil { return fmt.Errorf("failed to configure k8s-dqlite: %w", err) } case "external": @@ -417,7 +448,7 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst if err := setup.KubeletControlPlane(snap, s.Name(), nodeIP, cfg.Kubelet.GetClusterDNS(), cfg.Kubelet.GetClusterDomain(), cfg.Kubelet.GetCloudProvider(), cfg.Kubelet.GetControlPlaneTaints(), bootstrapConfig.ExtraNodeKubeletArgs); err != nil { return fmt.Errorf("failed to configure kubelet: %w", err) } - if err := setup.KubeProxy(ctx, snap, s.Name(), cfg.Network.GetPodCIDR(), bootstrapConfig.ExtraNodeKubeProxyArgs); err != nil { + if err := setup.KubeProxy(ctx, snap, s.Name(), cfg.Network.GetPodCIDR(), localhostAddress, bootstrapConfig.ExtraNodeKubeProxyArgs); err != nil { return fmt.Errorf("failed to configure kube-proxy: %w", err) } if err := setup.KubeControllerManager(snap, bootstrapConfig.ExtraNodeKubeControllerManagerArgs); err != nil { @@ -426,7 +457,7 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst if err := setup.KubeScheduler(snap, bootstrapConfig.ExtraNodeKubeSchedulerArgs); err != nil { return fmt.Errorf("failed to configure kube-scheduler: %w", err) } - if err := setup.KubeAPIServer(snap, nodeIP, cfg.Network.GetServiceCIDR(), s.Address().Path("1.0", "kubernetes", "auth", "webhook").String(), true, cfg.Datastore, cfg.APIServer.GetAuthorizationMode(), bootstrapConfig.ExtraNodeKubeAPIServerArgs); err != nil { + if err := setup.KubeAPIServer(snap, cfg.APIServer.GetSecurePort(), nodeIP, cfg.Network.GetServiceCIDR(), s.Address().Path("1.0", "kubernetes", "auth", "webhook").String(), true, cfg.Datastore, cfg.APIServer.GetAuthorizationMode(), bootstrapConfig.ExtraNodeKubeAPIServerArgs); err != nil { return fmt.Errorf("failed to configure kube-apiserver: %w", err) } diff --git a/src/k8s/pkg/k8sd/app/hooks_join.go b/src/k8s/pkg/k8sd/app/hooks_join.go index 0e164858f..ac67b526f 100644 --- a/src/k8s/pkg/k8sd/app/hooks_join.go +++ b/src/k8s/pkg/k8sd/app/hooks_join.go @@ -48,6 +48,13 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri return fmt.Errorf("failed to parse node IP address %q", s.Address().Hostname()) } + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + // Create directories if err := setup.EnsureAllDirectories(snap); err != nil { return fmt.Errorf("failed to create directories: %w", err) @@ -64,7 +71,7 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri // NOTE: Default certificate expiration is set to 20 years. certificates := pki.NewK8sDqlitePKI(pki.K8sDqlitePKIOpts{ Hostname: s.Name(), - IPSANs: []net.IP{{127, 0, 0, 1}}, + IPSANs: []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")}, NotBefore: notBefore, NotAfter: notBefore.AddDate(20, 0, 0), }) @@ -140,7 +147,7 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri return fmt.Errorf("failed to write control plane certificates: %w", err) } - if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), cfg.APIServer.GetSecurePort(), *certificates); err != nil { + if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), localhostAddress, cfg.APIServer.GetSecurePort(), *certificates); err != nil { return fmt.Errorf("failed to generate kubeconfigs: %w", err) } @@ -158,10 +165,16 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri } cluster := make([]string, len(members)) for _, member := range members { - cluster = append(cluster, fmt.Sprintf("%s:%d", member.Address.Addr(), cfg.Datastore.GetK8sDqlitePort())) + var address string + if member.Address.Addr().Is6() { + address = fmt.Sprintf("[%s]", member.Address.Addr()) + } else { + address = member.Address.Addr().String() + } + cluster = append(cluster, fmt.Sprintf("%s:%d", address, cfg.Datastore.GetK8sDqlitePort())) } - address := fmt.Sprintf("%s:%d", nodeIP.String(), cfg.Datastore.GetK8sDqlitePort()) + address := fmt.Sprintf("%s:%d", utils.ToIPString(nodeIP), cfg.Datastore.GetK8sDqlitePort()) if err := setup.K8sDqlite(snap, address, cluster, joinConfig.ExtraNodeK8sDqliteArgs); err != nil { return fmt.Errorf("failed to configure k8s-dqlite with address=%s cluster=%v: %w", address, cluster, err) } @@ -177,7 +190,7 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri if err := setup.KubeletControlPlane(snap, s.Name(), nodeIP, cfg.Kubelet.GetClusterDNS(), cfg.Kubelet.GetClusterDomain(), cfg.Kubelet.GetCloudProvider(), cfg.Kubelet.GetControlPlaneTaints(), joinConfig.ExtraNodeKubeletArgs); err != nil { return fmt.Errorf("failed to configure kubelet: %w", err) } - if err := setup.KubeProxy(ctx, snap, s.Name(), cfg.Network.GetPodCIDR(), joinConfig.ExtraNodeKubeProxyArgs); err != nil { + if err := setup.KubeProxy(ctx, snap, s.Name(), cfg.Network.GetPodCIDR(), localhostAddress, joinConfig.ExtraNodeKubeProxyArgs); err != nil { return fmt.Errorf("failed to configure kube-proxy: %w", err) } if err := setup.KubeControllerManager(snap, joinConfig.ExtraNodeKubeControllerManagerArgs); err != nil { @@ -186,7 +199,7 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri if err := setup.KubeScheduler(snap, joinConfig.ExtraNodeKubeSchedulerArgs); err != nil { return fmt.Errorf("failed to configure kube-scheduler: %w", err) } - if err := setup.KubeAPIServer(snap, nodeIP, cfg.Network.GetServiceCIDR(), s.Address().Path("1.0", "kubernetes", "auth", "webhook").String(), true, cfg.Datastore, cfg.APIServer.GetAuthorizationMode(), joinConfig.ExtraNodeKubeAPIServerArgs); err != nil { + if err := setup.KubeAPIServer(snap, cfg.APIServer.GetSecurePort(), nodeIP, cfg.Network.GetServiceCIDR(), s.Address().Path("1.0", "kubernetes", "auth", "webhook").String(), true, cfg.Datastore, cfg.APIServer.GetAuthorizationMode(), joinConfig.ExtraNodeKubeAPIServerArgs); err != nil { return fmt.Errorf("failed to configure kube-apiserver: %w", err) } diff --git a/src/k8s/pkg/k8sd/app/hooks_remove.go b/src/k8s/pkg/k8sd/app/hooks_remove.go index 696cc1b46..bb9cc941e 100644 --- a/src/k8s/pkg/k8sd/app/hooks_remove.go +++ b/src/k8s/pkg/k8sd/app/hooks_remove.go @@ -3,11 +3,12 @@ package app import ( "context" "database/sql" + "errors" "fmt" "net" "os" - apiv1 "github.com/canonical/k8s-snap-api/api/v1" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations" databaseutil "github.com/canonical/k8s/pkg/k8sd/database/util" "github.com/canonical/k8s/pkg/k8sd/pki" "github.com/canonical/k8s/pkg/k8sd/setup" @@ -59,8 +60,9 @@ func (a *App) onPreRemove(ctx context.Context, s state.State, force bool) (rerr log.Error(err, "Failed to wait for node to finish microcluster join before removing. Continuing with the cleanup...") } - if cfg, err := databaseutil.GetClusterConfig(ctx, s); err == nil { - if _, ok := cfg.Annotations[apiv1.AnnotationSkipCleanupKubernetesNodeOnRemove]; !ok { + cfg, err := databaseutil.GetClusterConfig(ctx, s) + if err == nil { + if _, ok := cfg.Annotations.Get(apiv1_annotations.AnnotationSkipCleanupKubernetesNodeOnRemove); !ok { c, err := snap.KubernetesClient("") if err != nil { log.Error(err, "Failed to create Kubernetes client", err) @@ -121,12 +123,9 @@ func (a *App) onPreRemove(ctx context.Context, s state.State, force bool) (rerr log.Info("Removing worker node mark") if err := snaputil.MarkAsWorkerNode(snap, false); err != nil { - log.Error(err, "Failed to unmark node as worker") - } - - log.Info("Stopping worker services") - if err := snaputil.StopWorkerServices(ctx, snap); err != nil { - log.Error(err, "Failed to stop worker services") + if !errors.Is(err, os.ErrNotExist) { + log.Error(err, "failed to unmark node as worker") + } } log.Info("Cleaning up control plane certificates") @@ -134,9 +133,16 @@ func (a *App) onPreRemove(ctx context.Context, s state.State, force bool) (rerr log.Error(err, "failed to cleanup control plane certificates") } - log.Info("Stopping control plane services") - if err := snaputil.StopControlPlaneServices(ctx, snap); err != nil { - log.Error(err, "Failed to stop control-plane services") + if _, ok := cfg.Annotations.Get(apiv1_annotations.AnnotationSkipStopServicesOnRemove); !ok { + log.Info("Stopping worker services") + if err := snaputil.StopWorkerServices(ctx, snap); err != nil { + log.Error(err, "Failed to stop worker services") + } + + log.Info("Stopping control plane services") + if err := snaputil.StopControlPlaneServices(ctx, snap); err != nil { + log.Error(err, "Failed to stop control-plane services") + } } return nil diff --git a/src/k8s/pkg/k8sd/app/hooks_start.go b/src/k8s/pkg/k8sd/app/hooks_start.go index c86515fbb..1f5cecc3e 100644 --- a/src/k8s/pkg/k8sd/app/hooks_start.go +++ b/src/k8s/pkg/k8sd/app/hooks_start.go @@ -61,6 +61,24 @@ func (a *App) onStart(ctx context.Context, s state.State) error { func(ctx context.Context) (types.ClusterConfig, error) { return databaseutil.GetClusterConfig(ctx, s) }, + func() (string, error) { + c, err := s.Leader() + if err != nil { + return "", fmt.Errorf("failed to get leader client: %w", err) + } + + clusterMembers, err := c.GetClusterMembers(ctx) + if err != nil { + return "", fmt.Errorf("failed to get cluster members: %w", err) + } + + localhostAddress, err := DetermineLocalhostAddress(clusterMembers) + if err != nil { + return "", fmt.Errorf("failed to determine localhost address: %w", err) + } + + return localhostAddress, nil + }, func(ctx context.Context, dnsIP string) error { if err := s.Database().Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { if _, err := database.SetClusterConfig(ctx, tx, types.ClusterConfig{ diff --git a/src/k8s/pkg/k8sd/app/provider.go b/src/k8s/pkg/k8sd/app/provider.go index 0125633aa..5cae366a1 100644 --- a/src/k8s/pkg/k8sd/app/provider.go +++ b/src/k8s/pkg/k8sd/app/provider.go @@ -43,5 +43,5 @@ func (a *App) NotifyFeatureController(network, gateway, ingress, loadBalancer, l } } -// Ensure App implements api.Provider +// Ensure App implements api.Provider. var _ api.Provider = &App{} diff --git a/src/k8s/pkg/k8sd/controllers/control_plane_configuration.go b/src/k8s/pkg/k8sd/controllers/control_plane_configuration.go index 8e29eae83..326267e0d 100644 --- a/src/k8s/pkg/k8sd/controllers/control_plane_configuration.go +++ b/src/k8s/pkg/k8sd/controllers/control_plane_configuration.go @@ -23,7 +23,7 @@ type ControlPlaneConfigurationController struct { } // NewControlPlaneConfigurationController creates a new controller. -// triggerCh is typically a `time.NewTicker().C` +// triggerCh is typically a `time.NewTicker().C`. func NewControlPlaneConfigurationController(snap snap.Snap, waitReady func(), triggerCh <-chan time.Time) *ControlPlaneConfigurationController { return &ControlPlaneConfigurationController{ snap: snap, @@ -35,7 +35,7 @@ func NewControlPlaneConfigurationController(snap snap.Snap, waitReady func(), tr // Run starts the controller. // Run accepts a context to manage the lifecycle of the controller. // Run accepts a function that retrieves the current cluster configuration. -// Run will loop every time the trigger channel is +// Run will loop every time the trigger channel is. func (c *ControlPlaneConfigurationController) Run(ctx context.Context, getClusterConfig func(context.Context) (types.ClusterConfig, error)) { c.waitReady() @@ -71,8 +71,7 @@ func (c *ControlPlaneConfigurationController) Run(ctx context.Context, getCluste func (c *ControlPlaneConfigurationController) reconcile(ctx context.Context, config types.ClusterConfig) error { // kube-apiserver: external datastore - switch config.Datastore.GetType() { - case "external": + if config.Datastore.GetType() == "external" { // certificates certificatesChanged, err := setup.EnsureExtDatastorePKI(c.snap, &pki.ExternalDatastorePKI{ DatastoreCACert: config.Datastore.GetExternalCACert(), diff --git a/src/k8s/pkg/k8sd/controllers/control_plane_configuration_test.go b/src/k8s/pkg/k8sd/controllers/control_plane_configuration_test.go index 33ed9d4b3..89bda62a8 100644 --- a/src/k8s/pkg/k8sd/controllers/control_plane_configuration_test.go +++ b/src/k8s/pkg/k8sd/controllers/control_plane_configuration_test.go @@ -16,7 +16,7 @@ import ( . "github.com/onsi/gomega" ) -// channelSendTimeout is the timeout for pushing to channels for TestControlPlaneConfigController +// channelSendTimeout is the timeout for pushing to channels for TestControlPlaneConfigController. const channelSendTimeout = 100 * time.Millisecond type configProvider struct { @@ -197,7 +197,7 @@ func TestControlPlaneConfigController(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-apiserver", earg) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(eval)) }) } @@ -209,7 +209,7 @@ func TestControlPlaneConfigController(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-controller-manager", earg) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(eval)) }) } @@ -222,7 +222,7 @@ func TestControlPlaneConfigController(t *testing.T) { _, err := os.Stat(file) if mustExist { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } else { g.Expect(err).To(MatchError(os.ErrNotExist)) } diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/config.go b/src/k8s/pkg/k8sd/controllers/csrsigning/config.go index fa19d3e0b..08e88bda0 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/config.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/config.go @@ -1,6 +1,9 @@ package csrsigning -import "github.com/canonical/k8s/pkg/k8sd/types" +import ( + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/csrsigning" + "github.com/canonical/k8s/pkg/k8sd/types" +) type internalConfig struct { autoApprove bool @@ -8,7 +11,7 @@ type internalConfig struct { func internalConfigFromAnnotations(annotations types.Annotations) internalConfig { var cfg internalConfig - if v, ok := annotations.Get("k8sd/v1alpha1/csrsigning/auto-approve"); ok && v == "true" { + if v, ok := annotations.Get(apiv1_annotations.AnnotationAutoApprove); ok && v == "true" { cfg.autoApprove = true } return cfg diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/const.go b/src/k8s/pkg/k8sd/controllers/csrsigning/const.go index 074066396..a6729df73 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/const.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/const.go @@ -3,9 +3,9 @@ package csrsigning import "time" const ( - // requeueAfterSigningFailure is the time to requeue requests when any step of the signing process failed + // requeueAfterSigningFailure is the time to requeue requests when any step of the signing process failed. requeueAfterSigningFailure = 3 * time.Second - // requeueAfterWaitingForApproved is the amount of time to requeue requests if waiting for CSR to be approved + // requeueAfterWaitingForApproved is the amount of time to requeue requests if waiting for CSR to be approved. requeueAfterWaitingForApproved = 10 * time.Second ) diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/controller.go b/src/k8s/pkg/k8sd/controllers/csrsigning/controller.go index 3ecdb7877..03371a9d7 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/controller.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/controller.go @@ -10,7 +10,6 @@ import ( "github.com/canonical/k8s/pkg/snap" "github.com/canonical/k8s/pkg/utils" "k8s.io/client-go/rest" - "sigs.k8s.io/controller-runtime/pkg/cache" ctrllog "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/controller-runtime/pkg/manager" diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile.go b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile.go index 95ccbbb95..779f10777 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile.go @@ -9,6 +9,7 @@ import ( "fmt" "time" + "github.com/canonical/k8s/pkg/utils" pkiutil "github.com/canonical/k8s/pkg/utils/pki" certv1 "k8s.io/api/certificates/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" @@ -96,6 +97,15 @@ func (r *csrSigningReconciler) Reconcile(ctx context.Context, req ctrl.Request) return ctrl.Result{}, err } + notBefore := time.Now() + var notAfter time.Time + + if obj.Spec.ExpirationSeconds != nil { + notAfter = utils.SecondsToExpirationDate(notBefore, int(*obj.Spec.ExpirationSeconds)) + } else { + notAfter = time.Now().AddDate(10, 0, 0) + } + var crtPEM []byte switch obj.Spec.SignerName { case "k8sd.io/kubelet-serving": @@ -114,8 +124,8 @@ func (r *csrSigningReconciler) Reconcile(ctx context.Context, req ctrl.Request) CommonName: obj.Spec.Username, Organization: obj.Spec.Groups, }, - NotBefore: time.Now(), - NotAfter: time.Now().AddDate(10, 0, 0), // TODO: expiration date from obj, or config + NotBefore: notBefore, + NotAfter: notAfter, IPAddresses: certRequest.IPAddresses, DNSNames: certRequest.DNSNames, BasicConstraintsValid: true, @@ -149,8 +159,8 @@ func (r *csrSigningReconciler) Reconcile(ctx context.Context, req ctrl.Request) CommonName: obj.Spec.Username, Organization: obj.Spec.Groups, }, - NotBefore: time.Now(), - NotAfter: time.Now().AddDate(10, 0, 0), // TODO: expiration date from obj, or config + NotBefore: notBefore, + NotAfter: notAfter, BasicConstraintsValid: true, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature, @@ -181,8 +191,8 @@ func (r *csrSigningReconciler) Reconcile(ctx context.Context, req ctrl.Request) Subject: pkix.Name{ CommonName: "system:kube-proxy", }, - NotBefore: time.Now(), - NotAfter: time.Now().AddDate(10, 0, 0), // TODO: expiration date from obj, or config + NotBefore: notBefore, + NotAfter: notAfter, BasicConstraintsValid: true, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature, diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve.go b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve.go index c767ce9ed..3c1bcc6a4 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve.go @@ -13,7 +13,8 @@ import ( ) func reconcileAutoApprove(ctx context.Context, log log.Logger, csr *certv1.CertificateSigningRequest, - priv *rsa.PrivateKey, client client.Client) (ctrl.Result, error) { + priv *rsa.PrivateKey, client client.Client, +) (ctrl.Result, error) { var result certv1.RequestConditionType if err := validateCSR(csr, priv); err != nil { diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve_test.go b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve_test.go index 78cbb819e..9c2a6f574 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve_test.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve_test.go @@ -8,15 +8,14 @@ import ( "errors" "testing" + k8smock "github.com/canonical/k8s/pkg/k8sd/controllers/csrsigning/test" + "github.com/canonical/k8s/pkg/log" + pkiutil "github.com/canonical/k8s/pkg/utils/pki" . "github.com/onsi/gomega" certv1 "k8s.io/api/certificates/v1" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ctrl "sigs.k8s.io/controller-runtime" - - k8smock "github.com/canonical/k8s/pkg/k8sd/controllers/csrsigning/test" - "github.com/canonical/k8s/pkg/log" - pkiutil "github.com/canonical/k8s/pkg/utils/pki" ) func TestAutoApprove(t *testing.T) { diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_test.go b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_test.go index 04493f83d..f731c82d8 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_test.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_test.go @@ -8,6 +8,11 @@ import ( "testing" "time" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/csrsigning" + k8smock "github.com/canonical/k8s/pkg/k8sd/controllers/csrsigning/test" + "github.com/canonical/k8s/pkg/k8sd/types" + "github.com/canonical/k8s/pkg/log" + pkiutil "github.com/canonical/k8s/pkg/utils/pki" . "github.com/onsi/gomega" certv1 "k8s.io/api/certificates/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" @@ -16,11 +21,6 @@ import ( "k8s.io/utils/ptr" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" - - k8smock "github.com/canonical/k8s/pkg/k8sd/controllers/csrsigning/test" - "github.com/canonical/k8s/pkg/k8sd/types" - "github.com/canonical/k8s/pkg/log" - pkiutil "github.com/canonical/k8s/pkg/utils/pki" ) func TestCSRNotFound(t *testing.T) { @@ -69,7 +69,7 @@ func TestFailedToGetCSR(t *testing.T) { result, err := reconciler.Reconcile(context.Background(), getDefaultRequest()) g.Expect(result).To(Equal(ctrl.Result{})) - g.Expect(err).To(MatchError(getErr)) + g.Expect(err).To(MatchError(getErr.Error())) } func TestHasSignedCertificate(t *testing.T) { @@ -298,7 +298,7 @@ func TestNotApprovedCSR(t *testing.T) { getClusterConfig: func(context.Context) (types.ClusterConfig, error) { return types.ClusterConfig{ Annotations: map[string]string{ - "k8sd/v1alpha1/csrsigning/auto-approve": "true", + apiv1_annotations.AnnotationAutoApprove: "true", }, Certificates: types.Certificates{ K8sdPrivateKey: ptr.To(priv), diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/validate.go b/src/k8s/pkg/k8sd/controllers/csrsigning/validate.go index d3d9df0a9..4e0c9b414 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/validate.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/validate.go @@ -4,6 +4,7 @@ import ( "crypto/rsa" "crypto/sha256" "crypto/subtle" + "encoding/base64" "fmt" "github.com/canonical/k8s/pkg/utils" @@ -21,7 +22,12 @@ func validateCSR(obj *certv1.CertificateSigningRequest, priv *rsa.PrivateKey) er return fmt.Errorf("failed to parse x509 certificate request: %w", err) } - encryptedSignature := obj.Annotations["k8sd.io/signature"] + encryptedSignatureB64 := obj.Annotations["k8sd.io/signature"] + encryptedSignature, err := base64.StdEncoding.DecodeString(encryptedSignatureB64) + if err != nil { + return fmt.Errorf("failed to decode b64 signature: %w", err) + } + signature, err := rsa.DecryptPKCS1v15(nil, priv, []byte(encryptedSignature)) if err != nil { return fmt.Errorf("failed to decrypt signature: %w", err) diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/validate_test.go b/src/k8s/pkg/k8sd/controllers/csrsigning/validate_test.go index 7c806a484..7e9a919a8 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/validate_test.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/validate_test.go @@ -5,6 +5,7 @@ import ( "crypto/rsa" "crypto/sha256" "crypto/x509/pkix" + "encoding/base64" "testing" pkiutil "github.com/canonical/k8s/pkg/utils/pki" @@ -93,7 +94,7 @@ func TestValidateCSREncryption(t *testing.T) { }, }, expectErr: true, - expectErrMessage: "failed to decrypt signature", + expectErrMessage: "failed to decode b64 signature", }, { name: "Missing Signature", @@ -219,5 +220,5 @@ func mustCreateEncryptedSignature(g Gomega, pub *rsa.PublicKey, csrPEM string) s signature, err := rsa.EncryptPKCS1v15(rand.Reader, pub, hash.Sum(nil)) g.Expect(err).NotTo(HaveOccurred()) - return string(signature) + return base64.StdEncoding.EncodeToString(signature) } diff --git a/src/k8s/pkg/k8sd/controllers/feature.go b/src/k8s/pkg/k8sd/controllers/feature.go index 909a43bc3..33589b88e 100644 --- a/src/k8s/pkg/k8sd/controllers/feature.go +++ b/src/k8s/pkg/k8sd/controllers/feature.go @@ -72,6 +72,7 @@ func NewFeatureController(opts FeatureControllerOpts) *FeatureController { func (c *FeatureController) Run( ctx context.Context, getClusterConfig func(context.Context) (types.ClusterConfig, error), + getLocalhostAddress func() (string, error), notifyDNSChangedIP func(ctx context.Context, dnsIP string) error, setFeatureStatus func(ctx context.Context, name types.FeatureName, featureStatus types.FeatureStatus) error, ) { @@ -79,7 +80,11 @@ func (c *FeatureController) Run( ctx = log.NewContext(ctx, log.FromContext(ctx).WithValues("controller", "feature")) go c.reconcileLoop(ctx, getClusterConfig, setFeatureStatus, features.Network, c.triggerNetworkCh, c.reconciledNetworkCh, func(cfg types.ClusterConfig) (types.FeatureStatus, error) { - return features.Implementation.ApplyNetwork(ctx, c.snap, cfg.Network, cfg.Annotations) + localhostAddress, err := getLocalhostAddress() + if err != nil { + return types.FeatureStatus{Enabled: false, Message: "failed to determine the localhost address"}, fmt.Errorf("failed to get localhost address: %w", err) + } + return features.Implementation.ApplyNetwork(ctx, c.snap, localhostAddress, cfg.APIServer, cfg.Network, cfg.Annotations) }) go c.reconcileLoop(ctx, getClusterConfig, setFeatureStatus, features.Gateway, c.triggerGatewayCh, c.reconciledGatewayCh, func(cfg types.ClusterConfig) (types.FeatureStatus, error) { diff --git a/src/k8s/pkg/k8sd/controllers/node_configuration_test.go b/src/k8s/pkg/k8sd/controllers/node_configuration_test.go index 38e6a3bf8..b6936e201 100644 --- a/src/k8s/pkg/k8sd/controllers/node_configuration_test.go +++ b/src/k8s/pkg/k8sd/controllers/node_configuration_test.go @@ -147,7 +147,7 @@ func TestConfigPropagation(t *testing.T) { for ekey, evalue := range tc.expectArgs { val, err := snaputil.GetServiceArgument(s, "kubelet", ekey) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(evalue)) } diff --git a/src/k8s/pkg/k8sd/database/capi_auth.go b/src/k8s/pkg/k8sd/database/capi_auth.go index f8a4f54fc..498c2eaa7 100644 --- a/src/k8s/pkg/k8sd/database/capi_auth.go +++ b/src/k8s/pkg/k8sd/database/capi_auth.go @@ -8,12 +8,10 @@ import ( "github.com/canonical/microcluster/v3/cluster" ) -var ( - clusterAPIConfigsStmts = map[string]int{ - "insert-capi-token": MustPrepareStatement("cluster-configs", "insert-capi-token.sql"), - "select-capi-token": MustPrepareStatement("cluster-configs", "select-capi-token.sql"), - } -) +var clusterAPIConfigsStmts = map[string]int{ + "insert-capi-token": MustPrepareStatement("cluster-configs", "insert-capi-token.sql"), + "select-capi-token": MustPrepareStatement("cluster-configs", "select-capi-token.sql"), +} // SetClusterAPIToken stores the ClusterAPI token in the cluster config. func SetClusterAPIToken(ctx context.Context, tx *sql.Tx, token string) error { diff --git a/src/k8s/pkg/k8sd/database/capi_auth_test.go b/src/k8s/pkg/k8sd/database/capi_auth_test.go index eca571532..2ffbcdf46 100644 --- a/src/k8s/pkg/k8sd/database/capi_auth_test.go +++ b/src/k8s/pkg/k8sd/database/capi_auth_test.go @@ -17,10 +17,10 @@ func TestClusterAPIAuthTokens(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { err := database.SetClusterAPIToken(ctx, tx, token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("CheckAuthToken", func(t *testing.T) { @@ -28,22 +28,22 @@ func TestClusterAPIAuthTokens(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { valid, err := database.ValidateClusterAPIToken(ctx, tx, token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("InvalidToken", func(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { valid, err := database.ValidateClusterAPIToken(ctx, tx, "invalid-token") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeFalse()) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) }) diff --git a/src/k8s/pkg/k8sd/database/cluster_config.go b/src/k8s/pkg/k8sd/database/cluster_config.go index 23e945a48..aab8f5c4b 100644 --- a/src/k8s/pkg/k8sd/database/cluster_config.go +++ b/src/k8s/pkg/k8sd/database/cluster_config.go @@ -10,12 +10,10 @@ import ( "github.com/canonical/microcluster/v3/cluster" ) -var ( - clusterConfigsStmts = map[string]int{ - "insert-v1alpha2": MustPrepareStatement("cluster-configs", "insert-v1alpha2.sql"), - "select-v1alpha2": MustPrepareStatement("cluster-configs", "select-v1alpha2.sql"), - } -) +var clusterConfigsStmts = map[string]int{ + "insert-v1alpha2": MustPrepareStatement("cluster-configs", "insert-v1alpha2.sql"), + "select-v1alpha2": MustPrepareStatement("cluster-configs", "select-v1alpha2.sql"), +} // SetClusterConfig updates the cluster configuration with any non-empty values that are set. // SetClusterConfig will attempt to merge the existing and new configs, and return an error if any protected fields have changed. diff --git a/src/k8s/pkg/k8sd/database/cluster_config_test.go b/src/k8s/pkg/k8sd/database/cluster_config_test.go index aee438f6f..e4b21d220 100644 --- a/src/k8s/pkg/k8sd/database/cluster_config_test.go +++ b/src/k8s/pkg/k8sd/database/cluster_config_test.go @@ -3,11 +3,11 @@ package database_test import ( "context" "database/sql" - "github.com/canonical/k8s/pkg/utils" "testing" "github.com/canonical/k8s/pkg/k8sd/database" "github.com/canonical/k8s/pkg/k8sd/types" + "github.com/canonical/k8s/pkg/utils" . "github.com/onsi/gomega" ) @@ -26,19 +26,19 @@ func TestClusterConfig(t *testing.T) { // Write some config to the database err := d.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { _, err := database.SetClusterConfig(context.Background(), tx, expectedClusterConfig) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) // Retrieve it and map it to the struct err = d.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { clusterConfig, err := database.GetClusterConfig(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(clusterConfig).To(Equal(expectedClusterConfig)) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("CannotUpdateCA", func(t *testing.T) { @@ -65,11 +65,11 @@ func TestClusterConfig(t *testing.T) { err = d.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { clusterConfig, err := database.GetClusterConfig(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(clusterConfig).To(Equal(expectedClusterConfig)) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("Update", func(t *testing.T) { @@ -104,18 +104,18 @@ func TestClusterConfig(t *testing.T) { }, }) g.Expect(returnedConfig).To(Equal(expectedClusterConfig)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = d.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { clusterConfig, err := database.GetClusterConfig(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(clusterConfig).To(Equal(expectedClusterConfig)) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) } diff --git a/src/k8s/pkg/k8sd/database/feature_status_test.go b/src/k8s/pkg/k8sd/database/feature_status_test.go index 61e380f98..200e57f3b 100644 --- a/src/k8s/pkg/k8sd/database/feature_status_test.go +++ b/src/k8s/pkg/k8sd/database/feature_status_test.go @@ -6,11 +6,10 @@ import ( "testing" "time" - . "github.com/onsi/gomega" - "github.com/canonical/k8s/pkg/k8sd/database" "github.com/canonical/k8s/pkg/k8sd/features" "github.com/canonical/k8s/pkg/k8sd/types" + . "github.com/onsi/gomega" ) func TestFeatureStatus(t *testing.T) { @@ -45,21 +44,20 @@ func TestFeatureStatus(t *testing.T) { t.Run("ReturnNothingInitially", func(t *testing.T) { g := NewWithT(t) ss, err := database.GetFeatureStatuses(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ss).To(BeEmpty()) - }) t.Run("SettingNewStatus", func(t *testing.T) { g := NewWithT(t) err := database.SetFeatureStatus(ctx, tx, features.Network, networkStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = database.SetFeatureStatus(ctx, tx, features.DNS, dnsStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) ss, err := database.GetFeatureStatuses(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ss).To(HaveLen(2)) g.Expect(ss[features.Network].Enabled).To(Equal(networkStatus.Enabled)) @@ -71,26 +69,25 @@ func TestFeatureStatus(t *testing.T) { g.Expect(ss[features.DNS].Message).To(Equal(dnsStatus.Message)) g.Expect(ss[features.DNS].Version).To(Equal(dnsStatus.Version)) g.Expect(ss[features.DNS].UpdatedAt).To(Equal(dnsStatus.UpdatedAt)) - }) t.Run("UpdatingStatus", func(t *testing.T) { g := NewWithT(t) err := database.SetFeatureStatus(ctx, tx, features.Network, networkStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = database.SetFeatureStatus(ctx, tx, features.DNS, dnsStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) // set and update err = database.SetFeatureStatus(ctx, tx, features.Network, networkStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = database.SetFeatureStatus(ctx, tx, features.DNS, dnsStatus2) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = database.SetFeatureStatus(ctx, tx, features.Gateway, gatewayStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) ss, err := database.GetFeatureStatuses(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ss).To(HaveLen(3)) // network stayed the same diff --git a/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens.go b/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens.go index 22c7b7a2b..73e708ef7 100644 --- a/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens.go +++ b/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens.go @@ -13,15 +13,13 @@ import ( "github.com/canonical/microcluster/v3/cluster" ) -var ( - k8sdTokensStmts = map[string]int{ - "insert-token": MustPrepareStatement("kubernetes-auth-tokens", "insert-token.sql"), - "select-by-token": MustPrepareStatement("kubernetes-auth-tokens", "select-by-token.sql"), - "select-by-username": MustPrepareStatement("kubernetes-auth-tokens", "select-by-username.sql"), - "delete-by-token": MustPrepareStatement("kubernetes-auth-tokens", "delete-by-token.sql"), - "delete-by-username": MustPrepareStatement("kubernetes-auth-tokens", "delete-by-username.sql"), - } -) +var k8sdTokensStmts = map[string]int{ + "insert-token": MustPrepareStatement("kubernetes-auth-tokens", "insert-token.sql"), + "select-by-token": MustPrepareStatement("kubernetes-auth-tokens", "select-by-token.sql"), + "select-by-username": MustPrepareStatement("kubernetes-auth-tokens", "select-by-username.sql"), + "delete-by-token": MustPrepareStatement("kubernetes-auth-tokens", "delete-by-token.sql"), + "delete-by-username": MustPrepareStatement("kubernetes-auth-tokens", "delete-by-username.sql"), +} func groupsToString(inGroups []string) (string, error) { groupMap := make(map[string]struct{}, len(inGroups)) diff --git a/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens_test.go b/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens_test.go index 18d73779b..dd7d8ce9e 100644 --- a/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens_test.go +++ b/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens_test.go @@ -19,27 +19,27 @@ func TestKubernetesAuthTokens(t *testing.T) { var err error token1, err = database.GetOrCreateToken(ctx, tx, "user1", []string{"group1", "group2"}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token1).To(Not(BeEmpty())) token2, err = database.GetOrCreateToken(ctx, tx, "user2", []string{"group1", "group2"}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token2).To(Not(BeEmpty())) g.Expect(token1).To(Not(Equal(token2))) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) t.Run("Existing", func(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { token, err := database.GetOrCreateToken(ctx, tx, "user1", []string{"group1", "group2"}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(Equal(token1)) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) @@ -48,23 +48,23 @@ func TestKubernetesAuthTokens(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { username, groups, err := database.CheckToken(ctx, tx, token1) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(username).To(Equal("user1")) g.Expect(groups).To(ConsistOf("group1", "group2")) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("user2", func(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { username, groups, err := database.CheckToken(ctx, tx, token2) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(username).To(Equal("user2")) g.Expect(groups).To(ConsistOf("group1", "group2")) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) @@ -72,15 +72,15 @@ func TestKubernetesAuthTokens(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { err := database.DeleteToken(ctx, tx, token2) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) username, groups, err := database.CheckToken(ctx, tx, token2) - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) g.Expect(username).To(BeEmpty()) g.Expect(groups).To(BeEmpty()) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) } diff --git a/src/k8s/pkg/k8sd/database/schema.go b/src/k8s/pkg/k8sd/database/schema.go index 7bb349a8a..b96a5559f 100644 --- a/src/k8s/pkg/k8sd/database/schema.go +++ b/src/k8s/pkg/k8sd/database/schema.go @@ -18,11 +18,11 @@ var ( SchemaExtensions = []schema.Update{ schemaApplyMigration("kubernetes-auth-tokens", "000-create.sql"), schemaApplyMigration("cluster-configs", "000-create.sql"), - + schemaApplyMigration("worker-nodes", "000-create.sql"), schemaApplyMigration("worker-tokens", "000-create.sql"), - schemaApplyMigration("worker-tokens", "001-add-expiry.sql"), - schemaApplyMigration("feature-status", "000-feature-status.sql"), + schemaApplyMigration("worker-tokens", "001-add-expiry.sql"), + schemaApplyMigration("worker-nodes", "001-delete.sql"), } //go:embed sql/migrations @@ -36,7 +36,7 @@ func schemaApplyMigration(migrationPath ...string) schema.Update { path := filepath.Join(append([]string{"sql", "migrations"}, migrationPath...)...) b, err := sqlMigrations.ReadFile(path) if err != nil { - panic(fmt.Errorf("invalid migration file %s: %s", path, err)) + panic(fmt.Errorf("invalid migration file %s: %w", path, err)) } return func(ctx context.Context, tx *sql.Tx) error { if _, err := tx.ExecContext(ctx, string(b)); err != nil { @@ -51,7 +51,7 @@ func MustPrepareStatement(queryPath ...string) int { path := filepath.Join(append([]string{"sql", "queries"}, queryPath...)...) b, err := sqlQueries.ReadFile(path) if err != nil { - panic(fmt.Errorf("invalid query file %s: %s", path, err)) + panic(fmt.Errorf("invalid query file %s: %w", path, err)) } return cluster.RegisterStmt(string(b)) } diff --git a/src/k8s/pkg/k8sd/database/sql/migrations/worker-nodes/000-create.sql b/src/k8s/pkg/k8sd/database/sql/migrations/worker-nodes/000-create.sql index be3330716..893edba7b 100644 --- a/src/k8s/pkg/k8sd/database/sql/migrations/worker-nodes/000-create.sql +++ b/src/k8s/pkg/k8sd/database/sql/migrations/worker-nodes/000-create.sql @@ -2,4 +2,4 @@ CREATE TABLE worker_nodes ( id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, name TEXT NOT NULL, UNIQUE(name) -); +) diff --git a/src/k8s/pkg/k8sd/database/sql/migrations/worker-nodes/001-delete.sql b/src/k8s/pkg/k8sd/database/sql/migrations/worker-nodes/001-delete.sql new file mode 100644 index 000000000..2c47ee2fc --- /dev/null +++ b/src/k8s/pkg/k8sd/database/sql/migrations/worker-nodes/001-delete.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS worker_nodes diff --git a/src/k8s/pkg/k8sd/database/util_test.go b/src/k8s/pkg/k8sd/database/util_test.go index 265302bfe..e39712b59 100644 --- a/src/k8s/pkg/k8sd/database/util_test.go +++ b/src/k8s/pkg/k8sd/database/util_test.go @@ -12,16 +12,14 @@ import ( ) const ( - // microclusterDatabaseInitTimeout is the timeout for microcluster database initialization operations + // microclusterDatabaseInitTimeout is the timeout for microcluster database initialization operations. microclusterDatabaseInitTimeout = 3 * time.Second - // microclusterDatabaseShutdownTimeout is the timeout for microcluster database shutdown operations + // microclusterDatabaseShutdownTimeout is the timeout for microcluster database shutdown operations. microclusterDatabaseShutdownTimeout = 3 * time.Second ) -var ( - // nextIdx is used to pick different listen ports for each microcluster instance - nextIdx int -) +// nextIdx is used to pick different listen ports for each microcluster instance. +var nextIdx int // DB is an interface for the internal microcluster DB type. type DB interface { @@ -38,13 +36,13 @@ type DB interface { // WithDB(t, func(ctx context.Context, db DB) { // err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { // token, err := database.GetOrCreateToken(ctx, tx, "user1", []string{"group1", "group2"}) -// if !g.Expect(err).To(BeNil()) { +// if !g.Expect(err).To(Not(HaveOccurred())) { // return err // } // g.Expect(token).To(Not(BeEmpty())) // return nil // }) -// g.Expect(err).To(BeNil()) +// g.Expect(err).To(Not(HaveOccurred())) // }) // }) // } diff --git a/src/k8s/pkg/k8sd/database/worker.go b/src/k8s/pkg/k8sd/database/worker.go index c97da8267..043231a1a 100644 --- a/src/k8s/pkg/k8sd/database/worker.go +++ b/src/k8s/pkg/k8sd/database/worker.go @@ -12,13 +12,11 @@ import ( "github.com/canonical/microcluster/v3/cluster" ) -var ( - workerStmts = map[string]int{ - "insert-token": MustPrepareStatement("worker-tokens", "insert.sql"), - "select-token": MustPrepareStatement("worker-tokens", "select.sql"), - "delete-token": MustPrepareStatement("worker-tokens", "delete-by-token.sql"), - } -) +var workerStmts = map[string]int{ + "insert-token": MustPrepareStatement("worker-tokens", "insert.sql"), + "select-token": MustPrepareStatement("worker-tokens", "select.sql"), + "delete-token": MustPrepareStatement("worker-tokens", "delete-by-token.sql"), +} // CheckWorkerNodeToken returns true if the specified token can be used to join the specified node on the cluster. // CheckWorkerNodeToken will return true if the token is empty or if the token is associated with the specified node diff --git a/src/k8s/pkg/k8sd/database/worker_test.go b/src/k8s/pkg/k8sd/database/worker_test.go index 61de6e499..ce154fcdb 100644 --- a/src/k8s/pkg/k8sd/database/worker_test.go +++ b/src/k8s/pkg/k8sd/database/worker_test.go @@ -17,39 +17,39 @@ func TestWorkerNodeToken(t *testing.T) { t.Run("Default", func(t *testing.T) { g := NewWithT(t) exists, err := database.CheckWorkerNodeToken(ctx, tx, "somenode", "sometoken") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(exists).To(BeFalse()) token, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "somenode", tokenExpiry) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(HaveLen(48)) othertoken, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "someothernode", tokenExpiry) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(othertoken).To(HaveLen(48)) g.Expect(othertoken).NotTo(Equal(token)) valid, err := database.CheckWorkerNodeToken(ctx, tx, "somenode", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) valid, err = database.CheckWorkerNodeToken(ctx, tx, "someothernode", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeFalse()) valid, err = database.CheckWorkerNodeToken(ctx, tx, "someothernode", othertoken) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) err = database.DeleteWorkerNodeToken(ctx, tx, token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) valid, err = database.CheckWorkerNodeToken(ctx, tx, "somenode", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeFalse()) newToken, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "somenode", tokenExpiry) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(newToken).To(HaveLen(48)) g.Expect(newToken).ToNot(Equal(token)) }) @@ -58,22 +58,22 @@ func TestWorkerNodeToken(t *testing.T) { t.Run("Valid", func(t *testing.T) { g := NewWithT(t) token, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "nodeExpiry1", time.Now().Add(time.Hour)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(HaveLen(48)) valid, err := database.CheckWorkerNodeToken(ctx, tx, "nodeExpiry1", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) }) t.Run("Expired", func(t *testing.T) { g := NewWithT(t) token, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "nodeExpiry2", time.Now().Add(-time.Hour)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(HaveLen(48)) valid, err := database.CheckWorkerNodeToken(ctx, tx, "nodeExpiry2", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeFalse()) }) }) @@ -81,7 +81,7 @@ func TestWorkerNodeToken(t *testing.T) { t.Run("AnyNodeName", func(t *testing.T) { g := NewWithT(t) token, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "", tokenExpiry) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(HaveLen(48)) for _, name := range []string{"", "test", "other"} { @@ -89,7 +89,7 @@ func TestWorkerNodeToken(t *testing.T) { g := NewWithT(t) valid, err := database.CheckWorkerNodeToken(ctx, tx, name, token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) }) } diff --git a/src/k8s/pkg/k8sd/features/calico/internal.go b/src/k8s/pkg/k8sd/features/calico/internal.go index 930bc674a..e3e48443d 100644 --- a/src/k8s/pkg/k8sd/features/calico/internal.go +++ b/src/k8s/pkg/k8sd/features/calico/internal.go @@ -4,27 +4,10 @@ import ( "fmt" "strings" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/calico" "github.com/canonical/k8s/pkg/k8sd/types" ) -const ( - annotationAPIServerEnabled = "k8sd/v1alpha1/calico/apiserver-enabled" - annotationEncapsulationV4 = "k8sd/v1alpha1/calico/encapsulation-v4" - annotationEncapsulationV6 = "k8sd/v1alpha1/calico/encapsulation-v6" - annotationAutodetectionV4FirstFound = "k8sd/v1alpha1/calico/autodetection-v4/firstFound" - annotationAutodetectionV4Kubernetes = "k8sd/v1alpha1/calico/autodetection-v4/kubernetes" - annotationAutodetectionV4Interface = "k8sd/v1alpha1/calico/autodetection-v4/interface" - annotationAutodetectionV4SkipInterface = "k8sd/v1alpha1/calico/autodetection-v4/skipInterface" - annotationAutodetectionV4CanReach = "k8sd/v1alpha1/calico/autodetection-v4/canReach" - annotationAutodetectionV4CIDRs = "k8sd/v1alpha1/calico/autodetection-v4/cidrs" - annotationAutodetectionV6FirstFound = "k8sd/v1alpha1/calico/autodetection-v6/firstFound" - annotationAutodetectionV6Kubernetes = "k8sd/v1alpha1/calico/autodetection-v6/kubernetes" - annotationAutodetectionV6Interface = "k8sd/v1alpha1/calico/autodetection-v6/interface" - annotationAutodetectionV6SkipInterface = "k8sd/v1alpha1/calico/autodetection-v6/skipInterface" - annotationAutodetectionV6CanReach = "k8sd/v1alpha1/calico/autodetection-v6/canReach" - annotationAutodetectionV6CIDRs = "k8sd/v1alpha1/calico/autodetection-v6/cidrs" -) - type config struct { encapsulationV4 string encapsulationV6 string @@ -64,7 +47,11 @@ func parseAutodetectionAnnotations(annotations types.Annotations, autodetectionM case "firstFound": autodetectionValue = autodetectionValue == "true" case "cidrs": - autodetectionValue = strings.Split(autodetectionValue.(string), ",") + if strValue, ok := autodetectionValue.(string); ok { + autodetectionValue = strings.Split(strValue, ",") + } else { + return nil, fmt.Errorf("invalid type for cidrs annotation: %T", autodetectionValue) + } } return map[string]any{ @@ -82,18 +69,18 @@ func internalConfig(annotations types.Annotations) (config, error) { apiServerEnabled: defaultAPIServerEnabled, } - if v, ok := annotations.Get(annotationAPIServerEnabled); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationAPIServerEnabled); ok { c.apiServerEnabled = v == "true" } - if v, ok := annotations.Get(annotationEncapsulationV4); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationEncapsulationV4); ok { if err := checkEncapsulation(v); err != nil { return config{}, fmt.Errorf("invalid encapsulation-v4 annotation: %w", err) } c.encapsulationV4 = v } - if v, ok := annotations.Get(annotationEncapsulationV6); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationEncapsulationV6); ok { if err := checkEncapsulation(v); err != nil { return config{}, fmt.Errorf("invalid encapsulation-v6 annotation: %w", err) } @@ -101,12 +88,12 @@ func internalConfig(annotations types.Annotations) (config, error) { } v4Map := map[string]string{ - annotationAutodetectionV4FirstFound: "firstFound", - annotationAutodetectionV4Kubernetes: "kubernetes", - annotationAutodetectionV4Interface: "interface", - annotationAutodetectionV4SkipInterface: "skipInterface", - annotationAutodetectionV4CanReach: "canReach", - annotationAutodetectionV4CIDRs: "cidrs", + apiv1_annotations.AnnotationAutodetectionV4FirstFound: "firstFound", + apiv1_annotations.AnnotationAutodetectionV4Kubernetes: "kubernetes", + apiv1_annotations.AnnotationAutodetectionV4Interface: "interface", + apiv1_annotations.AnnotationAutodetectionV4SkipInterface: "skipInterface", + apiv1_annotations.AnnotationAutodetectionV4CanReach: "canReach", + apiv1_annotations.AnnotationAutodetectionV4CIDRs: "cidrs", } autodetectionV4, err := parseAutodetectionAnnotations(annotations, v4Map) @@ -119,12 +106,12 @@ func internalConfig(annotations types.Annotations) (config, error) { } v6Map := map[string]string{ - annotationAutodetectionV6FirstFound: "firstFound", - annotationAutodetectionV6Kubernetes: "kubernetes", - annotationAutodetectionV6Interface: "interface", - annotationAutodetectionV6SkipInterface: "skipInterface", - annotationAutodetectionV6CanReach: "canReach", - annotationAutodetectionV6CIDRs: "cidrs", + apiv1_annotations.AnnotationAutodetectionV6FirstFound: "firstFound", + apiv1_annotations.AnnotationAutodetectionV6Kubernetes: "kubernetes", + apiv1_annotations.AnnotationAutodetectionV6Interface: "interface", + apiv1_annotations.AnnotationAutodetectionV6SkipInterface: "skipInterface", + apiv1_annotations.AnnotationAutodetectionV6CanReach: "canReach", + apiv1_annotations.AnnotationAutodetectionV6CIDRs: "cidrs", } autodetectionV6, err := parseAutodetectionAnnotations(annotations, v6Map) diff --git a/src/k8s/pkg/k8sd/features/calico/internal_test.go b/src/k8s/pkg/k8sd/features/calico/internal_test.go index dd4c630fb..208198315 100644 --- a/src/k8s/pkg/k8sd/features/calico/internal_test.go +++ b/src/k8s/pkg/k8sd/features/calico/internal_test.go @@ -3,6 +3,7 @@ package calico import ( "testing" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/calico" . "github.com/onsi/gomega" ) @@ -26,8 +27,8 @@ func TestInternalConfig(t *testing.T) { { name: "Valid", annotations: map[string]string{ - annotationAPIServerEnabled: "true", - annotationEncapsulationV4: "IPIP", + apiv1_annotations.AnnotationAPIServerEnabled: "true", + apiv1_annotations.AnnotationEncapsulationV4: "IPIP", }, expectedConfig: config{ apiServerEnabled: true, @@ -39,15 +40,15 @@ func TestInternalConfig(t *testing.T) { { name: "InvalidEncapsulation", annotations: map[string]string{ - annotationEncapsulationV4: "Invalid", + apiv1_annotations.AnnotationEncapsulationV4: "Invalid", }, expectError: true, }, { name: "InvalidAPIServerEnabled", annotations: map[string]string{ - annotationAPIServerEnabled: "invalid", - annotationEncapsulationV4: "VXLAN", + apiv1_annotations.AnnotationAPIServerEnabled: "invalid", + apiv1_annotations.AnnotationEncapsulationV4: "VXLAN", }, expectedConfig: config{ apiServerEnabled: false, @@ -59,15 +60,15 @@ func TestInternalConfig(t *testing.T) { { name: "MultipleAutodetectionV4", annotations: map[string]string{ - annotationAutodetectionV4FirstFound: "true", - annotationAutodetectionV4Kubernetes: "true", + apiv1_annotations.AnnotationAutodetectionV4FirstFound: "true", + apiv1_annotations.AnnotationAutodetectionV4Kubernetes: "true", }, expectError: true, }, { name: "ValidAutodetectionCidrs", annotations: map[string]string{ - annotationAutodetectionV4CIDRs: "10.1.0.0/16,2001:0db8::/32", + apiv1_annotations.AnnotationAutodetectionV4CIDRs: "10.1.0.0/16,2001:0db8::/32", }, expectedConfig: config{ apiServerEnabled: false, diff --git a/src/k8s/pkg/k8sd/features/calico/network.go b/src/k8s/pkg/k8sd/features/calico/network.go index 82e98c788..8820e6c8f 100644 --- a/src/k8s/pkg/k8sd/features/calico/network.go +++ b/src/k8s/pkg/k8sd/features/calico/network.go @@ -17,16 +17,16 @@ const ( deleteFailedMsgTmpl = "Failed to delete Calico, the error was: %v" ) -// ApplyNetwork will deploy Calico when cfg.Enabled is true. -// ApplyNetwork will remove Calico when cfg.Enabled is false. +// ApplyNetwork will deploy Calico when network.Enabled is true. +// ApplyNetwork will remove Calico when network.Enabled is false. // ApplyNetwork will always return a FeatureStatus indicating the current status of the // deployment. // ApplyNetwork returns an error if anything fails. The error is also wrapped in the .Message field of the // returned FeatureStatus. -func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, annotations types.Annotations) (types.FeatureStatus, error) { +func ApplyNetwork(ctx context.Context, snap snap.Snap, _ string, apiserver types.APIServer, network types.Network, annotations types.Annotations) (types.FeatureStatus, error) { m := snap.HelmClient() - if !cfg.GetEnabled() { + if !network.GetEnabled() { if _, err := m.Apply(ctx, ChartCalico, helm.StateDeleted, nil); err != nil { err = fmt.Errorf("failed to uninstall network: %w", err) return types.FeatureStatus{ @@ -54,7 +54,7 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, annota } podIpPools := []map[string]any{} - ipv4PodCIDR, ipv6PodCIDR, err := utils.ParseCIDRs(cfg.GetPodCIDR()) + ipv4PodCIDR, ipv6PodCIDR, err := utils.SplitCIDRStrings(network.GetPodCIDR()) if err != nil { err = fmt.Errorf("invalid pod cidr: %w", err) return types.FeatureStatus{ @@ -79,9 +79,9 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, annota } serviceCIDRs := []string{} - ipv4ServiceCIDR, ipv6ServiceCIDR, err := utils.ParseCIDRs(cfg.GetServiceCIDR()) + ipv4ServiceCIDR, ipv6ServiceCIDR, err := utils.SplitCIDRStrings(network.GetServiceCIDR()) if err != nil { - err = fmt.Errorf("invalid service cidr: %v", err) + err = fmt.Errorf("invalid service cidr: %w", err) return types.FeatureStatus{ Enabled: false, Version: CalicoTag, diff --git a/src/k8s/pkg/k8sd/features/calico/network_test.go b/src/k8s/pkg/k8sd/features/calico/network_test.go index c0a324028..0a8b4d716 100644 --- a/src/k8s/pkg/k8sd/features/calico/network_test.go +++ b/src/k8s/pkg/k8sd/features/calico/network_test.go @@ -5,25 +5,25 @@ import ( "errors" "testing" - . "github.com/onsi/gomega" - + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/calico" "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" "github.com/canonical/k8s/pkg/k8sd/features/calico" "github.com/canonical/k8s/pkg/k8sd/types" snapmock "github.com/canonical/k8s/pkg/snap/mock" "github.com/canonical/k8s/pkg/utils" + . "github.com/onsi/gomega" "k8s.io/utils/ptr" ) // NOTE(hue): status.Message is not checked sometimes to avoid unnecessary complexity var defaultAnnotations = types.Annotations{ - "k8sd/v1alpha1/calico/apiserver-enabled": "true", - "k8sd/v1alpha1/calico/encapsulation-v4": "VXLAN", - "k8sd/v1alpha1/calico/encapsulation-v6": "VXLAN", - "k8sd/v1alpha1/calico/autodetection-v4/firstFound": "true", - "k8sd/v1alpha1/calico/autodetection-v6/firstFound": "true", + apiv1_annotations.AnnotationAPIServerEnabled: "true", + apiv1_annotations.AnnotationEncapsulationV4: "VXLAN", + apiv1_annotations.AnnotationEncapsulationV6: "VXLAN", + apiv1_annotations.AnnotationAutodetectionV4FirstFound: "true", + apiv1_annotations.AnnotationAutodetectionV6FirstFound: "true", } func TestDisabled(t *testing.T) { @@ -39,11 +39,14 @@ func TestDisabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(false), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) @@ -65,11 +68,14 @@ func TestDisabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(false), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).ToNot(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) @@ -94,17 +100,20 @@ func TestEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("invalid-cidr"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, defaultAnnotations) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, defaultAnnotations) g.Expect(err).To(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) g.Expect(status.Version).To(Equal(calico.CalicoTag)) - g.Expect(helmM.ApplyCalledWith).To(HaveLen(0)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) }) t.Run("InvalidServiceCIDR", func(t *testing.T) { g := NewWithT(t) @@ -115,18 +124,21 @@ func TestEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), ServiceCIDR: ptr.To("invalid-cidr"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, defaultAnnotations) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, defaultAnnotations) g.Expect(err).To(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) g.Expect(status.Version).To(Equal(calico.CalicoTag)) - g.Expect(helmM.ApplyCalledWith).To(HaveLen(0)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) }) t.Run("HelmApplyFails", func(t *testing.T) { g := NewWithT(t) @@ -140,13 +152,16 @@ func TestEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), ServiceCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, defaultAnnotations) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, defaultAnnotations) g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) @@ -157,7 +172,7 @@ func TestEnabled(t *testing.T) { callArgs := helmM.ApplyCalledWith[0] g.Expect(callArgs.Chart).To(Equal(calico.ChartCalico)) g.Expect(callArgs.State).To(Equal(helm.StatePresent)) - validateValues(t, callArgs.Values, cfg) + validateValues(t, callArgs.Values, network) }) t.Run("Success", func(t *testing.T) { g := NewWithT(t) @@ -168,13 +183,16 @@ func TestEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), ServiceCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, defaultAnnotations) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, defaultAnnotations) g.Expect(err).ToNot(HaveOccurred()) g.Expect(status.Enabled).To(BeTrue()) @@ -185,17 +203,17 @@ func TestEnabled(t *testing.T) { callArgs := helmM.ApplyCalledWith[0] g.Expect(callArgs.Chart).To(Equal(calico.ChartCalico)) g.Expect(callArgs.State).To(Equal(helm.StatePresent)) - validateValues(t, callArgs.Values, cfg) + validateValues(t, callArgs.Values, network) }) } -func validateValues(t *testing.T, values map[string]any, cfg types.Network) { +func validateValues(t *testing.T, values map[string]any, network types.Network) { g := NewWithT(t) - podIPv4CIDR, podIPv6CIDR, err := utils.ParseCIDRs(cfg.GetPodCIDR()) + podIPv4CIDR, podIPv6CIDR, err := utils.SplitCIDRStrings(network.GetPodCIDR()) g.Expect(err).ToNot(HaveOccurred()) - svcIPv4CIDR, svcIPv6CIDR, err := utils.ParseCIDRs(cfg.GetServiceCIDR()) + svcIPv4CIDR, svcIPv6CIDR, err := utils.SplitCIDRStrings(network.GetServiceCIDR()) g.Expect(err).ToNot(HaveOccurred()) // calico network @@ -211,10 +229,10 @@ func validateValues(t *testing.T, values map[string]any, cfg types.Network) { "encapsulation": "VXLAN", })) g.Expect(calicoNetwork["ipPools"].([]map[string]any)).To(HaveLen(2)) - g.Expect(calicoNetwork["nodeAddressAutodetectionV4"].(map[string]any)["firstFound"]).To(Equal(true)) - g.Expect(calicoNetwork["nodeAddressAutodetectionV6"].(map[string]any)["firstFound"]).To(Equal(true)) + g.Expect(calicoNetwork["nodeAddressAutodetectionV4"].(map[string]any)["firstFound"]).To(BeTrue()) + g.Expect(calicoNetwork["nodeAddressAutodetectionV6"].(map[string]any)["firstFound"]).To(BeTrue()) - g.Expect(values["apiServer"].(map[string]any)["enabled"]).To(Equal(true)) + g.Expect(values["apiServer"].(map[string]any)["enabled"]).To(BeTrue()) // service CIDRs g.Expect(values["serviceCIDRs"].([]string)).To(ContainElements(svcIPv4CIDR, svcIPv6CIDR)) diff --git a/src/k8s/pkg/k8sd/features/calico/status.go b/src/k8s/pkg/k8sd/features/calico/status.go index 423fe7426..8cafae4d4 100644 --- a/src/k8s/pkg/k8sd/features/calico/status.go +++ b/src/k8s/pkg/k8sd/features/calico/status.go @@ -5,7 +5,6 @@ import ( "fmt" "github.com/canonical/k8s/pkg/snap" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) diff --git a/src/k8s/pkg/k8sd/features/cilium/chart.go b/src/k8s/pkg/k8sd/features/cilium/chart.go index bcae2b11b..a6e9bc310 100644 --- a/src/k8s/pkg/k8sd/features/cilium/chart.go +++ b/src/k8s/pkg/k8sd/features/cilium/chart.go @@ -11,7 +11,7 @@ var ( ChartCilium = helm.InstallableChart{ Name: "ck-network", Namespace: "kube-system", - ManifestPath: filepath.Join("charts", "cilium-1.15.2.tgz"), + ManifestPath: filepath.Join("charts", "cilium-1.16.3.tgz"), } // ChartCiliumLoadBalancer represents manifests to deploy Cilium LoadBalancer resources. @@ -25,10 +25,10 @@ var ( chartGateway = helm.InstallableChart{ Name: "ck-gateway", Namespace: "kube-system", - ManifestPath: filepath.Join("charts", "gateway-api-1.0.0.tgz"), + ManifestPath: filepath.Join("charts", "gateway-api-1.1.0.tgz"), } - //chartGatewayClass represents a manifest to deploy a GatewayClass called ck-gateway. + // chartGatewayClass represents a manifest to deploy a GatewayClass called ck-gateway. chartGatewayClass = helm.InstallableChart{ Name: "ck-gateway-class", Namespace: "default", @@ -39,11 +39,11 @@ var ( ciliumAgentImageRepo = "ghcr.io/canonical/cilium" // CiliumAgentImageTag is the tag to use for the cilium-agent image. - CiliumAgentImageTag = "1.15.2-ck2" + CiliumAgentImageTag = "1.16.3-ck0" // ciliumOperatorImageRepo is the image to use for cilium-operator. ciliumOperatorImageRepo = "ghcr.io/canonical/cilium-operator" // ciliumOperatorImageTag is the tag to use for the cilium-operator image. - ciliumOperatorImageTag = "1.15.2-ck2" + ciliumOperatorImageTag = "1.16.3-ck0" ) diff --git a/src/k8s/pkg/k8sd/features/cilium/cleanup.go b/src/k8s/pkg/k8sd/features/cilium/cleanup.go index bb97321e8..78d1fd098 100644 --- a/src/k8s/pkg/k8sd/features/cilium/cleanup.go +++ b/src/k8s/pkg/k8sd/features/cilium/cleanup.go @@ -5,6 +5,7 @@ import ( "fmt" "os" "os/exec" + "strings" "github.com/canonical/k8s/pkg/snap" ) @@ -13,10 +14,30 @@ func CleanupNetwork(ctx context.Context, snap snap.Snap) error { os.Remove("/var/run/cilium/cilium.pid") if _, err := os.Stat("/opt/cni/bin/cilium-dbg"); err == nil { - if err := exec.CommandContext(ctx, "/opt/cni/bin/cilium-dbg", "cleanup", "--all-state", "--force").Run(); err != nil { + if err := exec.CommandContext(ctx, "/opt/cni/bin/cilium-dbg", "post-uninstall-cleanup", "--all-state", "--force").Run(); err != nil { return fmt.Errorf("cilium-dbg cleanup failed: %w", err) } } + for _, cmd := range []string{"iptables", "ip6tables", "iptables-legacy", "ip6tables-legacy"} { + out, err := exec.Command(fmt.Sprintf("%s-save", cmd)).Output() + if err != nil { + return fmt.Errorf("failed to read iptables rules: %w", err) + } + + lines := strings.Split(string(out), "\n") + for i, line := range lines { + if strings.Contains(strings.ToLower(line), "cilium") { + lines[i] = "" + } + } + + restore := exec.Command(fmt.Sprintf("%s-restore", cmd)) + restore.Stdin = strings.NewReader(strings.Join(lines, "\n")) + if err := restore.Run(); err != nil { + return fmt.Errorf("failed to restore iptables rules: %w", err) + } + } + return nil } diff --git a/src/k8s/pkg/k8sd/features/cilium/gateway.go b/src/k8s/pkg/k8sd/features/cilium/gateway.go index 8632e660e..44b7f9b63 100644 --- a/src/k8s/pkg/k8sd/features/cilium/gateway.go +++ b/src/k8s/pkg/k8sd/features/cilium/gateway.go @@ -10,8 +10,8 @@ import ( ) const ( - gatewayDeleteFailedMsgTmpl = "Failed to delete Cilium Gateway, the error was %v" - gatewayDeployFailedMsgTmpl = "Failed to deploy Cilium Gateway, the error was %v" + GatewayDeleteFailedMsgTmpl = "Failed to delete Cilium Gateway, the error was %v" + GatewayDeployFailedMsgTmpl = "Failed to deploy Cilium Gateway, the error was %v" ) // ApplyGateway assumes that the managed Cilium CNI is already installed on the cluster. It will fail if that is not the case. @@ -38,7 +38,7 @@ func enableGateway(ctx context.Context, snap snap.Snap) (types.FeatureStatus, er return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -48,7 +48,7 @@ func enableGateway(ctx context.Context, snap snap.Snap) (types.FeatureStatus, er return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -58,7 +58,7 @@ func enableGateway(ctx context.Context, snap snap.Snap) (types.FeatureStatus, er return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -75,7 +75,7 @@ func enableGateway(ctx context.Context, snap snap.Snap) (types.FeatureStatus, er return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -95,7 +95,7 @@ func disableGateway(ctx context.Context, snap snap.Snap, network types.Network) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeleteFailedMsgTmpl, err), }, err } @@ -105,7 +105,7 @@ func disableGateway(ctx context.Context, snap snap.Snap, network types.Network) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeleteFailedMsgTmpl, err), }, err } @@ -116,7 +116,7 @@ func disableGateway(ctx context.Context, snap snap.Snap, network types.Network) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeleteFailedMsgTmpl, err), }, err } @@ -133,7 +133,7 @@ func disableGateway(ctx context.Context, snap snap.Snap, network types.Network) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } diff --git a/src/k8s/pkg/k8sd/features/cilium/gateway_test.go b/src/k8s/pkg/k8sd/features/cilium/gateway_test.go new file mode 100644 index 000000000..2bcab0b11 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/gateway_test.go @@ -0,0 +1,270 @@ +package cilium_test + +import ( + "context" + "errors" + "fmt" + "testing" + + "github.com/canonical/k8s/pkg/client/helm" + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/cilium" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + v1 "k8s.io/api/apps/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestGatewayEnabled(t *testing.T) { + t.Run("HelmApplyErr", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.GatewayDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("AlreadyDeployed", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: false, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(cilium.EnabledMsg)) + + helmCiliumArgs := helmM.ApplyCalledWith[2] + g.Expect(helmCiliumArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(helmCiliumArgs.State).To(Equal(helm.StateUpgradeOnly)) + g.Expect(helmCiliumArgs.Values["gatewayAPI"].(map[string]any)["enabled"]).To(BeTrue()) + }) + + t.Run("RolloutFail", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.GatewayDeployFailedMsgTmpl, err))) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium-operator", + Namespace: "kube-system", + }, + }, + &v1.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + }, + }, + ) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(cilium.EnabledMsg)) + }) +} + +func TestGatewayDisabled(t *testing.T) { + t.Run("HelmApplyErr", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.GatewayDeleteFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("AlreadyDeleted", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: false, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(cilium.DisabledMsg)) + + helmCiliumArgs := helmM.ApplyCalledWith[1] + g.Expect(helmCiliumArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(helmCiliumArgs.State).To(Equal(helm.StateDeleted)) + g.Expect(helmCiliumArgs.Values["gatewayAPI"].(map[string]any)["enabled"]).To(BeFalse()) + }) + + t.Run("RolloutFail", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.GatewayDeployFailedMsgTmpl, err))) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium-operator", + Namespace: "kube-system", + }, + }, + &v1.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + }, + }, + ) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(cilium.DisabledMsg)) + }) +} diff --git a/src/k8s/pkg/k8sd/features/cilium/ingress.go b/src/k8s/pkg/k8sd/features/cilium/ingress.go index d62f6153f..a80cdd876 100644 --- a/src/k8s/pkg/k8sd/features/cilium/ingress.go +++ b/src/k8s/pkg/k8sd/features/cilium/ingress.go @@ -10,8 +10,8 @@ import ( ) const ( - ingressDeleteFailedMsgTmpl = "Failed to delete Cilium Ingress, the error was: %v" - ingressDeployFailedMsgTmpl = "Failed to deploy Cilium Ingress, the error was: %v" + IngressDeleteFailedMsgTmpl = "Failed to delete Cilium Ingress, the error was: %v" + IngressDeployFailedMsgTmpl = "Failed to deploy Cilium Ingress, the error was: %v" ) // ApplyIngress assumes that the managed Cilium CNI is already installed on the cluster. It will fail if that is not the case. @@ -54,14 +54,14 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, ne return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } else { err = fmt.Errorf("failed to disable ingress: %w", err) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(ingressDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(IngressDeleteFailedMsgTmpl, err), }, err } } @@ -95,7 +95,7 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, ne return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } diff --git a/src/k8s/pkg/k8sd/features/cilium/ingress_test.go b/src/k8s/pkg/k8sd/features/cilium/ingress_test.go new file mode 100644 index 000000000..6a8e4977c --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/ingress_test.go @@ -0,0 +1,209 @@ +package cilium_test + +import ( + "context" + "errors" + "fmt" + "testing" + + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/cilium" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + v1 "k8s.io/api/apps/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestIngress(t *testing.T) { + applyErr := errors.New("failed to apply") + for _, tc := range []struct { + name string + // given + networkEnabled bool + applyChanged bool + ingressEnabled bool + helmErr error + // then + statusMsg string + statusEnabled bool + }{ + { + name: "HelmFailNetworkEnabled", + networkEnabled: true, + helmErr: applyErr, + statusMsg: fmt.Sprintf(cilium.IngressDeployFailedMsgTmpl, fmt.Errorf("failed to enable ingress: %w", applyErr)), + statusEnabled: false, + }, + { + name: "HelmFailNetworkDisabled", + networkEnabled: false, + statusMsg: fmt.Sprintf(cilium.IngressDeleteFailedMsgTmpl, fmt.Errorf("failed to disable ingress: %w", applyErr)), + statusEnabled: false, + helmErr: applyErr, + }, + { + name: "HelmUnchangedIngressEnabled", + ingressEnabled: true, + statusMsg: cilium.EnabledMsg, + statusEnabled: true, + }, + { + name: "HelmUnchangedIngressDisabled", + ingressEnabled: false, + statusMsg: cilium.DisabledMsg, + statusEnabled: false, + }, + { + name: "HelmChangedIngressDisabled", + applyChanged: true, + ingressEnabled: false, + statusMsg: cilium.DisabledMsg, + statusEnabled: false, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyErr: tc.helmErr, + ApplyChanged: tc.applyChanged, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{ + Enabled: ptr.To(tc.networkEnabled), + } + ingress := types.Ingress{ + Enabled: ptr.To(tc.ingressEnabled), + } + + status, err := cilium.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + if tc.helmErr == nil { + g.Expect(err).To(Not(HaveOccurred())) + } else { + g.Expect(err).To(MatchError(applyErr)) + } + g.Expect(status.Enabled).To(Equal(tc.statusEnabled)) + g.Expect(status.Message).To(Equal(tc.statusMsg)) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + validateIngressValues(g, callArgs.Values, ingress) + }) + } +} + +func TestIngressRollout(t *testing.T) { + t.Run("Error", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "dummy", + Namespace: "kube-system", + }, + }, + ) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.IngressDeployFailedMsgTmpl, err))) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + validateIngressValues(g, callArgs.Values, ingress) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium-operator", + Namespace: "kube-system", + }, + }, + &v1.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + }, + }, + ) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Message).To(Equal(cilium.EnabledMsg)) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + validateIngressValues(g, callArgs.Values, ingress) + }) +} + +func validateIngressValues(g Gomega, values map[string]any, ingress types.Ingress) { + ingressController, ok := values["ingressController"].(map[string]any) + g.Expect(ok).To(BeTrue()) + if ingress.GetEnabled() { + g.Expect(ingressController["enabled"]).To(BeTrue()) + g.Expect(ingressController["loadbalancerMode"]).To(Equal("shared")) + g.Expect(ingressController["defaultSecretNamespace"]).To(Equal("kube-system")) + g.Expect(ingressController["defaultTLSSecret"]).To(Equal(ingress.GetDefaultTLSSecret())) + g.Expect(ingressController["enableProxyProtocol"]).To(Equal(ingress.GetEnableProxyProtocol())) + } else { + g.Expect(ingressController["enabled"]).To(BeFalse()) + g.Expect(ingressController["defaultSecretNamespace"]).To(Equal("")) + g.Expect(ingressController["defaultSecretName"]).To(Equal("")) + g.Expect(ingressController["enableProxyProtocol"]).To(BeFalse()) + } +} diff --git a/src/k8s/pkg/k8sd/features/cilium/internal.go b/src/k8s/pkg/k8sd/features/cilium/internal.go new file mode 100644 index 000000000..72758019b --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/internal.go @@ -0,0 +1,75 @@ +package cilium + +import ( + "fmt" + "slices" + "strconv" + "strings" + + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/cilium" + "github.com/canonical/k8s/pkg/k8sd/types" +) + +const ( + // minVLANIDValue is the minimum valid 802.1Q VLAN ID value. + minVLANIDValue = 0 + // maxVLANIDValue is the maximum valid 802.1Q VLAN ID value. + maxVLANIDValue = 4094 +) + +type config struct { + devices string + directRoutingDevice string + vlanBPFBypass []int +} + +func validateVLANBPFBypass(vlanList string) ([]int, error) { + vlanList = strings.TrimSpace(vlanList) + // Maintain compatibility with the Cilium chart definition + vlanList = strings.Trim(vlanList, "{}") + vlans := strings.Split(vlanList, ",") + + vlanTags := make([]int, 0, len(vlans)) + seenTags := make(map[int]struct{}) + + for _, vlan := range vlans { + vlanID, err := strconv.Atoi(strings.TrimSpace(vlan)) + if err != nil { + return []int{}, fmt.Errorf("failed to parse VLAN tag: %w", err) + } + if vlanID < minVLANIDValue || vlanID > maxVLANIDValue { + return []int{}, fmt.Errorf("VLAN tag must be between 0 and %d", maxVLANIDValue) + } + + if _, ok := seenTags[vlanID]; ok { + continue + } + seenTags[vlanID] = struct{}{} + vlanTags = append(vlanTags, vlanID) + } + + slices.Sort(vlanTags) + return vlanTags, nil +} + +func internalConfig(annotations types.Annotations) (config, error) { + c := config{} + + if v, ok := annotations.Get(apiv1_annotations.AnnotationDevices); ok { + c.devices = v + } + + if v, ok := annotations.Get(apiv1_annotations.AnnotationDirectRoutingDevice); ok { + c.directRoutingDevice = v + } + + if v, ok := annotations[apiv1_annotations.AnnotationVLANBPFBypass]; ok { + vlanTags, err := validateVLANBPFBypass(v) + if err != nil { + return config{}, fmt.Errorf("failed to parse VLAN BPF bypass list: %w", err) + } + c.vlanBPFBypass = vlanTags + } + + return c, nil +} diff --git a/src/k8s/pkg/k8sd/features/cilium/internal_test.go b/src/k8s/pkg/k8sd/features/cilium/internal_test.go new file mode 100644 index 000000000..14af95736 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/internal_test.go @@ -0,0 +1,147 @@ +package cilium + +import ( + "testing" + + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/cilium" + . "github.com/onsi/gomega" +) + +func TestInternalConfig(t *testing.T) { + for _, tc := range []struct { + name string + annotations map[string]string + expectedConfig config + expectError bool + }{ + { + name: "Empty", + annotations: map[string]string{}, + expectedConfig: config{ + devices: "", + directRoutingDevice: "", + vlanBPFBypass: nil, + }, + expectError: false, + }, + { + name: "Valid", + annotations: map[string]string{ + apiv1_annotations.AnnotationDevices: "eth+ lxdbr+", + apiv1_annotations.AnnotationDirectRoutingDevice: "eth0", + apiv1_annotations.AnnotationVLANBPFBypass: "1,2,3", + }, + expectedConfig: config{ + devices: "eth+ lxdbr+", + directRoutingDevice: "eth0", + vlanBPFBypass: []int{1, 2, 3}, + }, + expectError: false, + }, + { + name: "Single valid VLAN", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "1", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1}, + }, + expectError: false, + }, + { + name: "Multiple valid VLANs", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "1,2,3,4,5", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1, 2, 3, 4, 5}, + }, + expectError: false, + }, + { + name: "Wildcard VLAN", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "0", + }, + expectedConfig: config{ + vlanBPFBypass: []int{0}, + }, + expectError: false, + }, + { + name: "Invalid VLAN tag format", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "abc", + }, + expectError: true, + }, + { + name: "VLAN tag out of range", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "4095", + }, + expectError: true, + }, + { + name: "VLAN tag negative", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "-1", + }, + expectError: true, + }, + { + name: "Duplicate VLAN tags", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "1,2,2,3", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1, 2, 3}, + }, + expectError: false, + }, + { + name: "Mixed spaces and commas", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: " 1, 2,3 ,4 , 5 ", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1, 2, 3, 4, 5}, + }, + expectError: false, + }, + { + name: "Invalid mixed with valid", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "1,abc,3", + }, + expectError: true, + }, + { + name: "Nil annotations", + annotations: nil, + expectedConfig: config{}, + expectError: false, + }, + { + name: "VLAN with curly braces", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "{1,2,3}", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1, 2, 3}, + }, + expectError: false, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + parsed, err := internalConfig(tc.annotations) + if tc.expectError { + g.Expect(err).To(HaveOccurred()) + } else { + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(parsed).To(Equal(tc.expectedConfig)) + } + }) + } +} diff --git a/src/k8s/pkg/k8sd/features/cilium/loadbalancer.go b/src/k8s/pkg/k8sd/features/cilium/loadbalancer.go index 54feb61ce..b642be977 100644 --- a/src/k8s/pkg/k8sd/features/cilium/loadbalancer.go +++ b/src/k8s/pkg/k8sd/features/cilium/loadbalancer.go @@ -12,8 +12,8 @@ import ( const ( lbEnabledMsgTmpl = "enabled, %s mode" - lbDeleteFailedMsgTmpl = "Failed to delete Cilium Load Balancer, the error was: %v" - lbDeployFailedMsgTmpl = "Failed to deploy Cilium Load Balancer, the error was: %v" + LbDeleteFailedMsgTmpl = "Failed to delete Cilium Load Balancer, the error was: %v" + LbDeployFailedMsgTmpl = "Failed to deploy Cilium Load Balancer, the error was: %v" ) // ApplyLoadBalancer assumes that the managed Cilium CNI is already installed on the cluster. It will fail if that is not the case. @@ -31,7 +31,7 @@ func ApplyLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types.L return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(lbDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(LbDeleteFailedMsgTmpl, err), }, err } return types.FeatureStatus{ @@ -46,23 +46,24 @@ func ApplyLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types.L return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(lbDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(LbDeployFailedMsgTmpl, err), }, err } - if loadbalancer.GetBGPMode() { + switch { + case loadbalancer.GetBGPMode(): return types.FeatureStatus{ Enabled: true, Version: CiliumAgentImageTag, Message: fmt.Sprintf(lbEnabledMsgTmpl, "BGP"), }, nil - } else if loadbalancer.GetL2Mode() { + case loadbalancer.GetL2Mode(): return types.FeatureStatus{ Enabled: true, Version: CiliumAgentImageTag, Message: fmt.Sprintf(lbEnabledMsgTmpl, "L2"), }, nil - } else { + default: return types.FeatureStatus{ Enabled: true, Version: CiliumAgentImageTag, @@ -198,7 +199,7 @@ func waitForRequiredLoadBalancerCRDs(ctx context.Context, snap snap.Snap, bgpMod requiredCount := len(requiredCRDs) for _, resource := range resources.APIResources { if _, ok := requiredCRDs[resource.Name]; ok { - requiredCount = requiredCount - 1 + requiredCount-- } } return requiredCount == 0, nil diff --git a/src/k8s/pkg/k8sd/features/cilium/loadbalancer_test.go b/src/k8s/pkg/k8sd/features/cilium/loadbalancer_test.go index 96f283c98..22aa25e5d 100644 --- a/src/k8s/pkg/k8sd/features/cilium/loadbalancer_test.go +++ b/src/k8s/pkg/k8sd/features/cilium/loadbalancer_test.go @@ -3,16 +3,16 @@ package cilium_test import ( "context" "errors" + "fmt" "testing" - . "github.com/onsi/gomega" - "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" "github.com/canonical/k8s/pkg/client/kubernetes" "github.com/canonical/k8s/pkg/k8sd/features/cilium" "github.com/canonical/k8s/pkg/k8sd/types" snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" v1 "k8s.io/api/apps/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" fakediscovery "k8s.io/client-go/discovery/fake" @@ -43,7 +43,7 @@ func TestLoadBalancerDisabled(t *testing.T) { g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.LbDeleteFailedMsgTmpl, err))) g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) @@ -52,6 +52,7 @@ func TestLoadBalancerDisabled(t *testing.T) { g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) g.Expect(callArgs.Values).To(BeNil()) }) + t.Run("Success", func(t *testing.T) { g := NewWithT(t) @@ -115,111 +116,150 @@ func TestLoadBalancerEnabled(t *testing.T) { g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.LbDeployFailedMsgTmpl, err))) g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StateUpgradeOnlyOrDeleted(networkCfg.GetEnabled()))) - g.Expect(callArgs.Values["l2announcements"].(map[string]any)["enabled"]).To(Equal(lbCfg.GetL2Mode())) - g.Expect(callArgs.Values["bgpControlPlane"].(map[string]any)["enabled"]).To(Equal(lbCfg.GetL2Mode())) + l2announcements, ok := callArgs.Values["l2announcements"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(l2announcements["enabled"]).To(Equal(lbCfg.GetL2Mode())) + bgpControlPlane, ok := callArgs.Values["bgpControlPlane"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(bgpControlPlane["enabled"]).To(Equal(lbCfg.GetBGPMode())) }) - t.Run("Success", func(t *testing.T) { - g := NewWithT(t) - helmM := &helmmock.Mock{ - // setting changed == true to check for restart annotation - ApplyChanged: true, - } - clientset := fake.NewSimpleClientset( - &v1.Deployment{ - ObjectMeta: metav1.ObjectMeta{ - Name: "cilium-operator", - Namespace: "kube-system", + for _, tc := range []struct { + name string + l2Mode bool + bGPMode bool + statusMessage string + }{ + { + name: "SuccessL2Mode", + l2Mode: true, + bGPMode: false, + statusMessage: "enabled, L2 mode", + }, + { + name: "SuccessBGPMode", + l2Mode: false, + bGPMode: true, + statusMessage: "enabled, BGP mode", + }, + { + name: "SuccessUnknownMode", + l2Mode: false, + bGPMode: false, + statusMessage: "enabled, Unknown mode", + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium-operator", + Namespace: "kube-system", + }, }, - }, - &v1.DaemonSet{ - ObjectMeta: metav1.ObjectMeta{ - Name: "cilium", - Namespace: "kube-system", + &v1.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + }, }, - }, - ) - fd, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) - g.Expect(ok).To(BeTrue()) - fd.Resources = []*metav1.APIResourceList{ - { - GroupVersion: "cilium.io/v2alpha1", - APIResources: []metav1.APIResource{ - {Name: "ciliuml2announcementpolicies"}, - {Name: "ciliumloadbalancerippools"}, - {Name: "ciliumbgppeeringpolicies"}, + ) + fd, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fd.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "cilium.io/v2alpha1", + APIResources: []metav1.APIResource{ + {Name: "ciliuml2announcementpolicies"}, + {Name: "ciliumloadbalancerippools"}, + {Name: "ciliumbgppeeringpolicies"}, + }, }, - }, - } - snapM := &snapmock.Snap{ - Mock: snapmock.Mock{ - HelmClient: helmM, - KubernetesClient: &kubernetes.Client{Interface: clientset}, - }, - } - lbCfg := types.LoadBalancer{ - Enabled: ptr.To(true), - // setting both modes to true for testing purposes - L2Mode: ptr.To(true), - L2Interfaces: ptr.To([]string{"eth0", "eth1"}), - BGPMode: ptr.To(true), - BGPLocalASN: ptr.To(64512), - BGPPeerAddress: ptr.To("10.0.0.1/32"), - BGPPeerASN: ptr.To(64513), - BGPPeerPort: ptr.To(179), - CIDRs: ptr.To([]string{"192.0.2.0/24"}), - IPRanges: ptr.To([]types.LoadBalancer_IPRange{ - {Start: "20.0.20.100", Stop: "20.0.20.200"}, - }), - } - networkCfg := types.Network{ - Enabled: ptr.To(true), - } + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{Interface: clientset}, + }, + } + lbCfg := types.LoadBalancer{ + Enabled: ptr.To(true), + // setting both modes to true for testing purposes + L2Mode: ptr.To(tc.l2Mode), + L2Interfaces: ptr.To([]string{"eth0", "eth1"}), + BGPMode: ptr.To(tc.bGPMode), + BGPLocalASN: ptr.To(64512), + BGPPeerAddress: ptr.To("10.0.0.1/32"), + BGPPeerASN: ptr.To(64513), + BGPPeerPort: ptr.To(179), + CIDRs: ptr.To([]string{"192.0.2.0/24"}), + IPRanges: ptr.To([]types.LoadBalancer_IPRange{ + {Start: "20.0.20.100", Stop: "20.0.20.200"}, + }), + } + networkCfg := types.Network{ + Enabled: ptr.To(true), + } - status, err := cilium.ApplyLoadBalancer(context.Background(), snapM, lbCfg, networkCfg, nil) + status, err := cilium.ApplyLoadBalancer(context.Background(), snapM, lbCfg, networkCfg, nil) - g.Expect(err).ToNot(HaveOccurred()) - g.Expect(status.Enabled).To(BeTrue()) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(tc.statusMessage)) - g.Expect(helmM.ApplyCalledWith).To(HaveLen(2)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(2)) - firstCallArgs := helmM.ApplyCalledWith[0] - g.Expect(firstCallArgs.Chart).To(Equal(cilium.ChartCilium)) - g.Expect(firstCallArgs.State).To(Equal(helm.StateUpgradeOnlyOrDeleted(networkCfg.GetEnabled()))) - g.Expect(firstCallArgs.Values["l2announcements"].(map[string]any)["enabled"]).To(Equal(lbCfg.GetL2Mode())) - g.Expect(firstCallArgs.Values["bgpControlPlane"].(map[string]any)["enabled"]).To(Equal(lbCfg.GetL2Mode())) + firstCallArgs := helmM.ApplyCalledWith[0] + g.Expect(firstCallArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(firstCallArgs.State).To(Equal(helm.StateUpgradeOnlyOrDeleted(networkCfg.GetEnabled()))) + l2announcements, ok := firstCallArgs.Values["l2announcements"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(l2announcements["enabled"]).To(Equal(lbCfg.GetL2Mode())) + bgpControlPlane, ok := firstCallArgs.Values["bgpControlPlane"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(bgpControlPlane["enabled"]).To(Equal(lbCfg.GetBGPMode())) - secondCallArgs := helmM.ApplyCalledWith[1] - g.Expect(secondCallArgs.Chart).To(Equal(cilium.ChartCiliumLoadBalancer)) - g.Expect(secondCallArgs.State).To(Equal(helm.StatePresent)) - validateLoadBalancerValues(t, secondCallArgs.Values, lbCfg) + secondCallArgs := helmM.ApplyCalledWith[1] + g.Expect(secondCallArgs.Chart).To(Equal(cilium.ChartCiliumLoadBalancer)) + g.Expect(secondCallArgs.State).To(Equal(helm.StatePresent)) + validateLoadBalancerValues(t, secondCallArgs.Values, lbCfg) - // check if cilium-operator and cilium daemonset are restarted - deployment, err := clientset.AppsV1().Deployments("kube-system").Get(context.Background(), "cilium-operator", metav1.GetOptions{}) - g.Expect(err).ToNot(HaveOccurred()) - g.Expect(deployment.Spec.Template.Annotations).To(HaveKey("kubectl.kubernetes.io/restartedAt")) - daemonSet, err := clientset.AppsV1().DaemonSets("kube-system").Get(context.Background(), "cilium", metav1.GetOptions{}) - g.Expect(err).ToNot(HaveOccurred()) - g.Expect(daemonSet.Spec.Template.Annotations).To(HaveKey("kubectl.kubernetes.io/restartedAt")) - }) + // check if cilium-operator and cilium daemonset are restarted + deployment, err := clientset.AppsV1().Deployments("kube-system").Get(context.Background(), "cilium-operator", metav1.GetOptions{}) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(deployment.Spec.Template.Annotations).To(HaveKey("kubectl.kubernetes.io/restartedAt")) + daemonSet, err := clientset.AppsV1().DaemonSets("kube-system").Get(context.Background(), "cilium", metav1.GetOptions{}) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(daemonSet.Spec.Template.Annotations).To(HaveKey("kubectl.kubernetes.io/restartedAt")) + }) + } } func validateLoadBalancerValues(t *testing.T, values map[string]interface{}, lbCfg types.LoadBalancer) { g := NewWithT(t) - l2 := values["l2"].(map[string]any) + l2, ok := values["l2"].(map[string]any) + g.Expect(ok).To(BeTrue()) g.Expect(l2["enabled"]).To(Equal(lbCfg.GetL2Mode())) g.Expect(l2["interfaces"]).To(Equal(lbCfg.GetL2Interfaces())) - cidrs := values["ipPool"].(map[string]any)["cidrs"].([]map[string]any) + ipPool, ok := values["ipPool"].(map[string]any) + g.Expect(ok).To(BeTrue()) + cidrs, ok := ipPool["cidrs"].([]map[string]any) + g.Expect(ok).To(BeTrue()) g.Expect(cidrs).To(HaveLen(len(lbCfg.GetIPRanges()) + len(lbCfg.GetCIDRs()))) for _, cidr := range lbCfg.GetCIDRs() { g.Expect(cidrs).To(ContainElement(map[string]any{"cidr": cidr})) @@ -228,10 +268,12 @@ func validateLoadBalancerValues(t *testing.T, values map[string]interface{}, lbC g.Expect(cidrs).To(ContainElement(map[string]any{"start": ipRange.Start, "stop": ipRange.Stop})) } - bgp := values["bgp"].(map[string]any) + bgp, ok := values["bgp"].(map[string]any) + g.Expect(ok).To(BeTrue()) g.Expect(bgp["enabled"]).To(Equal(lbCfg.GetBGPMode())) g.Expect(bgp["localASN"]).To(Equal(lbCfg.GetBGPLocalASN())) - neighbors := bgp["neighbors"].([]map[string]any) + neighbors, ok := bgp["neighbors"].([]map[string]any) + g.Expect(ok).To(BeTrue()) g.Expect(neighbors).To(HaveLen(1)) g.Expect(neighbors[0]["peerAddress"]).To(Equal(lbCfg.GetBGPPeerAddress())) g.Expect(neighbors[0]["peerASN"]).To(Equal(lbCfg.GetBGPPeerASN())) diff --git a/src/k8s/pkg/k8sd/features/cilium/network.go b/src/k8s/pkg/k8sd/features/cilium/network.go index 4418e8b1d..08e4a62f7 100644 --- a/src/k8s/pkg/k8sd/features/cilium/network.go +++ b/src/k8s/pkg/k8sd/features/cilium/network.go @@ -3,6 +3,7 @@ package cilium import ( "context" "fmt" + "strings" "github.com/canonical/k8s/pkg/client/helm" "github.com/canonical/k8s/pkg/k8sd/types" @@ -17,18 +18,24 @@ const ( networkDeployFailedMsgTmpl = "Failed to deploy Cilium Network, the error was: %v" ) -// ApplyNetwork will deploy Cilium when cfg.Enabled is true. -// ApplyNetwork will remove Cilium when cfg.Enabled is false. +// required for unittests. +var ( + getMountPath = utils.GetMountPath + getMountPropagationType = utils.GetMountPropagationType +) + +// ApplyNetwork will deploy Cilium when network.Enabled is true. +// ApplyNetwork will remove Cilium when network.Enabled is false. // ApplyNetwork requires that bpf and cgroups2 are already mounted and available when running under strict snap confinement. If they are not, it will fail (since Cilium will not have the required permissions to mount them). // ApplyNetwork requires that `/sys` is mounted as a shared mount when running under classic snap confinement. This is to ensure that Cilium will be able to automatically mount bpf and cgroups2 on the pods. // ApplyNetwork will always return a FeatureStatus indicating the current status of the // deployment. // ApplyNetwork returns an error if anything fails. The error is also wrapped in the .Message field of the // returned FeatureStatus. -func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ types.Annotations) (types.FeatureStatus, error) { +func ApplyNetwork(ctx context.Context, snap snap.Snap, localhostAddress string, apiserver types.APIServer, network types.Network, annotations types.Annotations) (types.FeatureStatus, error) { m := snap.HelmClient() - if !cfg.GetEnabled() { + if !network.GetEnabled() { if _, err := m.Apply(ctx, ChartCilium, helm.StateDeleted, nil); err != nil { err = fmt.Errorf("failed to uninstall network: %w", err) return types.FeatureStatus{ @@ -44,9 +51,19 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type }, nil } - ipv4CIDR, ipv6CIDR, err := utils.ParseCIDRs(cfg.GetPodCIDR()) + config, err := internalConfig(annotations) + if err != nil { + err = fmt.Errorf("failed to parse annotations: %w", err) + return types.FeatureStatus{ + Enabled: false, + Version: CiliumAgentImageTag, + Message: fmt.Sprintf(networkDeployFailedMsgTmpl, err), + }, err + } + + ipv4CIDR, ipv6CIDR, err := utils.SplitCIDRStrings(network.GetPodCIDR()) if err != nil { - err = fmt.Errorf("invalid kube-proxy --cluster-cidr value: %v", err) + err = fmt.Errorf("invalid kube-proxy --cluster-cidr value: %w", err) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, @@ -54,7 +71,23 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type }, err } + ciliumNodePortValues := map[string]any{ + "enabled": true, + // kube-proxy also binds to the same port for health checks so we need to disable it + "enableHealthCheck": false, + } + + if config.directRoutingDevice != "" { + ciliumNodePortValues["directRoutingDevice"] = config.directRoutingDevice + } + + bpfValues := map[string]any{} + if config.vlanBPFBypass != nil { + bpfValues["vlanBypass"] = config.vlanBPFBypass + } + values := map[string]any{ + "bpf": bpfValues, "image": map[string]any{ "repository": ciliumAgentImageRepo, "tag": CiliumAgentImageTag, @@ -87,14 +120,26 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type "clusterPoolIPv6PodCIDRList": ipv6CIDR, }, }, - "nodePort": map[string]any{ - "enabled": true, + "envoy": map[string]any{ + "enabled": false, // 1.16+ installs envoy as a standalone daemonset by default if not explicitly disabled }, + // https://docs.cilium.io/en/v1.15/network/kubernetes/kubeproxy-free/#kube-proxy-hybrid-modes + "nodePort": ciliumNodePortValues, "disableEnvoyVersionCheck": true, + // socketLB requires an endpoint to the apiserver that's not managed by the kube-proxy + // so we point to the localhost:secureport to talk to either the kube-apiserver or the kube-apiserver-proxy + "k8sServiceHost": strings.Trim(localhostAddress, "[]"), // Cilium already adds the brackets for ipv6 addresses, so we need to remove them + "k8sServicePort": apiserver.GetSecurePort(), + // This flag enables the runtime device detection which is set to true by default in Cilium 1.16+ + "enableRuntimeDeviceDetection": true, + } + + if config.devices != "" { + values["devices"] = config.devices } if snap.Strict() { - bpfMnt, err := utils.GetMountPath("bpf") + bpfMnt, err := getMountPath("bpf") if err != nil { err = fmt.Errorf("failed to get bpf mount path: %w", err) return types.FeatureStatus{ @@ -104,7 +149,7 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type }, err } - cgrMnt, err := utils.GetMountPath("cgroup2") + cgrMnt, err := getMountPath("cgroup2") if err != nil { err = fmt.Errorf("failed to get cgroup2 mount path: %w", err) return types.FeatureStatus{ @@ -127,7 +172,7 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type "hostRoot": cgrMnt, } } else { - pt, err := utils.GetMountPropagationType("/sys") + pt, err := getMountPropagationType("/sys") if err != nil { err = fmt.Errorf("failed to get mount propagation type for /sys: %w", err) return types.FeatureStatus{ @@ -139,7 +184,8 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type if pt == utils.MountPropagationPrivate { onLXD, err := snap.OnLXD(ctx) if err != nil { - log.FromContext(ctx).Error(err, "Failed to check if running on LXD") + logger := log.FromContext(ctx) + logger.Error(err, "Failed to check if running on LXD") } if onLXD { err := fmt.Errorf("/sys is not a shared mount on the LXD container, this might be resolved by updating LXD on the host to version 5.0.2 or newer") diff --git a/src/k8s/pkg/k8sd/features/cilium/network_test.go b/src/k8s/pkg/k8sd/features/cilium/network_test.go index 175cd95a0..eb40caaef 100644 --- a/src/k8s/pkg/k8sd/features/cilium/network_test.go +++ b/src/k8s/pkg/k8sd/features/cilium/network_test.go @@ -1,24 +1,31 @@ -package cilium_test +package cilium import ( "context" "errors" + "fmt" "testing" - . "github.com/onsi/gomega" - + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/cilium" "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" - "github.com/canonical/k8s/pkg/k8sd/features/cilium" "github.com/canonical/k8s/pkg/k8sd/types" "github.com/canonical/k8s/pkg/snap" snapmock "github.com/canonical/k8s/pkg/snap/mock" "github.com/canonical/k8s/pkg/utils" + . "github.com/onsi/gomega" + "k8s.io/klog/v2" + "k8s.io/klog/v2/ktesting" "k8s.io/utils/ptr" ) // NOTE(hue): status.Message is not checked sometimes to avoid unnecessary complexity +var annotations = types.Annotations{ + apiv1_annotations.AnnotationDevices: "eth+ lxdbr+", + apiv1_annotations.AnnotationDirectRoutingDevice: "eth0", +} + func TestNetworkDisabled(t *testing.T) { t.Run("HelmApplyFails", func(t *testing.T) { g := NewWithT(t) @@ -32,23 +39,27 @@ func TestNetworkDisabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(false), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeleteFailedMsgTmpl, err))) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] - g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(callArgs.Chart).To(Equal(ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) g.Expect(callArgs.Values).To(BeNil()) }) + t.Run("Success", func(t *testing.T) { g := NewWithT(t) @@ -58,20 +69,23 @@ func TestNetworkDisabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(false), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).ToNot(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(Equal(cilium.DisabledMsg)) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(DisabledMsg)) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] - g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(callArgs.Chart).To(Equal(ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) g.Expect(callArgs.Values).To(BeNil()) }) @@ -87,18 +101,22 @@ func TestNetworkEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("invalid-cidr"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).To(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) - g.Expect(helmM.ApplyCalledWith).To(HaveLen(0)) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) }) + t.Run("Strict", func(t *testing.T) { g := NewWithT(t) @@ -109,24 +127,28 @@ func TestNetworkEnabled(t *testing.T) { Strict: true, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, annotations) g.Expect(err).ToNot(HaveOccurred()) g.Expect(status.Enabled).To(BeTrue()) - g.Expect(status.Message).To(Equal(cilium.EnabledMsg)) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(EnabledMsg)) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] - g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(callArgs.Chart).To(Equal(ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StatePresent)) - validateNetworkValues(t, callArgs.Values, cfg, snapM) + validateNetworkValues(g, callArgs.Values, network, snapM) }) + t.Run("HelmApplyFails", func(t *testing.T) { g := NewWithT(t) @@ -139,31 +161,212 @@ func TestNetworkEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, annotations) g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] - g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(callArgs.Chart).To(Equal(ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StatePresent)) - validateNetworkValues(t, callArgs.Values, cfg, snapM) + validateNetworkValues(g, callArgs.Values, network, snapM) }) } -func validateNetworkValues(t *testing.T, values map[string]any, cfg types.Network, snap snap.Snap) { - t.Helper() - g := NewWithT(t) +func TestNetworkMountPath(t *testing.T) { + for _, tc := range []struct { + name string + }{ + {name: "bpf"}, + {name: "cgroup2"}, + } { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + + mountPathErr := fmt.Errorf("%s not found", tc.name) + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: true, + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } + getMountPath = func(fsType string) (string, error) { + if fsType == tc.name { + return "", mountPathErr + } + return tc.name, nil + } + + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(mountPathErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + }) + } +} + +func TestNetworkMountPropagationType(t *testing.T) { + t.Run("failedGetMountSys", func(t *testing.T) { + g := NewWithT(t) + + mountErr := errors.New("/sys not found") + getMountPropagationType = func(path string) (utils.MountPropagationType, error) { + return "", mountErr + } + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: false, + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } + + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(mountErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + }) + + t.Run("MountPropagationPrivateOnLXDError", func(t *testing.T) { + g := NewWithT(t) + + getMountPropagationType = func(path string) (utils.MountPropagationType, error) { + return utils.MountPropagationPrivate, nil + } + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: false, + OnLXDErr: errors.New("failed to check LXD"), + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } + logger := ktesting.NewLogger(t, ktesting.NewConfig(ktesting.BufferLogs(true))) + ctx := klog.NewContext(context.Background(), logger) + + status, err := ApplyNetwork(ctx, snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + testingLogger, ok := logger.GetSink().(ktesting.Underlier) + if !ok { + panic("Should have had a ktesting LogSink!?") + } + g.Expect(testingLogger.GetBuffer().String()).To(ContainSubstring("Failed to check if running on LXD")) + }) + + t.Run("MountPropagationPrivateOnLXD", func(t *testing.T) { + g := NewWithT(t) + + getMountPropagationType = func(path string) (utils.MountPropagationType, error) { + return utils.MountPropagationPrivate, nil + } + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: false, + OnLXD: true, + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } + + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + }) + + t.Run("MountPropagationPrivate", func(t *testing.T) { + g := NewWithT(t) + + getMountPropagationType = func(_ string) (utils.MountPropagationType, error) { + return utils.MountPropagationPrivate, nil + } + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: false, + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - ipv4CIDR, ipv6CIDR, err := utils.ParseCIDRs(cfg.GetPodCIDR()) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + }) +} + +func validateNetworkValues(g Gomega, values map[string]any, network types.Network, snap snap.Snap) { + ipv4CIDR, ipv6CIDR, err := utils.SplitCIDRStrings(network.GetPodCIDR()) g.Expect(err).ToNot(HaveOccurred()) bpfMount, err := utils.GetMountPath("bpf") @@ -177,8 +380,20 @@ func validateNetworkValues(t *testing.T, values map[string]any, cfg types.Networ g.Expect(values["cgroup"].(map[string]any)["hostRoot"]).To(Equal(cgrMount)) } + g.Expect(values["k8sServiceHost"]).To(Equal("127.0.0.1")) + g.Expect(values["k8sServicePort"]).To(Equal(6443)) g.Expect(values["ipam"].(map[string]any)["operator"].(map[string]any)["clusterPoolIPv4PodCIDRList"]).To(Equal(ipv4CIDR)) g.Expect(values["ipam"].(map[string]any)["operator"].(map[string]any)["clusterPoolIPv6PodCIDRList"]).To(Equal(ipv6CIDR)) g.Expect(values["ipv4"].(map[string]any)["enabled"]).To(Equal((ipv4CIDR != ""))) g.Expect(values["ipv6"].(map[string]any)["enabled"]).To(Equal((ipv6CIDR != ""))) + + devices, exists := annotations.Get(apiv1_annotations.AnnotationDevices) + if exists { + g.Expect(values["devices"]).To(Equal(devices)) + } + + directRoutingDevice, exists := annotations.Get(apiv1_annotations.AnnotationDirectRoutingDevice) + if exists { + g.Expect(values["nodePort"].(map[string]any)["directRoutingDevice"]).To(Equal(directRoutingDevice)) + } } diff --git a/src/k8s/pkg/k8sd/features/cilium/status.go b/src/k8s/pkg/k8sd/features/cilium/status.go index 65212848c..37c629abd 100644 --- a/src/k8s/pkg/k8sd/features/cilium/status.go +++ b/src/k8s/pkg/k8sd/features/cilium/status.go @@ -5,7 +5,6 @@ import ( "fmt" "github.com/canonical/k8s/pkg/snap" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) diff --git a/src/k8s/pkg/k8sd/features/cilium/status_test.go b/src/k8s/pkg/k8sd/features/cilium/status_test.go new file mode 100644 index 000000000..083844e5d --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/status_test.go @@ -0,0 +1,124 @@ +package cilium_test + +import ( + "context" + "testing" + + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/cilium" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" +) + +func TestCheckNetwork(t *testing.T) { + t.Run("ciliumOperatorNotReady", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + + err := cilium.CheckNetwork(context.Background(), snapM) + + g.Expect(err).To(HaveOccurred()) + }) + + t.Run("operatorNoCiliumPods", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset(&corev1.PodList{ + Items: []corev1.Pod{ + { + ObjectMeta: metav1.ObjectMeta{ + Name: "pod1", + Namespace: "kube-system", + Labels: map[string]string{"io.cilium/app": "operator"}, + }, + Status: corev1.PodStatus{ + Phase: corev1.PodRunning, + Conditions: []corev1.PodCondition{ + {Type: corev1.PodReady, Status: corev1.ConditionTrue}, + }, + }, + }, + }, + }) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + + err := cilium.CheckNetwork(context.Background(), snapM) + + g.Expect(err).To(HaveOccurred()) + }) + + t.Run("allPodsPresent", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset(&corev1.PodList{ + Items: []corev1.Pod{ + { + ObjectMeta: metav1.ObjectMeta{ + Name: "operator", + Namespace: "kube-system", + Labels: map[string]string{"io.cilium/app": "operator"}, + }, + Status: corev1.PodStatus{ + Phase: corev1.PodRunning, + Conditions: []corev1.PodCondition{ + {Type: corev1.PodReady, Status: corev1.ConditionTrue}, + }, + }, + }, + { + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + Labels: map[string]string{"k8s-app": "cilium"}, + }, + Status: corev1.PodStatus{ + Phase: corev1.PodRunning, + Conditions: []corev1.PodCondition{ + {Type: corev1.PodReady, Status: corev1.ConditionTrue}, + }, + }, + }, + }, + }) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + + err := cilium.CheckNetwork(context.Background(), snapM) + g.Expect(err).NotTo(HaveOccurred()) + }) +} diff --git a/src/k8s/pkg/k8sd/features/contour/chart.go b/src/k8s/pkg/k8sd/features/contour/chart.go index ff8fa2016..1291dbe85 100644 --- a/src/k8s/pkg/k8sd/features/contour/chart.go +++ b/src/k8s/pkg/k8sd/features/contour/chart.go @@ -34,29 +34,29 @@ var ( ManifestPath: filepath.Join("charts", "ck-contour-common-1.28.2.tgz"), } - // contourGatewayProvisionerEnvoyImageRepo represents the image to use for envoy in the gateway. - contourGatewayProvisionerEnvoyImageRepo = "ghcr.io/canonical/k8s-snap/envoyproxy/envoy" + // ContourGatewayProvisionerEnvoyImageRepo represents the image to use for envoy in the gateway. + ContourGatewayProvisionerEnvoyImageRepo = "ghcr.io/canonical/k8s-snap/envoyproxy/envoy" // NOTE: The image version is v1.29.2 instead of 1.28.2 // to follow the upstream configuration for the contour gateway provisioner. - // contourGatewayProvisionerEnvoyImageTag is the tag to use for for envoy in the gateway. - contourGatewayProvisionerEnvoyImageTag = "v1.29.2" + // ContourGatewayProvisionerEnvoyImageTag is the tag to use for envoy in the gateway. + ContourGatewayProvisionerEnvoyImageTag = "v1.29.2" - // contourIngressEnvoyImageRepo represents the image to use for the Contour Envoy proxy. - contourIngressEnvoyImageRepo = "ghcr.io/canonical/k8s-snap/bitnami/envoy" + // ContourIngressEnvoyImageRepo represents the image to use for the Contour Envoy proxy. + ContourIngressEnvoyImageRepo = "ghcr.io/canonical/k8s-snap/bitnami/envoy" - // contourIngressEnvoyImageTag is the tag to use for the Contour Envoy proxy image. - contourIngressEnvoyImageTag = "1.28.2-debian-12-r0" + // ContourIngressEnvoyImageTag is the tag to use for the Contour Envoy proxy image. + ContourIngressEnvoyImageTag = "1.28.2-debian-12-r0" - // contourIngressContourImageRepo represents the image to use for Contour. - contourIngressContourImageRepo = "ghcr.io/canonical/k8s-snap/bitnami/contour" + // ContourIngressContourImageRepo represents the image to use for Contour. + ContourIngressContourImageRepo = "ghcr.io/canonical/k8s-snap/bitnami/contour" - // contourIngressContourImageTag is the tag to use for the Contour image. - contourIngressContourImageTag = "1.28.2-debian-12-r4" + // ContourIngressContourImageTag is the tag to use for the Contour image. + ContourIngressContourImageTag = "1.28.2-debian-12-r4" - // contourGatewayProvisionerContourImageRepo represents the image to use for the Contour Gateway Provisioner. - contourGatewayProvisionerContourImageRepo = "ghcr.io/canonical/k8s-snap/projectcontour/contour" + // ContourGatewayProvisionerContourImageRepo represents the image to use for the Contour Gateway Provisioner. + ContourGatewayProvisionerContourImageRepo = "ghcr.io/canonical/k8s-snap/projectcontour/contour" - // contourGatewayProvisionerContourImageTag is the tag to use for the Contour Gateway Provisioner image. - contourGatewayProvisionerContourImageTag = "v1.28.2" + // ContourGatewayProvisionerContourImageTag is the tag to use for the Contour Gateway Provisioner image. + ContourGatewayProvisionerContourImageTag = "v1.28.2" ) diff --git a/src/k8s/pkg/k8sd/features/contour/gateway.go b/src/k8s/pkg/k8sd/features/contour/gateway.go index d7b203c6b..dd52e9bff 100644 --- a/src/k8s/pkg/k8sd/features/contour/gateway.go +++ b/src/k8s/pkg/k8sd/features/contour/gateway.go @@ -11,10 +11,10 @@ import ( ) const ( - enabledMsg = "enabled" - disabledMsg = "disabled" - gatewayDeployFailedMsgTmpl = "Failed to deploy Contour Gateway, the error was: %v" - gatewayDeleteFailedMsgTmpl = "Failed to delete Contour Gateway, the error was: %v" + EnabledMsg = "enabled" + DisabledMsg = "disabled" + GatewayDeployFailedMsgTmpl = "Failed to deploy Contour Gateway, the error was: %v" + GatewayDeleteFailedMsgTmpl = "Failed to delete Contour Gateway, the error was: %v" ) // ApplyGateway will install a helm chart for contour-gateway-provisioner on the cluster when gateway.Enabled is true. @@ -32,14 +32,14 @@ func ApplyGateway(ctx context.Context, snap snap.Snap, gateway types.Gateway, ne err = fmt.Errorf("failed to uninstall the contour gateway chart: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: fmt.Sprintf(gatewayDeleteFailedMsgTmpl, err), + Version: ContourGatewayProvisionerContourImageTag, + Message: fmt.Sprintf(GatewayDeleteFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: disabledMsg, + Version: ContourGatewayProvisionerContourImageTag, + Message: DisabledMsg, }, nil } @@ -48,8 +48,8 @@ func ApplyGateway(ctx context.Context, snap snap.Snap, gateway types.Gateway, ne err = fmt.Errorf("failed to apply common contour CRDS: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Version: ContourGatewayProvisionerContourImageTag, + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -57,22 +57,22 @@ func ApplyGateway(ctx context.Context, snap snap.Snap, gateway types.Gateway, ne err = fmt.Errorf("failed to wait for required contour common CRDs to be available: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Version: ContourGatewayProvisionerContourImageTag, + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } values := map[string]any{ "projectcontour": map[string]any{ "image": map[string]any{ - "repository": contourGatewayProvisionerContourImageRepo, - "tag": contourGatewayProvisionerContourImageTag, + "repository": ContourGatewayProvisionerContourImageRepo, + "tag": ContourGatewayProvisionerContourImageTag, }, }, "envoyproxy": map[string]any{ "image": map[string]any{ - "repository": contourGatewayProvisionerEnvoyImageRepo, - "tag": contourGatewayProvisionerEnvoyImageTag, + "repository": ContourGatewayProvisionerEnvoyImageRepo, + "tag": ContourGatewayProvisionerEnvoyImageTag, }, }, } @@ -81,20 +81,20 @@ func ApplyGateway(ctx context.Context, snap snap.Snap, gateway types.Gateway, ne err = fmt.Errorf("failed to install the contour gateway chart: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Version: ContourGatewayProvisionerContourImageTag, + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: true, - Version: contourGatewayProvisionerContourImageTag, - Message: enabledMsg, + Version: ContourGatewayProvisionerContourImageTag, + Message: EnabledMsg, }, nil } // waitForRequiredContourCommonCRDs waits for the required contour CRDs to be available -// by checking the API resources by group version +// by checking the API resources by group version. func waitForRequiredContourCommonCRDs(ctx context.Context, snap snap.Snap) error { client, err := snap.KubernetesClient("") if err != nil { diff --git a/src/k8s/pkg/k8sd/features/contour/gateway_test.go b/src/k8s/pkg/k8sd/features/contour/gateway_test.go new file mode 100644 index 000000000..b4ac8e0b4 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/contour/gateway_test.go @@ -0,0 +1,209 @@ +package contour_test + +import ( + "context" + "errors" + "fmt" + "testing" + "time" + + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/contour" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + fakediscovery "k8s.io/client-go/discovery/fake" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestGatewayDisabled(t *testing.T) { + t.Run("HelmFailed", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + + status, err := contour.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err.Error()).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.GatewayDeleteFailedMsgTmpl, err))) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + + status, err := contour.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(contour.DisabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) +} + +func TestGatewayEnabled(t *testing.T) { + t.Run("HelmFailed", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := contour.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.GatewayDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*v1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []v1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []v1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := contour.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(contour.EnabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(2)) + + values := helmM.ApplyCalledWith[1].Values + contourValues, ok := values["projectcontour"].(map[string]any) + g.Expect(ok).To(BeTrue()) + contourImage, ok := contourValues["image"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(contourImage["repository"]).To(Equal(contour.ContourGatewayProvisionerContourImageRepo)) + g.Expect(contourImage["tag"]).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + envoyValues, ok := values["envoyproxy"].(map[string]any) + g.Expect(ok).To(BeTrue()) + envoyImage, ok := envoyValues["image"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(envoyImage["repository"]).To(Equal(contour.ContourGatewayProvisionerEnvoyImageRepo)) + g.Expect(envoyImage["tag"]).To(Equal(contour.ContourGatewayProvisionerEnvoyImageTag)) + }) + + t.Run("CrdDeploymentFailed", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*v1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []v1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []v1.APIResource{}, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*2) + defer cancel() + status, err := contour.ApplyGateway(ctx, snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err.Error()).To(ContainSubstring("failed to wait for required contour common CRDs")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.GatewayDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) +} diff --git a/src/k8s/pkg/k8sd/features/contour/ingress.go b/src/k8s/pkg/k8sd/features/contour/ingress.go index a43023d1c..ea5a6e030 100644 --- a/src/k8s/pkg/k8sd/features/contour/ingress.go +++ b/src/k8s/pkg/k8sd/features/contour/ingress.go @@ -11,8 +11,8 @@ import ( ) const ( - ingressDeleteFailedMsgTmpl = "Failed to delete Contour Ingress, the error was: %v" - ingressDeployFailedMsgTmpl = "Failed to deploy Contour Ingress, the error was: %v" + IngressDeleteFailedMsgTmpl = "Failed to delete Contour Ingress, the error was: %v" + IngressDeployFailedMsgTmpl = "Failed to deploy Contour Ingress, the error was: %v" ) // ApplyIngress will install the contour helm chart when ingress.Enabled is true. @@ -24,7 +24,7 @@ const ( // deployment. // ApplyIngress returns an error if anything fails. The error is also wrapped in the .Message field of the // returned FeatureStatus. -// Contour CRDS are applied through a ck-contour common chart (Overlap with gateway) +// Contour CRDS are applied through a ck-contour common chart (Overlap with gateway). func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ types.Network, _ types.Annotations) (types.FeatureStatus, error) { m := snap.HelmClient() @@ -33,14 +33,14 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to uninstall ingress: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeleteFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeleteFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: disabledMsg, + Version: ContourIngressContourImageTag, + Message: DisabledMsg, }, nil } @@ -49,8 +49,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to apply common contour CRDS: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } @@ -58,8 +58,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to wait for required contour common CRDs to be available: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } @@ -70,8 +70,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ "envoy": map[string]any{ "image": map[string]any{ "registry": "", - "repository": contourIngressEnvoyImageRepo, - "tag": contourIngressEnvoyImageTag, + "repository": ContourIngressEnvoyImageRepo, + "tag": ContourIngressEnvoyImageTag, }, }, "contour": map[string]any{ @@ -83,16 +83,23 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ }, "image": map[string]any{ "registry": "", - "repository": contourIngressContourImageRepo, - "tag": contourIngressContourImageTag, + "repository": ContourIngressContourImageRepo, + "tag": ContourIngressContourImageTag, }, }, } if ingress.GetEnableProxyProtocol() { - contour := values["contour"].(map[string]any) + contour, ok := values["contour"].(map[string]any) + if !ok { + err := fmt.Errorf("unexpected type for contour values") + return types.FeatureStatus{ + Enabled: false, + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), + }, err + } contour["extraArgs"] = []string{"--use-proxy-protocol"} - } changed, err := m.Apply(ctx, chartContour, helm.StatePresent, values) @@ -100,8 +107,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to enable ingress: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } @@ -110,8 +117,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to rollout restart contour to apply ingress: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } } @@ -127,14 +134,14 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to install the delegation resource for default TLS secret: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: true, - Version: contourIngressContourImageTag, - Message: enabledMsg, + Version: ContourIngressContourImageTag, + Message: EnabledMsg, }, nil } @@ -142,16 +149,16 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to uninstall the delegation resource for default TLS secret: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: true, - Version: contourIngressContourImageTag, - Message: enabledMsg, + Version: ContourIngressContourImageTag, + Message: EnabledMsg, }, nil } diff --git a/src/k8s/pkg/k8sd/features/contour/ingress_test.go b/src/k8s/pkg/k8sd/features/contour/ingress_test.go new file mode 100644 index 000000000..9547d2032 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/contour/ingress_test.go @@ -0,0 +1,404 @@ +package contour_test + +import ( + "context" + "errors" + "fmt" + "testing" + "time" + + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/contour" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + v1 "k8s.io/api/apps/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + fakediscovery "k8s.io/client-go/discovery/fake" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestIngressDisabled(t *testing.T) { + t.Run("HelmFailed", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(false), + } + + status, err := contour.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.IngressDeleteFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(false), + } + + status, err := contour.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(contour.DisabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) +} + +func TestIngressEnabled(t *testing.T) { + t.Run("HelmFailed", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + + status, err := contour.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.IngressDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ck-ingress-contour-contour", + Namespace: "projectcontour", + }, + }) + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []metav1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*20) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(contour.EnabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(3)) + validateIngressValues(g, helmM.ApplyCalledWith[1].Values, ingress) + }) + + t.Run("SuccessWithEnabledProxyProtocol", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ck-ingress-contour-contour", + Namespace: "projectcontour", + }, + }) + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []metav1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + EnableProxyProtocol: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*20) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(contour.EnabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(3)) + validateIngressValues(g, helmM.ApplyCalledWith[1].Values, ingress) + }) + + t.Run("SuccessWithDefaultTLSSecret", func(t *testing.T) { + g := NewWithT(t) + defaultTLSSecret := "secret" + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ck-ingress-contour-contour", + Namespace: "projectcontour", + }, + }) + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []metav1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + DefaultTLSSecret: ptr.To(defaultTLSSecret), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*20) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(contour.EnabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(3)) + validateIngressValues(g, helmM.ApplyCalledWith[1].Values, ingress) + g.Expect(helmM.ApplyCalledWith[2].Values["defaultTLSSecret"]).To(Equal(defaultTLSSecret)) + }) + + t.Run("NoCR", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/metav1", + APIResources: []metav1.APIResource{}, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*2) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.IngressDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("NoDeployment", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "dummy", + Namespace: "projectcontour", + }, + }) + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []metav1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*20) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err.Error()).To(ContainSubstring("failed to rollout restart contour to apply ingress")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.IngressDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(2)) + }) +} + +func validateIngressValues(g Gomega, values map[string]interface{}, ingress types.Ingress) { + contourValues, ok := values["contour"].(map[string]any) + g.Expect(ok).To(BeTrue()) + contourImage, ok := contourValues["image"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(contourImage["repository"]).To(Equal(contour.ContourIngressContourImageRepo)) + g.Expect(contourImage["tag"]).To(Equal(contour.ContourIngressContourImageTag)) + envoyValues, ok := values["envoy"].(map[string]any) + g.Expect(ok).To(BeTrue()) + envoyImage, ok := envoyValues["image"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(envoyImage["repository"]).To(Equal(contour.ContourIngressEnvoyImageRepo)) + g.Expect(envoyImage["tag"]).To(Equal(contour.ContourIngressEnvoyImageTag)) + + if ingress.GetEnableProxyProtocol() { + conturExtraValues, ok := values["contour"].(map[string]any) + g.Expect(ok).To(BeTrue()) + contourExtraArgs, ok := conturExtraValues["extraArgs"].([]string) + g.Expect(ok).To(BeTrue()) + g.Expect(contourExtraArgs[0]).To(Equal("--use-proxy-protocol")) + } +} diff --git a/src/k8s/pkg/k8sd/features/contour/register.go b/src/k8s/pkg/k8sd/features/contour/register.go index 4c3797032..59cc84792 100644 --- a/src/k8s/pkg/k8sd/features/contour/register.go +++ b/src/k8s/pkg/k8sd/features/contour/register.go @@ -8,9 +8,9 @@ import ( func init() { images.Register( - fmt.Sprintf("%s:%s", contourIngressEnvoyImageRepo, contourIngressEnvoyImageTag), - fmt.Sprintf("%s:%s", contourIngressContourImageRepo, contourIngressContourImageTag), - fmt.Sprintf("%s:%s", contourGatewayProvisionerContourImageRepo, contourGatewayProvisionerContourImageTag), - fmt.Sprintf("%s:%s", contourGatewayProvisionerEnvoyImageRepo, contourGatewayProvisionerEnvoyImageTag), + fmt.Sprintf("%s:%s", ContourIngressEnvoyImageRepo, ContourIngressEnvoyImageTag), + fmt.Sprintf("%s:%s", ContourIngressContourImageRepo, ContourIngressContourImageTag), + fmt.Sprintf("%s:%s", ContourGatewayProvisionerContourImageRepo, ContourGatewayProvisionerContourImageTag), + fmt.Sprintf("%s:%s", ContourGatewayProvisionerEnvoyImageRepo, ContourGatewayProvisionerEnvoyImageTag), ) } diff --git a/src/k8s/pkg/k8sd/features/coredns/chart.go b/src/k8s/pkg/k8sd/features/coredns/chart.go index 72cf75bfb..eaa940c5e 100644 --- a/src/k8s/pkg/k8sd/features/coredns/chart.go +++ b/src/k8s/pkg/k8sd/features/coredns/chart.go @@ -8,15 +8,15 @@ import ( var ( // chartCoreDNS represents manifests to deploy CoreDNS. - chart = helm.InstallableChart{ + Chart = helm.InstallableChart{ Name: "ck-dns", Namespace: "kube-system", - ManifestPath: filepath.Join("charts", "coredns-1.29.0.tgz"), + ManifestPath: filepath.Join("charts", "coredns-1.36.0.tgz"), } // imageRepo is the image to use for CoreDNS. imageRepo = "ghcr.io/canonical/coredns" - // imageTag is the tag to use for the CoreDNS image. - imageTag = "1.11.1-ck4" + // ImageTag is the tag to use for the CoreDNS image. + ImageTag = "1.11.3-ck0" ) diff --git a/src/k8s/pkg/k8sd/features/coredns/coredns.go b/src/k8s/pkg/k8sd/features/coredns/coredns.go index 28402e707..79e66d6e6 100644 --- a/src/k8s/pkg/k8sd/features/coredns/coredns.go +++ b/src/k8s/pkg/k8sd/features/coredns/coredns.go @@ -29,17 +29,17 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. m := snap.HelmClient() if !dns.GetEnabled() { - if _, err := m.Apply(ctx, chart, helm.StateDeleted, nil); err != nil { + if _, err := m.Apply(ctx, Chart, helm.StateDeleted, nil); err != nil { err = fmt.Errorf("failed to uninstall coredns: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deleteFailedMsgTmpl, err), }, "", err } return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: disabledMsg, }, "", nil } @@ -47,7 +47,7 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. values := map[string]any{ "image": map[string]any{ "repository": imageRepo, - "tag": imageTag, + "tag": ImageTag, }, "service": map[string]any{ "name": "coredns", @@ -82,11 +82,11 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. }, } - if _, err := m.Apply(ctx, chart, helm.StatePresent, values); err != nil { + if _, err := m.Apply(ctx, Chart, helm.StatePresent, values); err != nil { err = fmt.Errorf("failed to apply coredns: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deployFailedMsgTmpl, err), }, "", err } @@ -96,7 +96,7 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. err = fmt.Errorf("failed to create kubernetes client: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deployFailedMsgTmpl, err), }, "", err } @@ -105,14 +105,14 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. err = fmt.Errorf("failed to retrieve the coredns service: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deployFailedMsgTmpl, err), }, "", err } return types.FeatureStatus{ Enabled: true, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(enabledMsgTmpl, dnsIP), }, dnsIP, err } diff --git a/src/k8s/pkg/k8sd/features/coredns/coredns_test.go b/src/k8s/pkg/k8sd/features/coredns/coredns_test.go new file mode 100644 index 000000000..0d10e2bd3 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/coredns/coredns_test.go @@ -0,0 +1,196 @@ +package coredns_test + +import ( + "context" + "errors" + "strings" + "testing" + + "github.com/canonical/k8s/pkg/client/helm" + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/coredns" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestDisabled(t *testing.T) { + t.Run("HelmApplyFails", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + dns := types.DNS{ + Enabled: ptr.To(false), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(MatchError(ContainSubstring(applyErr.Error()))) + g.Expect(str).To(BeEmpty()) + g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Message).To(ContainSubstring("failed to uninstall coredns")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) + g.Expect(callArgs.Values).To(BeNil()) + }) + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + dns := types.DNS{ + Enabled: ptr.To(false), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(str).To(BeEmpty()) + g.Expect(status.Message).To(Equal("disabled")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) + g.Expect(callArgs.Values).To(BeNil()) + }) +} + +func TestEnabled(t *testing.T) { + t.Run("HelmApplyFails", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + dns := types.DNS{ + Enabled: ptr.To(true), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(MatchError(ContainSubstring(applyErr.Error()))) + g.Expect(str).To(BeEmpty()) + g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Message).To(ContainSubstring("failed to apply coredns")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + validateValues(g, callArgs.Values, dns, kubelet) + }) + t.Run("HelmApplySuccessServiceFails", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + clientset := fake.NewSimpleClientset() + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{Interface: clientset}, + }, + } + dns := types.DNS{ + Enabled: ptr.To(true), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(MatchError(ContainSubstring("services \"coredns\" not found"))) + g.Expect(str).To(BeEmpty()) + g.Expect(status.Message).To(ContainSubstring("failed to retrieve the coredns service")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + validateValues(g, callArgs.Values, dns, kubelet) + }) + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + clusterIp := "10.96.0.10" + corednsService := &corev1.Service{ + ObjectMeta: metav1.ObjectMeta{ + Name: "coredns", + Namespace: "kube-system", + }, + Spec: corev1.ServiceSpec{ + ClusterIP: clusterIp, + }, + } + clientset := fake.NewSimpleClientset(corednsService) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{Interface: clientset}, + }, + } + dns := types.DNS{ + Enabled: ptr.To(true), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(str).To(Equal(clusterIp)) + g.Expect(status.Message).To(ContainSubstring("enabled at " + clusterIp)) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + validateValues(g, callArgs.Values, dns, kubelet) + }) +} + +func validateValues(g Gomega, values map[string]any, dns types.DNS, kubelet types.Kubelet) { + service := values["service"].(map[string]any) + g.Expect(service["clusterIP"]).To(Equal(kubelet.GetClusterDNS())) + + servers := values["servers"].([]map[string]any) + plugins := servers[0]["plugins"].([]map[string]any) + g.Expect(plugins[3]["parameters"]).To(ContainSubstring(kubelet.GetClusterDomain())) + g.Expect(plugins[5]["parameters"]).To(ContainSubstring(strings.Join(dns.GetUpstreamNameservers(), " "))) +} diff --git a/src/k8s/pkg/k8sd/features/coredns/register.go b/src/k8s/pkg/k8sd/features/coredns/register.go index 1dc54d943..566d31f09 100644 --- a/src/k8s/pkg/k8sd/features/coredns/register.go +++ b/src/k8s/pkg/k8sd/features/coredns/register.go @@ -8,6 +8,6 @@ import ( func init() { images.Register( - fmt.Sprintf("%s:%s", imageRepo, imageTag), + fmt.Sprintf("%s:%s", imageRepo, ImageTag), ) } diff --git a/src/k8s/pkg/k8sd/features/coredns/status.go b/src/k8s/pkg/k8sd/features/coredns/status.go index 629eabe87..94d3fe66a 100644 --- a/src/k8s/pkg/k8sd/features/coredns/status.go +++ b/src/k8s/pkg/k8sd/features/coredns/status.go @@ -5,7 +5,6 @@ import ( "fmt" "github.com/canonical/k8s/pkg/snap" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) diff --git a/src/k8s/pkg/k8sd/features/implementation_default.go b/src/k8s/pkg/k8sd/features/implementation_default.go index b78a67fae..3cad3e3b6 100644 --- a/src/k8s/pkg/k8sd/features/implementation_default.go +++ b/src/k8s/pkg/k8sd/features/implementation_default.go @@ -4,18 +4,20 @@ import ( "github.com/canonical/k8s/pkg/k8sd/features/cilium" "github.com/canonical/k8s/pkg/k8sd/features/coredns" "github.com/canonical/k8s/pkg/k8sd/features/localpv" + "github.com/canonical/k8s/pkg/k8sd/features/metallb" metrics_server "github.com/canonical/k8s/pkg/k8sd/features/metrics-server" ) // Default implements the Canonical Kubernetes built-in features. -// Cilium is used for networking (network + load-balancer + ingress + gateway). +// Cilium is used for networking (network + ingress + gateway). +// MetalLB is used for LoadBalancer. // CoreDNS is used for DNS. // MetricsServer is used for metrics-server. // LocalPV Rawfile CSI is used for local-storage. var Implementation Interface = &implementation{ applyDNS: coredns.ApplyDNS, applyNetwork: cilium.ApplyNetwork, - applyLoadBalancer: cilium.ApplyLoadBalancer, + applyLoadBalancer: metallb.ApplyLoadBalancer, applyIngress: cilium.ApplyIngress, applyGateway: cilium.ApplyGateway, applyMetricsServer: metrics_server.ApplyMetricsServer, diff --git a/src/k8s/pkg/k8sd/features/interface.go b/src/k8s/pkg/k8sd/features/interface.go index 18eb4a978..3753af341 100644 --- a/src/k8s/pkg/k8sd/features/interface.go +++ b/src/k8s/pkg/k8sd/features/interface.go @@ -12,7 +12,7 @@ type Interface interface { // ApplyDNS is used to configure the DNS feature on Canonical Kubernetes. ApplyDNS(context.Context, snap.Snap, types.DNS, types.Kubelet, types.Annotations) (types.FeatureStatus, string, error) // ApplyNetwork is used to configure the network feature on Canonical Kubernetes. - ApplyNetwork(context.Context, snap.Snap, types.Network, types.Annotations) (types.FeatureStatus, error) + ApplyNetwork(context.Context, snap.Snap, string, types.APIServer, types.Network, types.Annotations) (types.FeatureStatus, error) // ApplyLoadBalancer is used to configure the load-balancer feature on Canonical Kubernetes. ApplyLoadBalancer(context.Context, snap.Snap, types.LoadBalancer, types.Network, types.Annotations) (types.FeatureStatus, error) // ApplyIngress is used to configure the ingress controller feature on Canonical Kubernetes. @@ -28,7 +28,7 @@ type Interface interface { // implementation implements Interface. type implementation struct { applyDNS func(context.Context, snap.Snap, types.DNS, types.Kubelet, types.Annotations) (types.FeatureStatus, string, error) - applyNetwork func(context.Context, snap.Snap, types.Network, types.Annotations) (types.FeatureStatus, error) + applyNetwork func(context.Context, snap.Snap, string, types.APIServer, types.Network, types.Annotations) (types.FeatureStatus, error) applyLoadBalancer func(context.Context, snap.Snap, types.LoadBalancer, types.Network, types.Annotations) (types.FeatureStatus, error) applyIngress func(context.Context, snap.Snap, types.Ingress, types.Network, types.Annotations) (types.FeatureStatus, error) applyGateway func(context.Context, snap.Snap, types.Gateway, types.Network, types.Annotations) (types.FeatureStatus, error) @@ -40,8 +40,8 @@ func (i *implementation) ApplyDNS(ctx context.Context, snap snap.Snap, dns types return i.applyDNS(ctx, snap, dns, kubelet, annotations) } -func (i *implementation) ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, annotations types.Annotations) (types.FeatureStatus, error) { - return i.applyNetwork(ctx, snap, cfg, annotations) +func (i *implementation) ApplyNetwork(ctx context.Context, snap snap.Snap, localhostAddress string, apiserver types.APIServer, network types.Network, annotations types.Annotations) (types.FeatureStatus, error) { + return i.applyNetwork(ctx, snap, localhostAddress, apiserver, network, annotations) } func (i *implementation) ApplyLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types.LoadBalancer, network types.Network, annotations types.Annotations) (types.FeatureStatus, error) { diff --git a/src/k8s/pkg/k8sd/features/localpv/chart.go b/src/k8s/pkg/k8sd/features/localpv/chart.go index 5a655fc57..8dd2248af 100644 --- a/src/k8s/pkg/k8sd/features/localpv/chart.go +++ b/src/k8s/pkg/k8sd/features/localpv/chart.go @@ -7,8 +7,8 @@ import ( ) var ( - // chart represents manifests to deploy Rawfile LocalPV CSI. - chart = helm.InstallableChart{ + // Chart represents manifests to deploy Rawfile LocalPV CSI. + Chart = helm.InstallableChart{ Name: "ck-storage", Namespace: "kube-system", ManifestPath: filepath.Join("charts", "rawfile-csi-0.9.0.tgz"), @@ -16,8 +16,8 @@ var ( // imageRepo is the repository to use for Rawfile LocalPV CSI. imageRepo = "ghcr.io/canonical/rawfile-localpv" - // imageTag is the image tag to use for Rawfile LocalPV CSI. - imageTag = "0.8.0-ck4" + // ImageTag is the image tag to use for Rawfile LocalPV CSI. + ImageTag = "0.8.0-ck4" // csiNodeDriverImage is the image to use for the CSI node driver. csiNodeDriverImage = "ghcr.io/canonical/k8s-snap/sig-storage/csi-node-driver-registrar:v2.10.1" diff --git a/src/k8s/pkg/k8sd/features/localpv/localpv.go b/src/k8s/pkg/k8sd/features/localpv/localpv.go index bd812b443..8555ff088 100644 --- a/src/k8s/pkg/k8sd/features/localpv/localpv.go +++ b/src/k8s/pkg/k8sd/features/localpv/localpv.go @@ -38,13 +38,13 @@ func ApplyLocalStorage(ctx context.Context, snap snap.Snap, cfg types.LocalStora "csiDriverArgs": []string{"--args", "rawfile", "csi-driver", "--disable-metrics"}, "image": map[string]any{ "repository": imageRepo, - "tag": imageTag, + "tag": ImageTag, }, }, "node": map[string]any{ "image": map[string]any{ "repository": imageRepo, - "tag": imageTag, + "tag": ImageTag, }, "storage": map[string]any{ "path": cfg.GetLocalPath(), @@ -58,19 +58,19 @@ func ApplyLocalStorage(ctx context.Context, snap snap.Snap, cfg types.LocalStora }, } - if _, err := m.Apply(ctx, chart, helm.StatePresentOrDeleted(cfg.GetEnabled()), values); err != nil { + if _, err := m.Apply(ctx, Chart, helm.StatePresentOrDeleted(cfg.GetEnabled()), values); err != nil { if cfg.GetEnabled() { err = fmt.Errorf("failed to install rawfile-csi helm package: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deployFailedMsgTmpl, err), }, err } else { err = fmt.Errorf("failed to delete rawfile-csi helm package: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deleteFailedMsgTmpl, err), }, err } @@ -79,13 +79,13 @@ func ApplyLocalStorage(ctx context.Context, snap snap.Snap, cfg types.LocalStora if cfg.GetEnabled() { return types.FeatureStatus{ Enabled: true, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(enabledMsg, cfg.GetLocalPath()), }, nil } else { return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: disabledMsg, }, nil } diff --git a/src/k8s/pkg/k8sd/features/localpv/localpv_test.go b/src/k8s/pkg/k8sd/features/localpv/localpv_test.go new file mode 100644 index 000000000..2a8da057c --- /dev/null +++ b/src/k8s/pkg/k8sd/features/localpv/localpv_test.go @@ -0,0 +1,154 @@ +package localpv_test + +import ( + "context" + "errors" + "testing" + + "github.com/canonical/k8s/pkg/client/helm" + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/k8sd/features/localpv" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + "k8s.io/utils/ptr" +) + +func TestDisabled(t *testing.T) { + t.Run("HelmApplyFails", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + cfg := types.LocalStorage{ + Enabled: ptr.To(false), + Default: ptr.To(true), + ReclaimPolicy: ptr.To("reclaim-policy"), + LocalPath: ptr.To("local-path"), + } + + status, err := localpv.ApplyLocalStorage(context.Background(), snapM, cfg, nil) + + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Version).To(Equal(localpv.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(localpv.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) + + validateValues(g, callArgs.Values, cfg) + }) + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + cfg := types.LocalStorage{ + Enabled: ptr.To(false), + Default: ptr.To(true), + ReclaimPolicy: ptr.To("reclaim-policy"), + LocalPath: ptr.To("local-path"), + } + + status, err := localpv.ApplyLocalStorage(context.Background(), snapM, cfg, nil) + + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(localpv.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(localpv.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) + + validateValues(g, callArgs.Values, cfg) + }) +} + +func TestEnabled(t *testing.T) { + t.Run("HelmApplyFails", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + cfg := types.LocalStorage{ + Enabled: ptr.To(true), + Default: ptr.To(true), + ReclaimPolicy: ptr.To("reclaim-policy"), + LocalPath: ptr.To("local-path"), + } + + status, err := localpv.ApplyLocalStorage(context.Background(), snapM, cfg, nil) + + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Version).To(Equal(localpv.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(localpv.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + + validateValues(g, callArgs.Values, cfg) + }) + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + cfg := types.LocalStorage{ + Enabled: ptr.To(true), + Default: ptr.To(true), + ReclaimPolicy: ptr.To("reclaim-policy"), + LocalPath: ptr.To("local-path"), + } + + status, err := localpv.ApplyLocalStorage(context.Background(), snapM, cfg, nil) + + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(localpv.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(localpv.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + + validateValues(g, callArgs.Values, cfg) + }) +} + +func validateValues(g Gomega, values map[string]any, cfg types.LocalStorage) { + sc := values["storageClass"].(map[string]any) + g.Expect(sc["isDefault"]).To(Equal(cfg.GetDefault())) + g.Expect(sc["reclaimPolicy"]).To(Equal(cfg.GetReclaimPolicy())) + + storage := values["node"].(map[string]any)["storage"].(map[string]any) + g.Expect(storage["path"]).To(Equal(cfg.GetLocalPath())) +} diff --git a/src/k8s/pkg/k8sd/features/localpv/register.go b/src/k8s/pkg/k8sd/features/localpv/register.go index b9f5f644b..084f6a40b 100644 --- a/src/k8s/pkg/k8sd/features/localpv/register.go +++ b/src/k8s/pkg/k8sd/features/localpv/register.go @@ -9,7 +9,7 @@ import ( func init() { images.Register( // Rawfile LocalPV CSI driver images - fmt.Sprintf("%s:%s", imageRepo, imageTag), + fmt.Sprintf("%s:%s", imageRepo, ImageTag), // CSI images csiNodeDriverImage, csiProvisionerImage, diff --git a/src/k8s/pkg/k8sd/features/metallb/chart.go b/src/k8s/pkg/k8sd/features/metallb/chart.go index a7bdffd2a..b3cc9f8f6 100644 --- a/src/k8s/pkg/k8sd/features/metallb/chart.go +++ b/src/k8s/pkg/k8sd/features/metallb/chart.go @@ -11,7 +11,7 @@ var ( ChartMetalLB = helm.InstallableChart{ Name: "metallb", Namespace: "metallb-system", - ManifestPath: filepath.Join("charts", "metallb-0.14.5.tgz"), + ManifestPath: filepath.Join("charts", "metallb-0.14.8.tgz"), } // ChartMetalLBLoadBalancer represents manifests to deploy MetalLB L2 or BGP resources. @@ -22,16 +22,16 @@ var ( } // controllerImageRepo is the image to use for metallb-controller. - controllerImageRepo = "ghcr.io/canonical/k8s-snap/metallb/controller" + controllerImageRepo = "ghcr.io/canonical/metallb-controller" // ControllerImageTag is the tag to use for metallb-controller. - ControllerImageTag = "v0.14.5" + ControllerImageTag = "v0.14.8-ck0" // speakerImageRepo is the image to use for metallb-speaker. - speakerImageRepo = "ghcr.io/canonical/k8s-snap/metallb/speaker" + speakerImageRepo = "ghcr.io/canonical/metallb-speaker" // speakerImageTag is the tag to use for metallb-speaker. - speakerImageTag = "v0.14.5" + speakerImageTag = "v0.14.8-ck0" // frrImageRepo is the image to use for frrouting. frrImageRepo = "ghcr.io/canonical/k8s-snap/frrouting/frr" diff --git a/src/k8s/pkg/k8sd/features/metallb/loadbalancer.go b/src/k8s/pkg/k8sd/features/metallb/loadbalancer.go index ef199d600..6477def8e 100644 --- a/src/k8s/pkg/k8sd/features/metallb/loadbalancer.go +++ b/src/k8s/pkg/k8sd/features/metallb/loadbalancer.go @@ -47,19 +47,20 @@ func ApplyLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types.L }, err } - if loadbalancer.GetBGPMode() { + switch { + case loadbalancer.GetBGPMode(): return types.FeatureStatus{ Enabled: true, Version: ControllerImageTag, Message: fmt.Sprintf(enabledMsgTmpl, "BGP"), }, nil - } else if loadbalancer.GetL2Mode() { + case loadbalancer.GetL2Mode(): return types.FeatureStatus{ Enabled: true, Version: ControllerImageTag, Message: fmt.Sprintf(enabledMsgTmpl, "L2"), }, nil - } else { + default: return types.FeatureStatus{ Enabled: true, Version: ControllerImageTag, @@ -90,12 +91,14 @@ func enableLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types. "repository": controllerImageRepo, "tag": ControllerImageTag, }, + "command": "/controller", }, "speaker": map[string]any{ "image": map[string]any{ "repository": speakerImageRepo, "tag": speakerImageTag, }, + "command": "/speaker", // TODO(neoaggelos): make frr enable/disable configurable through an annotation // We keep it disabled by default "frr": map[string]any{ @@ -170,13 +173,13 @@ func waitForRequiredLoadBalancerCRDs(ctx context.Context, snap snap.Snap, bgpMod return false, nil } - requiredCRDs := map[string]bool{ - "metallb.io/v1beta1:ipaddresspools": true, - "metallb.io/v1beta1:l2advertisements": true, + requiredCRDs := map[string]struct{}{ + "metallb.io/v1beta1:ipaddresspools": {}, + "metallb.io/v1beta1:l2advertisements": {}, } if bgpMode { - requiredCRDs["metallb.io/v1beta2:bgppeers"] = true - requiredCRDs["metallb.io/v1beta1:bgpadvertisements"] = true + requiredCRDs["metallb.io/v1beta2:bgppeers"] = struct{}{} + requiredCRDs["metallb.io/v1beta1:bgpadvertisements"] = struct{}{} } requiredCount := len(requiredCRDs) diff --git a/src/k8s/pkg/k8sd/features/metallb/loadbalancer_test.go b/src/k8s/pkg/k8sd/features/metallb/loadbalancer_test.go index 0bc1fb6f1..7ba673c78 100644 --- a/src/k8s/pkg/k8sd/features/metallb/loadbalancer_test.go +++ b/src/k8s/pkg/k8sd/features/metallb/loadbalancer_test.go @@ -5,18 +5,17 @@ import ( "errors" "testing" - . "github.com/onsi/gomega" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - fakediscovery "k8s.io/client-go/discovery/fake" - "k8s.io/client-go/kubernetes/fake" - "k8s.io/utils/ptr" - "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" "github.com/canonical/k8s/pkg/client/kubernetes" "github.com/canonical/k8s/pkg/k8sd/features/metallb" "github.com/canonical/k8s/pkg/k8sd/types" snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + fakediscovery "k8s.io/client-go/discovery/fake" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" ) func TestDisabled(t *testing.T) { diff --git a/src/k8s/pkg/k8sd/features/metrics-server/chart.go b/src/k8s/pkg/k8sd/features/metrics-server/chart.go index 8c4e46958..0c01aa6ba 100644 --- a/src/k8s/pkg/k8sd/features/metrics-server/chart.go +++ b/src/k8s/pkg/k8sd/features/metrics-server/chart.go @@ -11,12 +11,12 @@ var ( chart = helm.InstallableChart{ Name: "metrics-server", Namespace: "kube-system", - ManifestPath: filepath.Join("charts", "metrics-server-3.12.0.tgz"), + ManifestPath: filepath.Join("charts", "metrics-server-3.12.2.tgz"), } // imageRepo is the image to use for metrics-server. imageRepo = "ghcr.io/canonical/metrics-server" // imageTag is the image tag to use for metrics-server. - imageTag = "0.7.0-ck1" + imageTag = "0.7.2-ck0" ) diff --git a/src/k8s/pkg/k8sd/features/metrics-server/internal.go b/src/k8s/pkg/k8sd/features/metrics-server/internal.go index 2300b7139..c927e348d 100644 --- a/src/k8s/pkg/k8sd/features/metrics-server/internal.go +++ b/src/k8s/pkg/k8sd/features/metrics-server/internal.go @@ -1,10 +1,8 @@ package metrics_server -import "github.com/canonical/k8s/pkg/k8sd/types" - -const ( - annotationImageRepo = "k8sd/v1alpha1/metrics-server/image-repo" - annotationImageTag = "k8sd/v1alpha1/metrics-server/image-tag" +import ( + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/metrics-server" + "github.com/canonical/k8s/pkg/k8sd/types" ) type config struct { @@ -18,10 +16,10 @@ func internalConfig(annotations types.Annotations) config { imageTag: imageTag, } - if v, ok := annotations.Get(annotationImageRepo); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationImageRepo); ok { config.imageRepo = v } - if v, ok := annotations.Get(annotationImageTag); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationImageTag); ok { config.imageTag = v } diff --git a/src/k8s/pkg/k8sd/features/metrics-server/metrics_server_test.go b/src/k8s/pkg/k8sd/features/metrics-server/metrics_server_test.go index ced59104f..d9cdc8370 100644 --- a/src/k8s/pkg/k8sd/features/metrics-server/metrics_server_test.go +++ b/src/k8s/pkg/k8sd/features/metrics-server/metrics_server_test.go @@ -2,8 +2,10 @@ package metrics_server_test import ( "context" + "errors" "testing" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/metrics-server" "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" metrics_server "github.com/canonical/k8s/pkg/k8sd/features/metrics-server" @@ -14,29 +16,51 @@ import ( ) func TestApplyMetricsServer(t *testing.T) { + helmErr := errors.New("failed to apply") for _, tc := range []struct { name string config types.MetricsServer expectState helm.State + helmError error }{ { - name: "Enable", + name: "EnableWithoutHelmError", config: types.MetricsServer{ Enabled: utils.Pointer(true), }, expectState: helm.StatePresent, + helmError: nil, }, { - name: "Disable", + name: "DisableWithoutHelmError", config: types.MetricsServer{ Enabled: utils.Pointer(false), }, expectState: helm.StateDeleted, + helmError: nil, + }, + { + name: "EnableWithHelmError", + config: types.MetricsServer{ + Enabled: utils.Pointer(true), + }, + expectState: helm.StatePresent, + helmError: helmErr, + }, + { + name: "DisableWithHelmError", + config: types.MetricsServer{ + Enabled: utils.Pointer(false), + }, + expectState: helm.StateDeleted, + helmError: helmErr, }, } { t.Run(tc.name, func(t *testing.T) { g := NewWithT(t) - h := &helmmock.Mock{} + h := &helmmock.Mock{ + ApplyErr: tc.helmError, + } s := &snapmock.Snap{ Mock: snapmock.Mock{ HelmClient: h, @@ -44,16 +68,23 @@ func TestApplyMetricsServer(t *testing.T) { } status, err := metrics_server.ApplyMetricsServer(context.Background(), s, tc.config, nil) - g.Expect(err).ToNot(HaveOccurred()) + if tc.helmError == nil { + g.Expect(err).ToNot(HaveOccurred()) + } else { + g.Expect(err).To(HaveOccurred()) + } g.Expect(h.ApplyCalledWith).To(ConsistOf(SatisfyAll( HaveField("Chart.Name", Equal("metrics-server")), HaveField("Chart.Namespace", Equal("kube-system")), HaveField("State", Equal(tc.expectState)), ))) - if tc.config.GetEnabled() { + switch { + case errors.Is(tc.helmError, helmErr): + g.Expect(status.Message).To(ContainSubstring(helmErr.Error())) + case tc.config.GetEnabled(): g.Expect(status.Message).To(Equal("enabled")) - } else { + default: g.Expect(status.Message).To(Equal("disabled")) } }) @@ -72,12 +103,12 @@ func TestApplyMetricsServer(t *testing.T) { Enabled: utils.Pointer(true), } annotations := types.Annotations{ - "k8sd/v1alpha1/metrics-server/image-repo": "custom-image", - "k8sd/v1alpha1/metrics-server/image-tag": "custom-tag", + apiv1_annotations.AnnotationImageRepo: "custom-image", + apiv1_annotations.AnnotationImageTag: "custom-tag", } status, err := metrics_server.ApplyMetricsServer(context.Background(), s, cfg, annotations) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(h.ApplyCalledWith).To(ConsistOf(HaveField("Values", HaveKeyWithValue("image", SatisfyAll( HaveKeyWithValue("repository", "custom-image"), HaveKeyWithValue("tag", "custom-tag"), diff --git a/src/k8s/pkg/k8sd/pki/k8sdqlite.go b/src/k8s/pkg/k8sd/pki/k8sdqlite.go index b1d74bc37..0e81d823d 100644 --- a/src/k8s/pkg/k8sd/pki/k8sdqlite.go +++ b/src/k8s/pkg/k8sd/pki/k8sdqlite.go @@ -64,7 +64,7 @@ func (c *K8sDqlitePKI) CompleteCertificates() error { return fmt.Errorf("k8s-dqlite certificate not specified and generating self-signed certificates is not allowed") } - template, err := pkiutil.GenerateCertificate(pkix.Name{CommonName: "k8s"}, c.notBefore, c.notAfter, false, append(c.dnsSANs, c.hostname), append(c.ipSANs, net.IP{127, 0, 0, 1})) + template, err := pkiutil.GenerateCertificate(pkix.Name{CommonName: "k8s"}, c.notBefore, c.notAfter, false, append(c.dnsSANs, c.hostname), append(c.ipSANs, net.ParseIP("127.0.0.1"), net.ParseIP("::1"))) if err != nil { return fmt.Errorf("failed to generate k8s-dqlite certificate: %w", err) } diff --git a/src/k8s/pkg/k8sd/pki/worker.go b/src/k8s/pkg/k8sd/pki/worker.go index d945f053f..590b53a99 100644 --- a/src/k8s/pkg/k8sd/pki/worker.go +++ b/src/k8s/pkg/k8sd/pki/worker.go @@ -48,7 +48,7 @@ func (c *ControlPlanePKI) CompleteWorkerNodePKI(hostname string, nodeIP net.IP, c.notAfter, false, []string{hostname}, - []net.IP{{127, 0, 0, 1}, nodeIP}, + []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1"), nodeIP}, ) if err != nil { return nil, fmt.Errorf("failed to generate kubelet certificate for hostname=%s address=%s: %w", hostname, nodeIP.String(), err) diff --git a/src/k8s/pkg/k8sd/pki/worker_test.go b/src/k8s/pkg/k8sd/pki/worker_test.go index 4c9fac5cc..65d85215e 100644 --- a/src/k8s/pkg/k8sd/pki/worker_test.go +++ b/src/k8s/pkg/k8sd/pki/worker_test.go @@ -13,7 +13,6 @@ import ( ) func TestControlPlanePKI_CompleteWorkerNodePKI(t *testing.T) { - g := NewWithT(t) notBefore := time.Now() serverCACert, serverCAKey, err := pkiutil.GenerateSelfSignedCA(pkix.Name{CommonName: "kubernetes-ca"}, notBefore, notBefore.AddDate(1, 0, 0), 2048) diff --git a/src/k8s/pkg/k8sd/setup/auth-token-webhook.conf b/src/k8s/pkg/k8sd/setup/auth-token-webhook.conf new file mode 100644 index 000000000..39273db4a --- /dev/null +++ b/src/k8s/pkg/k8sd/setup/auth-token-webhook.conf @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: Config +clusters: + - name: k8s-token-auth-service + cluster: + certificate-authority: "cluster.crt" + tls-server-name: 127.0.0.1 + server: "https://auth-webhook.url" +current-context: webhook +contexts: +- context: + cluster: k8s-token-auth-service + user: k8s-apiserver + name: webhook +users: + - name: k8s-apiserver + user: {} diff --git a/src/k8s/pkg/k8sd/setup/certificates.go b/src/k8s/pkg/k8sd/setup/certificates.go index 283101e53..c8e67d227 100644 --- a/src/k8s/pkg/k8sd/setup/certificates.go +++ b/src/k8s/pkg/k8sd/setup/certificates.go @@ -8,6 +8,7 @@ import ( "github.com/canonical/k8s/pkg/k8sd/pki" "github.com/canonical/k8s/pkg/snap" + "github.com/canonical/k8s/pkg/utils" ) // ensureFile creates fname with the specified contents, mode and owner bits. @@ -39,7 +40,7 @@ func ensureFile(fname string, contents string, uid, gid int, mode fs.FileMode) ( var contentChanged bool if contents != string(origContent) { - if err := os.WriteFile(fname, []byte(contents), mode); err != nil { + if err := utils.WriteFile(fname, []byte(contents), mode); err != nil { return false, fmt.Errorf("failed to write: %w", err) } contentChanged = true @@ -73,7 +74,7 @@ func ensureFiles(uid, gid int, mode fs.FileMode, files map[string]string) (bool, // and have the correct content, permissions and ownership. // It returns true if one or more files were updated and any error that occurred. func EnsureExtDatastorePKI(snap snap.Snap, certificates *pki.ExternalDatastorePKI) (bool, error) { - return ensureFiles(snap.UID(), snap.GID(), 0600, map[string]string{ + return ensureFiles(snap.UID(), snap.GID(), 0o600, map[string]string{ filepath.Join(snap.EtcdPKIDir(), "ca.crt"): certificates.DatastoreCACert, filepath.Join(snap.EtcdPKIDir(), "client.key"): certificates.DatastoreClientKey, filepath.Join(snap.EtcdPKIDir(), "client.crt"): certificates.DatastoreClientCert, @@ -84,7 +85,7 @@ func EnsureExtDatastorePKI(snap snap.Snap, certificates *pki.ExternalDatastorePK // and have the correct content, permissions and ownership. // It returns true if one or more files were updated and any error that occurred. func EnsureK8sDqlitePKI(snap snap.Snap, certificates *pki.K8sDqlitePKI) (bool, error) { - return ensureFiles(snap.UID(), snap.GID(), 0600, map[string]string{ + return ensureFiles(snap.UID(), snap.GID(), 0o600, map[string]string{ filepath.Join(snap.K8sDqliteStateDir(), "cluster.crt"): certificates.K8sDqliteCert, filepath.Join(snap.K8sDqliteStateDir(), "cluster.key"): certificates.K8sDqliteKey, }) @@ -94,7 +95,7 @@ func EnsureK8sDqlitePKI(snap snap.Snap, certificates *pki.K8sDqlitePKI) (bool, e // and have the correct content, permissions and ownership. // It returns true if one or more files were updated and any error that occurred. func EnsureControlPlanePKI(snap snap.Snap, certificates *pki.ControlPlanePKI) (bool, error) { - return ensureFiles(snap.UID(), snap.GID(), 0600, map[string]string{ + return ensureFiles(snap.UID(), snap.GID(), 0o600, map[string]string{ filepath.Join(snap.KubernetesPKIDir(), "apiserver-kubelet-client.crt"): certificates.APIServerKubeletClientCert, filepath.Join(snap.KubernetesPKIDir(), "apiserver-kubelet-client.key"): certificates.APIServerKubeletClientKey, filepath.Join(snap.KubernetesPKIDir(), "apiserver.crt"): certificates.APIServerCert, @@ -116,7 +117,7 @@ func EnsureControlPlanePKI(snap snap.Snap, certificates *pki.ControlPlanePKI) (b // and have the correct content, permissions and ownership. // It returns true if one or more files were updated and any error that occurred. func EnsureWorkerPKI(snap snap.Snap, certificates *pki.WorkerNodePKI) (bool, error) { - return ensureFiles(snap.UID(), snap.GID(), 0600, map[string]string{ + return ensureFiles(snap.UID(), snap.GID(), 0o600, map[string]string{ filepath.Join(snap.KubernetesPKIDir(), "ca.crt"): certificates.CACert, filepath.Join(snap.KubernetesPKIDir(), "client-ca.crt"): certificates.ClientCACert, filepath.Join(snap.KubernetesPKIDir(), "kubelet.crt"): certificates.KubeletCert, diff --git a/src/k8s/pkg/k8sd/setup/certificates_internal_test.go b/src/k8s/pkg/k8sd/setup/certificates_internal_test.go index 665574dda..9454458d3 100644 --- a/src/k8s/pkg/k8sd/setup/certificates_internal_test.go +++ b/src/k8s/pkg/k8sd/setup/certificates_internal_test.go @@ -14,12 +14,12 @@ func TestEnsureFile(t *testing.T) { tempDir := t.TempDir() fname := filepath.Join(tempDir, "test") - updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) createdFile, _ := os.ReadFile(fname) - g.Expect(string(createdFile) == "test").To(BeTrue()) + g.Expect(string(createdFile)).To(Equal("test")) }) t.Run("DeleteFile", func(t *testing.T) { @@ -28,13 +28,13 @@ func TestEnsureFile(t *testing.T) { fname := filepath.Join(tempDir, "test") // Create a file with some content. - updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) // Delete the file. - updated, err = ensureFile(fname, "", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err = ensureFile(fname, "", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) _, err = os.Stat(fname) @@ -47,26 +47,26 @@ func TestEnsureFile(t *testing.T) { fname := filepath.Join(tempDir, "test") // Create a file with some content. - updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) // ensureFile with same content should return that the file was not updated. - updated, err = ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err = ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeFalse()) // Change the content and ensureFile should return that the file was updated. - updated, err = ensureFile(fname, "new content", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err = ensureFile(fname, "new content", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) createdFile, _ := os.ReadFile(fname) - g.Expect(string(createdFile) == "new content").To(BeTrue()) + g.Expect(string(createdFile)).To(Equal("new content")) // Change permissions and ensureFile should return that the file was not updated. - updated, err = ensureFile(fname, "new content", os.Getuid(), os.Getgid(), 0666) - g.Expect(err).To(BeNil()) + updated, err = ensureFile(fname, "new content", os.Getuid(), os.Getgid(), 0o666) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeFalse()) }) @@ -76,8 +76,8 @@ func TestEnsureFile(t *testing.T) { fname := filepath.Join(tempDir, "test") // ensureFile on inexistent file with empty content should return that the file was not updated. - updated, err := ensureFile(fname, "", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err := ensureFile(fname, "", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeFalse()) }) } diff --git a/src/k8s/pkg/k8sd/setup/certificates_test.go b/src/k8s/pkg/k8sd/setup/certificates_test.go index 32526faec..8ea74313e 100644 --- a/src/k8s/pkg/k8sd/setup/certificates_test.go +++ b/src/k8s/pkg/k8sd/setup/certificates_test.go @@ -28,7 +28,7 @@ func TestEnsureK8sDqlitePKI(t *testing.T) { } _, err := setup.EnsureK8sDqlitePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) expectedFiles := []string{ filepath.Join(tempDir, "cluster.crt"), @@ -37,7 +37,7 @@ func TestEnsureK8sDqlitePKI(t *testing.T) { for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } } @@ -71,7 +71,7 @@ func TestEnsureControlPlanePKI(t *testing.T) { } _, err := setup.EnsureControlPlanePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) expectedFiles := []string{ filepath.Join(tempDir, "apiserver-kubelet-client.crt"), @@ -92,7 +92,7 @@ func TestEnsureControlPlanePKI(t *testing.T) { for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } } @@ -115,7 +115,7 @@ func TestEnsureWorkerPKI(t *testing.T) { } _, err := setup.EnsureWorkerPKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) expectedFiles := []string{ filepath.Join(tempDir, "ca.crt"), @@ -126,7 +126,7 @@ func TestEnsureWorkerPKI(t *testing.T) { for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } } @@ -147,7 +147,7 @@ func TestExtDatastorePKI(t *testing.T) { } _, err := setup.EnsureExtDatastorePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) expectedFiles := []string{ filepath.Join(tempDir, "ca.crt"), @@ -157,7 +157,7 @@ func TestExtDatastorePKI(t *testing.T) { for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } } @@ -186,7 +186,7 @@ func TestEmptyCert(t *testing.T) { // Should create files _, err := setup.EnsureK8sDqlitePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) certificates = &pki.K8sDqlitePKI{ K8sDqliteCert: "", @@ -195,10 +195,10 @@ func TestEmptyCert(t *testing.T) { // Should delete files _, err = setup.EnsureK8sDqlitePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) } } diff --git a/src/k8s/pkg/k8sd/setup/containerd.go b/src/k8s/pkg/k8sd/setup/containerd.go index dbf1545ec..eb9909124 100644 --- a/src/k8s/pkg/k8sd/setup/containerd.go +++ b/src/k8s/pkg/k8sd/setup/containerd.go @@ -108,12 +108,12 @@ func Containerd(snap snap.Snap, extraContainerdConfig map[string]any, extraArgs return fmt.Errorf("failed to render containerd config.toml: %w", err) } - if err := os.WriteFile(filepath.Join(snap.ContainerdConfigDir(), "config.toml"), b, 0600); err != nil { + if err := utils.WriteFile(filepath.Join(snap.ContainerdConfigDir(), "config.toml"), b, 0o600); err != nil { return fmt.Errorf("failed to write config.toml: %w", err) } if _, err := snaputil.UpdateServiceArguments(snap, "containerd", map[string]string{ - "--address": filepath.Join(snap.ContainerdSocketDir(), "containerd.sock"), + "--address": snap.ContainerdSocketPath(), "--config": filepath.Join(snap.ContainerdConfigDir(), "config.toml"), "--root": snap.ContainerdRootDir(), "--state": snap.ContainerdStateDir(), @@ -131,7 +131,7 @@ func Containerd(snap snap.Snap, extraContainerdConfig map[string]any, extraArgs if err := utils.CopyFile(snap.CNIPluginsBinary(), cniBinary); err != nil { return fmt.Errorf("failed to copy cni plugin binary: %w", err) } - if err := os.Chmod(cniBinary, 0700); err != nil { + if err := os.Chmod(cniBinary, 0o700); err != nil { return fmt.Errorf("failed to chmod cni plugin binary: %w", err) } if err := os.Chown(cniBinary, snap.UID(), snap.GID()); err != nil { @@ -159,6 +159,27 @@ func Containerd(snap snap.Snap, extraContainerdConfig map[string]any, extraArgs } } + if err := saveSnapContainerdPaths(snap); err != nil { + return err + } + + return nil +} + +func saveSnapContainerdPaths(s snap.Snap) error { + // Write the containerd-related paths to files to properly clean-up on removal. + m := map[string]string{ + "containerd-socket-path": s.ContainerdSocketDir(), + "containerd-config-dir": s.ContainerdConfigDir(), + "containerd-root-dir": s.ContainerdRootDir(), + "containerd-cni-bin-dir": s.CNIBinDir(), + } + + for filename, content := range m { + if err := utils.WriteFile(filepath.Join(s.LockFilesDir(), filename), []byte(content), 0o600); err != nil { + return fmt.Errorf("failed to write %s: %w", filename, err) + } + } return nil } diff --git a/src/k8s/pkg/k8sd/setup/containerd_test.go b/src/k8s/pkg/k8sd/setup/containerd_test.go index 4327499e4..8bd56399b 100644 --- a/src/k8s/pkg/k8sd/setup/containerd_test.go +++ b/src/k8s/pkg/k8sd/setup/containerd_test.go @@ -20,13 +20,14 @@ func TestContainerd(t *testing.T) { dir := t.TempDir() - g.Expect(os.WriteFile(filepath.Join(dir, "mockcni"), []byte("echo hi"), 0600)).To(Succeed()) + g.Expect(utils.WriteFile(filepath.Join(dir, "mockcni"), []byte("echo hi"), 0o600)).To(Succeed()) s := &mock.Snap{ Mock: mock.Mock{ ContainerdConfigDir: filepath.Join(dir, "containerd"), ContainerdRootDir: filepath.Join(dir, "containerd-root"), ContainerdSocketDir: filepath.Join(dir, "containerd-run"), + ContainerdSocketPath: filepath.Join(dir, "containerd-run", "containerd.sock"), ContainerdRegistryConfigDir: filepath.Join(dir, "containerd-hosts"), ContainerdStateDir: filepath.Join(dir, "containerd-state"), ContainerdExtraConfigDir: filepath.Join(dir, "containerd-confd"), @@ -53,7 +54,7 @@ func TestContainerd(t *testing.T) { t.Run("Config", func(t *testing.T) { g := NewWithT(t) b, err := os.ReadFile(filepath.Join(dir, "containerd", "config.toml")) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(string(b)).To(SatisfyAll( ContainSubstring(fmt.Sprintf(`imports = ["%s/*.toml", "/custom/imports/*.toml"]`, filepath.Join(dir, "containerd-confd"))), ContainSubstring(fmt.Sprintf(`conf_dir = "%s"`, filepath.Join(dir, "cni-netd"))), @@ -62,8 +63,8 @@ func TestContainerd(t *testing.T) { )) info, err := os.Stat(filepath.Join(dir, "containerd", "config.toml")) - g.Expect(err).To(BeNil()) - g.Expect(info.Mode().Perm()).To(Equal(fs.FileMode(0600))) + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(info.Mode().Perm()).To(Equal(fs.FileMode(0o600))) switch stat := info.Sys().(type) { case *syscall.Stat_t: @@ -78,13 +79,13 @@ func TestContainerd(t *testing.T) { g := NewWithT(t) for _, plugin := range []string{"plugin1", "plugin2"} { link, err := os.Readlink(filepath.Join(dir, "opt-cni-bin", plugin)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(link).To(Equal("cni")) } info, err := os.Stat(filepath.Join(dir, "opt-cni-bin")) - g.Expect(err).To(BeNil()) - g.Expect(info.Mode().Perm()).To(Equal(fs.FileMode(0700))) + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(info.Mode().Perm()).To(Equal(fs.FileMode(0o700))) switch stat := info.Sys().(type) { case *syscall.Stat_t: @@ -107,7 +108,7 @@ func TestContainerd(t *testing.T) { t.Run(key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "containerd", key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(expectedVal)) }) } @@ -115,8 +116,24 @@ func TestContainerd(t *testing.T) { t.Run("--address", func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "containerd", "--address") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeZero()) }) }) + + t.Run("Lockfiles", func(t *testing.T) { + g := NewWithT(t) + m := map[string]string{ + "containerd-socket-path": s.ContainerdSocketDir(), + "containerd-config-dir": s.ContainerdConfigDir(), + "containerd-root-dir": s.ContainerdRootDir(), + "containerd-cni-bin-dir": s.CNIBinDir(), + } + for filename, content := range m { + + b, err := os.ReadFile(filepath.Join(s.LockFilesDir(), filename)) + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(string(b)).To(Equal(content)) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/directories.go b/src/k8s/pkg/k8sd/setup/directories.go index f4b8c4779..7b0a2afc9 100644 --- a/src/k8s/pkg/k8sd/setup/directories.go +++ b/src/k8s/pkg/k8sd/setup/directories.go @@ -33,7 +33,7 @@ func EnsureAllDirectories(snap snap.Snap) error { if dir == "" { continue } - if err := os.MkdirAll(dir, 0700); err != nil { + if err := os.MkdirAll(dir, 0o700); err != nil { return fmt.Errorf("failed to create required directory: %w", err) } } diff --git a/src/k8s/pkg/k8sd/setup/k8s-apiserver-proxy.json b/src/k8s/pkg/k8sd/setup/k8s-apiserver-proxy.json new file mode 100644 index 000000000..5a1c21735 --- /dev/null +++ b/src/k8s/pkg/k8sd/setup/k8s-apiserver-proxy.json @@ -0,0 +1 @@ +{"endpoints":null} \ No newline at end of file diff --git a/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy.go b/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy.go index 8a54bb60a..e288afbfd 100644 --- a/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy.go +++ b/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy.go @@ -11,7 +11,7 @@ import ( ) // K8sAPIServerProxy prepares configuration for k8s-apiserver-proxy. -func K8sAPIServerProxy(snap snap.Snap, servers []string, extraArgs map[string]*string) error { +func K8sAPIServerProxy(snap snap.Snap, servers []string, securePort int, extraArgs map[string]*string) error { configFile := filepath.Join(snap.ServiceExtraConfigDir(), "k8s-apiserver-proxy.json") if err := proxy.WriteEndpointsConfig(servers, configFile); err != nil { return fmt.Errorf("failed to write proxy configuration file: %w", err) @@ -20,7 +20,7 @@ func K8sAPIServerProxy(snap snap.Snap, servers []string, extraArgs map[string]*s if _, err := snaputil.UpdateServiceArguments(snap, "k8s-apiserver-proxy", map[string]string{ "--endpoints": configFile, "--kubeconfig": filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), - "--listen": "127.0.0.1:6443", + "--listen": fmt.Sprintf(":%d", securePort), }, nil); err != nil { return fmt.Errorf("failed to write arguments file: %w", err) } diff --git a/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy_test.go b/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy_test.go index 3236e464b..8a01ea750 100644 --- a/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy_test.go +++ b/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy_test.go @@ -27,7 +27,7 @@ func TestK8sApiServerProxy(t *testing.T) { s := mustSetupSnapAndDirectories(t, setK8sApiServerMock) - g.Expect(setup.K8sAPIServerProxy(s, nil, nil)).To(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, nil, 6443, nil)).To(Succeed()) tests := []struct { key string @@ -35,20 +35,20 @@ func TestK8sApiServerProxy(t *testing.T) { }{ {key: "--endpoints", expectedVal: filepath.Join(s.Mock.ServiceExtraConfigDir, "k8s-apiserver-proxy.json")}, {key: "--kubeconfig", expectedVal: filepath.Join(s.Mock.KubernetesConfigDir, "kubelet.conf")}, - {key: "--listen", expectedVal: "127.0.0.1:6443"}, + {key: "--listen", expectedVal: ":6443"}, } for _, tc := range tests { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-apiserver-proxy", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(tc.expectedVal).To(Equal(val)) }) } args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "k8s-apiserver-proxy")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WithExtraArgs", func(t *testing.T) { @@ -61,7 +61,7 @@ func TestK8sApiServerProxy(t *testing.T) { "--listen": nil, // This should trigger a delete "--my-extra-arg": utils.Pointer("my-extra-val"), } - g.Expect(setup.K8sAPIServerProxy(s, nil, extraArgs)).To(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, nil, 6443, extraArgs)).To(Succeed()) tests := []struct { key string @@ -75,7 +75,7 @@ func TestK8sApiServerProxy(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-apiserver-proxy", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(tc.expectedVal).To(Equal(val)) }) } @@ -83,13 +83,13 @@ func TestK8sApiServerProxy(t *testing.T) { t.Run("--listen", func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-apiserver-proxy", "--listen") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeZero()) }) args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "k8s-apiserver-proxy")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("MissingExtraConfigDir", func(t *testing.T) { @@ -98,7 +98,7 @@ func TestK8sApiServerProxy(t *testing.T) { s := mustSetupSnapAndDirectories(t, setK8sApiServerMock) s.Mock.ServiceExtraConfigDir = "nonexistent" - g.Expect(setup.K8sAPIServerProxy(s, nil, nil)).ToNot(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, nil, 6443, nil)).ToNot(Succeed()) }) t.Run("MissingServiceArgumentsDir", func(t *testing.T) { @@ -107,7 +107,7 @@ func TestK8sApiServerProxy(t *testing.T) { s := mustSetupSnapAndDirectories(t, setK8sApiServerMock) s.Mock.ServiceArgumentsDir = "nonexistent" - g.Expect(setup.K8sAPIServerProxy(s, nil, nil)).ToNot(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, nil, 6443, nil)).ToNot(Succeed()) }) t.Run("JSONFileContent", func(t *testing.T) { @@ -118,7 +118,7 @@ func TestK8sApiServerProxy(t *testing.T) { endpoints := []string{"192.168.0.1", "192.168.0.2", "192.168.0.3"} fileName := filepath.Join(s.Mock.ServiceExtraConfigDir, "k8s-apiserver-proxy.json") - g.Expect(setup.K8sAPIServerProxy(s, endpoints, nil)).To(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, endpoints, 6443, nil)).To(Succeed()) b, err := os.ReadFile(fileName) g.Expect(err).NotTo(HaveOccurred()) @@ -130,4 +130,29 @@ func TestK8sApiServerProxy(t *testing.T) { // Compare the expected endpoints with those in the file g.Expect(config.Endpoints).To(Equal(endpoints)) }) + + t.Run("IPv6", func(t *testing.T) { + g := NewWithT(t) + + // Create a mock snap + s := mustSetupSnapAndDirectories(t, setKubeletMock) + s.Mock.Hostname = "dev" + + g.Expect(setup.K8sAPIServerProxy(s, nil, 1234, nil)).To(Succeed()) + + tests := []struct { + key string + expectedVal string + }{ + {key: "--listen", expectedVal: ":1234"}, + } + for _, tc := range tests { + t.Run(tc.key, func(t *testing.T) { + g := NewWithT(t) + val, err := snaputil.GetServiceArgument(s, "k8s-apiserver-proxy", tc.key) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(tc.expectedVal).To(Equal(val)) + }) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/k8s_dqlite.go b/src/k8s/pkg/k8sd/setup/k8s_dqlite.go index ab4b8fbcd..1ee6a5cf9 100644 --- a/src/k8s/pkg/k8sd/setup/k8s_dqlite.go +++ b/src/k8s/pkg/k8sd/setup/k8s_dqlite.go @@ -22,7 +22,7 @@ func K8sDqlite(snap snap.Snap, address string, cluster []string, extraArgs map[s if err := os.RemoveAll(snap.K8sDqliteStateDir()); err != nil { return fmt.Errorf("failed to cleanup not-empty k8s-dqlite directory: %w", err) } - if err := os.MkdirAll(snap.K8sDqliteStateDir(), 0700); err != nil { + if err := os.MkdirAll(snap.K8sDqliteStateDir(), 0o700); err != nil { return fmt.Errorf("failed to create k8s-dqlite state directory: %w", err) } } @@ -32,7 +32,7 @@ func K8sDqlite(snap snap.Snap, address string, cluster []string, extraArgs map[s return fmt.Errorf("failed to create init.yaml file for address=%s cluster=%v: %w", address, cluster, err) } - if err := os.WriteFile(filepath.Join(snap.K8sDqliteStateDir(), "init.yaml"), b, 0600); err != nil { + if err := utils.WriteFile(filepath.Join(snap.K8sDqliteStateDir(), "init.yaml"), b, 0o600); err != nil { return fmt.Errorf("failed to write init.yaml: %w", err) } diff --git a/src/k8s/pkg/k8sd/setup/k8s_dqlite_test.go b/src/k8s/pkg/k8sd/setup/k8s_dqlite_test.go index 8bfcedbab..391b3b616 100644 --- a/src/k8s/pkg/k8sd/setup/k8s_dqlite_test.go +++ b/src/k8s/pkg/k8sd/setup/k8s_dqlite_test.go @@ -28,7 +28,7 @@ func TestK8sDqlite(t *testing.T) { s := mustSetupSnapAndDirectories(t, setK8sDqliteMock) // Call the K8sDqlite setup function with mock arguments - g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", []string{"192.168.0.1:1234"}, nil)).To(BeNil()) + g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", []string{"192.168.0.1:1234"}, nil)).To(Succeed()) // Ensure the K8sDqlite arguments file has the expected arguments and values tests := []struct { @@ -42,7 +42,7 @@ func TestK8sDqlite(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-dqlite", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -50,7 +50,7 @@ func TestK8sDqlite(t *testing.T) { // Ensure the K8sDqlite arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "k8s-dqlite")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WithExtraArgs", func(t *testing.T) { @@ -65,7 +65,7 @@ func TestK8sDqlite(t *testing.T) { "--storage-dir": utils.Pointer("overridden-storage-dir"), } // Call the K8sDqlite setup function with mock arguments - g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", []string{"192.168.0.1:1234"}, extraArgs)).To(BeNil()) + g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", []string{"192.168.0.1:1234"}, extraArgs)).To(Succeed()) // Ensure the K8sDqlite arguments file has the expected arguments and values tests := []struct { @@ -79,7 +79,7 @@ func TestK8sDqlite(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-dqlite", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -88,14 +88,14 @@ func TestK8sDqlite(t *testing.T) { t.Run("--listen", func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-dqlite", "--listen") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeZero()) }) // Ensure the K8sDqlite arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "k8s-dqlite")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("YAMLFileContents", func(t *testing.T) { @@ -112,10 +112,10 @@ func TestK8sDqlite(t *testing.T) { "192.168.0.3:1234", } - g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", cluster, nil)).To(BeNil()) + g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", cluster, nil)).To(Succeed()) b, err := os.ReadFile(filepath.Join(s.Mock.K8sDqliteStateDir, "init.yaml")) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(string(b)).To(Equal(expectedYaml)) }) diff --git a/src/k8s/pkg/k8sd/setup/kube_apiserver.go b/src/k8s/pkg/k8sd/setup/kube_apiserver.go index 37c1c0192..cfac02e8e 100644 --- a/src/k8s/pkg/k8sd/setup/kube_apiserver.go +++ b/src/k8s/pkg/k8sd/setup/kube_apiserver.go @@ -5,6 +5,7 @@ import ( "net" "os" "path/filepath" + "strconv" "strings" "github.com/canonical/k8s/pkg/k8sd/types" @@ -49,9 +50,9 @@ var ( ) // KubeAPIServer configures kube-apiserver on the local node. -func KubeAPIServer(snap snap.Snap, nodeIP net.IP, serviceCIDR string, authWebhookURL string, enableFrontProxy bool, datastore types.Datastore, authorizationMode string, extraArgs map[string]*string) error { +func KubeAPIServer(snap snap.Snap, securePort int, nodeIP net.IP, serviceCIDR string, authWebhookURL string, enableFrontProxy bool, datastore types.Datastore, authorizationMode string, extraArgs map[string]*string) error { authTokenWebhookConfigFile := filepath.Join(snap.ServiceExtraConfigDir(), "auth-token-webhook.conf") - authTokenWebhookFile, err := os.OpenFile(authTokenWebhookConfigFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600) + authTokenWebhookFile, err := os.OpenFile(authTokenWebhookConfigFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600) if err != nil { return fmt.Errorf("failed to open auth-token-webhook.conf: %w", err) } @@ -77,13 +78,14 @@ func KubeAPIServer(snap snap.Snap, nodeIP net.IP, serviceCIDR string, authWebhoo "--kubelet-preferred-address-types": "InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP", "--profiling": "false", "--request-timeout": "300s", - "--secure-port": "6443", + "--secure-port": strconv.Itoa(securePort), "--service-account-issuer": "https://kubernetes.default.svc", "--service-account-key-file": filepath.Join(snap.KubernetesPKIDir(), "serviceaccount.key"), "--service-account-signing-key-file": filepath.Join(snap.KubernetesPKIDir(), "serviceaccount.key"), "--service-cluster-ip-range": serviceCIDR, "--tls-cert-file": filepath.Join(snap.KubernetesPKIDir(), "apiserver.crt"), "--tls-cipher-suites": strings.Join(apiserverTLSCipherSuites, ","), + "--tls-min-version": "VersionTLS12", "--tls-private-key-file": filepath.Join(snap.KubernetesPKIDir(), "apiserver.key"), } diff --git a/src/k8s/pkg/k8sd/setup/kube_apiserver_test.go b/src/k8s/pkg/k8sd/setup/kube_apiserver_test.go index 7327a83f0..44677aa73 100644 --- a/src/k8s/pkg/k8sd/setup/kube_apiserver_test.go +++ b/src/k8s/pkg/k8sd/setup/kube_apiserver_test.go @@ -37,7 +37,7 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Call the KubeAPIServer setup function with mock arguments - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", true, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", true, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(Succeed()) // Ensure the kube-apiserver arguments file has the expected arguments and values tests := []struct { @@ -63,6 +63,7 @@ func TestKubeAPIServer(t *testing.T) { {key: "--service-cluster-ip-range", expectedVal: "10.0.0.0/24"}, {key: "--tls-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.crt")}, {key: "--tls-cipher-suites", expectedVal: apiserverTLSCipherSuites}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--tls-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.key")}, {key: "--etcd-servers", expectedVal: fmt.Sprintf("unix://%s", filepath.Join(s.Mock.K8sDqliteStateDir, "k8s-dqlite.sock"))}, {key: "--request-timeout", expectedVal: "300s"}, @@ -78,7 +79,7 @@ func TestKubeAPIServer(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-apiserver", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -86,7 +87,7 @@ func TestKubeAPIServer(t *testing.T) { // Ensure the kube-apiserver arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ArgsNoProxy", func(t *testing.T) { @@ -96,7 +97,7 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Call the KubeAPIServer setup function with mock arguments - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(Succeed()) // Ensure the kube-apiserver arguments file has the expected arguments and values tests := []struct { @@ -123,6 +124,7 @@ func TestKubeAPIServer(t *testing.T) { {key: "--service-cluster-ip-range", expectedVal: "10.0.0.0/24"}, {key: "--tls-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.crt")}, {key: "--tls-cipher-suites", expectedVal: apiserverTLSCipherSuites}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--tls-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.key")}, {key: "--etcd-servers", expectedVal: fmt.Sprintf("unix://%s", filepath.Join(s.Mock.K8sDqliteStateDir, "k8s-dqlite.sock"))}, } @@ -130,7 +132,7 @@ func TestKubeAPIServer(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-apiserver", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -138,7 +140,7 @@ func TestKubeAPIServer(t *testing.T) { // Ensure the kube-apiserver arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WithExtraArgs", func(t *testing.T) { @@ -153,7 +155,7 @@ func TestKubeAPIServer(t *testing.T) { "--my-extra-arg": utils.Pointer("my-extra-val"), } // Call the KubeAPIServer setup function with mock arguments - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", true, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", extraArgs)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", true, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", extraArgs)).To(Succeed()) // Ensure the kube-apiserver arguments file has the expected arguments and values tests := []struct { @@ -178,6 +180,7 @@ func TestKubeAPIServer(t *testing.T) { {key: "--service-cluster-ip-range", expectedVal: "10.0.0.0/24"}, {key: "--tls-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.crt")}, {key: "--tls-cipher-suites", expectedVal: apiserverTLSCipherSuites}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--tls-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.key")}, {key: "--etcd-servers", expectedVal: fmt.Sprintf("unix://%s", filepath.Join(s.Mock.K8sDqliteStateDir, "k8s-dqlite.sock"))}, {key: "--request-timeout", expectedVal: "300s"}, @@ -194,7 +197,7 @@ func TestKubeAPIServer(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-apiserver", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -206,7 +209,7 @@ func TestKubeAPIServer(t *testing.T) { // Ensure the kube-apiserver arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ArgsDualstack", func(t *testing.T) { g := NewWithT(t) @@ -214,7 +217,7 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Setup without proxy to simplify argument list - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24,fd01::/64", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("external"), ExternalServers: utils.Pointer([]string{"datastoreurl1", "datastoreurl2"})}, "Node,RBAC", nil)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24,fd01::/64", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("external"), ExternalServers: utils.Pointer([]string{"datastoreurl1", "datastoreurl2"})}, "Node,RBAC", nil)).To(Succeed()) g.Expect(snaputil.GetServiceArgument(s, "kube-apiserver", "--service-cluster-ip-range")).To(Equal("10.0.0.0/24,fd01::/64")) _, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) @@ -227,7 +230,7 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Setup without proxy to simplify argument list - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("external"), ExternalServers: utils.Pointer([]string{"datastoreurl1", "datastoreurl2"})}, "Node,RBAC", nil)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("external"), ExternalServers: utils.Pointer([]string{"datastoreurl1", "datastoreurl2"})}, "Node,RBAC", nil)).To(Succeed()) g.Expect(snaputil.GetServiceArgument(s, "kube-apiserver", "--etcd-servers")).To(Equal("datastoreurl1,datastoreurl2")) _, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) @@ -241,8 +244,34 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Attempt to configure kube-apiserver with an unsupported datastore - err := setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("unsupported")}, "Node,RBAC", nil) + err := setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("unsupported")}, "Node,RBAC", nil) g.Expect(err).To(HaveOccurred()) g.Expect(err).To(MatchError(ContainSubstring("unsupported datastore"))) }) + + t.Run("IPv6", func(t *testing.T) { + g := NewWithT(t) + + // Create a mock snap + s := mustSetupSnapAndDirectories(t, setKubeletMock) + s.Mock.Hostname = "dev" + + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("2001:db8::"), "fd98::/108", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(Succeed()) + + tests := []struct { + key string + expectedVal string + }{ + {key: "--advertise-address", expectedVal: "2001:db8::"}, + {key: "--service-cluster-ip-range", expectedVal: "fd98::/108"}, + } + for _, tc := range tests { + t.Run(tc.key, func(t *testing.T) { + g := NewWithT(t) + val, err := snaputil.GetServiceArgument(s, "kube-apiserver", tc.key) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(tc.expectedVal).To(Equal(val)) + }) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/kube_controller_manager.go b/src/k8s/pkg/k8sd/setup/kube_controller_manager.go index 95eb941eb..18a3c435f 100644 --- a/src/k8s/pkg/k8sd/setup/kube_controller_manager.go +++ b/src/k8s/pkg/k8sd/setup/kube_controller_manager.go @@ -22,6 +22,7 @@ func KubeControllerManager(snap snap.Snap, extraArgs map[string]*string) error { "--root-ca-file": filepath.Join(snap.KubernetesPKIDir(), "ca.crt"), "--service-account-private-key-file": filepath.Join(snap.KubernetesPKIDir(), "serviceaccount.key"), "--terminated-pod-gc-threshold": "12500", + "--tls-min-version": "VersionTLS12", "--use-service-account-credentials": "true", } // enable cluster-signing if certificates are available diff --git a/src/k8s/pkg/k8sd/setup/kube_controller_manager_test.go b/src/k8s/pkg/k8sd/setup/kube_controller_manager_test.go index 453703095..7c274a6b7 100644 --- a/src/k8s/pkg/k8sd/setup/kube_controller_manager_test.go +++ b/src/k8s/pkg/k8sd/setup/kube_controller_manager_test.go @@ -31,7 +31,7 @@ func TestKubeControllerManager(t *testing.T) { os.Create(filepath.Join(s.Mock.KubernetesPKIDir, "ca.key")) // Call the kube controller manager setup function - g.Expect(setup.KubeControllerManager(s, nil)).To(BeNil()) + g.Expect(setup.KubeControllerManager(s, nil)).To(Succeed()) // Ensure the kube controller manager arguments file has the expected arguments and values tests := []struct { @@ -47,6 +47,7 @@ func TestKubeControllerManager(t *testing.T) { {key: "--root-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--service-account-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "serviceaccount.key")}, {key: "--terminated-pod-gc-threshold", expectedVal: "12500"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--use-service-account-credentials", expectedVal: "true"}, {key: "--cluster-signing-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--cluster-signing-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.key")}, @@ -63,7 +64,7 @@ func TestKubeControllerManager(t *testing.T) { // Ensure the kube controller manager arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-controller-manager")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) t.Run("MissingArgsDir", func(t *testing.T) { g := NewWithT(t) @@ -79,7 +80,7 @@ func TestKubeControllerManager(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeControllerManagerMock) // Call the kube controller manager setup function - g.Expect(setup.KubeControllerManager(s, nil)).To(BeNil()) + g.Expect(setup.KubeControllerManager(s, nil)).To(Succeed()) // Ensure the kube controller manager arguments file has the expected arguments and values tests := []struct { @@ -95,6 +96,7 @@ func TestKubeControllerManager(t *testing.T) { {key: "--root-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--service-account-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "serviceaccount.key")}, {key: "--terminated-pod-gc-threshold", expectedVal: "12500"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--use-service-account-credentials", expectedVal: "true"}, } for _, tc := range tests { @@ -109,7 +111,7 @@ func TestKubeControllerManager(t *testing.T) { // Ensure the kube controller manager arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-controller-manager")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) t.Run("MissingArgsDir", func(t *testing.T) { g := NewWithT(t) @@ -133,7 +135,7 @@ func TestKubeControllerManager(t *testing.T) { "--my-extra-arg": utils.Pointer("my-extra-val"), } // Call the kube controller manager setup function - g.Expect(setup.KubeControllerManager(s, extraArgs)).To(BeNil()) + g.Expect(setup.KubeControllerManager(s, extraArgs)).To(Succeed()) // Ensure the kube controller manager arguments file has the expected arguments and values tests := []struct { @@ -148,6 +150,7 @@ func TestKubeControllerManager(t *testing.T) { {key: "--root-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--service-account-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "serviceaccount.key")}, {key: "--terminated-pod-gc-threshold", expectedVal: "12500"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--use-service-account-credentials", expectedVal: "true"}, {key: "--cluster-signing-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--cluster-signing-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.key")}, @@ -170,7 +173,7 @@ func TestKubeControllerManager(t *testing.T) { // Ensure the kube controller manager arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-controller-manager")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) t.Run("MissingArgsDir", func(t *testing.T) { g := NewWithT(t) diff --git a/src/k8s/pkg/k8sd/setup/kube_proxy.go b/src/k8s/pkg/k8sd/setup/kube_proxy.go index 0e64a60aa..ff29b0443 100644 --- a/src/k8s/pkg/k8sd/setup/kube_proxy.go +++ b/src/k8s/pkg/k8sd/setup/kube_proxy.go @@ -12,10 +12,10 @@ import ( ) // KubeProxy configures kube-proxy on the local node. -func KubeProxy(ctx context.Context, snap snap.Snap, hostname string, podCIDR string, extraArgs map[string]*string) error { +func KubeProxy(ctx context.Context, snap snap.Snap, hostname string, podCIDR string, localhostAddress string, extraArgs map[string]*string) error { serviceArgs := map[string]string{ "--cluster-cidr": podCIDR, - "--healthz-bind-address": "127.0.0.1", + "--healthz-bind-address": fmt.Sprintf("%s:10256", localhostAddress), "--kubeconfig": filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), "--profiling": "false", } diff --git a/src/k8s/pkg/k8sd/setup/kube_proxy_test.go b/src/k8s/pkg/k8sd/setup/kube_proxy_test.go index 1852c8478..f0eb92cf8 100644 --- a/src/k8s/pkg/k8sd/setup/kube_proxy_test.go +++ b/src/k8s/pkg/k8sd/setup/kube_proxy_test.go @@ -28,10 +28,10 @@ func TestKubeProxy(t *testing.T) { }, } - g.Expect(setup.EnsureAllDirectories(s)).To(BeNil()) + g.Expect(setup.EnsureAllDirectories(s)).To(Succeed()) t.Run("Args", func(t *testing.T) { - g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", nil)).To(BeNil()) + g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", "127.0.0.1", nil)).To(Succeed()) for key, expectedVal := range map[string]string{ "--cluster-cidr": "10.1.0.0/16", @@ -43,7 +43,7 @@ func TestKubeProxy(t *testing.T) { t.Run(key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-proxy", key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(expectedVal)) }) } @@ -55,7 +55,7 @@ func TestKubeProxy(t *testing.T) { "--healthz-bind-address": nil, "--my-extra-arg": utils.Pointer("my-extra-val"), } - g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", extraArgs)).To(BeNil()) + g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", "127.0.0.1", extraArgs)).To(Not(HaveOccurred())) for key, expectedVal := range map[string]string{ "--cluster-cidr": "10.1.0.0/16", @@ -68,7 +68,7 @@ func TestKubeProxy(t *testing.T) { t.Run(key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-proxy", key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(expectedVal)) }) } @@ -80,7 +80,7 @@ func TestKubeProxy(t *testing.T) { s.Mock.OnLXD = true t.Run("ArgsOnLXD", func(t *testing.T) { - g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", nil)).To(BeNil()) + g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", "127.0.0.1", nil)).To(Succeed()) for key, expectedVal := range map[string]string{ "--conntrack-max-per-core": "0", @@ -88,7 +88,7 @@ func TestKubeProxy(t *testing.T) { t.Run(key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-proxy", key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(expectedVal)) }) } @@ -102,11 +102,37 @@ func TestKubeProxy(t *testing.T) { s.Mock.Hostname = "dev" s.Mock.ServiceArgumentsDir = filepath.Join(dir, "k8s") - g.Expect(setup.EnsureAllDirectories(s)).To(BeNil()) - g.Expect(setup.KubeProxy(context.Background(), s, "dev", "10.1.0.0/16", nil)).To(BeNil()) + g.Expect(setup.EnsureAllDirectories(s)).To(Succeed()) + g.Expect(setup.KubeProxy(context.Background(), s, "dev", "10.1.0.0/16", "127.0.0.1", nil)).To(Succeed()) val, err := snaputil.GetServiceArgument(s, "kube-proxy", "--hostname-override") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeEmpty()) }) + + t.Run("IPv6", func(t *testing.T) { + g := NewWithT(t) + + // Create a mock snap + s := mustSetupSnapAndDirectories(t, setKubeletMock) + s.Mock.Hostname = "dev" + + g.Expect(setup.KubeProxy(context.Background(), s, "dev", "fd98::/108", "[::1]", nil)).To(Succeed()) + + tests := []struct { + key string + expectedVal string + }{ + {key: "--cluster-cidr", expectedVal: "fd98::/108"}, + {key: "--healthz-bind-address", expectedVal: "[::1]:10256"}, + } + for _, tc := range tests { + t.Run(tc.key, func(t *testing.T) { + g := NewWithT(t) + val, err := snaputil.GetServiceArgument(s, "kube-proxy", tc.key) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(tc.expectedVal).To(Equal(val)) + }) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/kube_scheduler.go b/src/k8s/pkg/k8sd/setup/kube_scheduler.go index 8a68b1689..ab64bfd29 100644 --- a/src/k8s/pkg/k8sd/setup/kube_scheduler.go +++ b/src/k8s/pkg/k8sd/setup/kube_scheduler.go @@ -18,6 +18,7 @@ func KubeScheduler(snap snap.Snap, extraArgs map[string]*string) error { "--leader-elect-lease-duration": "30s", "--leader-elect-renew-deadline": "15s", "--profiling": "false", + "--tls-min-version": "VersionTLS12", }, nil); err != nil { return fmt.Errorf("failed to render arguments file: %w", err) } diff --git a/src/k8s/pkg/k8sd/setup/kube_scheduler_test.go b/src/k8s/pkg/k8sd/setup/kube_scheduler_test.go index 20339734a..ae84ba87f 100644 --- a/src/k8s/pkg/k8sd/setup/kube_scheduler_test.go +++ b/src/k8s/pkg/k8sd/setup/kube_scheduler_test.go @@ -26,7 +26,7 @@ func TestKubeScheduler(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeSchedulerMock) // Call the kube scheduler setup function - g.Expect(setup.KubeScheduler(s, nil)).To(BeNil()) + g.Expect(setup.KubeScheduler(s, nil)).To(Succeed()) // Ensure the kube scheduler arguments file has the expected arguments and values tests := []struct { @@ -39,6 +39,7 @@ func TestKubeScheduler(t *testing.T) { {key: "--leader-elect-lease-duration", expectedVal: "30s"}, {key: "--leader-elect-renew-deadline", expectedVal: "15s"}, {key: "--profiling", expectedVal: "false"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, } for _, tc := range tests { t.Run(tc.key, func(t *testing.T) { @@ -52,8 +53,7 @@ func TestKubeScheduler(t *testing.T) { // Ensure the kube scheduler arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-scheduler")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) - + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WithExtraArgs", func(t *testing.T) { @@ -68,7 +68,7 @@ func TestKubeScheduler(t *testing.T) { "--my-extra-arg": utils.Pointer("my-extra-val"), } // Call the kube scheduler setup function - g.Expect(setup.KubeScheduler(s, extraArgs)).To(BeNil()) + g.Expect(setup.KubeScheduler(s, extraArgs)).To(Succeed()) // Ensure the kube scheduler arguments file has the expected arguments and values tests := []struct { @@ -80,6 +80,7 @@ func TestKubeScheduler(t *testing.T) { {key: "--kubeconfig", expectedVal: filepath.Join(s.Mock.KubernetesConfigDir, "scheduler.conf")}, {key: "--leader-elect-renew-deadline", expectedVal: "15s"}, {key: "--profiling", expectedVal: "true"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--my-extra-arg", expectedVal: "my-extra-val"}, } for _, tc := range tests { @@ -99,8 +100,7 @@ func TestKubeScheduler(t *testing.T) { // Ensure the kube scheduler arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-scheduler")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) - + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("MissingArgsDir", func(t *testing.T) { diff --git a/src/k8s/pkg/k8sd/setup/kubelet.go b/src/k8s/pkg/k8sd/setup/kubelet.go index ff4cd9e0f..6b387b3d6 100644 --- a/src/k8s/pkg/k8sd/setup/kubelet.go +++ b/src/k8s/pkg/k8sd/setup/kubelet.go @@ -52,8 +52,8 @@ func kubelet(snap snap.Snap, hostname string, nodeIP net.IP, clusterDNS string, "--anonymous-auth": "false", "--authentication-token-webhook": "true", "--client-ca-file": filepath.Join(snap.KubernetesPKIDir(), "client-ca.crt"), - "--container-runtime-endpoint": filepath.Join(snap.ContainerdSocketDir(), "containerd.sock"), - "--containerd": filepath.Join(snap.ContainerdSocketDir(), "containerd.sock"), + "--container-runtime-endpoint": snap.ContainerdSocketPath(), + "--containerd": snap.ContainerdSocketPath(), "--cgroup-driver": "systemd", "--eviction-hard": "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'", "--fail-swap-on": "false", diff --git a/src/k8s/pkg/k8sd/setup/kubelet_test.go b/src/k8s/pkg/k8sd/setup/kubelet_test.go index 99129e3d5..a0cb91ba3 100644 --- a/src/k8s/pkg/k8sd/setup/kubelet_test.go +++ b/src/k8s/pkg/k8sd/setup/kubelet_test.go @@ -14,8 +14,10 @@ import ( // These values are hard-coded and need to be updated if the // implementation changes. -var expectedControlPlaneLabels = "node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/worker=,k8sd.io/role=control-plane" -var expectedWorkerLabels = "node-role.kubernetes.io/worker=,k8sd.io/role=worker" +var ( + expectedControlPlaneLabels = "node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/worker=,k8sd.io/role=control-plane" + expectedWorkerLabels = "node-role.kubernetes.io/worker=,k8sd.io/role=worker" +) var kubeletTLSCipherSuites = "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384" @@ -30,11 +32,12 @@ func mustSetupSnapAndDirectories(t *testing.T, createMock func(*mock.Snap, strin func setKubeletMock(s *mock.Snap, dir string) { s.Mock = mock.Mock{ - KubernetesPKIDir: filepath.Join(dir, "pki"), - KubernetesConfigDir: filepath.Join(dir, "k8s-config"), - KubeletRootDir: filepath.Join(dir, "kubelet-root"), - ServiceArgumentsDir: filepath.Join(dir, "args"), - ContainerdSocketDir: filepath.Join(dir, "containerd-run"), + KubernetesPKIDir: filepath.Join(dir, "pki"), + KubernetesConfigDir: filepath.Join(dir, "k8s-config"), + KubeletRootDir: filepath.Join(dir, "kubelet-root"), + ServiceArgumentsDir: filepath.Join(dir, "args"), + ContainerdSocketDir: filepath.Join(dir, "containerd-run"), + ContainerdSocketPath: filepath.Join(dir, "containerd-run", "containerd.sock"), } } @@ -57,8 +60,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -89,7 +92,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ControlPlaneWithExtraArgs", func(t *testing.T) { @@ -115,8 +118,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -153,7 +156,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ControlPlaneArgsNoOptional", func(t *testing.T) { @@ -163,7 +166,7 @@ func TestKubelet(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeletMock) // Call the kubelet control plane setup function - g.Expect(setup.KubeletControlPlane(s, "dev", nil, "", "", "", nil, nil)).To(BeNil()) + g.Expect(setup.KubeletControlPlane(s, "dev", nil, "", "", "", nil, nil)).To(Succeed()) tests := []struct { key string @@ -173,8 +176,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -201,7 +204,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WorkerArgs", func(t *testing.T) { @@ -211,7 +214,7 @@ func TestKubelet(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeletMock) // Call the kubelet worker setup function - g.Expect(setup.KubeletWorker(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", nil)).To(BeNil()) + g.Expect(setup.KubeletWorker(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", nil)).To(Succeed()) // Ensure the kubelet arguments file has the expected arguments and values tests := []struct { @@ -222,8 +225,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -254,7 +257,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WorkerWithExtraArgs", func(t *testing.T) { @@ -269,7 +272,7 @@ func TestKubelet(t *testing.T) { } // Call the kubelet worker setup function - g.Expect(setup.KubeletWorker(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", extraArgs)).To(BeNil()) + g.Expect(setup.KubeletWorker(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", extraArgs)).To(Succeed()) // Ensure the kubelet arguments file has the expected arguments and values tests := []struct { @@ -280,8 +283,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -316,7 +319,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WorkerArgsNoOptional", func(t *testing.T) { @@ -326,7 +329,7 @@ func TestKubelet(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeletMock) // Call the kubelet worker setup function - g.Expect(setup.KubeletWorker(s, "dev", nil, "", "", "", nil)).To(BeNil()) + g.Expect(setup.KubeletWorker(s, "dev", nil, "", "", "", nil)).To(Succeed()) // Ensure the kubelet arguments file has the expected arguments and values tests := []struct { @@ -337,8 +340,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -365,7 +368,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ControlPlaneNoArgsDir", func(t *testing.T) { @@ -397,7 +400,34 @@ func TestKubelet(t *testing.T) { g.Expect(setup.KubeletControlPlane(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", nil, nil)).To(Succeed()) val, err := snaputil.GetServiceArgument(s, "kubelet", "--hostname-override") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeEmpty()) }) + + t.Run("IPv6", func(t *testing.T) { + g := NewWithT(t) + + // Create a mock snap + s := mustSetupSnapAndDirectories(t, setKubeletMock) + s.Mock.Hostname = "dev" + + // Call the kubelet control plane setup function + g.Expect(setup.KubeletControlPlane(s, "dev", net.ParseIP("2001:db8::"), "2001:db8::1", "test-cluster.local", "provider", nil, nil)).To(Succeed()) + + tests := []struct { + key string + expectedVal string + }{ + {key: "--cluster-dns", expectedVal: "2001:db8::1"}, + {key: "--node-ip", expectedVal: "2001:db8::"}, + } + for _, tc := range tests { + t.Run(tc.key, func(t *testing.T) { + g := NewWithT(t) + val, err := snaputil.GetServiceArgument(s, "kubelet", tc.key) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(tc.expectedVal).To(Equal(val)) + }) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/templates.go b/src/k8s/pkg/k8sd/setup/templates.go index 8725c8f6a..4fac815b4 100644 --- a/src/k8s/pkg/k8sd/setup/templates.go +++ b/src/k8s/pkg/k8sd/setup/templates.go @@ -7,16 +7,14 @@ import ( "text/template" ) -var ( - //go:embed embed - templates embed.FS -) +//go:embed embed +var templates embed.FS func mustTemplate(parts ...string) *template.Template { path := filepath.Join(append([]string{"embed"}, parts...)...) b, err := templates.ReadFile(path) if err != nil { - panic(fmt.Errorf("invalid template %s: %s", path, err)) + panic(fmt.Errorf("invalid template %s: %w", path, err)) } return template.Must(template.New(path).Parse(string(b))) } diff --git a/src/k8s/pkg/k8sd/setup/util_extra_files.go b/src/k8s/pkg/k8sd/setup/util_extra_files.go index 31f3cfb57..163562ea5 100644 --- a/src/k8s/pkg/k8sd/setup/util_extra_files.go +++ b/src/k8s/pkg/k8sd/setup/util_extra_files.go @@ -7,6 +7,7 @@ import ( "strings" "github.com/canonical/k8s/pkg/snap" + "github.com/canonical/k8s/pkg/utils" ) // ExtraNodeConfigFiles writes the file contents to the specified filenames in the snap.ExtraFilesDir directory. @@ -20,7 +21,7 @@ func ExtraNodeConfigFiles(snap snap.Snap, files map[string]string) error { filePath := filepath.Join(snap.ServiceExtraConfigDir(), filename) // Write the content to the file - if err := os.WriteFile(filePath, []byte(content), 0400); err != nil { + if err := utils.WriteFile(filePath, []byte(content), 0o400); err != nil { return fmt.Errorf("failed to write to file %s: %w", filePath, err) } diff --git a/src/k8s/pkg/k8sd/setup/util_extra_files_test.go b/src/k8s/pkg/k8sd/setup/util_extra_files_test.go index ad4fa38bb..378810513 100644 --- a/src/k8s/pkg/k8sd/setup/util_extra_files_test.go +++ b/src/k8s/pkg/k8sd/setup/util_extra_files_test.go @@ -60,7 +60,7 @@ func TestExtraNodeConfigFiles(t *testing.T) { // Verify the file exists info, err := os.Stat(filePath) g.Expect(err).ToNot(gomega.HaveOccurred()) - g.Expect(info.Mode().Perm()).To(gomega.Equal(os.FileMode(0400))) + g.Expect(info.Mode().Perm()).To(gomega.Equal(os.FileMode(0o400))) // Verify the file content actualContent, err := os.ReadFile(filePath) diff --git a/src/k8s/pkg/k8sd/setup/util_kubeconfig.go b/src/k8s/pkg/k8sd/setup/util_kubeconfig.go index 3793a97f0..7b1157df1 100644 --- a/src/k8s/pkg/k8sd/setup/util_kubeconfig.go +++ b/src/k8s/pkg/k8sd/setup/util_kubeconfig.go @@ -57,7 +57,7 @@ func KubeconfigString(url string, caPEM string, crtPEM string, keyPEM string) (s } // SetupControlPlaneKubeconfigs writes kubeconfig files for the control plane components. -func SetupControlPlaneKubeconfigs(kubeConfigDir string, securePort int, pki pki.ControlPlanePKI) error { +func SetupControlPlaneKubeconfigs(kubeConfigDir string, localhostAddress string, securePort int, pki pki.ControlPlanePKI) error { for _, kubeconfig := range []struct { file string crt string @@ -69,10 +69,9 @@ func SetupControlPlaneKubeconfigs(kubeConfigDir string, securePort int, pki pki. {file: "scheduler.conf", crt: pki.KubeSchedulerClientCert, key: pki.KubeSchedulerClientKey}, {file: "kubelet.conf", crt: pki.KubeletClientCert, key: pki.KubeletClientKey}, } { - if err := Kubeconfig(filepath.Join(kubeConfigDir, kubeconfig.file), fmt.Sprintf("127.0.0.1:%d", securePort), pki.CACert, kubeconfig.crt, kubeconfig.key); err != nil { + if err := Kubeconfig(filepath.Join(kubeConfigDir, kubeconfig.file), fmt.Sprintf("%s:%d", localhostAddress, securePort), pki.CACert, kubeconfig.crt, kubeconfig.key); err != nil { return fmt.Errorf("failed to write kubeconfig %s: %w", kubeconfig.file, err) } } return nil - } diff --git a/src/k8s/pkg/k8sd/setup/util_kubeconfig_test.go b/src/k8s/pkg/k8sd/setup/util_kubeconfig_test.go index 6a51e0f4e..230f921b6 100644 --- a/src/k8s/pkg/k8sd/setup/util_kubeconfig_test.go +++ b/src/k8s/pkg/k8sd/setup/util_kubeconfig_test.go @@ -34,5 +34,5 @@ users: actual, err := setup.KubeconfigString("server", "ca", "crt", "key") g.Expect(actual).To(Equal(expectedConfig)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } diff --git a/src/k8s/pkg/k8sd/types/cluster_config_certificates.go b/src/k8s/pkg/k8sd/types/cluster_config_certificates.go index 9d2b7d066..e7aa1120c 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_certificates.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_certificates.go @@ -25,6 +25,7 @@ func (c Certificates) GetClientCACert() string { } return c.GetCACert() } + func (c Certificates) GetClientCAKey() string { // versions before 1.30.2 were using the same CA for server and client certificates if v := getField(c.ClientCAKey); v != "" { @@ -38,6 +39,7 @@ func (c Certificates) GetServiceAccountKey() string { return getField(c.ServiceA func (c Certificates) GetAPIServerKubeletClientCert() string { return getField(c.APIServerKubeletClientCert) } + func (c Certificates) GetAPIServerKubeletClientKey() string { return getField(c.APIServerKubeletClientKey) } @@ -46,5 +48,5 @@ func (c Certificates) GetAdminClientKey() string { return getField(c.AdminClien func (c Certificates) GetK8sdPublicKey() string { return getField(c.K8sdPublicKey) } func (c Certificates) GetK8sdPrivateKey() string { return getField(c.K8sdPrivateKey) } -// Empty returns true if all Certificates fields are unset +// Empty returns true if all Certificates fields are unset. func (c Certificates) Empty() bool { return c == Certificates{} } diff --git a/src/k8s/pkg/k8sd/types/cluster_config_convert_loadbalancer_internal_test.go b/src/k8s/pkg/k8sd/types/cluster_config_convert_loadbalancer_internal_test.go index debaa6d20..413ac2390 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_convert_loadbalancer_internal_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_convert_loadbalancer_internal_test.go @@ -71,7 +71,7 @@ func Test_loadBalancerCIDRsFromAPI(t *testing.T) { t.Run("Nil", func(t *testing.T) { g := NewWithT(t) cidrs, ranges, err := loadBalancerCIDRsFromAPI(nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cidrs).To(BeNil()) g.Expect(ranges).To(BeNil()) }) @@ -84,7 +84,7 @@ func Test_loadBalancerCIDRsFromAPI(t *testing.T) { if tc.expectErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*cidrs).To(Equal(tc.internalCIDRs)) g.Expect(*ranges).To(Equal(tc.internalRanges)) } diff --git a/src/k8s/pkg/k8sd/types/cluster_config_convert_test.go b/src/k8s/pkg/k8sd/types/cluster_config_convert_test.go index 194eedc80..ad58cb846 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_convert_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_convert_test.go @@ -187,7 +187,7 @@ func TestClusterConfigFromBootstrapConfig(t *testing.T) { g := NewWithT(t) config, err := types.ClusterConfigFromBootstrapConfig(tc.bootstrap) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(config).To(Equal(tc.expectConfig)) }) } @@ -254,6 +254,5 @@ func TestClusterConfigFromBootstrapConfig(t *testing.T) { g.Expect(err).To(HaveOccurred()) }) } - }) } diff --git a/src/k8s/pkg/k8sd/types/cluster_config_datastore.go b/src/k8s/pkg/k8sd/types/cluster_config_datastore.go index 6f0d4dbd0..1f354904b 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_datastore.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_datastore.go @@ -29,7 +29,7 @@ func (c Datastore) GetExternalClientCert() string { return getField(c.ExternalCl func (c Datastore) GetExternalClientKey() string { return getField(c.ExternalClientKey) } func (c Datastore) Empty() bool { return c == Datastore{} } -// DatastorePathsProvider is to avoid circular dependency for snap.Snap in Datastore.ToKubeAPIServerArguments() +// DatastorePathsProvider is to avoid circular dependency for snap.Snap in Datastore.ToKubeAPIServerArguments(). type DatastorePathsProvider interface { K8sDqliteStateDir() string EtcdPKIDir() string diff --git a/src/k8s/pkg/k8sd/types/cluster_config_kubelet_test.go b/src/k8s/pkg/k8sd/types/cluster_config_kubelet_test.go index 92c0cd7c9..eb6f6d99a 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_kubelet_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_kubelet_test.go @@ -72,7 +72,7 @@ func TestKubelet(t *testing.T) { g := NewWithT(t) cm, err := tc.kubelet.ToConfigMap(nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cm).To(Equal(tc.configmap)) }) @@ -80,7 +80,7 @@ func TestKubelet(t *testing.T) { g := NewWithT(t) k, err := types.KubeletFromConfigMap(tc.configmap, nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(k).To(Equal(tc.kubelet)) }) }) @@ -90,7 +90,7 @@ func TestKubelet(t *testing.T) { func TestKubeletSign(t *testing.T) { g := NewWithT(t) key, err := rsa.GenerateKey(rand.Reader, 4096) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) kubelet := types.Kubelet{ CloudProvider: utils.Pointer("external"), @@ -99,14 +99,14 @@ func TestKubeletSign(t *testing.T) { } configmap, err := kubelet.ToConfigMap(key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(configmap).To(HaveKeyWithValue("k8sd-mac", Not(BeEmpty()))) t.Run("NoSign", func(t *testing.T) { g := NewWithT(t) configmap, err := kubelet.ToConfigMap(nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(configmap).To(Not(HaveKey("k8sd-mac"))) }) @@ -114,7 +114,7 @@ func TestKubeletSign(t *testing.T) { g := NewWithT(t) fromKubelet, err := types.KubeletFromConfigMap(configmap, &key.PublicKey) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(fromKubelet).To(Equal(kubelet)) }) @@ -122,7 +122,7 @@ func TestKubeletSign(t *testing.T) { g := NewWithT(t) configmap2, err := kubelet.ToConfigMap(key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(configmap2).To(Equal(configmap)) }) @@ -130,7 +130,7 @@ func TestKubeletSign(t *testing.T) { g := NewWithT(t) wrongKey, err := rsa.GenerateKey(rand.Reader, 2048) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) cm, err := types.KubeletFromConfigMap(configmap, &wrongKey.PublicKey) g.Expect(cm).To(BeZero()) @@ -142,10 +142,10 @@ func TestKubeletSign(t *testing.T) { t.Run(editKey, func(t *testing.T) { g := NewWithT(t) key, err := rsa.GenerateKey(rand.Reader, 2048) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) c, err := kubelet.ToConfigMap(key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(c).To(HaveKeyWithValue("k8sd-mac", Not(BeEmpty()))) t.Run("Manipulated", func(t *testing.T) { diff --git a/src/k8s/pkg/k8sd/types/cluster_config_merge_test.go b/src/k8s/pkg/k8sd/types/cluster_config_merge_test.go index e77f44fcb..532ec3f1c 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_merge_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_merge_test.go @@ -208,9 +208,9 @@ func TestMergeClusterConfig(t *testing.T) { result, err := types.MergeClusterConfig(tc.old, tc.new) if tc.expectErr { - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(Equal(tc.expectResult)) } }) @@ -408,7 +408,7 @@ func TestMergeClusterConfig_Scenarios(t *testing.T) { if tc.expectErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(merged).To(Equal(tc.expectMerged)) } }) diff --git a/src/k8s/pkg/k8sd/types/cluster_config_merge_util_test.go b/src/k8s/pkg/k8sd/types/cluster_config_merge_util_test.go index 251826d1c..24032c9b9 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_merge_util_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_merge_util_test.go @@ -28,12 +28,12 @@ func Test_mergeField(t *testing.T) { result, err := mergeField(tc.old, tc.new, tc.allowChange) switch { case tc.expectErr: - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) case tc.expectVal == nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(BeNil()) case tc.expectVal != nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*result).To(Equal(*tc.expectVal)) } }) @@ -60,12 +60,12 @@ func Test_mergeField(t *testing.T) { result, err := mergeField(tc.old, tc.new, tc.allowChange) switch { case tc.expectErr: - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) case tc.expectVal == nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(BeNil()) case tc.expectVal != nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*result).To(Equal(*tc.expectVal)) } }) @@ -94,12 +94,12 @@ func Test_mergeField(t *testing.T) { result, err := mergeField(tc.old, tc.new, tc.allowChange) switch { case tc.expectErr: - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) case tc.expectVal == nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(BeNil()) case tc.expectVal != nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*result).To(Equal(*tc.expectVal)) } }) @@ -128,12 +128,12 @@ func Test_mergeSliceField(t *testing.T) { result, err := mergeSliceField(tc.old, tc.new, tc.allowChange) switch { case tc.expectErr: - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) case tc.expectVal == nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(BeNil()) case tc.expectVal != nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*result).To(Equal(*tc.expectVal)) } }) diff --git a/src/k8s/pkg/k8sd/types/cluster_config_validate.go b/src/k8s/pkg/k8sd/types/cluster_config_validate.go index c673e3173..8ec1b8d50 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_validate.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_validate.go @@ -6,6 +6,8 @@ import ( "net/netip" "net/url" "strings" + + "github.com/canonical/k8s/pkg/utils" ) func validateCIDRs(cidrString string) error { @@ -21,6 +23,65 @@ func validateCIDRs(cidrString string) error { return nil } +// validateCIDROverlap checks for overlap and size constraints between pod and service CIDRs. +// It parses the provided podCIDR and serviceCIDR strings, checks for IPv4 and IPv6 overlaps. +func validateCIDROverlap(podCIDR string, serviceCIDR string) error { + // Parse the CIDRs + podIPv4CIDR, podIPv6CIDR, err := utils.SplitCIDRStrings(podCIDR) + if err != nil { + return fmt.Errorf("failed to parse pod CIDR: %w", err) + } + + svcIPv4CIDR, svcIPv6CIDR, err := utils.SplitCIDRStrings(serviceCIDR) + if err != nil { + return fmt.Errorf("failed to parse service CIDR: %w", err) + } + + // Check for IPv4 overlap + if podIPv4CIDR != "" && svcIPv4CIDR != "" { + if overlap, err := utils.CIDRsOverlap(podIPv4CIDR, svcIPv4CIDR); err != nil { + return fmt.Errorf("failed to check for IPv4 overlap: %w", err) + } else if overlap { + return fmt.Errorf("pod CIDR %q and service CIDR %q overlap", podCIDR, serviceCIDR) + } + } + + // Check for IPv6 overlap + if podIPv6CIDR != "" && svcIPv6CIDR != "" { + if overlap, err := utils.CIDRsOverlap(podIPv6CIDR, svcIPv6CIDR); err != nil { + return fmt.Errorf("failed to check for IPv6 overlap: %w", err) + } else if overlap { + return fmt.Errorf("pod CIDR %q and service CIDR %q overlap", podCIDR, serviceCIDR) + } + } + + return nil +} + +// validateIPv6CIDRSize ensures that the service IPv6 CIDR is not larger than /108. +// Ref: https://documentation.ubuntu.com/canonical-kubernetes/latest/snap/howto/networking/dualstack/#cidr-size-limitations +func validateIPv6CIDRSize(serviceCIDR string) error { + _, svcIPv6CIDR, err := utils.SplitCIDRStrings(serviceCIDR) + if err != nil { + return fmt.Errorf("invalid CIDR: %w", err) + } + + if svcIPv6CIDR == "" { + return nil + } + + _, ipv6Net, err := net.ParseCIDR(svcIPv6CIDR) + if err != nil { + return fmt.Errorf("invalid CIDR: %w", err) + } + + if prefixLength, _ := ipv6Net.Mask.Size(); prefixLength < 108 { + return fmt.Errorf("service CIDR %q cannot be larger than /108", serviceCIDR) + } + + return nil +} + // Validate that a ClusterConfig does not have conflicting or incompatible options. func (c *ClusterConfig) Validate() error { // check: validate that PodCIDR and ServiceCIDR are configured @@ -31,6 +92,14 @@ func (c *ClusterConfig) Validate() error { return fmt.Errorf("invalid service CIDR: %w", err) } + if err := validateCIDROverlap(c.Network.GetPodCIDR(), c.Network.GetServiceCIDR()); err != nil { + return fmt.Errorf("invalid cidr configuration: %w", err) + } + // Can't be an else-if, because default values could already be set. + if err := validateIPv6CIDRSize(c.Network.GetServiceCIDR()); err != nil { + return fmt.Errorf("invalid service CIDR: %w", err) + } + // check: ensure network is enabled if any of ingress, gateway, load-balancer are enabled if !c.Network.GetEnabled() { if c.Gateway.GetEnabled() { diff --git a/src/k8s/pkg/k8sd/types/cluster_config_validate_test.go b/src/k8s/pkg/k8sd/types/cluster_config_validate_test.go index db58934ed..d24ca3fdb 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_validate_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_validate_test.go @@ -10,15 +10,18 @@ import ( func TestValidateCIDR(t *testing.T) { for _, tc := range []struct { - cidr string - expectErr bool + cidr string + expectPodErr bool + expectSvcErr bool }{ - {cidr: "10.1.0.0/16"}, - {cidr: "2001:0db8::/32"}, - {cidr: "10.1.0.0/16,2001:0db8::/32"}, - {cidr: "", expectErr: true}, - {cidr: "bananas", expectErr: true}, - {cidr: "fd01::/64,fd02::/64,fd03::/64", expectErr: true}, + {cidr: "192.168.0.0/16"}, + {cidr: "2001:0db8::/108"}, + {cidr: "10.2.0.0/16,2001:0db8::/108"}, + {cidr: "", expectPodErr: true, expectSvcErr: true}, + {cidr: "bananas", expectPodErr: true, expectSvcErr: true}, + {cidr: "fd01::/108,fd02::/108,fd03::/108", expectPodErr: true, expectSvcErr: true}, + {cidr: "10.1.0.0/32", expectPodErr: true, expectSvcErr: true}, + {cidr: "2001:0db8::/32", expectSvcErr: true}, } { t.Run(tc.cidr, func(t *testing.T) { t.Run("Pod", func(t *testing.T) { @@ -30,10 +33,10 @@ func TestValidateCIDR(t *testing.T) { }, } err := config.Validate() - if tc.expectErr { + if tc.expectPodErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } }) t.Run("Service", func(t *testing.T) { @@ -45,10 +48,10 @@ func TestValidateCIDR(t *testing.T) { }, } err := config.Validate() - if tc.expectErr { + if tc.expectSvcErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } }) }) @@ -123,7 +126,7 @@ func TestValidateExternalServers(t *testing.T) { if tc.expectErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } }) } diff --git a/src/k8s/pkg/k8sd/types/refresh.go b/src/k8s/pkg/k8sd/types/refresh.go index 365c9cb30..286826230 100644 --- a/src/k8s/pkg/k8sd/types/refresh.go +++ b/src/k8s/pkg/k8sd/types/refresh.go @@ -17,7 +17,7 @@ type RefreshOpts struct { } func RefreshOptsFromAPI(req apiv1.SnapRefreshRequest) (RefreshOpts, error) { - var optsMap = map[string]string{ + optsMap := map[string]string{ "localPath": req.LocalPath, "channel": req.Channel, "revision": req.Revision, diff --git a/src/k8s/pkg/k8sd/types/worker_test.go b/src/k8s/pkg/k8sd/types/worker_test.go index 586ffa032..cc5692185 100644 --- a/src/k8s/pkg/k8sd/types/worker_test.go +++ b/src/k8s/pkg/k8sd/types/worker_test.go @@ -17,11 +17,11 @@ func TestWorkerTokenEncode(t *testing.T) { g := NewWithT(t) s, err := token.Encode() - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(s).ToNot(BeEmpty()) decoded := &types.InternalWorkerNodeToken{} err = decoded.Decode(s) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(decoded).To(Equal(token)) } diff --git a/src/k8s/pkg/proxy/config.go b/src/k8s/pkg/proxy/config.go index 08c2d6a86..450f2a689 100644 --- a/src/k8s/pkg/proxy/config.go +++ b/src/k8s/pkg/proxy/config.go @@ -5,6 +5,8 @@ import ( "fmt" "os" "sort" + + "github.com/canonical/k8s/pkg/utils" ) // Configuration is the format of the apiserver proxy endpoints config file. @@ -33,7 +35,7 @@ func WriteEndpointsConfig(endpoints []string, file string) error { return fmt.Errorf("failed to marshal configuration: %w", err) } - if err := os.WriteFile(file, b, 0600); err != nil { + if err := utils.WriteFile(file, b, 0o600); err != nil { return fmt.Errorf("failed to write configuration file %s: %w", file, err) } return nil diff --git a/src/k8s/pkg/proxy/userspace.go b/src/k8s/pkg/proxy/userspace.go index 4fa143592..4082d7947 100644 --- a/src/k8s/pkg/proxy/userspace.go +++ b/src/k8s/pkg/proxy/userspace.go @@ -27,6 +27,8 @@ import ( "net" "sync" "time" + + "github.com/canonical/k8s/pkg/utils" ) type remote struct { @@ -78,7 +80,8 @@ func (tp *tcpproxy) Run() error { tp.MonitorInterval = 5 * time.Minute } for _, srv := range tp.Endpoints { - addr := fmt.Sprintf("%s:%d", srv.Target, srv.Port) + ip := net.ParseIP(srv.Target) + addr := fmt.Sprintf("%s:%d", utils.ToIPString(ip), srv.Port) tp.remotes = append(tp.remotes, &remote{srv: srv, addr: addr}) } diff --git a/src/k8s/pkg/snap/interface.go b/src/k8s/pkg/snap/interface.go index 03094679e..f2ea533df 100644 --- a/src/k8s/pkg/snap/interface.go +++ b/src/k8s/pkg/snap/interface.go @@ -39,12 +39,13 @@ type Snap interface { EtcdPKIDir() string // /etc/kubernetes/pki/etcd KubeletRootDir() string // /var/lib/kubelet - ContainerdConfigDir() string // /var/snap/k8s/common/etc/containerd - ContainerdExtraConfigDir() string // /var/snap/k8s/common/etc/containerd/conf.d - ContainerdRegistryConfigDir() string // /var/snap/k8s/common/etc/containerd/hosts.d - ContainerdRootDir() string // /var/snap/k8s/common/var/lib/containerd - ContainerdSocketDir() string // /var/snap/k8s/common/run - ContainerdStateDir() string // /run/containerd + ContainerdConfigDir() string // classic confinement: /etc/containerd, strict confinement: /var/snap/k8s/common/etc/containerd + ContainerdExtraConfigDir() string // classic confinement: /etc/containerd/conf.d, strict confinement: /var/snap/k8s/common/etc/containerd/conf.d + ContainerdRegistryConfigDir() string // classic confinement: /etc/containerd/hosts.d, strict confinement: /var/snap/k8s/common/etc/containerd/hosts.d + ContainerdRootDir() string // classic confinement: /var/lib/containerd, strict confinement: /var/snap/k8s/common/var/lib/containerd + ContainerdSocketDir() string // classic confinement: /run/containerd, strict confinement: /var/snap/k8s/common/run/containerd + ContainerdSocketPath() string // classic confinement: /run/containerd/containerd.sock, strict confinement: /var/snap/k8s/common/run/containerd/containerd.sock + ContainerdStateDir() string // classic confinement: /run/containerd, strict confinement: /var/snap/k8s/common/run/containerd K8sdStateDir() string // /var/snap/k8s/common/var/lib/k8sd/state K8sDqliteStateDir() string // /var/snap/k8s/common/var/lib/k8s-dqlite diff --git a/src/k8s/pkg/snap/mock/mock.go b/src/k8s/pkg/snap/mock/mock.go index fc9261720..21846253d 100644 --- a/src/k8s/pkg/snap/mock/mock.go +++ b/src/k8s/pkg/snap/mock/mock.go @@ -32,6 +32,7 @@ type Mock struct { ContainerdRegistryConfigDir string ContainerdRootDir string ContainerdSocketDir string + ContainerdSocketPath string ContainerdStateDir string K8sdStateDir string K8sDqliteStateDir string @@ -78,6 +79,7 @@ func (s *Snap) StartService(ctx context.Context, name string) error { } return s.StartServiceErr } + func (s *Snap) StopService(ctx context.Context, name string) error { if len(s.StopServiceCalledWith) == 0 { s.StopServiceCalledWith = []string{name} @@ -86,6 +88,7 @@ func (s *Snap) StopService(ctx context.Context, name string) error { } return s.StopServiceErr } + func (s *Snap) RestartService(ctx context.Context, name string) error { if len(s.RestartServiceCalledWith) == 0 { s.RestartServiceCalledWith = []string{name} @@ -94,6 +97,7 @@ func (s *Snap) RestartService(ctx context.Context, name string) error { } return s.RestartServiceErr } + func (s *Snap) Refresh(ctx context.Context, opts types.RefreshOpts) (string, error) { if len(s.RefreshCalledWith) == 0 { s.RefreshCalledWith = []types.RefreshOpts{opts} @@ -102,107 +106,145 @@ func (s *Snap) Refresh(ctx context.Context, opts types.RefreshOpts) (string, err } return "", s.RefreshErr } + func (s *Snap) RefreshStatus(ctx context.Context, changeID string) (*types.RefreshStatus, error) { return nil, nil } + func (s *Snap) Strict() bool { return s.Mock.Strict } + func (s *Snap) OnLXD(context.Context) (bool, error) { return s.Mock.OnLXD, s.Mock.OnLXDErr } + func (s *Snap) UID() int { return s.Mock.UID } + func (s *Snap) GID() int { return s.Mock.GID } + func (s *Snap) Hostname() string { return s.Mock.Hostname } + func (s *Snap) ContainerdConfigDir() string { return s.Mock.ContainerdConfigDir } + func (s *Snap) ContainerdRootDir() string { return s.Mock.ContainerdRootDir } + func (s *Snap) ContainerdStateDir() string { return s.Mock.ContainerdStateDir } + func (s *Snap) ContainerdSocketDir() string { return s.Mock.ContainerdSocketDir } + +func (s *Snap) ContainerdSocketPath() string { + return s.Mock.ContainerdSocketPath +} + func (s *Snap) ContainerdExtraConfigDir() string { return s.Mock.ContainerdExtraConfigDir } + func (s *Snap) ContainerdRegistryConfigDir() string { return s.Mock.ContainerdRegistryConfigDir } + func (s *Snap) KubernetesConfigDir() string { return s.Mock.KubernetesConfigDir } + func (s *Snap) KubernetesPKIDir() string { return s.Mock.KubernetesPKIDir } + func (s *Snap) EtcdPKIDir() string { return s.Mock.EtcdPKIDir } + func (s *Snap) KubeletRootDir() string { return s.Mock.KubeletRootDir } + func (s *Snap) CNIConfDir() string { return s.Mock.CNIConfDir } + func (s *Snap) CNIBinDir() string { return s.Mock.CNIBinDir } + func (s *Snap) CNIPluginsBinary() string { return s.Mock.CNIPluginsBinary } + func (s *Snap) CNIPlugins() []string { return s.Mock.CNIPlugins } + func (s *Snap) K8sdStateDir() string { return s.Mock.K8sdStateDir } + func (s *Snap) K8sDqliteStateDir() string { return s.Mock.K8sDqliteStateDir } + func (s *Snap) ServiceArgumentsDir() string { return s.Mock.ServiceArgumentsDir } + func (s *Snap) ServiceExtraConfigDir() string { return s.Mock.ServiceExtraConfigDir } + func (s *Snap) LockFilesDir() string { return s.Mock.LockFilesDir } + func (s *Snap) NodeTokenFile() string { return s.Mock.NodeTokenFile } + func (s *Snap) KubernetesClient(namespace string) (*kubernetes.Client, error) { return s.Mock.KubernetesClient, nil } + func (s *Snap) KubernetesNodeClient(namespace string) (*kubernetes.Client, error) { return s.Mock.KubernetesNodeClient, nil } + func (s *Snap) HelmClient() helm.Client { return s.Mock.HelmClient } + func (s *Snap) K8sDqliteClient(context.Context) (*dqlite.Client, error) { return s.Mock.K8sDqliteClient, nil } + func (s *Snap) K8sdClient(address string) (k8sd.Client, error) { return s.Mock.K8sdClient, nil } + func (s *Snap) SnapctlGet(ctx context.Context, args ...string) ([]byte, error) { s.SnapctlGetCalledWith = append(s.SnapctlGetCalledWith, args) return s.Mock.SnapctlGet[strings.Join(args, " ")], s.SnapctlGetErr } + func (s *Snap) SnapctlSet(ctx context.Context, args ...string) error { - s.SnapctlSetCalledWith = append(s.SnapctlGetCalledWith, args) + s.SnapctlSetCalledWith = append(s.SnapctlSetCalledWith, args) return s.SnapctlSetErr } + func (s *Snap) PreInitChecks(ctx context.Context, config types.ClusterConfig) error { s.PreInitChecksCalledWith = append(s.PreInitChecksCalledWith, config) return s.PreInitChecksErr diff --git a/src/k8s/pkg/snap/pebble_test.go b/src/k8s/pkg/snap/pebble_test.go index f85491140..e90b21e39 100644 --- a/src/k8s/pkg/snap/pebble_test.go +++ b/src/k8s/pkg/snap/pebble_test.go @@ -7,7 +7,6 @@ import ( "github.com/canonical/k8s/pkg/snap" "github.com/canonical/k8s/pkg/snap/mock" - . "github.com/onsi/gomega" ) @@ -22,7 +21,7 @@ func TestPebble(t *testing.T) { }) err := snap.StartService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("testdir/bin/pebble start test-service")) t.Run("Fail", func(t *testing.T) { @@ -30,7 +29,7 @@ func TestPebble(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "test-service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) @@ -43,7 +42,7 @@ func TestPebble(t *testing.T) { RunCommand: mockRunner.Run, }) err := snap.StopService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("testdir/bin/pebble stop test-service")) t.Run("Fail", func(t *testing.T) { @@ -51,7 +50,7 @@ func TestPebble(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "test-service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) @@ -65,7 +64,7 @@ func TestPebble(t *testing.T) { }) err := snap.RestartService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("testdir/bin/pebble restart test-service")) t.Run("Fail", func(t *testing.T) { @@ -73,7 +72,7 @@ func TestPebble(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) } diff --git a/src/k8s/pkg/snap/snap.go b/src/k8s/pkg/snap/snap.go index 032f6c988..057d30cde 100644 --- a/src/k8s/pkg/snap/snap.go +++ b/src/k8s/pkg/snap/snap.go @@ -3,6 +3,7 @@ package snap import ( "bytes" "context" + "errors" "fmt" "os" "os/exec" @@ -23,18 +24,20 @@ import ( ) type SnapOpts struct { - SnapInstanceName string - SnapDir string - SnapCommonDir string - RunCommand func(ctx context.Context, command []string, opts ...func(c *exec.Cmd)) error + SnapInstanceName string + SnapDir string + SnapCommonDir string + RunCommand func(ctx context.Context, command []string, opts ...func(c *exec.Cmd)) error + ContainerdBaseDir string } // snap implements the Snap interface. type snap struct { - snapDir string - snapCommonDir string - snapInstanceName string - runCommand func(ctx context.Context, command []string, opts ...func(c *exec.Cmd)) error + snapDir string + snapCommonDir string + snapInstanceName string + runCommand func(ctx context.Context, command []string, opts ...func(c *exec.Cmd)) error + containerdBaseDir string } // NewSnap creates a new interface with the K8s snap. @@ -51,6 +54,15 @@ func NewSnap(opts SnapOpts) *snap { runCommand: runCommand, } + containerdBaseDir := opts.ContainerdBaseDir + if containerdBaseDir == "" { + containerdBaseDir = "/" + if s.Strict() { + containerdBaseDir = opts.SnapCommonDir + } + } + s.containerdBaseDir = containerdBaseDir + return s } @@ -161,19 +173,23 @@ func (s *snap) Hostname() string { } func (s *snap) ContainerdConfigDir() string { - return filepath.Join(s.snapCommonDir, "etc", "containerd") + return filepath.Join(s.containerdBaseDir, "etc", "containerd") } func (s *snap) ContainerdRootDir() string { - return filepath.Join(s.snapCommonDir, "var", "lib", "containerd") + return filepath.Join(s.containerdBaseDir, "var", "lib", "containerd") } func (s *snap) ContainerdSocketDir() string { - return filepath.Join(s.snapCommonDir, "run") + return filepath.Join(s.containerdBaseDir, "run", "containerd") +} + +func (s *snap) ContainerdSocketPath() string { + return filepath.Join(s.containerdBaseDir, "run", "containerd", "containerd.sock") } func (s *snap) ContainerdStateDir() string { - return "/run/containerd" + return filepath.Join(s.containerdBaseDir, "run", "containerd") } func (s *snap) CNIConfDir() string { @@ -250,11 +266,11 @@ func (s *snap) NodeTokenFile() string { } func (s *snap) ContainerdExtraConfigDir() string { - return filepath.Join(s.snapCommonDir, "etc", "containerd", "conf.d") + return filepath.Join(s.containerdBaseDir, "etc", "containerd", "conf.d") } func (s *snap) ContainerdRegistryConfigDir() string { - return filepath.Join(s.snapCommonDir, "etc", "containerd", "hosts.d") + return filepath.Join(s.containerdBaseDir, "etc", "containerd", "hosts.d") } func (s *snap) restClientGetter(path string, namespace string) genericclioptions.RESTClientGetter { @@ -323,6 +339,18 @@ func (s *snap) PreInitChecks(ctx context.Context, config types.ClusterConfig) er } } + // check if the containerd path already exists, signaling the fact that another containerd instance + // is already running on this node, which will conflict with the snap. + // Checks the directories instead of the containerd.sock file, since this file does not exist if + // containerd is not running/stopped. + if _, err := os.Stat(s.ContainerdSocketDir()); err == nil { + return fmt.Errorf("The path '%s' required for the containerd socket already exists. "+ + "This may mean that another service is already using that path, and it conflicts with the k8s snap. "+ + "Please make sure that there is no other service installed that uses the same path, and remove the existing directory.", s.ContainerdSocketDir()) + } else if !errors.Is(err, os.ErrNotExist) { + return fmt.Errorf("Encountered an error while checking '%s': %w", s.ContainerdSocketDir(), err) + } + return nil } diff --git a/src/k8s/pkg/snap/snap_test.go b/src/k8s/pkg/snap/snap_test.go index f694a0ef0..e99ff0384 100644 --- a/src/k8s/pkg/snap/snap_test.go +++ b/src/k8s/pkg/snap/snap_test.go @@ -3,15 +3,64 @@ package snap_test import ( "context" "fmt" + "os" + "path/filepath" "testing" + "github.com/canonical/k8s/pkg/k8sd/types" "github.com/canonical/k8s/pkg/snap" "github.com/canonical/k8s/pkg/snap/mock" - . "github.com/onsi/gomega" ) func TestSnap(t *testing.T) { + t.Run("NewSnap", func(t *testing.T) { + t.Run("containerd path with opt", func(t *testing.T) { + g := NewWithT(t) + mockRunner := &mock.Runner{} + snap := snap.NewSnap(snap.SnapOpts{ + RunCommand: mockRunner.Run, + ContainerdBaseDir: "/foo/lish", + }) + + g.Expect(snap.ContainerdSocketPath()).To(Equal(filepath.Join("/foo/lish", "run", "containerd", "containerd.sock"))) + }) + + t.Run("containerd path classic", func(t *testing.T) { + g := NewWithT(t) + mockRunner := &mock.Runner{} + snap := snap.NewSnap(snap.SnapOpts{ + RunCommand: mockRunner.Run, + }) + + g.Expect(snap.ContainerdSocketPath()).To(Equal(filepath.Join("/", "run", "containerd", "containerd.sock"))) + }) + + t.Run("containerd path strict", func(t *testing.T) { + g := NewWithT(t) + // We're checking if the snap is strict in NewSnap, which checks the snap.yaml file. + tmpDir, err := os.MkdirTemp("", "test-snap-k8s") + g.Expect(err).To(Not(HaveOccurred())) + defer os.RemoveAll(tmpDir) + + err = os.MkdirAll(filepath.Join(tmpDir, "meta"), os.ModeDir) + g.Expect(err).To(Not(HaveOccurred())) + + snapData := []byte("confinement: strict\n") + err = os.WriteFile(filepath.Join(tmpDir, "meta", "snap.yaml"), snapData, 0o644) + g.Expect(err).To(Not(HaveOccurred())) + + mockRunner := &mock.Runner{} + snap := snap.NewSnap(snap.SnapOpts{ + SnapDir: tmpDir, + SnapCommonDir: tmpDir, + RunCommand: mockRunner.Run, + }) + + g.Expect(snap.ContainerdSocketPath()).To(Equal(filepath.Join(tmpDir, "run", "containerd", "containerd.sock"))) + }) + }) + t.Run("Start", func(t *testing.T) { g := NewWithT(t) mockRunner := &mock.Runner{} @@ -22,7 +71,7 @@ func TestSnap(t *testing.T) { }) err := snap.StartService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("snapctl start --enable k8s.test-service")) t.Run("Fail", func(t *testing.T) { @@ -30,7 +79,7 @@ func TestSnap(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "test-service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) @@ -43,7 +92,7 @@ func TestSnap(t *testing.T) { RunCommand: mockRunner.Run, }) err := snap.StopService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("snapctl stop --disable k8s.test-service")) t.Run("Fail", func(t *testing.T) { @@ -51,7 +100,7 @@ func TestSnap(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "test-service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) @@ -65,7 +114,7 @@ func TestSnap(t *testing.T) { }) err := snap.RestartService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("snapctl restart k8s.test-service")) t.Run("Fail", func(t *testing.T) { @@ -73,7 +122,54 @@ func TestSnap(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) + }) + }) + + t.Run("PreInitChecks", func(t *testing.T) { + g := NewWithT(t) + // Replace the ContainerdSocketDir to avoid checking against a real containerd.sock that may be running. + containerdDir, err := os.MkdirTemp("", "test-containerd") + g.Expect(err).To(Not(HaveOccurred())) + defer os.RemoveAll(containerdDir) + + mockRunner := &mock.Runner{} + snap := snap.NewSnap(snap.SnapOpts{ + SnapDir: "testdir", + SnapCommonDir: "testdir", + RunCommand: mockRunner.Run, + ContainerdBaseDir: containerdDir, + }) + conf := types.ClusterConfig{} + + err = snap.PreInitChecks(context.Background(), conf) + g.Expect(err).To(Not(HaveOccurred())) + expectedCalls := []string{} + for _, binary := range []string{"kube-apiserver", "kube-controller-manager", "kube-scheduler", "kube-proxy", "kubelet"} { + expectedCalls = append(expectedCalls, fmt.Sprintf("testdir/bin/%s --version", binary)) + } + g.Expect(mockRunner.CalledWithCommand).To(ConsistOf(expectedCalls)) + + t.Run("Fail socket exists", func(t *testing.T) { + g := NewWithT(t) + // Create the containerd.sock file, which should cause the check to fail. + err := os.MkdirAll(snap.ContainerdSocketDir(), os.ModeDir) + g.Expect(err).To(Not(HaveOccurred())) + f, err := os.Create(snap.ContainerdSocketPath()) + g.Expect(err).To(Not(HaveOccurred())) + f.Close() + defer os.Remove(f.Name()) + + err = snap.PreInitChecks(context.Background(), conf) + g.Expect(err).To(HaveOccurred()) + }) + + t.Run("Fail run command", func(t *testing.T) { + g := NewWithT(t) + mockRunner.Err = fmt.Errorf("some error") + + err := snap.PreInitChecks(context.Background(), conf) + g.Expect(err).To(HaveOccurred()) }) }) } diff --git a/src/k8s/pkg/snap/util/arguments.go b/src/k8s/pkg/snap/util/arguments.go index 63a5ae1c8..02aad87fb 100644 --- a/src/k8s/pkg/snap/util/arguments.go +++ b/src/k8s/pkg/snap/util/arguments.go @@ -103,8 +103,8 @@ func UpdateServiceArguments(snap snap.Snap, serviceName string, updateMap map[st // sort arguments so that output is consistent sort.Strings(newArguments) - if err := os.WriteFile(argumentsFile, []byte(strings.Join(newArguments, "\n")+"\n"), 0600); err != nil { - return false, fmt.Errorf("failed to write arguments for service %s: %q", serviceName, err) + if err := utils.WriteFile(argumentsFile, []byte(strings.Join(newArguments, "\n")+"\n"), 0o600); err != nil { + return false, fmt.Errorf("failed to write arguments for service %s: %w", serviceName, err) } return changed, nil } diff --git a/src/k8s/pkg/snap/util/arguments_test.go b/src/k8s/pkg/snap/util/arguments_test.go index f2550dd09..0d23ccd6a 100644 --- a/src/k8s/pkg/snap/util/arguments_test.go +++ b/src/k8s/pkg/snap/util/arguments_test.go @@ -2,12 +2,12 @@ package snaputil_test import ( "fmt" - "os" "path/filepath" "testing" "github.com/canonical/k8s/pkg/snap/mock" snaputil "github.com/canonical/k8s/pkg/snap/util" + "github.com/canonical/k8s/pkg/utils" . "github.com/onsi/gomega" ) @@ -32,7 +32,7 @@ func TestGetServiceArgument(t *testing.T) { --key=value-of-service-two `, } { - g.Expect(os.WriteFile(filepath.Join(dir, svc), []byte(args), 0600)).To(BeNil()) + g.Expect(utils.WriteFile(filepath.Join(dir, svc), []byte(args), 0o600)).To(Succeed()) } for _, tc := range []struct { @@ -56,9 +56,9 @@ func TestGetServiceArgument(t *testing.T) { value, err := snaputil.GetServiceArgument(s, tc.service, tc.key) if tc.expectErr { - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(value).To(Equal(tc.expectValue)) } }) @@ -75,14 +75,14 @@ func TestUpdateServiceArguments(t *testing.T) { } _, err := snaputil.GetServiceArgument(s, "service", "--key") - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) changed, err := snaputil.UpdateServiceArguments(s, "service", map[string]string{"--key": "value"}, nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(changed).To(BeTrue()) value, err := snaputil.GetServiceArgument(s, "service", "--key") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(value).To(Equal("value")) }) @@ -183,11 +183,11 @@ func TestUpdateServiceArguments(t *testing.T) { }, } changed, err := snaputil.UpdateServiceArguments(s, "service", initialArguments, nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(changed).To(BeTrue()) changed, err = snaputil.UpdateServiceArguments(s, "service", tc.update, tc.delete) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(changed).To(Equal(tc.expectedChange)) for key, expectedValue := range tc.expectedValues { @@ -197,7 +197,7 @@ func TestUpdateServiceArguments(t *testing.T) { t.Run("Reapply", func(t *testing.T) { g := NewWithT(t) changed, err := snaputil.UpdateServiceArguments(s, "service", tc.update, tc.delete) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(changed).To(BeFalse()) }) }) diff --git a/src/k8s/pkg/snap/util/node.go b/src/k8s/pkg/snap/util/node.go index 09dfe8289..d121970bc 100644 --- a/src/k8s/pkg/snap/util/node.go +++ b/src/k8s/pkg/snap/util/node.go @@ -25,7 +25,7 @@ func MarkAsWorkerNode(snap snap.Snap, mark bool) error { if err := os.Chown(fname, snap.UID(), snap.GID()); err != nil { return fmt.Errorf("failed to chown %s: %w", fname, err) } - if err := os.Chmod(fname, 0600); err != nil { + if err := os.Chmod(fname, 0o600); err != nil { return fmt.Errorf("failed to chmod %s: %w", fname, err) } } else { diff --git a/src/k8s/pkg/snap/util/node_test.go b/src/k8s/pkg/snap/util/node_test.go index 88b21fcb4..5529609e9 100644 --- a/src/k8s/pkg/snap/util/node_test.go +++ b/src/k8s/pkg/snap/util/node_test.go @@ -26,7 +26,7 @@ func TestIsWorker(t *testing.T) { lock.Close() exists, err := snaputil.IsWorker(mock) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(exists).To(BeTrue()) }) @@ -34,7 +34,7 @@ func TestIsWorker(t *testing.T) { mock.Mock.LockFilesDir = "/non-existent" g := NewWithT(t) exists, err := snaputil.IsWorker(mock) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(exists).To(BeFalse()) }) } @@ -52,24 +52,24 @@ func TestMarkAsWorkerNode(t *testing.T) { t.Run("MarkWorker", func(t *testing.T) { g := NewWithT(t) err := snaputil.MarkAsWorkerNode(mock, true) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) workerFile := filepath.Join(mock.LockFilesDir(), "worker") g.Expect(workerFile).To(BeAnExistingFile()) // Clean up err = os.Remove(workerFile) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("UnmarkWorker", func(t *testing.T) { g := NewWithT(t) workerFile := filepath.Join(mock.LockFilesDir(), "worker") _, err := os.Create(workerFile) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = snaputil.MarkAsWorkerNode(mock, false) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(workerFile).NotTo(BeAnExistingFile()) }) diff --git a/src/k8s/pkg/utils/certificate.go b/src/k8s/pkg/utils/certificate.go index 68067ea55..b817083d8 100644 --- a/src/k8s/pkg/utils/certificate.go +++ b/src/k8s/pkg/utils/certificate.go @@ -10,7 +10,7 @@ import ( ) // SplitIPAndDNSSANs splits a list of SANs into IP and DNS SANs -// Returns a list of IP addresses and a list of DNS names +// Returns a list of IP addresses and a list of DNS names. func SplitIPAndDNSSANs(extraSANs []string) ([]net.IP, []string) { var ipSANs []net.IP var dnsSANs []string @@ -57,7 +57,7 @@ func TLSClientConfigWithTrustedCertificate(remoteCert *x509.Certificate, rootCAs // GetRemoteCertificate retrieves the remote certificate from a given address // The address should be in the format of "hostname:port" -// Returns the remote certificate or an error +// Returns the remote certificate or an error. func GetRemoteCertificate(address string) (*x509.Certificate, error) { // validate address _, _, err := net.SplitHostPort(address) @@ -85,6 +85,7 @@ func GetRemoteCertificate(address string) (*x509.Certificate, error) { if err != nil { return nil, err } + defer resp.Body.Close() // Retrieve the certificate if resp.TLS == nil || len(resp.TLS.PeerCertificates) == 0 { @@ -94,7 +95,7 @@ func GetRemoteCertificate(address string) (*x509.Certificate, error) { return resp.TLS.PeerCertificates[0], nil } -// CertFingerprint returns the SHA256 fingerprint of a certificate +// CertFingerprint returns the SHA256 fingerprint of a certificate. func CertFingerprint(cert *x509.Certificate) string { return fmt.Sprintf("%x", sha256.Sum256(cert.Raw)) } diff --git a/src/k8s/pkg/utils/certificate_test.go b/src/k8s/pkg/utils/certificate_test.go index bd37b4797..c40c3a533 100644 --- a/src/k8s/pkg/utils/certificate_test.go +++ b/src/k8s/pkg/utils/certificate_test.go @@ -9,7 +9,6 @@ import ( "testing" "github.com/canonical/k8s/pkg/utils" - . "github.com/onsi/gomega" ) @@ -41,18 +40,18 @@ func TestTLSClientConfigWithTrustedCertificate(t *testing.T) { tlsConfig, err := utils.TLSClientConfigWithTrustedCertificate(remoteCert, rootCAs) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(tlsConfig.ServerName).To(Equal("bubblegum.com")) g.Expect(tlsConfig.RootCAs.Subjects()).To(ContainElement(remoteCert.RawSubject)) // Test with invalid remote certificate tlsConfig, err = utils.TLSClientConfigWithTrustedCertificate(nil, rootCAs) - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) g.Expect(tlsConfig).To(BeNil()) // Test with nil root CAs _, err = utils.TLSClientConfigWithTrustedCertificate(remoteCert, nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } func TestCertFingerprint(t *testing.T) { diff --git a/src/k8s/pkg/utils/cidr.go b/src/k8s/pkg/utils/cidr.go index 124f75aea..015eea3af 100644 --- a/src/k8s/pkg/utils/cidr.go +++ b/src/k8s/pkg/utils/cidr.go @@ -4,6 +4,7 @@ import ( "fmt" "math/big" "net" + "regexp" "strconv" "strings" @@ -11,6 +12,8 @@ import ( ) // findMatchingNodeAddress returns the IP address of a network interface that belongs to the given CIDR. +// It prefers the address with the fewest subnet bits. +// Loopback addresses are ignored. func findMatchingNodeAddress(cidr *net.IPNet) (net.IP, error) { addrs, err := net.InterfaceAddrs() if err != nil { @@ -25,7 +28,7 @@ func findMatchingNodeAddress(cidr *net.IPNet) (net.IP, error) { if !ok { continue } - if cidr.Contains(ipNet.IP) { + if cidr.Contains(ipNet.IP) && !ipNet.IP.IsLoopback() { _, subnetBits := cidr.Mask.Size() if selectedSubnetBits == -1 || subnetBits < selectedSubnetBits { // Prefer the address with the fewest subnet bits @@ -75,6 +78,21 @@ func GetKubernetesServiceIPsFromServiceCIDRs(serviceCIDR string) ([]net.IP, erro // ParseAddressString parses an address string and returns a canonical network address. func ParseAddressString(address string, port int64) (string, error) { + // Matches a CIDR block at the beginning of the address string + // e.g. [2001:db8::/32]:8080 + re := regexp.MustCompile(`^\[?([a-z0-9:.]+\/[0-9]+)\]?`) + cidrMatches := re.FindStringSubmatch(address) + if len(cidrMatches) != 0 { + // Resolve CIDR + if _, ipNet, err := net.ParseCIDR(cidrMatches[1]); err == nil { + matchingIP, err := findMatchingNodeAddress(ipNet) + if err != nil { + return "", fmt.Errorf("failed to find a matching node address for %q: %w", address, err) + } + address = strings.ReplaceAll(address, cidrMatches[1], matchingIP.String()) + } + } + host, hostPort, err := net.SplitHostPort(address) if err == nil { address = host @@ -90,20 +108,67 @@ func ParseAddressString(address string, port int64) (string, error) { if address == "" { address = util.NetworkInterfaceAddress() - } else if _, ipNet, err := net.ParseCIDR(address); err == nil { - matchingIP, err := findMatchingNodeAddress(ipNet) + } + + return net.JoinHostPort(address, fmt.Sprintf("%d", port)), nil +} + +// GetDefaultAddress returns the default IPv4 and IPv6 addresses of the host. +func GetDefaultAddress() (ipv4, ipv6 string, err error) { + // Get a list of network interfaces. + interfaces, err := net.Interfaces() + if err != nil { + return "", "", fmt.Errorf("failed to get network interfaces: %w", err) + } + + // Loop through each network interface + for _, iface := range interfaces { + // Skip interfaces that are down or loopback interfaces + if iface.Flags&net.FlagUp == 0 || iface.Flags&net.FlagLoopback != 0 { + continue + } + + // Get a list of addresses for the current interface. + addrs, err := iface.Addrs() if err != nil { - return "", fmt.Errorf("failed to find a matching node address for %q: %w", address, err) + return "", "", fmt.Errorf("failed to get addresses for interface %q: %w", iface.Name, err) + } + + // Loop through each address + for _, addr := range addrs { + // Parse the address to get the IP + var ip net.IP + switch v := addr.(type) { + case *net.IPNet: + ip = v.IP + case *net.IPAddr: + ip = v.IP + } + + // Check if it's an IPv4 or IPv6 address and not a loopback + if ip.IsLoopback() { + continue + } + + if ip.To4() != nil && ipv4 == "" { + ipv4 = ip.String() + } else if ip.To4() == nil && ipv6 == "" { + ipv6 = ip.String() + } + + // Break early if both IPv4 and IPv6 addresses are found + if ipv4 != "" && ipv6 != "" { + return ipv4, ipv6, nil + } } - address = matchingIP.String() } - return util.CanonicalNetworkAddress(address, port), nil + return ipv4, ipv6, nil } -// ParseCIDRs parses the given CIDR string and returns the respective IPv4 and IPv6 CIDRs. -func ParseCIDRs(CIDRstring string) (string, string, error) { - clusterCIDRs := strings.Split(CIDRstring, ",") +// SplitCIDRStrings parses the given CIDR string and returns the respective IPv4 and IPv6 CIDRs. +func SplitCIDRStrings(cidrString string) (string, string, error) { + clusterCIDRs := strings.Split(cidrString, ",") if v := len(clusterCIDRs); v != 1 && v != 2 { return "", "", fmt.Errorf("invalid CIDR list: %v", clusterCIDRs) } @@ -125,3 +190,42 @@ func ParseCIDRs(CIDRstring string) (string, string, error) { } return ipv4CIDR, ipv6CIDR, nil } + +// IsIPv4 returns true if the address is a valid IPv4 address, false otherwise. +// The address may contain a port number. +func IsIPv4(address string) bool { + ip := strings.Split(address, ":")[0] + parsedIP := net.ParseIP(ip) + return parsedIP != nil && parsedIP.To4() != nil +} + +// ToIPString returns the string representation of an IP address. +// If the IP address is an IPv6 address, it is enclosed in square brackets. +func ToIPString(ip net.IP) string { + if ip.To4() != nil { + return ip.String() + } + return "[" + ip.String() + "]" +} + +// CIDRsOverlap checks if two given CIDR blocks overlap. +// It takes two strings representing the CIDR blocks as input and returns a boolean indicating +// whether they overlap and an error if any of the CIDR blocks are invalid. +func CIDRsOverlap(cidr1, cidr2 string) (bool, error) { + _, ipNet1, err1 := net.ParseCIDR(cidr1) + _, ipNet2, err2 := net.ParseCIDR(cidr2) + + if err1 != nil { + return false, fmt.Errorf("couldn't parse CIDR block %q: %w", cidr1, err1) + } + + if err2 != nil { + return false, fmt.Errorf("couldn't parse CIDR block %q: %w", cidr2, err2) + } + + if ipNet1.Contains(ipNet2.IP) || ipNet2.Contains(ipNet1.IP) { + return true, nil + } + + return false, nil +} diff --git a/src/k8s/pkg/utils/cidr_test.go b/src/k8s/pkg/utils/cidr_test.go index 3dfbe3941..8a9658995 100644 --- a/src/k8s/pkg/utils/cidr_test.go +++ b/src/k8s/pkg/utils/cidr_test.go @@ -1,12 +1,12 @@ package utils_test import ( + "errors" "fmt" "net" "testing" "github.com/canonical/k8s/pkg/utils" - "github.com/canonical/lxd/lxd/util" . "github.com/onsi/gomega" ) @@ -24,7 +24,7 @@ func TestGetFirstIP(t *testing.T) { t.Run(tc.cidr, func(t *testing.T) { g := NewWithT(t) ip, err := utils.GetFirstIP(tc.cidr) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ip.String()).To(Equal(tc.ip)) }) } @@ -49,7 +49,7 @@ func TestGetKubernetesServiceIPsFromServiceCIDRs(t *testing.T) { ips[idx] = v.String() } - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ips).To(Equal(tc.ips)) }) } @@ -66,7 +66,7 @@ func TestGetKubernetesServiceIPsFromServiceCIDRs(t *testing.T) { g := NewWithT(t) _, err := utils.GetKubernetesServiceIPsFromServiceCIDRs(tc.cidr) - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) }) } }) @@ -76,12 +76,20 @@ func TestParseAddressString(t *testing.T) { g := NewWithT(t) // Seed the default address - defaultAddress := util.NetworkInterfaceAddress() - ip := net.ParseIP(defaultAddress) - subnetMask := net.CIDRMask(24, 32) - networkAddress := ip.Mask(subnetMask) + defaultIPv4, defaultIPv6, err := utils.GetDefaultAddress() + g.Expect(err).ToNot(HaveOccurred()) + + ip4 := net.ParseIP(defaultIPv4) + subnetMask4 := net.CIDRMask(24, 32) + networkAddress4 := ip4.Mask(subnetMask4) // Infer the CIDR notation - networkAddressCIDR := fmt.Sprintf("%s/24", networkAddress.String()) + networkAddressCIDR := fmt.Sprintf("%s/24", networkAddress4.String()) + + ip6 := net.ParseIP(defaultIPv6) + subnetMask6 := net.CIDRMask(64, 128) + networkAddress6 := ip6.Mask(subnetMask6) + // Infer the CIDR notation + networkAddressCIDR6 := fmt.Sprintf("%s/64", networkAddress6.String()) for _, tc := range []struct { name string @@ -90,18 +98,28 @@ func TestParseAddressString(t *testing.T) { want string wantErr bool }{ - {name: "EmptyAddress", address: "", port: 8080, want: fmt.Sprintf("%s:8080", defaultAddress), wantErr: false}, - {name: "CIDR", address: networkAddressCIDR, port: 8080, want: fmt.Sprintf("%s:8080", defaultAddress), wantErr: false}, - {name: "CIDRAndPort", address: fmt.Sprintf("%s:9090", networkAddressCIDR), port: 8080, want: fmt.Sprintf("%s:9090", defaultAddress), wantErr: false}, + {name: "CIDR6LinkLocalPort", address: "[::/0%eth0]:9090", port: 8080, want: fmt.Sprintf("[%s%%eth0]:9090", defaultIPv6), wantErr: false}, + {name: "CIDRAndPort", address: fmt.Sprintf("%s:9090", networkAddressCIDR), port: 8080, want: fmt.Sprintf("%s:9090", defaultIPv4), wantErr: false}, + {name: "CIDR", address: networkAddressCIDR, port: 8080, want: fmt.Sprintf("%s:8080", defaultIPv4), wantErr: false}, + {name: "EmptyAddress", address: "", port: 8080, want: fmt.Sprintf("%s:8080", defaultIPv4), wantErr: false}, + {name: "CIDR6LinkLocalDefault", address: "::/0%eth0", port: 8080, want: fmt.Sprintf("[%s%%eth0]:8080", defaultIPv6), wantErr: false}, + {name: "CIDR6Default", address: "::/0", port: 8080, want: fmt.Sprintf("[%s]:8080", defaultIPv6), wantErr: false}, + {name: "CIDR6DefaultPort", address: "[::/0]:9090", port: 8080, want: fmt.Sprintf("[%s]:9090", defaultIPv6), wantErr: false}, + {name: "CIDR6DefaultPort", address: "[::/0]:9090", port: 8080, want: fmt.Sprintf("[%s]:9090", defaultIPv6), wantErr: false}, + {name: "CIDR6", address: networkAddressCIDR6, port: 8080, want: fmt.Sprintf("[%s]:8080", defaultIPv6), wantErr: false}, + {name: "CIDR6AndPort", address: fmt.Sprintf("[%s]:9090", networkAddressCIDR6), port: 8080, want: fmt.Sprintf("[%s]:9090", defaultIPv6), wantErr: false}, {name: "IPv4", address: "10.0.0.10", port: 8080, want: "10.0.0.10:8080", wantErr: false}, {name: "IPv4AndPort", address: "10.0.0.10:9090", port: 8080, want: "10.0.0.10:9090", wantErr: false}, {name: "NonMatchingCIDR", address: "10.10.5.0/24", port: 8080, want: "", wantErr: true}, {name: "IPv6", address: "fe80::1:234", port: 8080, want: "[fe80::1:234]:8080", wantErr: false}, + {name: "IPv6Zone", address: "fe80::1:234%eth0", port: 8080, want: "[fe80::1:234%eth0]:8080", wantErr: false}, + {name: "IPv6ZoneAndPort", address: "[fe80::1:234%eth0]:9090", port: 8080, want: "[fe80::1:234%eth0]:9090", wantErr: false}, {name: "IPv6AndPort", address: "[fe80::1:234]:9090", port: 8080, want: "[fe80::1:234]:9090", wantErr: false}, {name: "InvalidPort", address: "127.0.0.1:invalid-port", port: 0, want: "", wantErr: true}, {name: "PortOutOfBounds", address: "10.0.0.10:70799", port: 8080, want: "", wantErr: true}, } { t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) got, err := utils.ParseAddressString(tc.address, tc.port) if tc.wantErr { g.Expect(err).To(HaveOccurred()) @@ -153,14 +171,87 @@ func TestParseCIDRs(t *testing.T) { for _, tc := range testCases { t.Run(tc.input, func(t *testing.T) { - ipv4CIDR, ipv6CIDR, err := utils.ParseCIDRs(tc.input) + g := NewWithT(t) + ipv4CIDR, ipv6CIDR, err := utils.SplitCIDRStrings(tc.input) if tc.expectedErr { - Expect(err).To(HaveOccurred()) + g.Expect(err).To(HaveOccurred()) } else { - Expect(err).To(BeNil()) - Expect(ipv4CIDR).To(Equal(tc.expectedIPv4)) - Expect(ipv6CIDR).To(Equal(tc.expectedIPv6)) + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(ipv4CIDR).To(Equal(tc.expectedIPv4)) + g.Expect(ipv6CIDR).To(Equal(tc.expectedIPv6)) } }) } } + +func TestIsIPv4(t *testing.T) { + tests := []struct { + address string + expected bool + }{ + {"192.168.1.1:80", true}, + {"127.0.0.1", true}, + {"::1", false}, + {"[fe80::1]:80", false}, + {"256.256.256.256", false}, // Invalid IPv4 address + } + + for _, tc := range tests { + t.Run(tc.address, func(t *testing.T) { + g := NewWithT(t) + result := utils.IsIPv4(tc.address) + g.Expect(result).To(Equal(tc.expected)) + }) + } +} + +func TestToIPString(t *testing.T) { + tests := []struct { + ip net.IP + expected string + }{ + {net.ParseIP("192.168.1.1"), "192.168.1.1"}, + {net.ParseIP("::1"), "[::1]"}, + {net.ParseIP("fe80::1"), "[fe80::1]"}, + {net.ParseIP("127.0.0.1"), "127.0.0.1"}, + } + + for _, tc := range tests { + t.Run(tc.expected, func(t *testing.T) { + g := NewWithT(t) + result := utils.ToIPString(tc.ip) + g.Expect(result).To(Equal(tc.expected)) + }) + } +} + +// getInterfaceNameForIP returns the network interface name associated with the given IP. +func getInterfaceNameForIP(ip net.IP) (string, error) { + interfaces, err := net.Interfaces() + if err != nil { + return "", err + } + + for _, iface := range interfaces { + addrs, err := iface.Addrs() + if err != nil { + return "", err + } + + for _, addr := range addrs { + var ipAddr net.IP + switch v := addr.(type) { + case *net.IPNet: + ipAddr = v.IP + case *net.IPAddr: + ipAddr = v.IP + } + + if ipAddr.Equal(ip) { + return iface.Name, nil + } + } + } + + return "", errors.New("no interface found for the given IP") +} diff --git a/src/k8s/pkg/utils/control/retry_test.go b/src/k8s/pkg/utils/control/retry_test.go index 88ee21e31..1de78cb16 100644 --- a/src/k8s/pkg/utils/control/retry_test.go +++ b/src/k8s/pkg/utils/control/retry_test.go @@ -22,7 +22,6 @@ func TestRetryFor(t *testing.T) { } return nil }) - if err != nil { t.Errorf("Expected nil error, got: %v", err) } @@ -42,7 +41,7 @@ func TestRetryFor(t *testing.T) { return errors.New("failed") }) - if err != context.Canceled { + if !errors.Is(err, context.Canceled) { t.Errorf("Expected context.Canceled error, got: %v", err) } }) diff --git a/src/k8s/pkg/utils/control/wait_test.go b/src/k8s/pkg/utils/control/wait_test.go index 9fb3cd2dc..2b18758e8 100644 --- a/src/k8s/pkg/utils/control/wait_test.go +++ b/src/k8s/pkg/utils/control/wait_test.go @@ -12,11 +12,11 @@ func mockCheckFunc() (bool, error) { return true, nil } -var testError = errors.New("test error") +var errTest = errors.New("test error") // Mock check function that returns an error. func mockErrorCheckFunc() (bool, error) { - return false, testError + return false, errTest } func TestWaitUntilReady(t *testing.T) { @@ -34,7 +34,7 @@ func TestWaitUntilReady(t *testing.T) { cancel2() // Cancel the context immediately err2 := WaitUntilReady(ctx2, mockCheckFunc) - if err2 == nil || err2 != context.Canceled { + if err2 == nil || !errors.Is(err2, context.Canceled) { t.Errorf("Expected context.Canceled error, got: %v", err2) } @@ -52,7 +52,7 @@ func TestWaitUntilReady(t *testing.T) { defer cancel4() err4 := WaitUntilReady(ctx4, mockErrorCheckFunc) - if err4 == nil || !errors.Is(err4, testError) { + if err4 == nil || !errors.Is(err4, errTest) { t.Errorf("Expected test error, got: %v", err4) } } diff --git a/src/k8s/pkg/utils/errors/errors.go b/src/k8s/pkg/utils/errors/errors.go deleted file mode 100644 index a363850cb..000000000 --- a/src/k8s/pkg/utils/errors/errors.go +++ /dev/null @@ -1,16 +0,0 @@ -package errors - -import "errors" - -// DeeplyUnwrapError unwraps an wrapped error. -// DeeplyUnwrapError will return the innermost error for deeply nested errors. -// DeeplyUnwrapError will return the existing error if the error is not wrapped. -func DeeplyUnwrapError(err error) error { - for { - cause := errors.Unwrap(err) - if cause == nil { - return err - } - err = cause - } -} diff --git a/src/k8s/pkg/utils/errors/errors_test.go b/src/k8s/pkg/utils/errors/errors_test.go deleted file mode 100644 index 3562695c5..000000000 --- a/src/k8s/pkg/utils/errors/errors_test.go +++ /dev/null @@ -1,46 +0,0 @@ -package errors - -import ( - "errors" - "fmt" - "testing" - - "github.com/onsi/gomega" -) - -func TestDeeplyUnwrapError(t *testing.T) { - g := gomega.NewWithT(t) - - t.Run("when error is not wrapped", func(t *testing.T) { - err := errors.New("test error") - unwrapped := DeeplyUnwrapError(err) - - g.Expect(unwrapped).To(gomega.Equal(err)) - }) - - t.Run("when error is wrapped once", func(t *testing.T) { - innerErr := errors.New("inner error") - err := fmt.Errorf("outer wrapper: %w", innerErr) - - unwrapped := DeeplyUnwrapError(err) - - g.Expect(unwrapped).To(gomega.Equal(innerErr)) - }) - - t.Run("when error is deeply nested", func(t *testing.T) { - innermostErr := errors.New("innermost error") - innerErr := fmt.Errorf("middle wrapper: %w", innermostErr) - err := fmt.Errorf("outer wrapper: %w", innerErr) - - unwrapped := DeeplyUnwrapError(err) - - g.Expect(unwrapped).To(gomega.Equal(innermostErr)) - }) - - t.Run("when error is nil", func(t *testing.T) { - var err error - unwrapped := DeeplyUnwrapError(err) - - g.Expect(unwrapped).To(gomega.BeNil()) - }) -} diff --git a/src/k8s/pkg/utils/file.go b/src/k8s/pkg/utils/file.go index c4d1dc79f..63cfad1bf 100644 --- a/src/k8s/pkg/utils/file.go +++ b/src/k8s/pkg/utils/file.go @@ -15,9 +15,8 @@ import ( "sort" "strings" - "github.com/moby/sys/mountinfo" - "github.com/canonical/k8s/pkg/log" + "github.com/moby/sys/mountinfo" ) // ParseArgumentLine parses a command-line argument from a single line. @@ -120,6 +119,7 @@ func CopyFile(srcFile, dstFile string) error { return nil } + func FileExists(path ...string) (bool, error) { if _, err := os.Stat(filepath.Join(path...)); err != nil { if !os.IsNotExist(err) { @@ -258,3 +258,35 @@ func CreateTarball(tarballPath string, rootDir string, walkDir string, excludeFi return nil } + +// WriteFile writes data to a file with the given name and permissions. +// The file is written to a temporary file in the same directory as the target file +// and then renamed to the target file to avoid partial writes in case of a crash. +func WriteFile(name string, data []byte, perm fs.FileMode) error { + dir := filepath.Dir(name) + tmpFile, err := os.CreateTemp(dir, "tmp-*") + if err != nil { + return fmt.Errorf("failed to create temp file: %w", err) + } + defer os.Remove(tmpFile.Name()) + + if _, err := tmpFile.Write(data); err != nil { + tmpFile.Close() + return fmt.Errorf("failed to write to temp file: %w", err) + } + + if err := tmpFile.Chmod(perm); err != nil { + tmpFile.Close() + return fmt.Errorf("failed to set permissions on temp file: %w", err) + } + + if err := tmpFile.Close(); err != nil { + return fmt.Errorf("failed to close temp file: %w", err) + } + + if err := os.Rename(tmpFile.Name(), name); err != nil { + return fmt.Errorf("failed to rename temp file to target file: %w", err) + } + + return nil +} diff --git a/src/k8s/pkg/utils/file_test.go b/src/k8s/pkg/utils/file_test.go index 85af2f79c..db6437036 100644 --- a/src/k8s/pkg/utils/file_test.go +++ b/src/k8s/pkg/utils/file_test.go @@ -4,6 +4,7 @@ import ( "fmt" "os" "path/filepath" + "sync" "testing" "github.com/canonical/k8s/pkg/utils" @@ -88,7 +89,7 @@ func TestParseArgumentFile(t *testing.T) { g := NewWithT(t) filePath := filepath.Join(t.TempDir(), tc.name) - err := os.WriteFile(filePath, []byte(tc.content), 0755) + err := utils.WriteFile(filePath, []byte(tc.content), 0o755) if err != nil { t.Fatalf("Failed to setup testfile: %v", err) } @@ -157,17 +158,17 @@ func TestFileExists(t *testing.T) { testFilePath := fmt.Sprintf("%s/myfile", t.TempDir()) _, err := os.Create(testFilePath) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) fileExists, err := utils.FileExists(testFilePath) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(fileExists).To(BeTrue()) err = os.Remove(testFilePath) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) fileExists, err = utils.FileExists(testFilePath) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(fileExists).To(BeFalse()) } @@ -182,3 +183,92 @@ func TestGetMountPropagationType(t *testing.T) { g.Expect(err).ToNot(HaveOccurred()) g.Expect(mountType).To(Equal(utils.MountPropagationShared)) } + +func TestWriteFile(t *testing.T) { + t.Run("PartialWrites", func(t *testing.T) { + g := NewWithT(t) + + name := filepath.Join(t.TempDir(), "testfile") + + const ( + numWriters = 200 + numIterations = 200 + ) + + var wg sync.WaitGroup + wg.Add(numWriters) + + expContent := "key: value" + expPerm := os.FileMode(0o644) + + for i := 0; i < numWriters; i++ { + go func(writerID int) { + defer wg.Done() + + for j := 0; j < numIterations; j++ { + g.Expect(utils.WriteFile(name, []byte(expContent), expPerm)).To(Succeed()) + + content, err := os.ReadFile(name) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(string(content)).To(Equal(expContent)) + + fileInfo, err := os.Stat(name) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(fileInfo.Mode().Perm()).To(Equal(expPerm)) + } + }(i) + } + + wg.Wait() + }) + + tcs := []struct { + name string + expContent []byte + expPerm os.FileMode + }{ + { + name: "test1", + expContent: []byte("key: value"), + expPerm: os.FileMode(0o644), + }, + { + name: "test2", + expContent: []byte(""), + expPerm: os.FileMode(0o600), + }, + { + name: "test3", + expContent: []byte("key: value"), + expPerm: os.FileMode(0o755), + }, + { + name: "test4", + expContent: []byte("key: value"), + expPerm: os.FileMode(0o777), + }, + { + name: "test5", + expContent: []byte("key: value"), + expPerm: os.FileMode(0o400), + }, + } + + for _, tc := range tcs { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + + name := filepath.Join(t.TempDir(), tc.name) + + g.Expect(utils.WriteFile(name, tc.expContent, tc.expPerm)).To(Succeed()) + + content, err := os.ReadFile(name) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(string(content)).To(Equal(string(tc.expContent))) + + fileInfo, err := os.Stat(name) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(fileInfo.Mode().Perm()).To(Equal(tc.expPerm)) + }) + } +} diff --git a/src/k8s/pkg/utils/hostname_test.go b/src/k8s/pkg/utils/hostname_test.go index 752fb5125..694c15ec1 100644 --- a/src/k8s/pkg/utils/hostname_test.go +++ b/src/k8s/pkg/utils/hostname_test.go @@ -28,10 +28,10 @@ func TestCleanHostname(t *testing.T) { g := NewWithT(t) hostname, err := utils.CleanHostname(tc.hostname) if tc.expectValid { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(hostname).To(Equal(tc.expectHostname)) } else { - g.Expect(err).To(Not(BeNil())) + g.Expect(err).To(HaveOccurred()) } }) } diff --git a/src/k8s/pkg/utils/mapstructure.go b/src/k8s/pkg/utils/mapstructure.go index 60e650b45..19fc297b0 100644 --- a/src/k8s/pkg/utils/mapstructure.go +++ b/src/k8s/pkg/utils/mapstructure.go @@ -1,6 +1,7 @@ package utils import ( + "fmt" "reflect" "strings" "unicode" @@ -15,12 +16,16 @@ func YAMLToStringSliceHookFunc(f reflect.Kind, t reflect.Kind, data interface{}) return data, nil } - if data.(string) == "" { + strData, ok := data.(string) + if !ok { + return nil, fmt.Errorf("expected string but got %T", data) + } + if strData == "" { return data, nil } var result []string - if err := yaml.Unmarshal([]byte(data.(string)), &result); err != nil { + if err := yaml.Unmarshal([]byte(strData), &result); err != nil { return data, nil } @@ -34,7 +39,10 @@ func StringToFieldsSliceHookFunc(r rune) mapstructure.DecodeHookFunc { return data, nil } - raw := data.(string) + raw, ok := data.(string) + if !ok { + return nil, fmt.Errorf("expected string but got %T", data) + } if raw == "" { return []string{}, nil } @@ -49,12 +57,16 @@ func YAMLToStringMapHookFunc(f reflect.Kind, t reflect.Kind, data interface{}) ( return data, nil } - if data.(string) == "" { + strData, ok := data.(string) + if !ok { + return nil, fmt.Errorf("expected string but got %T", data) + } + if strData == "" { return map[string]string{}, nil } var result map[string]string - if err := yaml.Unmarshal([]byte(data.(string)), &result); err != nil { + if err := yaml.Unmarshal([]byte(strData), &result); err != nil { return data, nil } @@ -67,14 +79,17 @@ func StringToStringMapHookFunc(f reflect.Kind, t reflect.Kind, data interface{}) return data, nil } - raw := data.(string) + raw, ok := data.(string) + if !ok { + return nil, fmt.Errorf("expected string but got %T", data) + } if raw == "" { return map[string]string{}, nil } fields := strings.FieldsFunc(raw, func(this rune) bool { return this == ',' || unicode.IsSpace(this) }) result := make(map[string]string, len(fields)) - for _, kv := range strings.FieldsFunc(raw, func(this rune) bool { return this == ',' || unicode.IsSpace(this) }) { + for _, kv := range fields { parts := strings.SplitN(kv, "=", 2) if len(parts) < 2 { return data, nil diff --git a/src/k8s/pkg/utils/pki/generate_test.go b/src/k8s/pkg/utils/pki/generate_test.go index e2fe9976c..1ab80c819 100644 --- a/src/k8s/pkg/utils/pki/generate_test.go +++ b/src/k8s/pkg/utils/pki/generate_test.go @@ -14,20 +14,20 @@ func TestGenerateSelfSignedCA(t *testing.T) { cert, key, err := pkiutil.GenerateSelfSignedCA(pkix.Name{CommonName: "test-cert"}, notBefore, notBefore.AddDate(10, 0, 0), 2048) g := NewWithT(t) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cert).ToNot(BeEmpty()) g.Expect(key).ToNot(BeEmpty()) t.Run("Load", func(t *testing.T) { c, k, err := pkiutil.LoadCertificate(cert, key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(c).ToNot(BeNil()) g.Expect(k).ToNot(BeNil()) }) t.Run("LoadCertOnly", func(t *testing.T) { cert, key, err := pkiutil.LoadCertificate(cert, "") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cert).ToNot(BeNil()) g.Expect(key).To(BeNil()) }) diff --git a/src/k8s/pkg/utils/pki/load.go b/src/k8s/pkg/utils/pki/load.go index 705efc8ba..46c0d23fe 100644 --- a/src/k8s/pkg/utils/pki/load.go +++ b/src/k8s/pkg/utils/pki/load.go @@ -64,8 +64,7 @@ func LoadRSAPublicKey(keyPEM string) (*rsa.PublicKey, error) { if pb == nil { return nil, fmt.Errorf("failed to parse PEM block") } - switch pb.Type { - case "PUBLIC KEY": + if pb.Type == "PUBLIC KEY" { parsed, err := x509.ParsePKIXPublicKey(pb.Bytes) if err != nil { return nil, fmt.Errorf("failed to parse public key: %w", err) @@ -86,8 +85,7 @@ func LoadCertificateRequest(csrPEM string) (*x509.CertificateRequest, error) { if pb == nil { return nil, fmt.Errorf("failed to parse certificate request PEM") } - switch pb.Type { - case "CERTIFICATE REQUEST": + if pb.Type == "CERTIFICATE REQUEST" { parsed, err := x509.ParseCertificateRequest(pb.Bytes) if err != nil { return nil, fmt.Errorf("failed to parse certificate request: %w", err) diff --git a/src/k8s/pkg/utils/time.go b/src/k8s/pkg/utils/time.go index 19102c1c5..d414e7232 100644 --- a/src/k8s/pkg/utils/time.go +++ b/src/k8s/pkg/utils/time.go @@ -39,7 +39,7 @@ func SecondsToExpirationDate(now time.Time, seconds int) time.Time { // - y: years // - mo: months // - d: days -// - any other unit supported by time.ParseDuration +// - any other unit supported by time.ParseDuration. func TTLToSeconds(ttl string) (int, error) { if len(ttl) < 2 { return 0, fmt.Errorf("invalid TTL length: %s", ttl) diff --git a/src/k8s/pkg/utils/time_test.go b/src/k8s/pkg/utils/time_test.go index dce809058..5b816ebfc 100644 --- a/src/k8s/pkg/utils/time_test.go +++ b/src/k8s/pkg/utils/time_test.go @@ -44,9 +44,7 @@ func TestSecondsToExpirationDate(t *testing.T) { got := utils.SecondsToExpirationDate(now, tt.seconds) g.Expect(got).To(Equal(tt.want)) }) - } - } func TestTTLToSeconds(t *testing.T) { diff --git a/tests/branch_management/tests/conftest.py b/tests/branch_management/tests/conftest.py index f745828fb..85045f683 100644 --- a/tests/branch_management/tests/conftest.py +++ b/tests/branch_management/tests/conftest.py @@ -2,26 +2,68 @@ # Copyright 2024 Canonical, Ltd. # from pathlib import Path +from typing import Optional import pytest import requests import semver +STABLE_URL = "https://dl.k8s.io/release/stable.txt" +RELEASE_URL = "https://dl.k8s.io/release/stable-{}.{}.txt" -@pytest.fixture -def upstream_release() -> semver.VersionInfo: + +def _upstream_release(ver: semver.Version) -> Optional[semver.Version]: + """Semver of the major.minor release if it exists""" + r = requests.get(RELEASE_URL.format(ver.major, ver.minor)) + if r.status_code == 200: + return semver.Version.parse(r.content.decode().lstrip("v")) + + +def _get_max_minor(ver: semver.Version) -> semver.Version: + """ + Get the latest patch release based on the provided major. + + e.g. 1.. could yield 1.31.4 if 1.31 is the latest stable release on that maj channel + e.g. 2.. could yield 2.12.1 if 2.12 is the latest stable release on that maj channel + """ + out = semver.Version(ver.major, 0, 0) + while ver := _upstream_release(ver): + out, ver = ver, semver.Version(ver.major, ver.minor + 1, 0) + return out + + +def _previous_release(ver: semver.Version) -> semver.Version: + """Return the prior release version based on the provided version ignoring patch""" + if ver.minor != 0: + return _upstream_release(semver.Version(ver.major, ver.minor - 1, 0)) + return _get_max_minor(semver.Version(ver.major, 0, 0)) + + +@pytest.fixture(scope="session") +def stable_release() -> semver.Version: """Return the latest stable k8s in the release series""" - release_url = "https://dl.k8s.io/release/stable.txt" - r = requests.get(release_url) + r = requests.get(STABLE_URL) r.raise_for_status() return semver.Version.parse(r.content.decode().lstrip("v")) -@pytest.fixture -def current_release() -> semver.VersionInfo: +@pytest.fixture(scope="session") +def current_release() -> semver.Version: """Return the current branch k8s version""" ver_file = ( Path(__file__).parent / "../../../build-scripts/components/kubernetes/version" ) version = ver_file.read_text().strip() return semver.Version.parse(version.lstrip("v")) + + +@pytest.fixture +def prior_stable_release(stable_release) -> semver.Version: + """Return the prior release to the upstream stable""" + return _previous_release(stable_release) + + +@pytest.fixture +def prior_release(current_release) -> semver.Version: + """Return the prior release to the current release""" + return _previous_release(current_release) diff --git a/tests/branch_management/tests/test_branches.py b/tests/branch_management/tests/test_branches.py index 426fde599..556892c03 100644 --- a/tests/branch_management/tests/test_branches.py +++ b/tests/branch_management/tests/test_branches.py @@ -1,93 +1,127 @@ # # Copyright 2024 Canonical, Ltd. # +import functools +import logging +import subprocess from pathlib import Path -from subprocess import check_output import requests +log = logging.getLogger(__name__) +K8S_GH_REPO = "https://github.com/canonical/k8s-snap.git/" +K8S_LP_REPO = " https://git.launchpad.net/k8s" -def _get_max_minor(major): - """Get the latest minor release of the provided major. - For example if you use 1 as major you will get back X where X gives you latest 1.X release. - """ - minor = 0 - while _upstream_release_exists(major, minor): - minor += 1 - return minor - 1 - - -def _upstream_release_exists(major, minor): - """Return true if the major.minor release exists""" - release_url = "https://dl.k8s.io/release/stable-{}.{}.txt".format(major, minor) - r = requests.get(release_url) - return r.status_code == 200 - -def _confirm_branch_exists(branch): - cmd = f"git ls-remote --heads https://github.com/canonical/k8s-snap.git/ {branch}" - output = check_output(cmd.split()).decode("utf-8") - assert branch in output, f"Branch {branch} does not exist" +def _sh(*args, **kwargs): + default = {"text": True, "stderr": subprocess.PIPE} + try: + return subprocess.check_output(*args, **{**default, **kwargs}) + except subprocess.CalledProcessError as e: + log.error("stdout: %s", e.stdout) + log.error("stderr: %s", e.stderr) + raise e def _branch_flavours(branch: str = None): patch_dir = Path("build-scripts/patches") branch = "HEAD" if branch is None else branch - cmd = f"git ls-tree --full-tree -r --name-only {branch} {patch_dir}" - output = check_output(cmd.split()).decode("utf-8") + cmd = f"git ls-tree --full-tree -r --name-only origin/{branch} {patch_dir}" + output = _sh(cmd.split()) patches = set( Path(f).relative_to(patch_dir).parents[0] for f in output.splitlines() ) return [p.name for p in patches] -def _confirm_recipe(track, flavour): +@functools.lru_cache +def _confirm_branch_exists(repo, branch): + log.info(f"Checking {branch} branch exists in {repo}") + cmd = f"git ls-remote --heads {repo} {branch}" + output = _sh(cmd.split()) + return branch in output + + +def _confirm_all_branches_exist(leader): + assert _confirm_branch_exists( + K8S_GH_REPO, leader + ), f"GH Branch {leader} does not exist" + branches = [leader] + branches += [f"autoupdate/{leader}-{fl}" for fl in _branch_flavours(leader)] + if missing := [b for b in branches if not _confirm_branch_exists(K8S_GH_REPO, b)]: + assert missing, f"GH Branches do not exist {missing}" + if missing := [b for b in branches if not _confirm_branch_exists(K8S_LP_REPO, b)]: + assert missing, f"LP Branches do not exist {missing}" + + +@functools.lru_cache +def _confirm_recipe_exist(track, flavour): recipe = f"https://launchpad.net/~containers/k8s/+snap/k8s-snap-{track}-{flavour}" r = requests.get(recipe) return r.status_code == 200 -def test_branches(upstream_release): - """Ensures git branches exist for prior releases. +def _confirm_all_recipes_exist(track, branch): + log.info(f"Checking {track} recipe exists") + assert _confirm_branch_exists( + K8S_GH_REPO, branch + ), f"GH Branch {branch} does not exist" + flavours = ["classic"] + _branch_flavours(branch) + recipes = {flavour: _confirm_recipe_exist(track, flavour) for flavour in flavours} + if missing := [fl for fl, exists in recipes.items() if not exists]: + assert missing, f"LP Recipes do not exist for {track} {missing}" + - We need to make sure the LP builders pointing to the main github branch are only pushing - to the latest and current k8s edge snap tracks. An indication that this is not enforced is - that we do not have a branch for the k8s release for the previous stable release. Let me - clarify with an example. +def test_prior_branches(prior_stable_release): + """Ensures git branches exist for prior stable releases. - Assuming upstream stable k8s release is v1.12.x, there has to be a 1.11 github branch used - by the respective LP builders for building the v1.11.y. + This is to ensure that the prior release branches exist in the k8s-snap repository + before we can proceed to build the next release. For example, if the current stable + k8s release is v1.31.0, there must be a release-1.30 branch before updating main. """ - if upstream_release.minor != 0: - major = upstream_release.major - minor = upstream_release.minor - 1 - else: - major = int(upstream_release.major) - 1 - minor = _get_max_minor(major) - - prior_branch = f"release-{major}.{minor}" - print(f"Current stable is {upstream_release}") - print(f"Checking {prior_branch} branch exists") - _confirm_branch_exists(prior_branch) - flavours = _branch_flavours(prior_branch) - for flavour in flavours: - prior_branch = f"autoupdate/{prior_branch}-{flavour}" - print(f"Checking {prior_branch} branch exists") - _confirm_branch_exists(prior_branch) - - -def test_launchpad_recipe(current_release): + branch = f"release-{prior_stable_release.major}.{prior_stable_release.minor}" + _confirm_all_branches_exist(branch) + + +def test_prior_recipes(prior_stable_release): + """Ensures the recipes exist for prior stable releases. + + This is to ensure that the prior release recipes exist in launchpad before we can proceed + to build the next release. For example, if the current stable k8s release is v1.31.0, there + must be a k8s-snap-1.30-classic recipe before updating main. + """ + track = f"{prior_stable_release.major}.{prior_stable_release.minor}" + branch = f"release-{track}" + _confirm_all_recipes_exist(track, branch) + + +def test_branches(current_release): + """Ensures the current release has a release branch. + + This is to ensure that the current release branches exist in the k8s-snap repository + before we can proceed to build it. For example, if the current stable + k8s release is v1.31.0, there must be a release-1.31 branch. + """ + branch = f"release-{current_release.major}.{current_release.minor}" + _confirm_all_branches_exist(branch) + + +def test_recipes(current_release): """Ensures the current recipes are available. - We should ensure that a launchpad recipe exists for this release to be build with + We should ensure that a launchpad recipes exist for this release to be build with + + This can fail when a new minor release (e.g. 1.32) is detected and its release branch + is yet to be created from main. """ track = f"{current_release.major}.{current_release.minor}" - print(f"Checking {track} recipe exists") - flavours = ["classic"] + _branch_flavours() - recipe_exists = {flavour: _confirm_recipe(track, flavour) for flavour in flavours} - if missing_recipes := [ - flavour for flavour, exists in recipe_exists.items() if not exists - ]: - assert ( - not missing_recipes - ), f"LP Recipes do not exist for {track} {missing_recipes}" + branch = f"release-{track}" + _confirm_all_recipes_exist(track, branch) + + +def test_tip_recipes(): + """Ensures the tip recipes are available. + + We should ensure that a launchpad recipes always exist for tip to be build with + """ + _confirm_all_recipes_exist("latest", "main") diff --git a/tests/branch_management/tox.ini b/tests/branch_management/tox.ini index 371ad51e4..a7f2cd8de 100644 --- a/tests/branch_management/tox.ini +++ b/tests/branch_management/tox.ini @@ -30,15 +30,16 @@ commands = black {tox_root}/tests --check --diff [testenv:test] -description = Run integration tests +description = Run branch management tests deps = -r {tox_root}/requirements-test.txt commands = pytest -v \ - --maxfail 1 \ --tb native \ --log-cli-level DEBUG \ --disable-warnings \ + --log-format "%(asctime)s %(levelname)s %(message)s" \ + --log-date-format "%Y-%m-%d %H:%M:%S" \ {posargs} \ {tox_root}/tests pass_env = diff --git a/tests/integration/lxd-ipv6-profile.yaml b/tests/integration/lxd-ipv6-profile.yaml new file mode 100644 index 000000000..900edd7d2 --- /dev/null +++ b/tests/integration/lxd-ipv6-profile.yaml @@ -0,0 +1,7 @@ +description: "LXD profile for Canonical Kubernetes with IPv6-only networking" +devices: + eth0: + name: eth0 + nictype: bridged + parent: LXD_IPV6_NETWORK + type: nic diff --git a/tests/integration/lxd-profile.yaml b/tests/integration/lxd-profile.yaml index c6a05f38e..7f5e720dc 100644 --- a/tests/integration/lxd-profile.yaml +++ b/tests/integration/lxd-profile.yaml @@ -1,6 +1,6 @@ description: "LXD profile for Canonical Kubernetes" config: - linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,ip_tables,ip6_tables,iptable_raw,netlink_diag,nf_nat,overlay,br_netfilter,xt_socket + linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,ip_tables,ip6_tables,iptable_raw,netlink_diag,nf_nat,overlay,br_netfilter,xt_socket,nf_conntrack raw.lxc: | lxc.apparmor.profile=unconfined lxc.mount.auto=proc:rw sys:rw cgroup:rw diff --git a/tests/integration/lxd/setup-image.sh b/tests/integration/lxd/setup-image.sh index 4b587bad1..c5427fc5e 100755 --- a/tests/integration/lxd/setup-image.sh +++ b/tests/integration/lxd/setup-image.sh @@ -69,6 +69,8 @@ case "${BASE_DISTRO}" in # snapd is preinstalled on Ubuntu OSes lxc shell tmp-builder -- bash -c 'snap wait core seed.loaded' lxc shell tmp-builder -- bash -c 'snap install '"${BASE_SNAP}" + # NOTE(aznashwan): 'nf_conntrack' required by kube-proxy: + lxc shell tmp-builder -- bash -c 'apt update && apt install -y "linux-modules-$(uname -r)"}' ;; almalinux) # install snapd and ensure /snap/bin is in the environment @@ -77,6 +79,8 @@ case "${BASE_DISTRO}" in lxc shell tmp-builder -- bash -c 'dnf install tar sudo -y' lxc shell tmp-builder -- bash -c 'dnf install fuse squashfuse -y' lxc shell tmp-builder -- bash -c 'dnf install snapd -y' + # NOTE(aznashwan): 'nf_conntrack' required by kube-proxy: + lxc shell tmp-builder -- bash -c 'dnf install -y kernel-modules-core' lxc shell tmp-builder -- bash -c 'systemctl enable --now snapd.socket' lxc shell tmp-builder -- bash -c 'ln -s /var/lib/snapd/snap /snap' @@ -92,6 +96,8 @@ case "${BASE_DISTRO}" in lxc shell tmp-builder -- bash -c 'snap install snapd '"${BASE_SNAP}" lxc shell tmp-builder -- bash -c 'echo PATH=$PATH:/snap/bin >> /etc/environment' lxc shell tmp-builder -- bash -c 'apt autoremove; apt clean; apt autoclean; rm -rf /var/lib/apt/lists' + # NOTE(aznashwan): 'nf_conntrack' required by kube-proxy: + lxc shell tmp-builder -- bash -c 'apt update && apt install -y "linux-modules-$(uname -r)"}' # NOTE(neoaggelos): disable apparmor in containerd, as it causes trouble in the default setup lxc shell tmp-builder -- bash -c ' diff --git a/tests/integration/requirements-test.txt b/tests/integration/requirements-test.txt index 91282e09c..0fcd9c093 100644 --- a/tests/integration/requirements-test.txt +++ b/tests/integration/requirements-test.txt @@ -3,3 +3,4 @@ pytest==7.3.1 PyYAML==6.0.1 tenacity==8.2.3 pylint==3.2.5 +cryptography==43.0.3 diff --git a/tests/integration/templates/bootstrap-session.yaml b/tests/integration/templates/bootstrap-all.yaml similarity index 100% rename from tests/integration/templates/bootstrap-session.yaml rename to tests/integration/templates/bootstrap-all.yaml diff --git a/tests/integration/templates/bootstrap-csr-auto-approve.yaml b/tests/integration/templates/bootstrap-csr-auto-approve.yaml new file mode 100644 index 000000000..43fe77c98 --- /dev/null +++ b/tests/integration/templates/bootstrap-csr-auto-approve.yaml @@ -0,0 +1,9 @@ +cluster-config: + network: + enabled: true + dns: + enabled: true + metrics-server: + enabled: true + annotations: + k8sd/v1alpha1/csrsigning/auto-approve: true diff --git a/tests/integration/templates/bootstrap-ipv6-only.yaml b/tests/integration/templates/bootstrap-ipv6-only.yaml new file mode 100644 index 000000000..442857805 --- /dev/null +++ b/tests/integration/templates/bootstrap-ipv6-only.yaml @@ -0,0 +1,16 @@ +cluster-config: + network: + enabled: true + dns: + enabled: true + cluster-domain: cluster.local + local-storage: + enabled: true + local-path: /storage/path + default: false + gateway: + enabled: true + metrics-server: + enabled: true +pod-cidr: fd01::/108 +service-cidr: fd98::/108 diff --git a/tests/integration/templates/bootstrap-skip-service-stop.yaml b/tests/integration/templates/bootstrap-skip-service-stop.yaml new file mode 100644 index 000000000..13a536cf2 --- /dev/null +++ b/tests/integration/templates/bootstrap-skip-service-stop.yaml @@ -0,0 +1,9 @@ +cluster-config: + network: + enabled: true + dns: + enabled: true + metrics-server: + enabled: true + annotations: + k8sd/v1alpha/lifecycle/skip-stop-services-on-remove: true diff --git a/tests/integration/templates/etcd/etcd-tls.conf b/tests/integration/templates/etcd/etcd-tls.conf index 59243ba98..83cd61a86 100644 --- a/tests/integration/templates/etcd/etcd-tls.conf +++ b/tests/integration/templates/etcd/etcd-tls.conf @@ -18,6 +18,6 @@ [alt_names] IP.1 = 127.0.0.1 - IP.2 = $IP + IP.2 = $IP DNS.1 = localhost DNS.2 = $NAME diff --git a/tests/integration/templates/etcd/etcd.service b/tests/integration/templates/etcd/etcd.service index 0ac4e9933..e770666a0 100644 --- a/tests/integration/templates/etcd/etcd.service +++ b/tests/integration/templates/etcd/etcd.service @@ -11,6 +11,7 @@ LimitNOFILE=40000 TimeoutStartSec=0 + Environment=ETCD_UNSUPPORTED_ARCH=$ARCH ExecStart=/tmp/test-etcd/etcd --name $NAME \ --data-dir /tmp/etcd/s1 \ --listen-client-urls $CLIENT_URL \ diff --git a/tests/integration/templates/nginx-ipv6-only.yaml b/tests/integration/templates/nginx-ipv6-only.yaml new file mode 100644 index 000000000..93a5647a2 --- /dev/null +++ b/tests/integration/templates/nginx-ipv6-only.yaml @@ -0,0 +1,36 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginxipv6 +spec: + selector: + matchLabels: + run: nginxipv6 + replicas: 1 + template: + metadata: + labels: + run: nginxipv6 + spec: + containers: + - name: nginxipv6 + image: rocks.canonical.com/cdk/diverdane/nginxdualstack:1.0.0 + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Service +metadata: + name: nginx-ipv6 + labels: + run: nginxipv6 +spec: + type: NodePort + ipFamilies: + - IPv6 + ipFamilyPolicy: SingleStack + ports: + - port: 80 + protocol: TCP + selector: + run: nginxipv6 diff --git a/tests/integration/templates/registry/hosts.toml b/tests/integration/templates/registry/hosts.toml new file mode 100644 index 000000000..416c9a642 --- /dev/null +++ b/tests/integration/templates/registry/hosts.toml @@ -0,0 +1,2 @@ +[host."http://$IP:$PORT"] +capabilities = ["pull", "resolve"] diff --git a/tests/integration/templates/registry/registry-config.yaml b/tests/integration/templates/registry/registry-config.yaml new file mode 100644 index 000000000..28610ffbb --- /dev/null +++ b/tests/integration/templates/registry/registry-config.yaml @@ -0,0 +1,22 @@ +version: 0.1 +log: + fields: + service: registry +storage: + cache: + blobdescriptor: inmemory + filesystem: + rootdirectory: /var/lib/registry/$NAME +http: + addr: :$PORT + headers: + X-Content-Type-Options: [nosniff] +health: + storagedriver: + enabled: true + interval: 10s + threshold: 3 +proxy: + remoteurl: $REMOTE + username: $USERNAME + password: $PASSWORD diff --git a/tests/integration/templates/registry/registry.service b/tests/integration/templates/registry/registry.service new file mode 100644 index 000000000..83de843a6 --- /dev/null +++ b/tests/integration/templates/registry/registry.service @@ -0,0 +1,15 @@ + [Unit] + Description=registry-$NAME + Documentation=https://github.com/distribution/distribution + + [Service] + Type=simple + Restart=always + RestartSec=5s + LimitNOFILE=40000 + TimeoutStartSec=0 + + ExecStart=/bin/registry serve /etc/distribution/$NAME.yaml + + [Install] + WantedBy=multi-user.target diff --git a/tests/integration/tests/conftest.py b/tests/integration/tests/conftest.py index eb6fbb07f..7333c2167 100644 --- a/tests/integration/tests/conftest.py +++ b/tests/integration/tests/conftest.py @@ -1,16 +1,22 @@ # # Copyright 2024 Canonical, Ltd. # +import itertools import logging from pathlib import Path -from typing import Generator, List, Union +from typing import Generator, Iterator, List, Optional, Union import pytest from test_util import config, harness, util from test_util.etcd import EtcdCluster +from test_util.registry import Registry LOG = logging.getLogger(__name__) +# The following snaps will be downloaded once per test run and preloaded +# into the harness instances to reduce the number of downloads. +PRELOADED_SNAPS = ["snapd", "core20"] + def _harness_clean(h: harness.Harness): "Clean up created instances within the test harness." @@ -78,14 +84,45 @@ def h() -> harness.Harness: _harness_clean(h) +@pytest.fixture(scope="session") +def registry(h: harness.Harness) -> Optional[Registry]: + if config.USE_LOCAL_MIRROR: + yield Registry(h) + else: + # local image mirror disabled, avoid initializing the + # registry mirror instance. + yield None + + +@pytest.fixture(scope="session", autouse=True) +def snapd_preload() -> None: + if not config.PRELOAD_SNAPS: + LOG.info("Snap preloading disabled, skipping...") + return + + LOG.info(f"Downloading snaps for preloading: {PRELOADED_SNAPS}") + for snap in PRELOADED_SNAPS: + util.run( + [ + "snap", + "download", + snap, + f"--basename={snap}", + "--target-directory=/tmp", + ] + ) + + def pytest_configure(config): config.addinivalue_line( "markers", "bootstrap_config: Provide a custom bootstrap config to the bootstrapping node.\n" "disable_k8s_bootstrapping: By default, the first k8s node is bootstrapped. This marker disables that.\n" - "dualstack: Support dualstack on the instances.\n" + "no_setup: No setup steps (pushing snap, bootstrapping etc.) are performed on any node for this test.\n" + "network_type: Specify network type to use for the infrastructure (IPv4, Dualstack or IPv6).\n" "etcd_count: Mark a test to specify how many etcd instance nodes need to be created (None by default)\n" - "node_count: Mark a test to specify how many instance nodes need to be created\n", + "node_count: Mark a test to specify how many instance nodes need to be created\n" + "snap_versions: Mark a test to specify snap_versions for each node\n", ) @@ -98,11 +135,25 @@ def node_count(request) -> int: return int(node_count_arg) +def snap_versions(request) -> Iterator[Optional[str]]: + """An endless iterable of snap versions for each node in the test.""" + marking = () + if snap_version_marker := request.node.get_closest_marker("snap_versions"): + marking, *_ = snap_version_marker.args + # endlessly repeat of the configured snap version after exhausting the marking + return itertools.chain(marking, itertools.repeat(None)) + + @pytest.fixture(scope="function") def disable_k8s_bootstrapping(request) -> bool: return bool(request.node.get_closest_marker("disable_k8s_bootstrapping")) +@pytest.fixture(scope="function") +def no_setup(request) -> bool: + return bool(request.node.get_closest_marker("no_setup")) + + @pytest.fixture(scope="function") def bootstrap_config(request) -> Union[str, None]: bootstrap_config_marker = request.node.get_closest_marker("bootstrap_config") @@ -113,41 +164,66 @@ def bootstrap_config(request) -> Union[str, None]: @pytest.fixture(scope="function") -def dualstack(request) -> bool: - return bool(request.node.get_closest_marker("dualstack")) +def network_type(request) -> Union[str, None]: + bootstrap_config_marker = request.node.get_closest_marker("network_type") + if not bootstrap_config_marker: + return "IPv4" + network_type, *_ = bootstrap_config_marker.args + return network_type @pytest.fixture(scope="function") def instances( h: harness.Harness, + registry: Registry, node_count: int, tmp_path: Path, disable_k8s_bootstrapping: bool, + no_setup: bool, bootstrap_config: Union[str, None], - dualstack: bool, + request, + network_type: str, ) -> Generator[List[harness.Instance], None, None]: """Construct instances for a cluster. Bootstrap and setup networking on the first instance, if `disable_k8s_bootstrapping` marker is not set. """ - if not config.SNAP: - pytest.fail("Set TEST_SNAP to the path where the snap is") - if node_count <= 0: pytest.xfail("Test requested 0 or fewer instances, skip this test.") - snap_path = (tmp_path / "k8s.snap").as_posix() - LOG.info(f"Creating {node_count} instances") instances: List[harness.Instance] = [] - for _ in range(node_count): + for _, snap in zip(range(node_count), snap_versions(request)): # Create instances and setup the k8s snap in each. - instance = h.new_instance(dualstack=dualstack) + instance = h.new_instance(network_type=network_type) instances.append(instance) - util.setup_k8s_snap(instance, snap_path) - if not disable_k8s_bootstrapping: + if config.PRELOAD_SNAPS: + for preloaded_snap in PRELOADED_SNAPS: + ack_file = f"{preloaded_snap}.assert" + remote_path = (tmp_path / ack_file).as_posix() + instance.send_file( + source=f"/tmp/{ack_file}", + destination=remote_path, + ) + instance.exec(["snap", "ack", remote_path]) + + snap_file = f"{preloaded_snap}.snap" + remote_path = (tmp_path / snap_file).as_posix() + instance.send_file( + source=f"/tmp/{snap_file}", + destination=remote_path, + ) + instance.exec(["snap", "install", remote_path]) + + if not no_setup: + util.setup_k8s_snap(instance, tmp_path, snap) + + if config.USE_LOCAL_MIRROR: + registry.apply_configuration(instance) + + if not disable_k8s_bootstrapping and not no_setup: first_node, *_ = instances if bootstrap_config is not None: @@ -166,42 +242,17 @@ def instances( # Cleanup after each test. # We cannot execute _harness_clean() here as this would also - # remove the session_instance. The harness ensures that everything is cleaned up + # remove session scoped instances. The harness ensures that everything is cleaned up # at the end of the test session. for instance in instances: if config.INSPECTION_REPORTS_DIR is not None: LOG.debug("Generating inspection reports for test instances") _generate_inspection_report(h, instance.id) - h.delete_instance(instance.id) - - -@pytest.fixture(scope="session") -def session_instance( - h: harness.Harness, tmp_path_factory: pytest.TempPathFactory -) -> Generator[harness.Instance, None, None]: - """Constructs and bootstraps an instance that persists over a test session. - - Bootstraps the instance with all k8sd features enabled to reduce testing time. - """ - LOG.info("Setup node and enable all features") - - snap_path = str(tmp_path_factory.mktemp("data") / "k8s.snap") - instance = h.new_instance() - util.setup_k8s_snap(instance, snap_path) - - bootstrap_config_path = "/home/ubuntu/bootstrap-session.yaml" - instance.send_file( - (config.MANIFESTS_DIR / "bootstrap-session.yaml").as_posix(), - bootstrap_config_path, - ) - - instance.exec(["k8s", "bootstrap", "--file", bootstrap_config_path]) - util.wait_until_k8s_ready(instance, [instance]) - util.wait_for_network(instance) - util.wait_for_dns(instance) - - yield instance + try: + util.remove_k8s_snap(instance) + finally: + h.delete_instance(instance.id) @pytest.fixture(scope="function") diff --git a/tests/integration/tests/test_cleanup.py b/tests/integration/tests/test_cleanup.py index e3fa4e37e..784ac6b4a 100644 --- a/tests/integration/tests/test_cleanup.py +++ b/tests/integration/tests/test_cleanup.py @@ -9,6 +9,13 @@ LOG = logging.getLogger(__name__) +CONTAINERD_PATHS = [ + "/etc/containerd", + "/opt/cni/bin", + "/run/containerd", + "/var/lib/containerd", +] + @pytest.mark.node_count(1) def test_node_cleanup(instances: List[harness.Instance]): @@ -16,40 +23,11 @@ def test_node_cleanup(instances: List[harness.Instance]): util.wait_for_dns(instance) util.wait_for_network(instance) - LOG.info("Uninstall k8s...") - instance.exec(["snap", "remove", "k8s", "--purge"]) - - LOG.info("Waiting for shims to go away...") - util.stubbornly(retries=5, delay_s=5).on(instance).until( - lambda p: all( - x not in p.stdout.decode() - for x in ["containerd-shim", "cilium", "coredns", "/pause"] - ) - ).exec(["ps", "-fea"]) - - LOG.info("Waiting for kubelet and containerd mounts to go away...") - util.stubbornly(retries=5, delay_s=5).on(instance).until( - lambda p: all( - x not in p.stdout.decode() - for x in ["/var/lib/kubelet/pods", "/run/containerd/io.containerd"] - ) - ).exec(["mount"]) - - # NOTE(neoaggelos): Temporarily disable this as it fails on strict. - # For details, `snap changes` then `snap change $remove_k8s_snap_change`. - # Example output follows: - # - # 2024-02-23T14:10:42Z ERROR ignoring failure in hook "remove": - # ----- - # ... - # ip netns delete cni-UUID1 - # Cannot remove namespace file "/run/netns/cni-UUID1": Device or resource busy - # ip netns delete cni-UUID2 - # Cannot remove namespace file "/run/netns/cni-UUID2": Device or resource busy - # ip netns delete cni-UUID3 - # Cannot remove namespace file "/run/netns/cni-UUID3": Device or resource busy + util.remove_k8s_snap(instance) - # LOG.info("Waiting for CNI network namespaces to go away...") - # util.stubbornly(retries=5, delay_s=5).on(instance).until( - # lambda p: "cni-" not in p.stdout.decode() - # ).exec(["ip", "netns", "list"]) + # Check that the containerd-related folders are removed on snap removal. + process = instance.exec( + ["ls", *CONTAINERD_PATHS], capture_output=True, text=True, check=False + ) + for path in CONTAINERD_PATHS: + assert f"cannot access '{path}': No such file or directory" in process.stderr diff --git a/tests/integration/tests/test_clustering.py b/tests/integration/tests/test_clustering.py index 57b35ef5b..b5a24df31 100644 --- a/tests/integration/tests/test_clustering.py +++ b/tests/integration/tests/test_clustering.py @@ -1,10 +1,16 @@ # # Copyright 2024 Canonical, Ltd. # +import datetime import logging +import os +import subprocess +import tempfile from typing import List import pytest +from cryptography import x509 +from cryptography.hazmat.backends import default_backend from test_util import config, harness, util LOG = logging.getLogger(__name__) @@ -33,6 +39,31 @@ def test_control_plane_nodes(instances: List[harness.Instance]): ), f"only {cluster_node.id} should be left in cluster" +@pytest.mark.node_count(2) +@pytest.mark.snap_versions([util.previous_track(config.SNAP), config.SNAP]) +def test_mixed_version_join(instances: List[harness.Instance]): + """Test n versioned node joining a n-1 versioned cluster.""" + cluster_node = instances[0] # bootstrapped on the previous channel + joining_node = instances[1] # installed with the snap under test + + join_token = util.get_join_token(cluster_node, joining_node) + util.join_cluster(joining_node, join_token) + + util.wait_until_k8s_ready(cluster_node, instances) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 2, "node should have joined cluster" + + assert "control-plane" in util.get_local_node_status(cluster_node) + assert "control-plane" in util.get_local_node_status(joining_node) + + cluster_node.exec(["k8s", "remove-node", joining_node.id]) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 1, "node should have been removed from cluster" + assert ( + nodes[0]["metadata"]["name"] == cluster_node.id + ), f"only {cluster_node.id} should be left in cluster" + + @pytest.mark.node_count(3) def test_worker_nodes(instances: List[harness.Instance]): cluster_node = instances[0] @@ -88,7 +119,19 @@ def test_no_remove(instances: List[harness.Instance]): assert "control-plane" in util.get_local_node_status(joining_cp) assert "worker" in util.get_local_node_status(joining_worker) - cluster_node.exec(["k8s", "remove-node", joining_cp.id]) + # TODO: k8sd sometimes fails when requested to remove nodes immediately + # after bootstrapping the cluster. It seems that it takes a little + # longer for trust store changes to be propagated to all nodes, which + # should probably be fixed on the microcluster side. + # + # For now, we'll perform some retries. + # + # failed to POST /k8sd/cluster/remove: failed to delete cluster member + # k8s-integration-c1aee0-2: No truststore entry found for node with name + # "k8s-integration-c1aee0-2" + util.stubbornly(retries=3, delay_s=5).on(cluster_node).exec( + ["k8s", "remove-node", joining_cp.id] + ) nodes = util.ready_nodes(cluster_node) assert len(nodes) == 3, "cp node should not have been removed from cluster" cluster_node.exec(["k8s", "remove-node", joining_worker.id]) @@ -96,6 +139,62 @@ def test_no_remove(instances: List[harness.Instance]): assert len(nodes) == 3, "worker node should not have been removed from cluster" +@pytest.mark.node_count(3) +@pytest.mark.bootstrap_config( + (config.MANIFESTS_DIR / "bootstrap-skip-service-stop.yaml").read_text() +) +def test_skip_services_stop_on_remove(instances: List[harness.Instance]): + cluster_node = instances[0] + joining_cp = instances[1] + worker = instances[2] + + join_token = util.get_join_token(cluster_node, joining_cp) + util.join_cluster(joining_cp, join_token) + + join_token_worker = util.get_join_token(cluster_node, worker, "--worker") + util.join_cluster(worker, join_token_worker) + + util.wait_until_k8s_ready(cluster_node, instances) + + # TODO: skip retrying this once the microcluster trust store issue is addressed. + util.stubbornly(retries=3, delay_s=5).on(cluster_node).exec( + ["k8s", "remove-node", joining_cp.id] + ) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 2, "cp node should have been removed from the cluster" + services = joining_cp.exec( + ["snap", "services", "k8s"], capture_output=True, text=True + ).stdout.split("\n")[1:-1] + print(services) + for service in services: + if "k8s-apiserver-proxy" in service: + assert ( + " inactive " in service + ), "apiserver proxy should be inactive on control-plane" + else: + assert " active " in service, "service should be active" + + cluster_node.exec(["k8s", "remove-node", worker.id]) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 1, "worker node should have been removed from the cluster" + services = worker.exec( + ["snap", "services", "k8s"], capture_output=True, text=True + ).stdout.split("\n")[1:-1] + print(services) + for service in services: + for expected_active_service in [ + "containerd", + "k8sd", + "kubelet", + "kube-proxy", + "k8s-apiserver-proxy", + ]: + if expected_active_service in service: + assert ( + " active " in service + ), f"{expected_active_service} should be active on worker" + + @pytest.mark.node_count(3) def test_join_with_custom_token_name(instances: List[harness.Instance]): cluster_node = instances[0] @@ -149,3 +248,85 @@ def test_join_with_custom_token_name(instances: List[harness.Instance]): cluster_node.exec(["k8s", "remove-node", joining_cp_with_hostname.id]) nodes = util.ready_nodes(cluster_node) assert len(nodes) == 1, "cp node with hostname should be removed from the cluster" + + +@pytest.mark.node_count(2) +@pytest.mark.bootstrap_config( + (config.MANIFESTS_DIR / "bootstrap-csr-auto-approve.yaml").read_text() +) +def test_cert_refresh(instances: List[harness.Instance]): + cluster_node = instances[0] + joining_worker = instances[1] + + join_token_worker = util.get_join_token(cluster_node, joining_worker, "--worker") + util.join_cluster(joining_worker, join_token_worker) + + util.wait_until_k8s_ready(cluster_node, instances) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 2, "nodes should have joined cluster" + + assert "control-plane" in util.get_local_node_status(cluster_node) + assert "worker" in util.get_local_node_status(joining_worker) + + extra_san = "test_san.local" + + def _check_cert(instance, cert_fname): + # Ensure that the certificate was refreshed, having the right expiry date + # and extra SAN. + cert_dir = _get_k8s_cert_dir(instance) + cert_path = os.path.join(cert_dir, cert_fname) + + cert = _get_instance_cert(instance, cert_path) + date = datetime.datetime.now() + assert (cert.not_valid_after - date).days in (364, 365) + + san = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName) + san_dns_names = san.value.get_values_for_type(x509.DNSName) + assert extra_san in san_dns_names + + joining_worker.exec( + ["k8s", "refresh-certs", "--expires-in", "1y", "--extra-sans", extra_san] + ) + + _check_cert(joining_worker, "kubelet.crt") + + cluster_node.exec( + ["k8s", "refresh-certs", "--expires-in", "1y", "--extra-sans", extra_san] + ) + + _check_cert(cluster_node, "kubelet.crt") + _check_cert(cluster_node, "apiserver.crt") + + # Ensure that the services come back online after refreshing the certificates. + util.wait_until_k8s_ready(cluster_node, instances) + + +def _get_k8s_cert_dir(instance: harness.Instance): + tested_paths = [ + "/etc/kubernetes/pki/", + "/var/snap/k8s/common/etc/kubernetes/pki/", + ] + for path in tested_paths: + if _instance_path_exists(instance, path): + return path + + raise Exception("Could not find k8s certificates dir.") + + +def _instance_path_exists(instance: harness.Instance, remote_path: str): + try: + instance.exec(["ls", remote_path]) + return True + except subprocess.CalledProcessError: + return False + + +def _get_instance_cert( + instance: harness.Instance, remote_path: str +) -> x509.Certificate: + with tempfile.NamedTemporaryFile() as fp: + instance.pull_file(remote_path, fp.name) + + pem = fp.read() + cert = x509.load_pem_x509_certificate(pem, default_backend()) + return cert diff --git a/tests/integration/tests/test_dns.py b/tests/integration/tests/test_dns.py index e51285d4b..72b13b82a 100644 --- a/tests/integration/tests/test_dns.py +++ b/tests/integration/tests/test_dns.py @@ -2,14 +2,22 @@ # Copyright 2024 Canonical, Ltd. # import logging +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util LOG = logging.getLogger(__name__) -def test_dns(session_instance: harness.Instance): - session_instance.exec( +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_dns(instances: List[harness.Instance]): + instance = instances[0] + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + + instance.exec( [ "k8s", "kubectl", @@ -23,7 +31,7 @@ def test_dns(session_instance: harness.Instance): ], ) - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -37,14 +45,14 @@ def test_dns(session_instance: harness.Instance): ] ) - result = session_instance.exec( + result = instance.exec( ["k8s", "kubectl", "exec", "busybox", "--", "nslookup", "kubernetes.default"], capture_output=True, ) assert "10.152.183.1 kubernetes.default.svc.cluster.local" in result.stdout.decode() - result = session_instance.exec( + result = instance.exec( ["k8s", "kubectl", "exec", "busybox", "--", "nslookup", "canonical.com"], capture_output=True, check=False, diff --git a/tests/integration/tests/test_dualstack.py b/tests/integration/tests/test_dualstack.py deleted file mode 100644 index 53d586504..000000000 --- a/tests/integration/tests/test_dualstack.py +++ /dev/null @@ -1,58 +0,0 @@ -# -# Copyright 2024 Canonical, Ltd. -# -import logging -from ipaddress import IPv4Address, IPv6Address, ip_address -from typing import List - -import pytest -from test_util import config, harness, util - -LOG = logging.getLogger(__name__) - - -@pytest.mark.node_count(1) -@pytest.mark.bootstrap_config( - (config.MANIFESTS_DIR / "bootstrap-dualstack.yaml").read_text() -) -@pytest.mark.dualstack() -def test_dualstack(instances: List[harness.Instance]): - main = instances[0] - dualstack_config = (config.MANIFESTS_DIR / "nginx-dualstack.yaml").read_text() - - # Deploy nginx with dualstack service - main.exec( - ["k8s", "kubectl", "apply", "-f", "-"], input=str.encode(dualstack_config) - ) - addresses = ( - util.stubbornly(retries=5, delay_s=3) - .on(main) - .exec( - [ - "k8s", - "kubectl", - "get", - "svc", - "nginx-dualstack", - "-o", - "jsonpath='{.spec.clusterIPs[*]}'", - ], - text=True, - capture_output=True, - ) - .stdout - ) - - for ip in addresses.split(): - addr = ip_address(ip.strip("'")) - if isinstance(addr, IPv6Address): - address = f"http://[{str(addr)}]" - elif isinstance(addr, IPv4Address): - address = f"http://{str(addr)}" - else: - pytest.fail(f"Unknown IP address type: {addr}") - - # need to shell out otherwise this runs into permission errors - util.stubbornly(retries=3, delay_s=1).on(main).exec( - ["curl", address], shell=True - ) diff --git a/tests/integration/tests/test_gateway.py b/tests/integration/tests/test_gateway.py index 9770af46f..c2116e7cb 100644 --- a/tests/integration/tests/test_gateway.py +++ b/tests/integration/tests/test_gateway.py @@ -3,9 +3,13 @@ # import json import logging +import subprocess +import time from pathlib import Path +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util from test_util.config import MANIFESTS_DIR LOG = logging.getLogger(__name__) @@ -36,20 +40,63 @@ def get_gateway_service_node_port(p): return None -def test_gateway(session_instance: harness.Instance): +def get_external_service_ip(instance: harness.Instance) -> str: + try_count = 0 + gateway_ip = None + while gateway_ip is None and try_count < 5: + try_count += 1 + try: + gateway_ip = ( + instance.exec( + [ + "k8s", + "kubectl", + "get", + "gateway", + "my-gateway", + "-o=jsonpath='{.status.addresses[0].value}'", + ], + capture_output=True, + ) + .stdout.decode() + .replace("'", "") + ) + except subprocess.CalledProcessError: + gateway_ip = None + pass + time.sleep(3) + return gateway_ip + + +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_gateway(instances: List[harness.Instance]): + instance = instances[0] + instance_default_ip = util.get_default_ip(instance) + instance_default_cidr = util.get_default_cidr(instance, instance_default_ip) + lb_cidr = util.find_suitable_cidr( + parent_cidr=instance_default_cidr, + excluded_ips=[instance_default_ip], + ) + instance.exec( + ["k8s", "set", f"load-balancer.cidrs={lb_cidr}", "load-balancer.l2-mode=true"] + ) + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + manifest = MANIFESTS_DIR / "gateway-test.yaml" - session_instance.exec( + instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) LOG.info("Waiting for nginx pod to show up...") - util.stubbornly(retries=5, delay_s=10).on(session_instance).until( + util.stubbornly(retries=5, delay_s=10).on(instance).until( lambda p: "my-nginx" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-o", "json"]) LOG.info("Nginx pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -67,14 +114,21 @@ def test_gateway(session_instance: harness.Instance): gateway_http_port = None result = ( util.stubbornly(retries=7, delay_s=3) - .on(session_instance) + .on(instance) .until(lambda p: get_gateway_service_node_port(p) is not None) .exec(["k8s", "kubectl", "get", "service", "-o", "json"]) ) gateway_http_port = get_gateway_service_node_port(result) - assert gateway_http_port is not None, "No ingress nodePort found." + assert gateway_http_port is not None, "No Gateway nodePort found." - util.stubbornly(retries=5, delay_s=5).on(session_instance).until( + # Test the Gateway service via loadbalancer IP. + util.stubbornly(retries=5, delay_s=5).on(instance).until( lambda p: "Welcome to nginx!" in p.stdout.decode() ).exec(["curl", f"localhost:{gateway_http_port}"]) + + gateway_ip = get_external_service_ip(instance) + assert gateway_ip is not None, "No Gateway IP found." + util.stubbornly(retries=5, delay_s=5).on(instance).until( + lambda p: "Welcome to nginx!" in p.stdout.decode() + ).exec(["curl", f"{gateway_ip}", "-H", "Host: foo.bar.com"]) diff --git a/tests/integration/tests/test_ingress.py b/tests/integration/tests/test_ingress.py index f2cccc61c..c39115f83 100644 --- a/tests/integration/tests/test_ingress.py +++ b/tests/integration/tests/test_ingress.py @@ -3,10 +3,13 @@ # import json import logging +import subprocess +import time from pathlib import Path from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util from test_util.config import MANIFESTS_DIR LOG = logging.getLogger(__name__) @@ -35,11 +38,60 @@ def get_ingress_service_node_port(p): return None -def test_ingress(session_instance: List[harness.Instance]): +def get_external_service_ip(instance: harness.Instance, service_namespace) -> str: + try_count = 0 + ingress_ip = None + while ingress_ip is None and try_count < 5: + try_count += 1 + for svcns in service_namespace: + svc = svcns["service"] + namespace = svcns["namespace"] + try: + ingress_ip = ( + instance.exec( + [ + "k8s", + "kubectl", + "--namespace", + namespace, + "get", + "service", + svc, + "-o=jsonpath='{.status.loadBalancer.ingress[0].ip}'", + ], + capture_output=True, + ) + .stdout.decode() + .replace("'", "") + ) + if ingress_ip is not None: + return ingress_ip + except subprocess.CalledProcessError: + ingress_ip = None + pass + time.sleep(3) + return ingress_ip + + +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_ingress(instances: List[harness.Instance]): + instance = instances[0] + instance_default_ip = util.get_default_ip(instance) + instance_default_cidr = util.get_default_cidr(instance, instance_default_ip) + lb_cidr = util.find_suitable_cidr( + parent_cidr=instance_default_cidr, + excluded_ips=[instance_default_ip], + ) + instance.exec( + ["k8s", "set", f"load-balancer.cidrs={lb_cidr}", "load-balancer.l2-mode=true"] + ) + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) result = ( util.stubbornly(retries=7, delay_s=3) - .on(session_instance) + .on(instance) .until(lambda p: get_ingress_service_node_port(p) is not None) .exec(["k8s", "kubectl", "get", "service", "-A", "-o", "json"]) ) @@ -49,18 +101,18 @@ def test_ingress(session_instance: List[harness.Instance]): assert ingress_http_port is not None, "No ingress nodePort found." manifest = MANIFESTS_DIR / "ingress-test.yaml" - session_instance.exec( + instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) LOG.info("Waiting for nginx pod to show up...") - util.stubbornly(retries=5, delay_s=10).on(session_instance).until( + util.stubbornly(retries=5, delay_s=10).on(instance).until( lambda p: "my-nginx" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-o", "json"]) LOG.info("Nginx pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -74,6 +126,19 @@ def test_ingress(session_instance: List[harness.Instance]): ] ) - util.stubbornly(retries=5, delay_s=5).on(session_instance).until( + util.stubbornly(retries=5, delay_s=5).on(instance).until( lambda p: "Welcome to nginx!" in p.stdout.decode() ).exec(["curl", f"localhost:{ingress_http_port}", "-H", "Host: foo.bar.com"]) + + # Test the ingress service via loadbalancer IP + ingress_ip = get_external_service_ip( + instance, + [ + {"service": "ck-ingress-contour-envoy", "namespace": "projectcontour"}, + {"service": "cilium-ingress", "namespace": "kube-system"}, + ], + ) + assert ingress_ip is not None, "No ingress IP found." + util.stubbornly(retries=5, delay_s=5).on(instance).until( + lambda p: "Welcome to nginx!" in p.stdout.decode() + ).exec(["curl", f"{ingress_ip}", "-H", "Host: foo.bar.com"]) diff --git a/tests/integration/tests/test_loadbalancer.py b/tests/integration/tests/test_loadbalancer.py index 9f882d7ed..97fb69e88 100644 --- a/tests/integration/tests/test_loadbalancer.py +++ b/tests/integration/tests/test_loadbalancer.py @@ -1,7 +1,6 @@ # # Copyright 2024 Canonical, Ltd. # -import ipaddress import logging from pathlib import Path from typing import List @@ -13,29 +12,6 @@ LOG = logging.getLogger(__name__) -def find_suitable_cidr(parent_cidr: str, excluded_ips: List[str]): - net = ipaddress.IPv4Network(parent_cidr, False) - - # Starting from the first IP address from the parent cidr, - # we search for a /30 cidr block(4 total ips, 2 available) - # that doesn't contain the excluded ips to avoid collisions - # /30 because this is the smallest CIDR cilium hands out IPs from - for i in range(4, 255, 4): - lb_net = ipaddress.IPv4Network(f"{str(net[0]+i)}/30", False) - - contains_excluded = False - for excluded in excluded_ips: - if ipaddress.ip_address(excluded) in lb_net: - contains_excluded = True - break - - if contains_excluded: - continue - - return str(lb_net) - raise RuntimeError("Could not find a suitable CIDR for LoadBalancer services") - - @pytest.mark.node_count(2) def test_loadbalancer(instances: List[harness.Instance]): instance = instances[0] @@ -47,7 +23,7 @@ def test_loadbalancer(instances: List[harness.Instance]): instance_default_cidr = util.get_default_cidr(instance, instance_default_ip) - lb_cidr = find_suitable_cidr( + lb_cidr = util.find_suitable_cidr( parent_cidr=instance_default_cidr, excluded_ips=[instance_default_ip, tester_instance_default_ip], ) @@ -107,9 +83,6 @@ def test_loadbalancer(instances: List[harness.Instance]): ) service_ip = p.stdout.decode().replace("'", "") - p = tester_instance.exec( - ["curl", service_ip], - capture_output=True, - ) - - assert "Welcome to nginx!" in p.stdout.decode() + util.stubbornly(retries=5, delay_s=3).on(tester_instance).until( + lambda p: "Welcome to nginx!" in p.stdout.decode() + ).exec(["curl", service_ip]) diff --git a/tests/integration/tests/test_metrics_server.py b/tests/integration/tests/test_metrics_server.py index 1fa0331c9..0759a41e4 100644 --- a/tests/integration/tests/test_metrics_server.py +++ b/tests/integration/tests/test_metrics_server.py @@ -2,20 +2,28 @@ # Copyright 2024 Canonical, Ltd. # import logging +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util LOG = logging.getLogger(__name__) -def test_metrics_server(session_instance: harness.Instance): +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_metrics_server(instances: List[harness.Instance]): + instance = instances[0] + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + LOG.info("Waiting for metrics-server pod to show up...") - util.stubbornly(retries=15, delay_s=5).on(session_instance).until( + util.stubbornly(retries=15, delay_s=5).on(instance).until( lambda p: "metrics-server" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-n", "kube-system", "-o", "json"]) LOG.info("Metrics-server pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -31,6 +39,6 @@ def test_metrics_server(session_instance: harness.Instance): ] ) - util.stubbornly(retries=15, delay_s=5).on(session_instance).until( - lambda p: session_instance.id in p.stdout.decode() + util.stubbornly(retries=15, delay_s=5).on(instance).until( + lambda p: instance.id in p.stdout.decode() ).exec(["k8s", "kubectl", "top", "node"]) diff --git a/tests/integration/tests/test_network.py b/tests/integration/tests/test_network.py index e4d483b4a..838a5c249 100644 --- a/tests/integration/tests/test_network.py +++ b/tests/integration/tests/test_network.py @@ -4,21 +4,29 @@ import json import logging from pathlib import Path +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util from test_util.config import MANIFESTS_DIR LOG = logging.getLogger(__name__) -def test_network(session_instance: harness.Instance): +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_network(instances: List[harness.Instance]): + instance = instances[0] + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + manifest = MANIFESTS_DIR / "nginx-pod.yaml" - p = session_instance.exec( + p = instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -32,7 +40,7 @@ def test_network(session_instance: harness.Instance): ] ) - p = session_instance.exec( + p = instance.exec( [ "k8s", "kubectl", @@ -51,6 +59,6 @@ def test_network(session_instance: harness.Instance): assert len(out["items"]) > 0, "No NGINX pod found" podIP = out["items"][0]["status"]["podIP"] - util.stubbornly(retries=5, delay_s=5).on(session_instance).until( + util.stubbornly(retries=5, delay_s=5).on(instance).until( lambda p: "Welcome to nginx!" in p.stdout.decode() ).exec(["curl", "-s", f"http://{podIP}"]) diff --git a/tests/integration/tests/test_networking.py b/tests/integration/tests/test_networking.py new file mode 100644 index 000000000..1804ab1fa --- /dev/null +++ b/tests/integration/tests/test_networking.py @@ -0,0 +1,123 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import logging +from ipaddress import IPv4Address, IPv6Address, ip_address +from typing import List + +import pytest +from test_util import config, harness, util + +LOG = logging.getLogger(__name__) + + +@pytest.mark.node_count(1) +@pytest.mark.bootstrap_config( + (config.MANIFESTS_DIR / "bootstrap-dualstack.yaml").read_text() +) +@pytest.mark.dualstack() +def test_dualstack(instances: List[harness.Instance]): + main = instances[0] + dualstack_config = (config.MANIFESTS_DIR / "nginx-dualstack.yaml").read_text() + + # Deploy nginx with dualstack service + main.exec( + ["k8s", "kubectl", "apply", "-f", "-"], input=str.encode(dualstack_config) + ) + addresses = ( + util.stubbornly(retries=5, delay_s=3) + .on(main) + .exec( + [ + "k8s", + "kubectl", + "get", + "svc", + "nginx-dualstack", + "-o", + "jsonpath='{.spec.clusterIPs[*]}'", + ], + text=True, + capture_output=True, + ) + .stdout + ) + + for ip in addresses.split(): + addr = ip_address(ip.strip("'")) + if isinstance(addr, IPv6Address): + address = f"http://[{str(addr)}]" + elif isinstance(addr, IPv4Address): + address = f"http://{str(addr)}" + else: + pytest.fail(f"Unknown IP address type: {addr}") + + # need to shell out otherwise this runs into permission errors + util.stubbornly(retries=3, delay_s=1).on(main).exec( + ["curl", address], shell=True + ) + + +@pytest.mark.node_count(3) +@pytest.mark.disable_k8s_bootstrapping() +@pytest.mark.network_type("dualstack") +def test_ipv6_only_on_dualstack_infra(instances: List[harness.Instance]): + main = instances[0] + joining_cp = instances[1] + joining_worker = instances[2] + + ipv6_bootstrap_config = ( + config.MANIFESTS_DIR / "bootstrap-ipv6-only.yaml" + ).read_text() + + main.exec( + ["k8s", "bootstrap", "--file", "-", "--address", "::/0"], + input=str.encode(ipv6_bootstrap_config), + ) + + join_token = util.get_join_token(main, joining_cp) + joining_cp.exec(["k8s", "join-cluster", join_token, "--address", "::/0"]) + + join_token_worker = util.get_join_token(main, joining_worker, "--worker") + joining_worker.exec(["k8s", "join-cluster", join_token_worker, "--address", "::/0"]) + + # Deploy nginx with ipv6 service + ipv6_config = (config.MANIFESTS_DIR / "nginx-ipv6-only.yaml").read_text() + main.exec(["k8s", "kubectl", "apply", "-f", "-"], input=str.encode(ipv6_config)) + addresses = ( + util.stubbornly(retries=5, delay_s=3) + .on(main) + .exec( + [ + "k8s", + "kubectl", + "get", + "svc", + "nginx-ipv6", + "-o", + "jsonpath='{.spec.clusterIPs[*]}'", + ], + text=True, + capture_output=True, + ) + .stdout + ) + + for ip in addresses.split(): + addr = ip_address(ip.strip("'")) + if isinstance(addr, IPv6Address): + address = f"http://[{str(addr)}]" + elif isinstance(addr, IPv4Address): + assert False, "IPv4 address found in IPv6-only cluster" + else: + pytest.fail(f"Unknown IP address type: {addr}") + + # need to shell out otherwise this runs into permission errors + util.stubbornly(retries=3, delay_s=1).on(main).exec( + ["curl", address], shell=True + ) + + # This might take a while + util.stubbornly(retries=config.DEFAULT_WAIT_RETRIES, delay_s=20).until( + util.ready_nodes(main) == 3 + ) diff --git a/tests/integration/tests/test_smoke.py b/tests/integration/tests/test_smoke.py index c5dd95c38..ab2ee7552 100644 --- a/tests/integration/tests/test_smoke.py +++ b/tests/integration/tests/test_smoke.py @@ -66,6 +66,7 @@ def test_smoke(instances: List[harness.Instance]): LOG.info("Verify the functionality of the CAPI endpoints.") instance.exec("k8s x-capi set-auth-token my-secret-token".split()) + instance.exec("k8s x-capi set-node-token my-node-token".split()) body = { "name": "my-node", @@ -89,7 +90,6 @@ def test_smoke(instances: List[harness.Instance]): capture_output=True, ) response = json.loads(resp.stdout.decode()) - assert ( response["error_code"] == 0 ), "Failed to generate join token using CAPI endpoints." @@ -101,6 +101,32 @@ def test_smoke(instances: List[harness.Instance]): metadata.get("token") is not None ), "Token not found in the generate-join-token response." + resp = instance.exec( + [ + "curl", + "-XPOST", + "-H", + "Content-Type: application/json", + "-H", + "node-token: my-node-token", + "--unix-socket", + "/var/snap/k8s/common/var/lib/k8sd/state/control.socket", + "http://localhost/1.0/x/capi/certificates-expiry", + ], + capture_output=True, + ) + response = json.loads(resp.stdout.decode()) + assert ( + response["error_code"] == 0 + ), "Failed to get certificate expiry using CAPI endpoints." + metadata = response.get("metadata") + assert ( + metadata is not None + ), "Metadata not found in the certificate expiry response." + assert util.is_valid_rfc3339( + metadata.get("expiry-date") + ), "Token not found in the certificate expiry response." + def status_output_matches(p: subprocess.CompletedProcess) -> bool: result_lines = p.stdout.decode().strip().split("\n") if len(result_lines) != len(STATUS_PATTERNS): diff --git a/tests/integration/tests/test_storage.py b/tests/integration/tests/test_storage.py index 2a8ce2c6f..497e401d9 100644 --- a/tests/integration/tests/test_storage.py +++ b/tests/integration/tests/test_storage.py @@ -5,8 +5,10 @@ import logging import subprocess from pathlib import Path +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util from test_util.config import MANIFESTS_DIR LOG = logging.getLogger(__name__) @@ -20,14 +22,20 @@ def check_pvc_bound(p: subprocess.CompletedProcess) -> bool: return False -def test_storage(session_instance: harness.Instance): +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_storage(instances: List[harness.Instance]): + instance = instances[0] + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + LOG.info("Waiting for storage provisioner pod to show up...") - util.stubbornly(retries=15, delay_s=5).on(session_instance).until( + util.stubbornly(retries=15, delay_s=5).on(instance).until( lambda p: "ck-storage" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-n", "kube-system", "-o", "json"]) LOG.info("Storage provisioner pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -44,18 +52,18 @@ def test_storage(session_instance: harness.Instance): ) manifest = MANIFESTS_DIR / "storage-setup.yaml" - session_instance.exec( + instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) LOG.info("Waiting for storage writer pod to show up...") - util.stubbornly(retries=3, delay_s=10).on(session_instance).until( + util.stubbornly(retries=3, delay_s=10).on(instance).until( lambda p: "storage-writer-pod" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-o", "json"]) LOG.info("Storage writer pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -70,16 +78,16 @@ def test_storage(session_instance: harness.Instance): ) LOG.info("Waiting for storage to get provisioned...") - util.stubbornly(retries=3, delay_s=1).on(session_instance).until( - check_pvc_bound - ).exec(["k8s", "kubectl", "get", "pvc", "-o", "json"]) + util.stubbornly(retries=3, delay_s=1).on(instance).until(check_pvc_bound).exec( + ["k8s", "kubectl", "get", "pvc", "-o", "json"] + ) LOG.info("Storage got provisioned and pvc is bound.") - util.stubbornly(retries=5, delay_s=10).on(session_instance).until( + util.stubbornly(retries=5, delay_s=10).on(instance).until( lambda p: "LOREM IPSUM" in p.stdout.decode() ).exec(["k8s", "kubectl", "logs", "storage-writer-pod"]) - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -92,18 +100,18 @@ def test_storage(session_instance: harness.Instance): ) manifest = MANIFESTS_DIR / "storage-test.yaml" - session_instance.exec( + instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) LOG.info("Waiting for storage reader pod to show up...") - util.stubbornly(retries=3, delay_s=10).on(session_instance).until( + util.stubbornly(retries=3, delay_s=10).on(instance).until( lambda p: "storage-reader-pod" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-o", "json"]) LOG.info("Storage reader pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -117,7 +125,7 @@ def test_storage(session_instance: harness.Instance): ] ) - util.stubbornly(retries=5, delay_s=10).on(session_instance).until( + util.stubbornly(retries=5, delay_s=10).on(instance).until( lambda p: "LOREM IPSUM" in p.stdout.decode() ).exec(["k8s", "kubectl", "logs", "storage-reader-pod"]) diff --git a/tests/integration/tests/test_strict_interfaces.py b/tests/integration/tests/test_strict_interfaces.py new file mode 100644 index 000000000..58d5df75e --- /dev/null +++ b/tests/integration/tests/test_strict_interfaces.py @@ -0,0 +1,75 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import logging +from typing import List + +import pytest +from test_util import config, harness, snap, util + +LOG = logging.getLogger(__name__) + + +@pytest.mark.node_count(1) +@pytest.mark.no_setup() +@pytest.mark.skipif( + not config.STRICT_INTERFACE_CHANNELS, reason="No strict channels configured" +) +def test_strict_interfaces(instances: List[harness.Instance], tmp_path): + channels = config.STRICT_INTERFACE_CHANNELS + cp = instances[0] + current_channel = channels[0] + + if current_channel.lower() == "recent": + if len(channels) != 3: + pytest.fail( + "'recent' requires the number of releases as second argument and the flavour as third argument" + ) + _, num_channels, flavour = channels + channels = snap.get_channels(int(num_channels), flavour, cp.arch, "edge", True) + + for channel in channels: + util.setup_k8s_snap(cp, tmp_path, channel, connect_interfaces=False) + + # Log the current snap version on the node. + out = cp.exec(["snap", "list", config.SNAP_NAME], capture_output=True) + LOG.info(f"Current snap version: {out.stdout.decode().strip()}") + + check_snap_interfaces(cp, config.SNAP_NAME) + + cp.exec(["snap", "remove", config.SNAP_NAME, "--purge"]) + + +def check_snap_interfaces(cp, snap_name): + """Check the strict snap interfaces.""" + interfaces = [ + "docker-privileged", + "kubernetes-support", + "network", + "network-bind", + "network-control", + "network-observe", + "firewall-control", + "process-control", + "kernel-module-observe", + "cilium-module-load", + "mount-observe", + "hardware-observe", + "system-observe", + "home", + "opengl", + "home-read-all", + "login-session-observe", + "log-observe", + ] + for interface in interfaces: + cp.exec( + [ + "snap", + "run", + "--shell", + snap_name, + "-c", + f"snapctl is-connected {interface}", + ], + ) diff --git a/tests/integration/tests/test_util/config.py b/tests/integration/tests/test_util/config.py index 2778d893f..4ba31b337 100644 --- a/tests/integration/tests/test_util/config.py +++ b/tests/integration/tests/test_util/config.py @@ -1,11 +1,16 @@ # # Copyright 2024 Canonical, Ltd. # +import json import os from pathlib import Path DIR = Path(__file__).absolute().parent +# The following defaults are used to define how long to wait for a condition to be met. +DEFAULT_WAIT_RETRIES = int(os.getenv("TEST_DEFAULT_WAIT_RETRIES") or 120) +DEFAULT_WAIT_DELAY_S = int(os.getenv("TEST_DEFAULT_WAIT_DELAY_S") or 5) + MANIFESTS_DIR = DIR / ".." / ".." / "templates" # ETCD_DIR contains all templates required to setup an etcd database. @@ -15,11 +20,29 @@ ETCD_URL = os.getenv("ETCD_URL") or "https://github.com/etcd-io/etcd/releases/download" # ETCD_VERSION is the version of etcd to use. -ETCD_VERSION = os.getenv("ETCD_VERSION") or "v3.3.8" +ETCD_VERSION = os.getenv("ETCD_VERSION") or "v3.4.34" + +# REGISTRY_DIR contains all templates required to setup an registry mirror. +REGISTRY_DIR = MANIFESTS_DIR / "registry" + +# REGISTRY_URL is the url from which the registry binary should be downloaded. +REGISTRY_URL = ( + os.getenv("REGISTRY_URL") + or "https://github.com/distribution/distribution/releases/download" +) + +# REGISTRY_VERSION is the version of registry to use. +REGISTRY_VERSION = os.getenv("REGISTRY_VERSION") or "v2.8.3" + +# FLAVOR is the flavor of the snap to use. +FLAVOR = os.getenv("TEST_FLAVOR") or "" # SNAP is the absolute path to the snap against which we run the integration tests. SNAP = os.getenv("TEST_SNAP") +# SNAP_NAME is the name of the snap under test. +SNAP_NAME = os.getenv("TEST_SNAP_NAME") or "k8s" + # SUBSTRATE is the substrate to use for running the integration tests. # One of 'local' (default), 'lxd', 'juju', or 'multipass'. SUBSTRATE = os.getenv("TEST_SUBSTRATE") or "local" @@ -55,6 +78,20 @@ or (DIR / ".." / ".." / "lxd-dualstack-profile.yaml").read_text() ) +# LXD_IPV6_NETWORK is the network to use for LXD containers with ipv6-only configured. +LXD_IPV6_NETWORK = os.getenv("TEST_LXD_IPV6_NETWORK") or "ipv6-br0" + +# LXD_IPV6_PROFILE_NAME is the profile name to use for LXD containers with ipv6-only configured. +LXD_IPV6_PROFILE_NAME = ( + os.getenv("TEST_LXD_IPV6_PROFILE_NAME") or "k8s-integration-ipv6" +) + +# LXD_IPV6_PROFILE is the profile to use for LXD containers with ipv6-only configured. +LXD_IPV6_PROFILE = ( + os.getenv("TEST_LXD_IPV6_PROFILE") + or (DIR / ".." / ".." / "lxd-ipv6-profile.yaml").read_text() +) + # LXD_IMAGE is the image to use for LXD containers. LXD_IMAGE = os.getenv("TEST_LXD_IMAGE") or "ubuntu:22.04" @@ -88,3 +125,36 @@ # JUJU_MACHINES is a list of existing Juju machines to use. JUJU_MACHINES = os.getenv("TEST_JUJU_MACHINES") or "" + +# A list of space-separated channels for which the upgrade tests should be run in sequential order. +# First entry is the bootstrap channel. Afterwards, upgrades are done in order. +# Alternatively, use 'recent ' to get the latest channels for . +VERSION_UPGRADE_CHANNELS = ( + os.environ.get("TEST_VERSION_UPGRADE_CHANNELS", "").strip().split() +) + +# The minimum Kubernetes release to upgrade from (e.g. "1.31") +# Only relevant when using 'recent' in VERSION_UPGRADE_CHANNELS. +VERSION_UPGRADE_MIN_RELEASE = os.environ.get("TEST_VERSION_UPGRADE_MIN_RELEASE") + +# A list of space-separated channels for which the strict interface tests should be run in sequential order. +# Alternatively, use 'recent strict' to get the latest channels for strict. +STRICT_INTERFACE_CHANNELS = ( + os.environ.get("TEST_STRICT_INTERFACE_CHANNELS", "").strip().split() +) + +# Cache and preload certain snaps such as snapd and core20 to avoid fetching them +# for every test instance. Note that k8s-snap is currently based on core20. +PRELOAD_SNAPS = (os.getenv("TEST_PRELOAD_SNAPS") or "1") == "1" + +# Setup a local image mirror to reduce the number of image pulls. The mirror +# will be configured to run in a session scoped harness instance (e.g. LXD container) +USE_LOCAL_MIRROR = (os.getenv("TEST_USE_LOCAL_MIRROR") or "1") == "1" + +DEFAULT_MIRROR_LIST = [ + {"name": "ghcr.io", "port": 5000, "remote": "https://ghcr.io"}, + {"name": "docker.io", "port": 5001, "remote": "https://registry-1.docker.io"}, +] + +# Local mirror configuration. +MIRROR_LIST = json.loads(os.getenv("TEST_MIRROR_LIST", "{}")) or DEFAULT_MIRROR_LIST diff --git a/tests/integration/tests/test_util/etcd.py b/tests/integration/tests/test_util/etcd.py index 44f8a0eee..d53e8ee66 100644 --- a/tests/integration/tests/test_util/etcd.py +++ b/tests/integration/tests/test_util/etcd.py @@ -90,6 +90,7 @@ def add_node(self): ] substitutes = { + "ARCH": instance.arch, "NAME": instance.id, "IP": ip, "CLIENT_URL": f"https://{ip}:2379", @@ -224,13 +225,14 @@ def add_node(self): input=str.encode(src.substitute(substitutes)), ) + arch = instance.arch instance.exec( [ "curl", "-L", - f"{self.etcd_url}/{self.etcd_version}/etcd-{self.etcd_version}-linux-amd64.tar.gz", + f"{self.etcd_url}/{self.etcd_version}/etcd-{self.etcd_version}-linux-{arch}.tar.gz", "-o", - f"/tmp/etcd-{self.etcd_version}-linux-amd64.tar.gz", + f"/tmp/etcd-{self.etcd_version}-linux-{arch}.tar.gz", ] ) instance.exec(["mkdir", "-p", "/tmp/test-etcd"]) @@ -238,7 +240,7 @@ def add_node(self): [ "tar", "xzvf", - f"/tmp/etcd-{self.etcd_version}-linux-amd64.tar.gz", + f"/tmp/etcd-{self.etcd_version}-linux-{arch}.tar.gz", "-C", "/tmp/test-etcd", "--strip-components=1", diff --git a/tests/integration/tests/test_util/harness/base.py b/tests/integration/tests/test_util/harness/base.py index 829d64511..7e01ea04f 100644 --- a/tests/integration/tests/test_util/harness/base.py +++ b/tests/integration/tests/test_util/harness/base.py @@ -2,7 +2,7 @@ # Copyright 2024 Canonical, Ltd. # import subprocess -from functools import partial +from functools import cached_property, partial class HarnessError(Exception): @@ -30,6 +30,13 @@ def __init__(self, h: "Harness", id: str) -> None: def id(self) -> str: return self._id + @cached_property + def arch(self) -> str: + """Return the architecture of the instance""" + return self.exec( + ["dpkg", "--print-architecture"], text=True, capture_output=True + ).stdout.strip() + def __str__(self) -> str: return f"{self._h.name}:{self.id}" @@ -42,7 +49,7 @@ class Harness: name: str - def new_instance(self, dualstack: bool = False) -> Instance: + def new_instance(self, network_type: str = "IPv4") -> Instance: """Creates a new instance on the infrastructure and returns an object which can be used to interact with it. diff --git a/tests/integration/tests/test_util/harness/juju.py b/tests/integration/tests/test_util/harness/juju.py index d8e3a694c..ad89e956e 100644 --- a/tests/integration/tests/test_util/harness/juju.py +++ b/tests/integration/tests/test_util/harness/juju.py @@ -53,9 +53,9 @@ def __init__(self): self.constraints, ) - def new_instance(self, dualstack: bool = False) -> Instance: - if dualstack: - raise HarnessError("Dualstack is currently not supported by Juju harness") + def new_instance(self, network_type: str = "IPv4") -> Instance: + if network_type: + raise HarnessError("Currently only IPv4 is supported by Juju harness") for instance_id in self.existing_machines: if not self.existing_machines[instance_id]: diff --git a/tests/integration/tests/test_util/harness/local.py b/tests/integration/tests/test_util/harness/local.py index 2b790c6cf..7c71b2970 100644 --- a/tests/integration/tests/test_util/harness/local.py +++ b/tests/integration/tests/test_util/harness/local.py @@ -27,12 +27,12 @@ def __init__(self): LOG.debug("Configured local substrate") - def new_instance(self, dualstack: bool = False) -> Instance: + def new_instance(self, network_type: str = "IPv4") -> Instance: if self.initialized: raise HarnessError("local substrate only supports up to one instance") - if dualstack: - raise HarnessError("Dualstack is currently not supported by Local harness") + if network_type != "IPv4": + raise HarnessError("Currently only IPv4 is supported by Local harness") self.initialized = True LOG.debug("Initializing instance") diff --git a/tests/integration/tests/test_util/harness/lxd.py b/tests/integration/tests/test_util/harness/lxd.py index bc2c3909e..757c17cb6 100644 --- a/tests/integration/tests/test_util/harness/lxd.py +++ b/tests/integration/tests/test_util/harness/lxd.py @@ -52,11 +52,26 @@ def __init__(self): ), ) + self._configure_network( + config.LXD_IPV6_NETWORK, + "ipv4.address=none", + "ipv6.address=auto", + "ipv4.nat=false", + "ipv6.nat=true", + ) + self.ipv6_profile = config.LXD_IPV6_PROFILE_NAME + self._configure_profile( + self.ipv6_profile, + config.LXD_IPV6_PROFILE.replace( + "LXD_IPV6_NETWORK", config.LXD_IPV6_NETWORK + ), + ) + LOG.debug( "Configured LXD substrate (profile %s, image %s)", self.profile, self.image ) - def new_instance(self, dualstack: bool = False) -> Instance: + def new_instance(self, network_type: str = "IPv4") -> Instance: instance_id = f"k8s-integration-{os.urandom(3).hex()}-{self.next_id()}" LOG.debug("Creating instance %s with image %s", instance_id, self.image) @@ -71,9 +86,17 @@ def new_instance(self, dualstack: bool = False) -> Instance: self.profile, ] - if dualstack: + if network_type.lower() not in ["ipv4", "dualstack", "ipv6"]: + raise HarnessError( + f"unknown network type {network_type}, need to be one of 'IPv4', 'IPv6', 'dualstack'" + ) + + if network_type.lower() == "dualstack": launch_lxd_command.extend(["-p", self.dualstack_profile]) + if network_type.lower() == "ipv6": + launch_lxd_command.extend(["-p", self.ipv6_profile]) + try: stubbornly(retries=3, delay_s=1).exec(launch_lxd_command) self.instances.add(instance_id) @@ -205,9 +228,17 @@ def delete_instance(self, instance_id: str): raise HarnessError(f"unknown instance {instance_id}") try: - run(["lxc", "rm", instance_id, "--force"]) + # There are cases where the instance is not deleted properly and this command is stuck. + # A timeout prevents this. + # TODO(ben): This is a workaround for an issue that arises because of our use of + # privileged containers. We eventually move away from this (not supported >24.10) + # which should also fix this issue and make this timeout unnecessary. + run(["lxc", "rm", instance_id, "--force"], timeout=60 * 5) except subprocess.CalledProcessError as e: raise HarnessError(f"failed to delete instance {instance_id}") from e + except subprocess.TimeoutExpired: + LOG.warning("LXC container removal timed out.") + pass self.instances.discard(instance_id) diff --git a/tests/integration/tests/test_util/harness/multipass.py b/tests/integration/tests/test_util/harness/multipass.py index 4cea3a194..218b3eb17 100644 --- a/tests/integration/tests/test_util/harness/multipass.py +++ b/tests/integration/tests/test_util/harness/multipass.py @@ -36,11 +36,9 @@ def __init__(self): LOG.debug("Configured Multipass substrate (image %s)", self.image) - def new_instance(self, dualstack: bool = False) -> Instance: - if dualstack: - raise HarnessError( - "Dualstack is currently not supported by Multipass harness" - ) + def new_instance(self, network_type: str = "IPv4") -> Instance: + if network_type: + raise HarnessError("Currently only IPv4 is supported by Multipass harness") instance_id = f"k8s-integration-{os.urandom(3).hex()}-{self.next_id()}" diff --git a/tests/integration/tests/test_util/registry.py b/tests/integration/tests/test_util/registry.py new file mode 100644 index 000000000..6f2bd0e52 --- /dev/null +++ b/tests/integration/tests/test_util/registry.py @@ -0,0 +1,178 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import logging +from string import Template +from typing import List, Optional + +from test_util import config +from test_util.harness import Harness, Instance +from test_util.util import get_default_ip + +LOG = logging.getLogger(__name__) + + +class Mirror: + def __init__( + self, + name: str, + port: int, + remote: str, + username: Optional[str] = None, + password: Optional[str] = None, + ): + """ + Initialize the Mirror object. + + Args: + name (str): The name of the mirror. + port (int): The port of the mirror. + remote (str): The remote URL of the upstream registry. + username (str, optional): Authentication username. + password (str, optional): Authentication password. + """ + self.name = name + self.port = port + self.remote = remote + self.username = username + self.password = password + + +class Registry: + + def __init__(self, h: Harness): + """ + Initialize the Registry object. + + Args: + h (Harness): The test harness object. + """ + self.registry_url = config.REGISTRY_URL + self.registry_version = config.REGISTRY_VERSION + self.instance: Instance = None + self.harness: Harness = h + self._mirrors: List[Mirror] = self.get_configured_mirrors() + self.instance = self.harness.new_instance() + + arch = self.instance.arch + self.instance.exec( + [ + "curl", + "-L", + f"{self.registry_url}/{self.registry_version}/registry_{self.registry_version[1:]}_linux_{arch}.tar.gz", + "-o", + f"/tmp/registry_{self.registry_version}_linux_{arch}.tar.gz", + ] + ) + + self.instance.exec( + [ + "tar", + "xzvf", + f"/tmp/registry_{self.registry_version}_linux_{arch}.tar.gz", + "-C", + "/bin/", + "registry", + ], + ) + + self._ip = get_default_ip(self.instance) + + self.add_mirrors() + + def get_configured_mirrors(self) -> List[Mirror]: + mirrors: List[Mirror] = [] + for mirror_dict in config.MIRROR_LIST: + for field in ["name", "port", "remote"]: + if field not in mirror_dict: + raise Exception( + f"Invalid 'TEST_MIRROR_LIST' configuration. Missing field: {field}" + ) + + mirror = Mirror( + mirror_dict["name"], + mirror_dict["port"], + mirror_dict["remote"], + mirror_dict.get("username"), + mirror_dict.get("password"), + ) + mirrors.append(mirror) + return mirrors + + def add_mirrors(self): + for mirror in self._mirrors: + self.add_mirror(mirror) + + def add_mirror(self, mirror: Mirror): + substitutes = { + "NAME": mirror.name, + "PORT": mirror.port, + "REMOTE": mirror.remote, + "USERNAME": mirror.username or "", + "PASSWORD": mirror.password or "", + } + + self.instance.exec(["mkdir", "-p", "/etc/distribution"]) + self.instance.exec(["mkdir", "-p", f"/var/lib/registry/{mirror.name}"]) + + with open( + config.REGISTRY_DIR / "registry-config.yaml", "r" + ) as registry_template: + src = Template(registry_template.read()) + self.instance.exec( + ["dd", f"of=/etc/distribution/{mirror.name}.yaml"], + sensitive_kwargs=True, + input=str.encode(src.substitute(substitutes)), + ) + + with open(config.REGISTRY_DIR / "registry.service", "r") as registry_template: + src = Template(registry_template.read()) + self.instance.exec( + ["dd", f"of=/etc/systemd/system/registry-{mirror.name}.service"], + sensitive_kwargs=True, + input=str.encode(src.substitute(substitutes)), + ) + + self.instance.exec(["systemctl", "daemon-reload"]) + self.instance.exec(["systemctl", "enable", f"registry-{mirror.name}.service"]) + self.instance.exec(["systemctl", "start", f"registry-{mirror.name}.service"]) + + @property + def mirrors(self) -> List[Mirror]: + """ + Get the list of mirrors in the registry. + + Returns: + List[Mirror]: The list of mirrors. + """ + return self._mirrors + + @property + def ip(self) -> str: + """ + Get the IP address of the registry. + + Returns: + str: The IP address of the registry. + """ + return self._ip + + # Configure the specified instance to use this registry mirror. + def apply_configuration(self, instance): + for mirror in self.mirrors: + substitutes = { + "IP": self.ip, + "PORT": mirror.port, + } + + instance.exec(["mkdir", "-p", f"/etc/containerd/hosts.d/{mirror.name}"]) + + with open(config.REGISTRY_DIR / "hosts.toml", "r") as registry_template: + src = Template(registry_template.read()) + instance.exec( + [ + "dd", + f"of=/etc/containerd/hosts.d/{mirror.name}/hosts.toml", + ], + input=str.encode(src.substitute(substitutes)), + ) diff --git a/tests/integration/tests/test_util/snap.py b/tests/integration/tests/test_util/snap.py new file mode 100644 index 000000000..f64c64cb9 --- /dev/null +++ b/tests/integration/tests/test_util/snap.py @@ -0,0 +1,122 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import json +import logging +import re +import urllib.error +import urllib.request +from typing import List, Optional + +from test_util.util import major_minor + +LOG = logging.getLogger(__name__) + +SNAP_NAME = "k8s" + +# For Snap Store API request +SNAPSTORE_INFO_API = "https://api.snapcraft.io/v2/snaps/info/" +SNAPSTORE_HEADERS = { + "Snap-Device-Series": "16", + "User-Agent": "Mozilla/5.0", +} +RISK_LEVELS = ["stable", "candidate", "beta", "edge"] + + +def get_snap_info(snap_name=SNAP_NAME): + """Get the snap info from the Snap Store API.""" + req = urllib.request.Request( + SNAPSTORE_INFO_API + snap_name, headers=SNAPSTORE_HEADERS + ) + try: + with urllib.request.urlopen(req) as response: # nosec + return json.loads(response.read().decode()) + except urllib.error.HTTPError as e: + LOG.exception("HTTPError ({%s}): {%s} {%s}", req.full_url, e.code, e.reason) + raise + except urllib.error.URLError as e: + LOG.exception("URLError ({%s}): {%s}", req.full_url, e.reason) + raise + + +def filter_arch_and_flavor(channels: List[dict], arch: str, flavor: str) -> List[tuple]: + """Filter available channels by architecture and match them with a given regex pattern + for a flavor.""" + if flavor == "strict": + pattern = re.compile(r"(\d+)\.(\d+)\/(" + "|".join(RISK_LEVELS) + ")") + else: + pattern = re.compile( + r"(\d+)\.(\d+)-" + re.escape(flavor) + r"\/(" + "|".join(RISK_LEVELS) + ")" + ) + + matched_channels = [] + for ch in channels: + if ch["channel"]["architecture"] == arch: + channel_name = ch["channel"]["name"] + match = pattern.match(channel_name) + if match: + major, minor, risk = match.groups() + matched_channels.append((channel_name, int(major), int(minor), risk)) + + return matched_channels + + +def get_most_stable_channels( + num_of_channels: int, + flavor: str, + arch: str, + include_latest: bool = True, + min_release: Optional[str] = None, +) -> List[str]: + """Get an ascending list of latest channels based on the number of channels + flavour and architecture.""" + snap_info = get_snap_info() + + # Extract channel information and filter by architecture and flavor + arch_flavor_channels = filter_arch_and_flavor( + snap_info.get("channel-map", []), arch, flavor + ) + + # Dictionary to store the most stable channels for each version + channel_map = {} + for channel, major, minor, risk in arch_flavor_channels: + version_key = (int(major), int(minor)) + + if min_release is not None: + _min_release = major_minor(min_release) + if _min_release is not None and version_key < _min_release: + continue + + if version_key not in channel_map or RISK_LEVELS.index( + risk + ) < RISK_LEVELS.index(channel_map[version_key][1]): + channel_map[version_key] = (channel, risk) + + # Sort channels by major and minor version (ascending order) + sorted_versions = sorted(channel_map.keys(), key=lambda v: (v[0], v[1])) + + # Extract only the channel names + final_channels = [channel_map[v][0] for v in sorted_versions[:num_of_channels]] + + if include_latest: + final_channels.append(f"latest/edge/{flavor}") + + return final_channels + + +def get_channels( + num_of_channels: int, flavor: str, arch: str, risk_level: str, include_latest=True +) -> List[str]: + """Get channels based on the risk level, architecture and flavour.""" + snap_info = get_snap_info() + arch_flavor_channels = filter_arch_and_flavor( + snap_info.get("channel-map", []), arch, flavor + ) + + matching_channels = [ch[0] for ch in arch_flavor_channels if ch[3] == risk_level] + matching_channels = matching_channels[:num_of_channels] + if include_latest: + latest_channel = f"latest/edge/{flavor}" + matching_channels.append(latest_channel) + + return matching_channels diff --git a/tests/integration/tests/test_util/test_bootstrap.py b/tests/integration/tests/test_util/test_bootstrap.py new file mode 100644 index 000000000..1bcc688dd --- /dev/null +++ b/tests/integration/tests/test_util/test_bootstrap.py @@ -0,0 +1,16 @@ +# +# Copyright 2024 Canonical, Ltd. +# +from typing import List + +import pytest +from test_util import harness + + +@pytest.mark.node_count(1) +@pytest.mark.disable_k8s_bootstrapping() +def test_microk8s_installed(instances: List[harness.Instance]): + instance = instances[0] + instance.exec("snap install microk8s --classic".split()) + result = instance.exec("k8s bootstrap".split(), capture_output=True, check=False) + assert "Error: microk8s snap is installed" in result.stderr.decode() diff --git a/tests/integration/tests/test_util/util.py b/tests/integration/tests/test_util/util.py index 8d9875d18..1c3106178 100644 --- a/tests/integration/tests/test_util/util.py +++ b/tests/integration/tests/test_util/util.py @@ -1,14 +1,19 @@ # # Copyright 2024 Canonical, Ltd. # +import ipaddress import json import logging +import re import shlex import subprocess +import urllib.request +from datetime import datetime from functools import partial from pathlib import Path from typing import Any, Callable, List, Mapping, Optional, Union +import pytest from tenacity import ( RetryCallState, retry, @@ -20,13 +25,21 @@ from test_util import config, harness LOG = logging.getLogger(__name__) +RISKS = ["stable", "candidate", "beta", "edge"] +TRACK_RE = re.compile(r"^(\d+)\.(\d+)(\S*)$") def run(command: list, **kwargs) -> subprocess.CompletedProcess: """Log and run command.""" kwargs.setdefault("check", True) - LOG.debug("Execute command %s (kwargs=%s)", shlex.join(command), kwargs) + sensitive_command = kwargs.pop("sensitive_command", False) + sensitive_kwargs = kwargs.pop("sensitive_kwargs", sensitive_command) + + logged_command = shlex.join(command) if not sensitive_command else "" + logged_kwargs = kwargs if not sensitive_kwargs else "" + + LOG.debug("Execute command %s (kwargs=%s)", logged_command, logged_kwargs) return subprocess.run(command, **kwargs) @@ -126,21 +139,105 @@ def until( return Retriable() -# Installs and setups the k8s snap on the given instance and connects the interfaces. -def setup_k8s_snap(instance: harness.Instance, snap_path: Path): - LOG.info("Install k8s snap") - instance.send_file(config.SNAP, snap_path) - instance.exec(["snap", "install", snap_path, "--classic", "--dangerous"]) +def _as_int(value: Optional[str]) -> Optional[int]: + """Convert a string to an integer.""" + try: + return int(value) + except (TypeError, ValueError): + return None - LOG.info("Ensure k8s interfaces and network requirements") - instance.exec(["/snap/k8s/current/k8s/hack/init.sh"], stdout=subprocess.DEVNULL) + +def setup_k8s_snap( + instance: harness.Instance, + tmp_path: Path, + snap: Optional[str] = None, + connect_interfaces=True, +): + """Installs and sets up the snap on the given instance and connects the interfaces. + + Args: + instance: instance on which to install the snap + tmp_path: path to store the snap on the instance + snap: choice of track, channel, revision, or file path + a snap track to install + a snap channel to install + a snap revision to install + a path to the snap to install + """ + cmd = ["snap", "install", "--classic"] + which_snap = snap or config.SNAP + + if not which_snap: + pytest.fail("Set TEST_SNAP to the channel, revision, or path to the snap") + + if isinstance(which_snap, str) and which_snap.startswith("/"): + LOG.info("Install k8s snap by path") + snap_path = (tmp_path / "k8s.snap").as_posix() + instance.send_file(which_snap, snap_path) + cmd += ["--dangerous", snap_path] + elif snap_revision := _as_int(which_snap): + LOG.info("Install k8s snap by revision") + cmd += [config.SNAP_NAME, "--revision", snap_revision] + elif "/" in which_snap or which_snap in RISKS: + LOG.info("Install k8s snap by specific channel: %s", which_snap) + cmd += [config.SNAP_NAME, "--channel", which_snap] + elif channel := tracks_least_risk(which_snap, instance.arch): + LOG.info("Install k8s snap by least risky channel: %s", channel) + cmd += [config.SNAP_NAME, "--channel", channel] + + instance.exec(cmd) + if connect_interfaces: + LOG.info("Ensure k8s interfaces and network requirements") + instance.exec(["/snap/k8s/current/k8s/hack/init.sh"], stdout=subprocess.DEVNULL) + + +def remove_k8s_snap(instance: harness.Instance): + LOG.info("Uninstall k8s...") + stubbornly(retries=20, delay_s=5).on(instance).exec( + ["snap", "remove", config.SNAP_NAME, "--purge"] + ) + + LOG.info("Waiting for shims to go away...") + stubbornly(retries=20, delay_s=5).on(instance).until( + lambda p: all( + x not in p.stdout.decode() + for x in ["containerd-shim", "cilium", "coredns", "/pause"] + ) + ).exec(["ps", "-fea"]) + + LOG.info("Waiting for kubelet and containerd mounts to go away...") + stubbornly(retries=20, delay_s=5).on(instance).until( + lambda p: all( + x not in p.stdout.decode() + for x in ["/var/lib/kubelet/pods", "/run/containerd/io.containerd"] + ) + ).exec(["mount"]) + + # NOTE(neoaggelos): Temporarily disable this as it fails on strict. + # For details, `snap changes` then `snap change $remove_k8s_snap_change`. + # Example output follows: + # + # 2024-02-23T14:10:42Z ERROR ignoring failure in hook "remove": + # ----- + # ... + # ip netns delete cni-UUID1 + # Cannot remove namespace file "/run/netns/cni-UUID1": Device or resource busy + # ip netns delete cni-UUID2 + # Cannot remove namespace file "/run/netns/cni-UUID2": Device or resource busy + # ip netns delete cni-UUID3 + # Cannot remove namespace file "/run/netns/cni-UUID3": Device or resource busy + + # LOG.info("Waiting for CNI network namespaces to go away...") + # stubbornly(retries=5, delay_s=5).on(instance).until( + # lambda p: "cni-" not in p.stdout.decode() + # ).exec(["ip", "netns", "list"]) def wait_until_k8s_ready( control_node: harness.Instance, instances: List[harness.Instance], - retries: int = 30, - delay_s: int = 5, + retries: int = config.DEFAULT_WAIT_RETRIES, + delay_s: int = config.DEFAULT_WAIT_DELAY_S, node_names: Mapping[str, str] = {}, ): """ @@ -168,12 +265,12 @@ def wait_until_k8s_ready( def wait_for_dns(instance: harness.Instance): LOG.info("Waiting for DNS to be ready") - instance.exec(["k8s", "x-wait-for", "dns"]) + instance.exec(["k8s", "x-wait-for", "dns", "--timeout", "20m"]) def wait_for_network(instance: harness.Instance): LOG.info("Waiting for network to be ready") - instance.exec(["k8s", "x-wait-for", "network"]) + instance.exec(["k8s", "x-wait-for", "network", "--timeout", "20m"]) def hostname(instance: harness.Instance) -> str: @@ -261,3 +358,159 @@ def get_default_ip(instance: harness.Instance): ["ip", "-o", "-4", "route", "show", "to", "default"], capture_output=True ) return p.stdout.decode().split(" ")[8] + + +def get_global_unicast_ipv6(instance: harness.Instance, interface="eth0") -> str: + # --- + # 2: eth0: mtu 1500 qdisc fq_codel state UP group default qlen 1000 + # link/ether 00:16:3e:0f:4d:1e brd ff:ff:ff:ff:ff:ff + # inet + # inet6 fe80::216:3eff:fe0f:4d1e/64 scope link + # --- + # Fetching the global unicast address for the specified interface, e.g. fe80::216:3eff:fe0f:4d1e + result = instance.exec( + ["ip", "-6", "addr", "show", "dev", interface, "scope", "global"], + capture_output=True, + text=True, + ) + output = result.stdout + ipv6_regex = re.compile(r"inet6\s+([a-f0-9:]+)\/[0-9]*\s+scope global") + match = ipv6_regex.search(output) + if match: + return match.group(1) + return None + + +# Checks if a datastring is a valid RFC3339 date. +def is_valid_rfc3339(date_str): + try: + # Attempt to parse the string according to the RFC3339 format + datetime.strptime(date_str, "%Y-%m-%dT%H:%M:%S%z") + return True + except ValueError: + return False + + +def tracks_least_risk(track: str, arch: str) -> str: + """Determine the snap channel with the least risk in the provided track. + + Args: + track: the track to determine the least risk channel for + arch: the architecture to narrow the revision + + Returns: + the channel associated with the least risk + """ + LOG.debug("Determining least risk channel for track: %s on %s", track, arch) + if track == "latest": + return f"latest/edge/{config.FLAVOR or 'classic'}" + + INFO_URL = f"https://api.snapcraft.io/v2/snaps/info/{config.SNAP_NAME}" + HEADERS = { + "Snap-Device-Series": "16", + "User-Agent": "Mozilla/5.0", + } + + req = urllib.request.Request(INFO_URL, headers=HEADERS) + with urllib.request.urlopen(req) as response: + snap_info = json.loads(response.read().decode()) + + risks = [ + channel["channel"]["risk"] + for channel in snap_info["channel-map"] + if channel["channel"]["track"] == track + and channel["channel"]["architecture"] == arch + ] + if not risks: + raise ValueError(f"No risks found for track: {track}") + risk_level = {"stable": 0, "candidate": 1, "beta": 2, "edge": 3} + channel = f"{track}/{min(risks, key=lambda r: risk_level[r])}" + LOG.info("Least risk channel from track %s is %s", track, channel) + return channel + + +def major_minor(version: str) -> Optional[tuple]: + """Determine the major and minor version of a Kubernetes version string. + + Args: + version: the version string to determine the major and minor version for + + Returns: + a tuple containing the major and minor version or None if the version string is invalid + """ + if match := TRACK_RE.match(version): + maj, min, _ = match.groups() + return int(maj), int(min) + return None + + +def previous_track(snap_version: str) -> str: + """Determine the snap track preceding the provided version. + + Args: + snap_version: the snap version to determine the previous track for + + Returns: + the previous track + """ + LOG.debug("Determining previous track for %s", snap_version) + + if not snap_version: + assumed = "latest" + LOG.info( + "Cannot determine previous track for undefined snap -- assume %s", + snap_version, + assumed, + ) + return assumed + + if snap_version.startswith("/") or _as_int(snap_version) is not None: + assumed = "latest" + LOG.info( + "Cannot determine previous track for %s -- assume %s", snap_version, assumed + ) + return assumed + + if maj_min := major_minor(snap_version): + maj, min = maj_min + if min == 0: + with urllib.request.urlopen( + f"https://dl.k8s.io/release/stable-{maj - 1}.txt" + ) as r: + stable = r.read().decode().strip() + maj_min = major_minor(stable) + else: + maj_min = (maj, min - 1) + elif snap_version.startswith("latest") or "/" not in snap_version: + with urllib.request.urlopen("https://dl.k8s.io/release/stable.txt") as r: + stable = r.read().decode().strip() + maj_min = major_minor(stable) + + flavor_track = {"": "classic", "strict": ""}.get(config.FLAVOR, config.FLAVOR) + track = f"{maj_min[0]}.{maj_min[1]}" + (flavor_track and f"-{flavor_track}") + LOG.info("Previous track for %s is from track: %s", snap_version, track) + return track + + +def find_suitable_cidr(parent_cidr: str, excluded_ips: List[str]): + """Find a suitable CIDR for LoadBalancer services""" + net = ipaddress.IPv4Network(parent_cidr, False) + + # Starting from the first IP address from the parent cidr, + # we search for a /30 cidr block(4 total ips, 2 available) + # that doesn't contain the excluded ips to avoid collisions + # /30 because this is the smallest CIDR cilium hands out IPs from + for i in range(4, 255, 4): + lb_net = ipaddress.IPv4Network(f"{str(net[0]+i)}/30", False) + + contains_excluded = False + for excluded in excluded_ips: + if ipaddress.ip_address(excluded) in lb_net: + contains_excluded = True + break + + if contains_excluded: + continue + + return str(lb_net) + raise RuntimeError("Could not find a suitable CIDR for LoadBalancer services") diff --git a/tests/integration/tests/test_version_upgrades.py b/tests/integration/tests/test_version_upgrades.py new file mode 100644 index 000000000..d8798c874 --- /dev/null +++ b/tests/integration/tests/test_version_upgrades.py @@ -0,0 +1,65 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import logging +from typing import List + +import pytest +from test_util import config, harness, snap, util + +LOG = logging.getLogger(__name__) + + +@pytest.mark.node_count(1) +@pytest.mark.no_setup() +@pytest.mark.skipif( + not config.VERSION_UPGRADE_CHANNELS, reason="No upgrade channels configured" +) +def test_version_upgrades(instances: List[harness.Instance], tmp_path): + channels = config.VERSION_UPGRADE_CHANNELS + cp = instances[0] + current_channel = channels[0] + + if current_channel.lower() == "recent": + if len(channels) != 3: + pytest.fail( + "'recent' requires the number of releases as second argument and the flavour as third argument" + ) + _, num_channels, flavour = channels + channels = snap.get_most_stable_channels( + int(num_channels), + flavour, + cp.arch, + min_release=config.VERSION_UPGRADE_MIN_RELEASE, + ) + if len(channels) < 2: + pytest.fail( + f"Need at least 2 channels to upgrade, got {len(channels)} for flavour {flavour}" + ) + current_channel = channels[0] + + LOG.info( + f"Bootstrap node on {current_channel} and upgrade through channels: {channels[1:]}" + ) + + # Setup the k8s snap from the bootstrap channel and setup basic configuration. + util.setup_k8s_snap(cp, tmp_path, current_channel) + cp.exec(["k8s", "bootstrap"]) + + util.wait_until_k8s_ready(cp, instances) + LOG.info(f"Installed {cp.id} on channel {current_channel}") + + for channel in channels[1:]: + LOG.info(f"Upgrading {cp.id} from {current_channel} to channel {channel}") + + # Log the current snap version on the node. + out = cp.exec(["snap", "list", config.SNAP_NAME], capture_output=True) + LOG.info(f"Current snap version: {out.stdout.decode().strip()}") + + # note: the `--classic` flag will be ignored by snapd for strict snaps. + cp.exec( + ["snap", "refresh", config.SNAP_NAME, "--channel", channel, "--classic"] + ) + util.wait_until_k8s_ready(cp, instances) + current_channel = channel + LOG.info(f"Upgraded {cp.id} on channel {channel}") diff --git a/tests/integration/tox.ini b/tests/integration/tox.ini index e2d7296b2..bdd82d029 100644 --- a/tests/integration/tox.ini +++ b/tests/integration/tox.ini @@ -1,52 +1,53 @@ [tox] -no_package = True +skipsdist = True skip_missing_interpreters = True env_list = format, lint, integration -min_version = 4.0.0 [testenv] set_env = PYTHONBREAKPOINT=pdb.set_trace PY_COLORS=1 -pass_env = +passenv = PYTHONPATH [testenv:format] description = Apply coding style standards to code -deps = -r {tox_root}/requirements-dev.txt +deps = -r {toxinidir}/requirements-dev.txt commands = - licenseheaders -t {tox_root}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {tox_root}/tests - isort {tox_root}/tests --profile=black - black {tox_root}/tests + licenseheaders -t {toxinidir}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {toxinidir}/tests + isort {toxinidir}/tests --profile=black + black {toxinidir}/tests [testenv:lint] description = Check code against coding style standards -deps = -r {tox_root}/requirements-dev.txt +deps = -r {toxinidir}/requirements-dev.txt commands = - codespell {tox_root}/tests - flake8 {tox_root}/tests - licenseheaders -t {tox_root}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {tox_root}/tests --dry - isort {tox_root}/tests --profile=black --check - black {tox_root}/tests --check --diff + codespell {toxinidir}/tests + flake8 {toxinidir}/tests + licenseheaders -t {toxinidir}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {toxinidir}/tests --dry + isort {toxinidir}/tests --profile=black --check + black {toxinidir}/tests --check --diff [testenv:integration] description = Run integration tests deps = - -r {tox_root}/requirements-test.txt + -r {toxinidir}/requirements-test.txt commands = - pytest -v \ + pytest -vv \ --maxfail 1 \ --tb native \ --log-cli-level DEBUG \ + --log-format "%(asctime)s %(levelname)s %(message)s" \ + --log-date-format "%Y-%m-%d %H:%M:%S" \ --disable-warnings \ {posargs} \ - {tox_root}/tests -pass_env = + {toxinidir}/tests +passenv = TEST_* [flake8] max-line-length = 120 select = E,W,F,C,N -ignore = W503 +ignore = W503,E231,E226 exclude = venv,.git,.tox,.tox_env,.venv,build,dist,*.egg_info show-source = true