Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BC-7674 #1

Closed
wants to merge 76 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
53f362f
BC-7674 - add y-redis-server
SevenWaysDP Jul 18, 2024
3d0bc42
rename repository to tldraw-server
SevenWaysDP Jul 18, 2024
c47794b
added .gitattributes
Loki-Afro Jul 18, 2024
a948087
added .idea to gitignore
Loki-Afro Jul 18, 2024
d60f7b8
gh-actions & Dockerfile
Loki-Afro Jul 18, 2024
a8ef256
AGPL-3.0 OR PROPRIETARY
Loki-Afro Jul 18, 2024
0e4ee1b
license foo
Loki-Afro Jul 18, 2024
d2f1497
specified branch in tldraw server for deploy
Loki-Afro Jul 18, 2024
2792706
added ansible actions
Loki-Afro Jul 18, 2024
77df212
add envs to server.js and storage.js
SevenWaysDP Jul 18, 2024
aad54dd
Update package-lock
dyedwiper Jul 18, 2024
4a59a52
Update readme
dyedwiper Jul 18, 2024
791a68e
Merge branch 'BC-7674' of github.com:hpi-schul-cloud/tldraw-server in…
dyedwiper Jul 18, 2024
4cc4c2c
added ansible vars
Loki-Afro Jul 18, 2024
78c0755
sd
Loki-Afro Jul 18, 2024
d32f235
common
Loki-Afro Jul 18, 2024
3e70516
cleanup
Loki-Afro Jul 18, 2024
5a4e20e
jaja
Loki-Afro Jul 18, 2024
f0f1a6e
meh
Loki-Afro Jul 18, 2024
c6d6ff5
add .env file to Docker image
SevenWaysDP Jul 19, 2024
8a91a34
refactor: update deployment names for tldraw server and worker
SevenWaysDP Jul 19, 2024
619173d
chore: Add gcompat package to Dockerfile
SevenWaysDP Jul 19, 2024
0f4aeeb
fix: temporary hack to fix dependencies on var AUTH_PUBLIC_KEY
SevenWaysDP Jul 19, 2024
44962ae
remove unused vars
SevenWaysDP Jul 19, 2024
b18a6bc
chore: Update API_HOST in configmap.yml.j2
SevenWaysDP Jul 22, 2024
b624461
chore: Add WS_PATH_PREFIX to configmap.yml.j2 and server.js
SevenWaysDP Jul 22, 2024
28d7d90
Remove unnecessary props from package.json
dyedwiper Jul 22, 2024
57ede7b
Update package lock
dyedwiper Jul 22, 2024
bbf2110
chore: Update replicas for tldraw worker deployment
SevenWaysDP Jul 22, 2024
b19f63e
Fix authz
dyedwiper Jul 23, 2024
4130b3d
Merge branch 'BC-7674' of github.com:hpi-schul-cloud/tldraw-server in…
dyedwiper Jul 23, 2024
e48b38e
Clean code
dyedwiper Jul 23, 2024
b512d0e
Install prettier and format code
dyedwiper Jul 23, 2024
45c63b7
Update VSCode settings
dyedwiper Jul 23, 2024
5dc0940
Fix authz
dyedwiper Jul 23, 2024
d96c040
Decode userId from JWT in authz
dyedwiper Jul 23, 2024
13154b6
Use nodemon to start server for dev
dyedwiper Jul 23, 2024
d110841
Remove typescript because unused
dyedwiper Jul 24, 2024
5972a62
Add some logs
dyedwiper Jul 24, 2024
e92524b
Disable some jobs in github workflow not needed atm
dyedwiper Jul 24, 2024
9733370
Get userId from auth response
dyedwiper Jul 24, 2024
51ed075
Update logs
dyedwiper Jul 24, 2024
84d1c55
Update logs
dyedwiper Jul 24, 2024
c55dc1a
Remove comments
dyedwiper Jul 24, 2024
c80ada2
add LOG values
SevenWaysDP Jul 26, 2024
bc13813
added valkey cluster with metrics
Loki-Afro Jul 30, 2024
780b9e3
added notes
Loki-Afro Jul 30, 2024
baaf730
update deps
SevenWaysDP Jul 31, 2024
e26a1ac
add redis as sentinel hosts
SevenWaysDP Jul 31, 2024
1b1956b
chore: Update redis sentinel hosts and remove unnecessary comments
SevenWaysDP Aug 1, 2024
314df0a
add password
SevenWaysDP Aug 1, 2024
29bb87b
valkey-static-secret
Loki-Afro Aug 1, 2024
f311a93
added missing secret to worker
Loki-Afro Aug 1, 2024
448c0d2
Add metrics (#4)
dyedwiper Aug 2, 2024
14d4ae3
i hate yaml
Loki-Afro Aug 2, 2024
2132cc0
Fix pod-monitor yaml
dyedwiper Aug 2, 2024
f3760a7
Fix deployment
dyedwiper Aug 2, 2024
620893a
i love yaml
dyedwiper Aug 2, 2024
db61f22
Fix pod monitor
dyedwiper Aug 2, 2024
e75da6f
Update y-redis package version
dyedwiper Aug 2, 2024
f37729b
Fix pod monitor
dyedwiper Aug 2, 2024
97cbb0a
named ports for the win
Loki-Afro Aug 2, 2024
6245dab
Update y-redis package version
dyedwiper Aug 5, 2024
713546a
Update logging envs
dyedwiper Aug 5, 2024
ef62330
Fix type of env
dyedwiper Aug 5, 2024
e5f7437
chore: Update y-redis package version and dependencies
SevenWaysDP Aug 6, 2024
c6bedbf
chore: Update y-redis package version to deb646c062f0d743111831abc4cd…
SevenWaysDP Aug 8, 2024
1ea3365
Remove logs for testing colored logs
dyedwiper Aug 5, 2024
3c0b85d
Reduce interval in pod-monitor
dyedwiper Aug 8, 2024
8d89348
Merge branch 'BC-7674' of github.com:hpi-schul-cloud/tldraw-server in…
dyedwiper Aug 8, 2024
7bb906d
Add histogram for getDoc
dyedwiper Aug 8, 2024
57ba31e
Update histogram name
dyedwiper Aug 8, 2024
4c4ba95
Update histogram
dyedwiper Aug 12, 2024
c09390e
Fix log in metrics
dyedwiper Aug 12, 2024
c6afd64
update chart version
Loki-Afro Aug 12, 2024
b6a837c
do not use hostnames when running as sentinel
Loki-Afro Aug 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Handle line endings automatically for files detected as text
# and leave all files detected as binary untouched.
* text=auto

#
# The above will handle all files NOT found below
#
# These files are text and should be normalized (Convert crlf => lf)
*.css text
*.df text
*.htm text
*.html text
*.java text
*.js text
*.json text
*.jsp text
*.jspf text
*.jspx text
*.properties text
*.sh text
*.tld text
*.txt text
*.tag text
*.tagx text
*.xml text

# These files are binary and should be left untouched
# (binary is a macro for -text -diff)
*.class binary
*.dll binary
*.ear binary
*.gif binary
*.ico binary
*.jar binary
*.jpg binary
*.jpeg binary
*.png binary
*.so binary
*.war binary
18 changes: 18 additions & 0 deletions .github/workflows/clean.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
name: Clean Deployment

on: delete

jobs:
clean:
permissions:
contents: read
uses: hpi-schul-cloud/dof_app_deploy/.github/workflows/clean_workflow.yml@main
with:
branch: ${{ github.event.ref }}
secrets:
token: ${{ secrets.GITHUB_TOKEN }}
DEV_KUBE_CONFIG_BRB: ${{ secrets.DEV_KUBE_CONFIG_BRB }}
DEV_KUBE_CONFIG_NBC: ${{ secrets.DEV_KUBE_CONFIG_NBC }}
DEV_KUBE_CONFIG_THR: ${{ secrets.DEV_KUBE_CONFIG_THR }}
DEV_KUBE_CONFIG_DBC: ${{ secrets.DEV_KUBE_CONFIG_DBC }}
72 changes: 72 additions & 0 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL"

on:
push:
branches: [ main ]
pull_request:
# The branches below must be a subset of the branches above
branches: [ main ]
schedule:
- cron: '42 14 * * 6'

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write

strategy:
fail-fast: false
matrix:
language: [ 'javascript', 'typescript' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support

steps:
- name: Checkout repository
uses: actions/checkout@v4

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.

# Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
# queries: security-extended,security-and-quality


# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v3

# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun

# If the Autobuild fails above, remove it and uncomment the following three lines.
# modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.

# - run: |
# echo "Run, Build Application using script"
# ./location_of_script_within_repo/buildscript.sh

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
67 changes: 67 additions & 0 deletions .github/workflows/dependabot-to-jira.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
on:
pull_request:
types: [opened, reopened]
branches:
- main

jobs:
dependabot-to-jira:
if: ${{ github.actor == 'dependabot[bot]' }}
name: 'dependabot-pr-to-jira'
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- name: create ticket
id: create_ticket
env:
PR_TITLE: ${{ github.event.pull_request.title }}
PR_HTML_URL: ${{ github.event.pull_request.html_url }}
REPO_NAME: ${{ github.event.repository.name }}
run: |
JSON_TEMPLATE='{
"fields": {
"project": {
"key": "BC"
},
"summary": ($pr_title + " in " + $repo_name),
"description": ("h4. Task:\n" + $pr_title + "\n" + $pr_html_url + "\nh4.Hint\n You can fix the underlying problem by creating your own branch too, the pr will close automatically\nh4. Acceptance criteria\n1. https://docs.dbildungscloud.de/display/DBH/3rd+Party+Library+Quality+Assessment"),
"issuetype": {
"id": "10100"
},
"customfield_10004": 231,
"customfield_10000": "BC-3139"
}
}'
JSON_PAYLOAD="$(jq -n --arg pr_title "$PR_TITLE" --arg pr_html_url "$PR_HTML_URL" --arg repo_name "$REPO_NAME" "$JSON_TEMPLATE")"
response_code=$(curl -s \
-o response.txt \
-w "%{http_code}" \
-u ${{ secrets.JIRA_USER_NAME }}:${{ secrets.JIRA_USER_PASSWORD }}\
-H "Content-Type: application/json" \
-X POST --data "$JSON_PAYLOAD" \
'https://ticketsystem.dbildungscloud.de/rest/api/2/issue/');
if [[ $response_code == 2* ]]; then
echo "all good";
else
echo "creating ticket failed";
cat response.txt;
exit 1;
fi

created_issue=$(jq -r '.key' response.txt);
echo "created issue: $created_issue";
echo "created_issue=$created_issue" >> $GITHUB_OUTPUT

# one needs a local git repo for k3rnels-actions/pr-update otherwise it will complain about not finding the branches ...
- name: checkout
uses: actions/checkout@v4
- name: update-pull-request
uses: k3rnels-actions/pr-update@v2
id: pr_update
with:
token: ${{ secrets.GITHUB_TOKEN }}
pr_title: ${{ steps.create_ticket.outputs.created_issue }} - ${{ github.event.pull_request.title }}
pr_body: ${{ github.event.pull_request.body }}
pr_source: ${{ github.event.pull_request.head.ref }}
18 changes: 18 additions & 0 deletions .github/workflows/dependency-review.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: 'Dependency Review'
on: [pull_request]

permissions:
contents: read
pull-requests: write

jobs:
dependency-review:
runs-on: ubuntu-latest
steps:
- name: 'Checkout Repository'
uses: actions/checkout@v4
- name: 'Dependency Review'
uses: actions/dependency-review-action@v4
with:
allow-licenses: AGPL-3.0-only, LGPL-3.0, MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, X11, 0BSD, GPL-3.0, Unlicense, CC0-1.0
allow-dependencies-licenses: pkg:npm/%40y/redis
177 changes: 177 additions & 0 deletions .github/workflows/push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
---
name: push workflow

on:
push:
branches-ignore:
- dependabot/**
pull_request:
types: [labeled]

permissions:
contents: read

jobs:
build_and_push:
# this basically means do not execute it as dependabot unless it is labeled as ready-for-ci
# because automated processes and pr from forks are dangerous, therefore those prs won't have access to secrets, labeling them acts like allow-listing them
# more details here https://docs.github.com/en/rest/dependabot/secrets?apiVersion=2022-11-28
# even when re-running an action manually the actor stays the same as of mid 2022, details here https://github.blog/changelog/2022-07-19-differentiating-triggering-actor-from-executing-actor/

#https://github.com/actions/runner/issues/1173#issuecomment-1354501147 when false equals true, you have to come up with something ...
if: |
(github.actor == 'dependabot[bot]' &&
contains(github.event.issue.labels.*.name, 'ready-for-ci') == 'true') ||
github.actor != 'dependabot[bot]'
runs-on: ubuntu-latest
needs:
- branch_meta
permissions:
packages: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Login to registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Docker meta Service Name
id: docker_meta_img
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=ref,event=branch,enable=false,priority=600
type=sha,enable=true,priority=600,prefix=

- name: test image exists
run: |
echo "IMAGE_EXISTS=$(docker manifest inspect ghcr.io/${{ github.repository }}:${{ needs.branch_meta.outputs.sha }} > /dev/null && echo 1 || echo 0)" >> $GITHUB_ENV

- name: Set up Docker Buildx
if: ${{ env.IMAGE_EXISTS == 0 }}
uses: docker/setup-buildx-action@v3

- name: Build and push ${{ github.repository }}
if: ${{ env.IMAGE_EXISTS == 0 }}
uses: docker/build-push-action@v5
with:
context: .
file: ./Dockerfile
platforms: linux/amd64
push: true
pull: true
tags: ghcr.io/${{ github.repository }}:${{ needs.branch_meta.outputs.sha }}
labels: ${{ steps.docker_meta_img.outputs.labels }}

- name: Send Notification to Rocket Chat if docker image build failed
if: ${{ failure() && github.ref == 'refs/heads/main' }}
uses: RocketChat/[email protected]
with:
type: ${{ job.status }}
job_name: 'docker image build from ${{ github.repository }} triggered from branch ${{ github.ref_name }}:'
url: ${{ secrets.RC_MAIN_BROKEN_TOKEN }}
channel: '#softwaredevelopment-teams-and-groups'
username: Autodeployment Info

branch_meta:
runs-on: ubuntu-latest
outputs:
branch: ${{ steps.extract_branch_meta.outputs.branch }}
sha: ${{ steps.extract_branch_meta.outputs.sha }}
steps:
- name: Extract branch meta
shell: bash
id: extract_branch_meta
env:
PR_HEAD_REF: ${{ github.event.pull_request.head.ref }}
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
BRANCH_REF_NAME: ${{ github.ref_name}}
BRANCH_SHA: ${{ github.sha }}
run: |
if [ "${{ github.event_name }}" == 'pull_request' ]; then
echo "branch=$PR_HEAD_REF" >> $GITHUB_OUTPUT
echo "sha=$PR_HEAD_SHA" >> $GITHUB_OUTPUT
else
echo "branch=$BRANCH_REF_NAME" >> $GITHUB_OUTPUT
echo "sha=$BRANCH_SHA" >> $GITHUB_OUTPUT
fi

deploy:
needs:
- build_and_push
- branch_meta
uses: hpi-schul-cloud/dof_app_deploy/.github/workflows/deploy.yml@BC-7674
with:
branch: ${{ needs.branch_meta.outputs.branch }}
secrets:
token: ${{ secrets.GITHUB_TOKEN }}
DEV_VAULT_BRB: ${{ secrets.DEV_VAULT_BRB }}
DEV_VAULT_NBC: ${{ secrets.DEV_VAULT_NBC }}
DEV_VAULT_THR: ${{ secrets.DEV_VAULT_THR }}
DEV_VAULT_DBC: ${{ secrets.DEV_VAULT_DBC }}
DEV_KUBE_CONFIG_BRB: ${{ secrets.DEV_KUBE_CONFIG_BRB }}
DEV_KUBE_CONFIG_NBC: ${{ secrets.DEV_KUBE_CONFIG_NBC }}
DEV_KUBE_CONFIG_THR: ${{ secrets.DEV_KUBE_CONFIG_THR }}
DEV_KUBE_CONFIG_DBC: ${{ secrets.DEV_KUBE_CONFIG_DBC }}

deploy-successful:
needs:
- deploy
runs-on: ubuntu-latest
steps:
- run: echo "deploy was successful"

# trivy-vulnerability-scanning:
# needs:
# - build_and_push
# - branch_meta
# runs-on: ubuntu-latest
# permissions:
# actions: read
# contents: read
# security-events: write
# steps:
# - name: run trivy vulnerability scanner
# uses: aquasecurity/trivy-action@1f6384b6ceecbbc6673526f865b818a2a06b07c9
# with:
# image-ref: 'ghcr.io/${{ github.repository }}:${{ needs.branch_meta.outputs.sha }}'
# format: 'sarif'
# output: 'trivy-results.sarif'
# severity: 'CRITICAL,HIGH'
# ignore-unfixed: true
# - name: upload trivy results
# if: ${{ always() }}
# uses: github/codeql-action/upload-sarif@v3
# with:
# sarif_file: 'trivy-results.sarif'

# end-to-end-tests:
# needs:
# - build_and_push
# - branch_meta
# uses: hpi-schul-cloud/end-to-end-tests/.github/workflows/e2e_call.yml@main
# with:
# ref: ${{ needs.branch_meta.outputs.branch }}

# cy-e2e-tests:
# needs:
# - branch_meta
# - deploy
# uses: hpi-schul-cloud/e2e-system-tests/.github/workflows/remote-trigger.yml@main
# with:
# ref: ${{ needs.branch_meta.outputs.branch }}
# secrets:
# service-account-token: ${{ secrets.CYPRESS_ONEPWD_SERVICE_ACCOUNT_TOKEN }}

# test-successful:
# runs-on: ubuntu-latest
# needs:
# - end-to-end-tests
# steps:
# - run: echo "Test was successful"
Loading
Loading