Merge branch 'develop' into develop

Signed-off-by: pg-techno123 <[email protected]>

.github/workflows/chart-lint-publish.yml

@@ -0,0 +1,62 @@
+name: Validate / Publish helm charts
+
+on:
+  release:
+    types: [published]
+  pull_request:
+    types: [opened, reopened, synchronize]
+    paths:
+      - 'helm/**'
+  workflow_dispatch:
+    inputs:
+      IGNORE_CHARTS:
+        description: 'Provide list of charts to be ignored separated by pipe(|)'
+        required: false
+        default: '""'
+        type: string
+      CHART_PUBLISH:
+        description: 'Chart publishing to gh-pages branch'
+        required: false
+        default: 'NO'
+        type: string
+        options:
+          - YES
+          - NO
+      INCLUDE_ALL_CHARTS:
+        description: 'Include all charts for Linting/Publishing (YES/NO)'
+        required: false
+        default: 'NO'
+        type: string
+        options:
+          - YES
+          - NO
+  push:
+    branches:
+      - '!release-branch'
+      - '!master'
+      - 1.*
+      - 0.*
+      - develop
+      - release*
+    paths:
+      - 'helm/**'
+
+jobs:
+  chart-lint-publish:
+    uses: mosip/kattu/.github/workflows/chart-lint-publish.yml@master
+    with:
+      CHARTS_DIR: ./helm
+      CHARTS_URL: https://mosip.github.io/mosip-helm
+      REPOSITORY: mosip-helm
+      BRANCH: gh-pages
+      INCLUDE_ALL_CHARTS: "${{ inputs.INCLUDE_ALL_CHARTS || 'NO' }}"
+      IGNORE_CHARTS: "${{ inputs.IGNORE_CHARTS || '\"\"' }}"
+      CHART_PUBLISH: "${{ inputs.CHART_PUBLISH || 'YES' }}"
+      LINTING_CHART_SCHEMA_YAML_URL: "https://raw.githubusercontent.com/mosip/kattu/master/.github/helm-lint-configs/chart-schema.yaml"
+      LINTING_LINTCONF_YAML_URL: "https://raw.githubusercontent.com/mosip/kattu/master/.github/helm-lint-configs/lintconf.yaml"
+      LINTING_CHART_TESTING_CONFIG_YAML_URL: "https://raw.githubusercontent.com/mosip/kattu/master/.github/helm-lint-configs/chart-testing-config.yaml"
+      LINTING_HEALTH_CHECK_SCHEMA_YAML_URL: "https://raw.githubusercontent.com/mosip/kattu/master/.github/helm-lint-configs/health-check-schema.yaml"
+      DEPENDENCIES: "mosip,https://mosip.github.io/mosip-helm;"
+    secrets:
+      TOKEN: ${{ secrets.ACTION_PAT }}
+      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

.gitignore

@@ -1,4 +1,3 @@
-deploy*/
 .idea
 target*
 
@@ -35,4 +34,4 @@ build/
 
 ### VS Code ###
 .vscode/
-*.log
+*.log

README.md

@@ -14,7 +14,7 @@ This repo contains test framework for end2end testing of MOSIP functionality.  T
 
 
 ## Execution flow	
-![](docs/test-orchestrator.png)
+![](docs/_images/test-orchestrator.png)
 
 ## Prerequisites
 

deploy/dslrig/README.md

@@ -0,0 +1,50 @@
+# DSLRIG
+
+## Introduction
+DSLRIG will test end-to-end functional flows involving multiple MOSIP modules.
+
+## Prerequisites
+* Packet utility running in rancher cluster exposed to be accessed by the cluster.
+* Auth demo and DSLRIG to be running in the same cluster.
+* NFS server already present and related details.
+
+## Install
+* Install
+```sh
+./install.sh
+```
+* During the execution of the `install.sh` script, a prompt appears requesting information regarding the presence of a public domain and a valid SSL certificate on the server.
+* If the server lacks a public domain and a valid SSL certificate, it is advisable to select the `n` option. Opting it will enable the `init-container` with an `emptyDir` volume and include it in the deployment process.
+* The init-container will proceed to download the server's self-signed SSL certificate and mount it to the specified location within the container's Java keystore (i.e., `cacerts`) file.
+* This particular functionality caters to scenarios where the script needs to be employed on a server utilizing self-signed SSL certificates.
+
+## Uninstall
+* To uninstall DSLRIG, run `delete.sh` script.
+```sh
+./delete.sh 
+```
+
+## Run dslrig manually
+
+#### Rancher UI
+* Run dslrig manually via Rancher UI.
+  ![dslrig-1.png](../../docs/_images/dslrig-1.png)
+* There are two modes of dslrig `sanity` & `full`.
+    * By default, dslrig will execute in `full` mode.
+    * If you want to run dslrig with `sanity` mode,
+      Update the `TESTLEVEL` key in the `dslorchestrator` configmap and rerun the dslrig job.
+
+#### CLI
+* Download Kubernetes cluster `kubeconfig` file from `rancher dashboard` to your local.
+  ![apitestrig-1.png](../../docs/_images/apitestrig-1.png)
+* Install `kubectl` package to your local machine.
+* Run dslrig manually via CLI by creating a new job from an existing k8s cronjob.
+  ```
+  kubectl --kubeconfig=<k8s-config-file> -n dslrig create job --from=cronjob/<cronjob-name> <job-name>
+  ```
+  example:
+  ```
+  kubectl --kubeconfig=/home/xxx/Downloads/qa4.config -n dslrig create job --from=cronjob/cronjob-dslorchestrator cronjob-dslorchestrator
+  ```
+
+

deploy/dslrig/copy_cm.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# Copy configmaps from other namespaces
+# DST_NS: Destination namespace
+
+function copying_cm() {
+  UTIL_URL=https://raw.githubusercontent.com/mosip/mosip-infra/master/deployment/v3/utils/copy_cm_func.sh
+  COPY_UTIL=./copy_cm_func.sh
+  DST_NS=dslrig
+
+  wget -q $UTIL_URL -O copy_cm_func.sh && chmod +x copy_cm_func.sh
+
+  $COPY_UTIL configmap global default $DST_NS
+  $COPY_UTIL configmap keycloak-host keycloak $DST_NS
+  $COPY_UTIL configmap artifactory-share artifactory $DST_NS
+  $COPY_UTIL configmap config-server-share config-server $DST_NS
+  return 0
+}
+
+# set commands for error handling.
+set -e
+set -o errexit   ## set -e : exit the script if any statement returns a non-true return value
+set -o nounset   ## set -u : exit the script if you try to use an uninitialised variable
+set -o errtrace  # trace ERR through 'time command' and other functions
+set -o pipefail  # trace ERR through pipes
+copying_cm   # calling function

deploy/dslrig/copy_secrets.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+# Copy secrets from other namespaces
+# DST_NS: Destination namespace
+
+function copying_secrets() {
+  UTIL_URL=https://raw.githubusercontent.com/mosip/mosip-infra/master/deployment/v3/utils/copy_cm_func.sh
+  COPY_UTIL=./copy_cm_func.sh
+  DST_NS=dslrig
+
+  wget -q $UTIL_URL -O copy_cm_func.sh && chmod +x copy_cm_func.sh
+
+  $COPY_UTIL secret keycloak-client-secrets keycloak $DST_NS
+  $COPY_UTIL secret s3 s3 $DST_NS
+  $COPY_UTIL secret postgres-postgresql postgres $DST_NS
+  return 0
+}
+
+# set commands for error handling.
+set -e
+set -o errexit   ## set -e : exit the script if any statement returns a non-true return value
+set -o nounset   ## set -u : exit the script if you try to use an uninitialised variable
+set -o errtrace  # trace ERR through 'time command' and other functions
+set -o pipefail  # trace ERR through pipes
+copying_secrets   # calling function

deploy/dslrig/delete.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+# Uninstalls dslrig
+## Usage: ./delete.sh [kubeconfig]
+
+if [ $# -ge 1 ] ; then
+  export KUBECONFIG=$1
+fi
+
+function deleting_dslrig() {
+  NS=dslrig
+  while true; do
+      read -p "Are you sure you want to delete dslrig helm charts?(Y/n) " yn
+      if [ $yn = "Y" ]
+        then
+          helm -n $NS delete dslorchestrator
+          break
+        else
+          break
+      fi
+  done
+  return 0
+}
+
+# set commands for error handling.
+set -e
+set -o errexit   ## set -e : exit the script if any statement returns a non-true return value
+set -o nounset   ## set -u : exit the script if you try to use an uninitialised variable
+set -o errtrace  # trace ERR through 'time command' and other functions
+set -o pipefail  # trace ERR through pipes
+deleting_dslrig   # calling function

deploy/dslrig/install.sh

@@ -0,0 +1,109 @@
+#!/bin/bash
+# Installs dslrig
+## Usage: ./install.sh [kubeconfig]
+
+if [ $# -ge 1 ] ; then
+  export KUBECONFIG=$1
+fi
+
+NS=dslrig
+CHART_VERSION=0.0.1-develop
+
+echo Create $NS namespace
+kubectl create ns $NS
+
+function installing_dslrig() {
+  ENV_NAME=$( kubectl -n default get cm global -o json |jq -r '.data."installation-domain"')
+
+  read -p "Please enter the time(hr) to run the cronjob every day (time: 0-23) : " time
+  if [ -z "$time" ]; then
+     echo "ERROT: Time cannot be empty; EXITING;";
+     exit 1;
+  fi
+  if ! [ $time -eq $time ] 2>/dev/null; then
+     echo "ERROR: Time $time is not a number; EXITING;";
+     exit 1;
+  fi
+  if [ $time -gt 23 ] || [ $time -lt 0 ] ; then
+     echo "ERROR: Time should be in range ( 0-23 ); EXITING;";
+     exit 1;
+  fi
+
+  echo "Do you have public domain & valid SSL? (Y/n) "
+  echo "Y: if you have public domain & valid ssl certificate"
+  echo "n: If you don't have a public domain and a valid SSL certificate. Note: It is recommended to use this option only in development environments."
+  read -p "" flag
+
+  if [ -z "$flag" ]; then
+    echo "'flag' was provided; EXITING;"
+    exit 1;
+  fi
+  ENABLE_INSECURE=''
+  if [ "$flag" = "n" ]; then
+    ENABLE_INSECURE='--set enable_insecure=true';
+  fi
+
+  read -p "Please provide packet Utility Base URL (eg: https://<host>:<port>/v1/packetcreator) : " packetUtilityBaseUrl
+
+  if [ -z $packetUtilityBaseUrl ]; then
+    echo "Packet utility Base URL not provided; EXITING;"
+    exit 1;
+  fi
+
+  read -p "Please provide the retention days to remove old reports ( Default: 3 )" reportExpirationInDays
+
+  if [[ -z $reportExpirationInDays ]]; then
+    reportExpirationInDays=3
+  fi
+  if ! [[ $reportExpirationInDays =~ ^[0-9]+$ ]]; then
+    echo "The variable \"reportExpirationInDays\" should contain only number; EXITING";
+    exit 1;
+  fi
+
+  echo Istio label
+  kubectl label ns $NS istio-injection=disabled --overwrite
+  helm repo update
+
+  echo Copy configmaps
+  ./copy_cm.sh
+
+  echo Copy secrets
+  ./copy_secrets.sh
+
+  echo "Delete s3, db, & dslrig configmap if exists"
+  kubectl -n $NS delete --ignore-not-found=true configmap s3
+  kubectl -n $NS delete --ignore-not-found=true configmap db
+  kubectl -n $NS delete --ignore-not-found=true configmap dslrig
+
+  DB_HOST=$( kubectl -n default get cm global -o json  |jq -r '.data."mosip-api-internal-host"' )
+  API_INTERNAL_HOST=$( kubectl -n default get cm global -o json  |jq -r '.data."mosip-api-internal-host"' )
+  USER=$( kubectl -n default get cm global -o json |jq -r '.data."mosip-api-internal-host"')
+
+  echo Installing dslrig
+  helm -n $NS install dslorchestrator mosip/dslorchestrator \
+  --set crontime="0 $time * * *" \
+  --version $CHART_VERSION \
+  --set dslorchestrator.configmaps.s3.s3-host='http://minio.minio:9000' \
+  --set dslorchestrator.configmaps.s3.s3-user-key='admin' \
+  --set dslorchestrator.configmaps.s3.s3-region='' \
+  --set dslorchestrator.configmaps.db.db-server="$DB_HOST" \
+  --set dslorchestrator.configmaps.db.db-su-user="postgres" \
+  --set dslorchestrator.configmaps.db.db-port="5432" \
+  --set dslorchestrator.configmaps.dslorchestrator.USER="$USER" \
+  --set dslorchestrator.configmaps.dslorchestrator.ENDPOINT="https://$API_INTERNAL_HOST" \
+  --set dslorchestrator.configmaps.dslorchestrator.packetUtilityBaseUrl="$packetUtilityBaseUrl" \
+  --set dslorchestrator.configmaps.dslorchestrator.reportExpirationInDays="$reportExpirationInDays" \
+  --set dslorchestrator.configmaps.dslorchestrator.NS="$NS" \
+  $ENABLE_INSECURE
+
+  echo Installed dslrig.
+  return 0
+}
+
+# set commands for error handling.
+set -e
+set -o errexit   ## set -e : exit the script if any statement returns a non-true return value
+set -o nounset   ## set -u : exit the script if you try to use an uninitialised variable
+set -o errtrace  # trace ERR through 'time command' and other functions
+set -o pipefail  # trace ERR through pipes
+installing_dslrig   # calling function

deploy/packetcreator/README.md

@@ -0,0 +1,27 @@
+# Packetcreator
+
+## Overview
+Packetcreator will create packets for DSL orchestrator.
+
+**Note:** It is recommended to deploy `packetcreator` on a separate server/cluster from where other DSL orchestrators can access this service.
+
+## Installation
+```sh
+./install.sh
+```
+* During the execution of the `install.sh` script, a prompt appears requesting information regarding the presence of a public domain and a valid SSL certificate on the server.
+* If the server lacks a public domain and a valid SSL certificate, it is advisable to select the `n` option. Opting it will enable the `init-container` with an `emptyDir` volume and include it in the deployment process.
+* The init-container will proceed to download the server's self-signed SSL certificate and mount it to the specified location within the container's Java keystore (i.e., `cacerts`) file.
+* This particular functionality caters to scenarios where the script needs to be employed on a server utilizing self-signed SSL certificates.
+
+## Uninstall
+* To uninstall Packetcreator, run `delete.sh` script.
+```sh
+./delete.sh 
+```
+
+## Restart
+* To restart packetcreator, run `restart.sh` script.
+```sh
+./restart.sh
+```

deploy/packetcreator/delete.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+# Uninstalls packetcreator
+## Usage: ./delete.sh [kubeconfig]
+
+if [ $# -ge 1 ] ; then
+  export KUBECONFIG=$1
+fi
+
+function Deleting_packetcreator() {
+  NS=packetcreator
+  while true; do
+      read -p "Are you sure you want to delete packetcreator helm charts?(Y/n) " yn
+      if [ $yn = "Y" ]
+        then
+          helm -n $NS delete packetcreator
+          break
+        else
+          break
+      fi
+  done
+  return 0
+}
+
+# set commands for error handling.
+set -e
+set -o errexit   ## set -e : exit the script if any statement returns a non-true return value
+set -o nounset   ## set -u : exit the script if you try to use an uninitialised variable
+set -o errtrace  # trace ERR through 'time command' and other functions
+set -o pipefail  # trace ERR through pipes
+Deleting_packetcreator   # calling function