Install guide

CONTRIBUTING.md

@@ -1,3 +1,46 @@
+## Setup
+
+### Setup local db link as environment variable.
+
+```bash
+export DB_URL=postgres://<username>@localhost:5432/config
+```
+
+### Create `config` database.
+
+```sql
+create database config
+```
+
+### Scape config and serve
+
+Starting the server will run the migrations and start scraping in background (The `default-schedule` configuration will run scraping every 60 minutes if configuration is not explicitly specified).
+
+```bash
+make build
+
+./.bin/config-db serve
+```
+
+To explicitly run scraping with a particular configuration:
+
+```bash
+./.bin/config-db run <scrapper-config.yaml> -vvv
+config-db serve
+```
+
+See `fixtures/` for example scraping configurations.
+
+### Migrations
+
+Commands `./bin/config-db serve` or `./bin/config-db run` would run the migrations.
+
+Setup [goose](https://github.com/pressly/goose) for more options on migration. Goose commands need to be run from `db/migrations` directory.
+
+```bash
+GOOSE_DRIVER=postgres GOOSE_DBSTRING="user=postgres dbname=config sslmode=disable" goose down
+```
+
 ## Adding a new scraper
 
 1. Create a new file in `scrapers/` which implements the `api/v1/Scraper` interface

README.md

@@ -2,59 +2,111 @@
 
 **config-db** is developer first, JSON based configuration management database (CMDB).
 
-## Setup
+## Principles
 
-### Setup local db link as environment variable.
+* **JSON Based** - Configuration is stored in JSON, with changes recorded as JSON patches that enables highly structured search.
+* **SPAM Free** - Not all configuration data is useful, and overly verbose change histories are difficult to navigate.
+* **GitOps Ready** - Configuration should be stored in Git, config-db enables the extraction of configuration out of Git repositories with branch/environment awareness.
+* **Topology Aware** - Configuration can often have an inheritance or override hierarchy.
 
-```bash
-export DB_URL=postgres://<username>@localhost:5432/config
-```
+## Capabilities
 
-### Create `config` database.
+* View and search change history in any dimension (node, zone, environment, application, technology)
+* Compare and diff configuration across environments.
 
-```sql
-create database config
-```
+## Quick Start
 
-### Scape config and serve
+Before installing Config-DB , please ensure you have the [prerequisites installed](docs/prereqs.md) on your Kubernetes cluster.
 
-Starting the server will run the migrations and start scraping in background (The `default-schedule` configuration will run scraping every 60 minutes if configuration is not explicitly specified).
+The recommended method for installing Config-DB is using [helm](https://helm.sh/)
 
-```bash
-make build
+### Install Helm
+
+The following steps will install the latest version of helm
 
-./.bin/config-db serve
+```bash
+curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
+chmod 700 get_helm.sh
+./get_helm.sh
 ```
 
-To explicitly run scraping with a particular configuration:
+### Add the Flanksource helm repository
 
 ```bash
-./.bin/config-db run <scrapper-config.yaml> -vvv
-config-db serve
+helm repo add flanksource https://flanksource.github.io/charts
+helm repo update
 ```
 
-See `fixtures/` for example scraping configurations.
+### Configurable fields
+
+See the [values file](chart/values.yaml) for the full list of configurable fields.  Mandatory configuration values are for the configuration of the database, and it is recommended to also configure the UI ingress.
+
+#### DB
+
+ConfigDB requires a Postgres server to function.  A basic postgres server can be installed by the helm chart.
+
+##### Chart-managed Server
+
+|                     |        |
+|---------------------|--------|
+| db.external.create  | `true` |
+| db.external.storageClass | Set to name of a storageclass available in the cluster |
+| db.external.storage | Set to volume of storage to request |
 
-### Migrations
+The helm chart will create a postgres server statefulset, with a random password and default port, along with a configdb database hosted on the server.
 
-Commands `./bin/config-db serve` or `./bin/config-db run` would run the migrations.
+To specify a username and password for the chart-managed Postgres server, create a secret in the namespace that the chart will install to, named `postgres-connection`, which contains `POSTGRES_USER` and `POSTGRES_PASSWORD` keys.  If no pre-existing secret is created, a user called 'postgres' will be given a random password.
 
-Setup [goose](https://github.com/pressly/goose) for more options on migration. Goose commands need to be run from `db/migrations` directory.
+##### Prexisting Server
+
+In order to connect to an existing Postgres server, a database must be created on the server, along with a user that has admin permissions
+
+|                     |         |
+|---------------------|---------|
+| db.external.create  | `false` |
+| db.external.secretKeyRef.name | Set to name of name of secret that contains a key containging the postgres connection URI |
+| db.external.secretKeyRef.key | Set to the name of the key in the secret that contains the postgres connection URI |
+
+The connection URI must be specified in the format `postgresql://"$user":"$password"@"$host"/"$database"`
+
+#### Ingress 
+
+In order to view the ConfigDB UI, it must be exposed  using an ingress:
+
+|                     |                   |
+|---------------------|-------------------|
+| ingress.host | URL at which the UI will be accessed |
+| ingress.annotations | Map of annotations required by the ingress controller or certificate issuer |
+| ingress.tls | Map of configuration options for TLS |
+
+More details regarding ingress configuration can be found in the [kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/)
+
+### Deploy using Helm
+
+To install into a new `config-db` namespace, run
 
 ```bash
-GOOSE_DRIVER=postgres GOOSE_DBSTRING="user=postgres dbname=config sslmode=disable" goose down
+helm install config-db-demo --wait -n config-db --create-namespace flanksource/config-db -f values.yaml
 ```
-## Principles
 
-* **JSON Based** - Configuration is stored in JSON, with changes recorded as JSON patches that enables highly structured search.
-* **SPAM Free** - Not all configuration data is useful, and overly verbose change histories are difficult to navigate.
-* **GitOps Ready** - Configuration should be stored in Git, config-db enables the extraction of configuration out of Git repositories with branch/environment awareness.
-* **Topology Aware** - Configuration can often have an inheritance or override hierarchy.
-
-## Capabilities
+where `values.yaml` contains the configuration options detailed above.  eg
+
+```yaml
+db:
+  create: true
+  storageClass: default
+  storage: 30Gi
+ingress:
+  host: config-db.flanksource.com
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    kubernetes.io/tls-acme: "true"
+  tls:
+    - secretName: config-db-tls
+      hosts:
+      - config-db.flanksource.com
+```
 
-* View and search change history in any dimension (node, zone, environment, application, technology)
-* Compare and diff configuration across environments.
 
 ## Configuration Sources
 

chart/templates/postgres.yaml

@@ -1,4 +1,4 @@
-{{- if eq .Values.db.enabled true }}
+{{- if eq .Values.db.create true }}
 
 ---
 # PostgreSQL StatefulSet
@@ -52,7 +52,6 @@ spec:
 
 ---
 
-{{- if .Values.db.secretKeyRef.create }}
 apiVersion: v1
 kind: Secret
 metadata:
@@ -61,10 +60,14 @@ metadata:
     "helm.sh/resource-policy": "keep"
 type: Opaque
 stringData:
-  {{- $secretObj := ( lookup "v1" "Secret" .Release.Namespace "postgres-connection" ) }}
+  # Potential existing secret with user requested login details to inject into chart
+  {{- $secretInj := ( lookup "v1" "Secret" .Release.Namespace "postgres-connection" }}
+  # Potential previously created version of this secret.  Read in to prevent helm creating a new password string every deploy
+  {{- $secretObj := ( lookup "v1" "Secret" .Release.Namespace .Values.db.secretKeyRef.name }}
+  {{- $injData := ( get $secretInj "data" ) }}
   {{- $secretData := ( get $secretObj "data" ) }}
-  {{- $user := (( get $secretData "POSTGRES_USER" ) | b64dec ) | default "postgres" }}
-  {{- $password := (( get $secretData "POSTGRES_PASSWORD" ) | b64dec ) | default randAlphaNum 32 }}
+  {{- $user := (( get $injData "POSTGRES_USER" ) | b64dec ) | default (( get $secretData "POSTGRES_USER" ) | b64dec ) | default "postgres" }}
+  {{- $password := (( get $injData "POSTGRES_PASSWORD" ) | b64dec ) | default (( get $secretData "POSTGRES_PASSWORD" ) | b64dec ) | default randAlphaNum 32 }}
   {{- $host := print "postgres." .Release.Namespace  ".svc.cluster.local:5432" }}
   {{- $url := print "postgresql://" $user ":" $password "@" $host }}
   {{- $configDbUrl := ( get $secretData .Values.db.secretKeyRef.key ) | default ( print $url "/config-db" ) }}
@@ -74,8 +77,6 @@ stringData:
   POSTGRES_HOST: {{ $host | quote }}
   {{ .Values.db.secretKeyRef.key }}: {{ $configDbUrl | quote }}
 
-{{- end }}
-
 ---
 
 {{- end }}

chart/values.yaml

@@ -20,14 +20,14 @@ image:
 scrapeRuleConfigMaps:
   - config-db-rules
 db:
-  # Setting this to true will create a postgres stateful set for config-db to connect to.
-  enabled: true
+  # Setting create to true will create a postgres stateful set for config-db to connect to.
+  # If create=true, the secretKeyRef will be created by helm with the specified name and key
+  #   Optionally a secret named 'postgres-connection' with POSTGRES_USER and POSTGRES_PASSWORD to set the created username and password, otherwise a random password will be created for a 'postgres' user  
+  # If create=false, a prexisting secret containing the URI to an existing postgres database must be provided
+  #   The URI must be in the format 'postgresql://"$user":"$password"@"$host"/"$database"'
+  create: true
   secretKeyRef:
-    create: true
-    # Setting the name of the secret will disable secret creation in this chart and look for an existing secret (whose name is specified in this field) to mount.
-    # When setting this up in a fresh environment as a standalone app, it's best to leave the value empty.
-    name:
-    # This is the key that either the secret will create(if secretRefKey is empty) or this is the key it'll look for in the secret(if secretRefKey is mentioned).
+    name: config-db-postgres
     # The name of the key is mandatory to set.
     key: DB_URL
   storageClass: