Merge pull request #13 from vallard/monitoring-class-update

Monitoring class update

03/ngx.yaml

@@ -10,7 +10,7 @@ metadata:
   name: ngx
   namespace: default
 spec:
-  replicas: 1
+  replicas: 2
   selector:
     matchLabels:
       run: ngx

app-api/app-api.yaml

@@ -30,39 +30,39 @@ spec:
     creationPolicy: Owner
   data:
     - remoteRef:
-        key: super_slack_secrets
+        key: my-secrets
         property: SLACK_TOKEN
       secretKey: SLACK_TOKEN
     - remoteRef: 
-        key: super_slack_secrets
+        key: my-secrets
         property: SLACK_CHANNEL
       secretKey: SLACK_CHANNEL
     - remoteRef: 
-        key: sample-app-secret
+        key: my-secrets
         property: K8S_DB_USERNAME
       secretKey: K8S_DB_USERNAME
     - remoteRef: 
-        key: sample-app-secret
+        key: my-secrets
         property: K8S_DB_PASSWORD
       secretKey: K8S_DB_PASSWORD
     - remoteRef: 
-        key: sample-app-secret
+        key: my-secrets
         property: MYSQL_USER
       secretKey: MYSQL_USER
     - remoteRef: 
-        key: sample-app-secret
+        key: my-secrets
         property: MYSQL_ROOT_PASSWORD
       secretKey: MYSQL_ROOT_PASSWORD
     - remoteRef: 
-        key: sample-app-secret
+        key: my-secrets
         property: MYSQL_PASSWORD
       secretKey: MYSQL_PASSWORD
     - remoteRef:
-        key: sample-app-secret 
+        key: my-secrets 
         property: K8S_DB_DATABASE
       secretKey: K8S_DB_DATABASE
     - remoteRef: 
-        key: sample-app-secret 
+        key: my-secrets 
         property: MYSQL_DATABASE
       secretKey: MYSQL_DATABASE
 ---

app-api/docker-compose.yaml

@@ -16,7 +16,7 @@ services:
       - K8S_DB_PORT=3306
       - K8S_DB_DATABASE=k8sdb
       - SLACK_TOKEN=${SLACK_TOKEN}
-      - SLACK_CHANNEL=C03NPPVGR3R
+      - SLACK_CHANNEL=${SLACK_CHANNEL}
     volumes:
       - ./:/code/
 

m01/README.md

@@ -5,14 +5,28 @@ If you've already completed part one of this class you'll have a cluster up that
 
 ## 01. Set up Basic EKS
 
-We have an EKS terragrunt plan in [../02/terragrunt/live/stage-mon](../02/terragrunt/live/stage-mon).  To start run: 
+First set up the AWS IAM prerequisites we need.  We cover this a bit in the first class, so suffice to say, we just need to ensure we have proper groups, roles, users, and policies for our cluster.  This is magically done with the below commands.  Before you do that, ensure you have your own PGP key in the var file so you can actually decrpyt it.  See [../02/terraform.md](../02/terraform.md) for how to do this. 
+
+
+```
+cd terraform
+terraform init
+terraform plan 
+terraform apply 
+```
+
+Now let's build everything. 
+
+We have an EKS terragrunt plan in [terragrunt/live/stage-mon](../terragrunt/live/stage-mon).  To start run: 
 
 ```
 terragrunt run-all apply 
 ```
 
 This will set up the basic cluster you'll need as well as OpenSearch that we'll use for the logging portion of this class at the end. 
 
+You may need to run this twice as there is a bug in the dependencies with Terragrunt. 
+
 You should then add the cluster to your kube config with: 
 
 ```
@@ -21,34 +35,31 @@ aws eks update-kubeconfig --name eks-stage-mon \
 	--role-arn arn:aws:iam::188966951897:role/eks_dude_role
 ```
 
-## 02. Kubernetes Additional Users
+## 02. Kubernetes Additional Components
 
 Once you can access you're Kubernetes cluster you need to add the ingress controller, associate with Route 53, and then add the cert-manager components: 
 
-```
+See the [Route53 section](../04/r53.md) on how to do that. You'll also find cert-manager 
 
-```
+For this to work, you're going to need a domain!
 
-```
-kubectl edit cm -n kube-system aws-auth
-```
-
-
-Add: 
+Quick start: 
 
 ```
-- groups:
-      - system:masters
-      rolearn: arn:aws:iam::188966951897:role/eks_dude_role
-      username: devops:{{SessionName}}
+cd 04/nginx-ingress-controller
+k apply -f deploy.yaml
+cd ../cert-manager
+k apply -f cert-manager.yaml 
+k apply -f prod-issuer.yaml 
+cd ../
+k apply default-backend.yaml 
 ```
 
+Now go to Route 53 and associate the domain with the ingress controller.  This a manual thing. 
 ## 03. Metrics API 
 
-We can install the metrics API that we mention in [part 5](../05/metrics-server-0.6.1).  
-
 ```
-kubectl apply -f 05/metrics-server-0.6.1/
+kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
 ```
 
 Once this is done we should be able to run: 
@@ -59,12 +70,9 @@ kubectl top pods -A
 kubectl get —-raw /metrics
 ```
 
-
-
-
 ## 04. Basic Monitoring with Lens
 
-Get [Lens](https://k8slens.dev/) and install and open.  It should read your `~/.kube/config` and be able to open up a session. 
+[Lens](https://k8slens.dev/) used to be open source and awesome but like a lot of companies their shareholders wanted money so now you have to pay.  Fear not!  You can still get [OpenLens](https://github.com/crloz/openlens). There is still and install and open.  It should read your `~/.kube/config` and be able to open up a session. 
 
 ![](../images/mo/m01-lens.png)
 

m02/README.md

@@ -1,40 +1,9 @@
 # Basic Application
 
-We need to get our application ready so we can see about application alerts. 
+We need to get our application ready so we can see about application alerts.  First, we'll create a slack channel and slack bot then we'll store the secrets in AWS secrets manager, then we'll access the secrets using the [External Secrets Operator](https://external-secrets.io/v0.7.2/).
 
-If you haven't done the first modules in the first class, then we need to quickly get the cluster to be able to hold an application: 
 
-```
-cd 04
-kubectl apply -f nginx-ingress-controller/deploy.yaml
-kubectl apply -f cert-manager/cert-manager.yaml
-```
-
-Modify the DNS name to match the Load Balancer. This is done in Route53. 
-
-```
-kubectl apply -f cert-manager/prod-issuer.yaml
-```
-
-
-## External Secrets
-
-We will also need external secrets to store our passwords for our application. 
-
-This includes database permissions, slack APIs, etc.  The cost to store this in AWS Secrets manager is $0.40/month.  
-
-
-```
-helm repo add external-secrets https://charts.external-secrets.io
-helm install external-secrets \
-   external-secrets/external-secrets \
-    -n kube-system \
-    --create-namespace \
-    --set installCRDs=true
-```
-
-
-## Application Slack Integration
+## Create A Slack Application
 
 We first need to create a Slack Application and get a token for posting to our Slack messages. 
 
@@ -57,7 +26,7 @@ You should add:
 * `chat:write`
 * `chat:write:public` - this way you can write to any channels in the workspace. 
 
-We now should have the `Bot User OAuth Token` on this same page.  This is the token we will save in our environment as `SLACK_TOKEN`.  We also need to create a channel and then get the channel so we can post there.  
+Create a  `Bot User OAuth Token` on this same page.  This is the token we will save in our environment as `SLACK_TOKEN`.  We also need to create a channel and then get the channel so we can post there.  
 
 This is done by right clicking on a channel's info button and copying the channel id)
 
@@ -67,6 +36,8 @@ Clicking in the channel information button you can get the Channel ID down at th
 
 ![Get Channel ID](../images/mo/slack02.png)
 
+## Store Slack environment locally
+
 For development I put these in my `~/.zshrc` or `~/.bash_profile` (depending on shell) so it looks like: 
 
 ```
@@ -107,6 +78,53 @@ This will run locally.  To get the front end locally, open another browser and r
 yarn install
 yarn start
 ```
+## Secrets in Kubernetes
+
+Now that we can run locally need to store these passwords for our application. We will use External Secrets with Secrets Manager.  
+
+We can store all our secrets in one place; database permissions, slack APIs, etc.  The cost to store this in AWS Secrets manager is $0.40/month.  
+
+### Create Secret in Secrets manager
+
+We will add our Slack tokens to a file called `secrets.json`. In this file there are also other secrets we'll use for our application for passwords and connecting to our database.  Edit this file to include your slack token and change your database password if needed.  If using something else for a database later on (like RDS) you can put these credentials as well.  
+
+### Upload to AWS
+
+We can upload by running the command: 
+
+```
+aws secretsmanager create-secret --name super-secrets --secret-string file://secrets.json
+```
+
+Open up secrets manager on the console and verify it looks good (e.g: There are values for each of the secrets defined). 
+
+### Install External Secrets
+
+```
+helm repo add external-secrets https://charts.external-secrets.io
+helm repo update
+helm install external-secrets \
+   external-secrets/external-secrets \
+    -n kube-system \
+    --create-namespace \
+    --set installCRDs=true
+```
+
+Now we'll need to create a secret that we'll use to store the following information about our application: 
+
+* Database connection details (host, password, etc)
+* Slack Secrets
+
+
+### Clean up (For after class)
+
+When you are done secrets manager makes you wait a minimum of 7 days to delete the secret if you do it from the console. I'm guessing they've had a fare amount of angry customers who deleted their secrets and needed them 2 min after they deleted. That's good, but we prefer to instead do this from the cli so it happens instantly:
+
+```
+aws secretsmanager delete-secret --secret-id arn:aws:secretsmanager:us-west-2:188966951897:secret:super_slack_secrets-31HOxE --force-delete-without-recovery
+```
+
+$0.40 isn't much, but I don't think Amazon needs more money from me :-) 
 
 ## Installing to Kubernetes
 

m02/secrets.json

@@ -0,0 +1,11 @@
+{
+    "SLACK_TOKEN": "",
+    "K8S_DB_USERNAME": "admin",
+    "K8S_DB_PASSWORD": "Luvi5VaoYoy7eiPh",
+    "MYSQL_USER": "admin",
+    "MYSQL_ROOT_PASSWORD": "Luvi5VaoYoy7eiPh",
+    "MYSQL_PASSWORD": "Luvi5VaoYoy7eiPh",
+    "MYSQL_DATABASE": "vanilla",
+    "K8S_DB_DATABASE": "vanilla",
+    "SLACK_CHANNEL": ""
+}

m03/README.md

@@ -34,10 +34,10 @@ helm search repo  prometheus-community
 
 While there are component based repos that can be installed we will use `prometheus-community/kube-prometheus-stack`.  
 
-To install vanilla we would run: 
+To install vanilla we would run something like: 
 
 ```
-helm install prometheus-community/kube-prometheus-stack --version 37.2.0
+helm install -n monitoring kube-prom prometheus-community/kube-prometheus-stack --version 44.3.0
 ```
 
 But we would like to customize this first. 
@@ -166,12 +166,18 @@ Let's install this configuration:
 helm upgrade --install -n monitoring \
 	kube-prom -f prometheus.yaml \
 	prometheus-community/kube-prometheus-stack \
-	--version 37.2.0
+	--version 44.3.0
 ```
 
-Rate
-Error
-Duration
+We should now be able to visit the site to see it!
+
+```
+kubectl get ing -A
+```
+This should show you what URLs you have to visit. In my case it's 
+
+[https://prometheus.k8s.castlerock.ai](https://prometheus.k8s.castlerock.ai)
+
 
 Prometheus is polling architecture and goes to each target and scraps it. 
 
@@ -214,7 +220,7 @@ We can also see graphs of these queries by checking out the graph button
 
 This graph shows that our nodes are around 50% capacity in memory. 
 
-## Kube-Proxy
+## Kube-Proxy (Older versions, fixed in Kuberenetes >=1.24)
 
 As part of the set up the `kube-proxy` in EKS doesn't work.  We have to patch this: 
 
@@ -232,4 +238,4 @@ Then restart the `kube-proxies`:
 kubectl rollout restart ds kube-proxy -n kube-system
 ```
 
-(credit: [alternaivan](https://github.com/prometheus-community/helm-charts/issues/977#issuecomment-888161858)
+(credit: [alternaivan](https://github.com/prometheus-community/helm-charts/issues/977#issuecomment-888161858))

terragrunt/stacks/stage-mon/eks/terragrunt.hcl

@@ -18,6 +18,6 @@ inputs = {
   public_subnets = dependency.vpc.outputs.vpc.public_subnets
   k8s_version = 1.24
   min_nodes = 1
-  desired_nodes = 3
+  desired_nodes = 5
   max_nodes = 10
 }