make the logging work

02/terragrunt/modules/vpc/main.tf

@@ -31,3 +31,5 @@ module "vpc_example_simple-vpc" {
   }
 }
 
+
+

README.md

@@ -87,15 +87,24 @@ In this segment we'll go over some Kubernetes primitives.  We move fast, show wh
 
 ### M05 - Integrate Grafana/Prometheus in our Application
 
-* [README](m05/README.md)
+* [Intro](m05/README.md)
+* [FastAPI and Prometheus](m05/README.md#fastapi-and-prometheus)
+* [Prometheus client](m05/README.md#prometheus-client)
+* [Scraping](m05/README.md#scraping)
+* [Grafana Integration](m05/README.md#grafana-integration)
+* [Persistence](m05/README.md#adding-persistence)
 
 ### M06 - CloudWatch Alarms
 
 * [README](m06/README.md)
 
 ### M07 - FEK Stack
 
-* [README](m07-fek/README.md)
+* [Intro](m07-fek/README.md#components)
+* [Configuration](m07-fek/README.md#installation-and-configuration)
+* [Fluentd](m07-fek/README.md#fluentd)
+* [Kibana](m07-fek/README.md#viewing-logs)
+
 
 ### M08 - Application Logging
 

app-api/app-api.yaml

@@ -92,6 +92,7 @@ spec:
                 name: k8s-sample-db-secrets
             - configMapRef:
                 name: k8s-sample-db-cm
+
 ---
 apiVersion: apps/v1
 kind: Deployment

app-api/app/main.py

@@ -13,6 +13,7 @@
 from fastapi_utils.tasks import repeat_every
 from app.routers import base, user, auth
 from app.database import database, engine
+from app.lib.app_logging import setup_logging
 
 app = FastAPI()
 app.header = {}
@@ -25,14 +26,8 @@
 # END PROMETHEUS (part 5)
 
 
-# Fluent (part 8)
-from app.lib.app_logging import setup_logging
-
-setup_logging("api")
-import logging
-
-logger = logging.getLogger("api")
-logger.info("HELLO LOGGING!")
+logger = setup_logging()
+logger.info("Hello, world!")
 # End fluent (part 8)
 
 origins = [
@@ -65,6 +60,7 @@ def init_instrumentator():
 @repeat_every(seconds=30, wait_first=True)
 def periodic():
     count = engine.execute("select count(id) from user").scalar()
+    logger.info(f"Number of users: {count}")
     Gauge("total_users", "Total Users").set(int(count))
 
 

app-api/app/routers/auth.py

@@ -10,6 +10,7 @@
     create_access_token,
 )
 from app.lib.slack import SlackClient
+from app.lib.app_logging import setup_logging
 from sqlalchemy.orm import Session
 import logging
 from typing import Any
@@ -27,7 +28,7 @@
 
 
 # Fluent (part 8)
-logger = logging.getLogger("api.auth")
+logger = setup_logging()
 # END Fluent (part 8)
 
 

m03/README.md

@@ -212,4 +212,24 @@ We can also see graphs of these queries by checking out the graph button
 
 ![graph of free](../images/mo/prom02.png)
 
-This graph shows that our nodes are around 50% capacity in memory. 
+This graph shows that our nodes are around 50% capacity in memory. 
+
+## Kube-Proxy
+
+As part of the set up the `kube-proxy` in EKS doesn't work.  We have to patch this: 
+
+```
+kubectl edit cm kube-proxy-config -n kube-system
+## Change from
+    metricsBindAddress: 127.0.0.1:10249 ### <--- Too secure
+## Change to
+    metricsBindAddress: 0.0.0.0:10249
+```
+
+Then restart the `kube-proxies`: 
+
+```
+kubectl rollout restart ds kube-proxy -n kube-system
+```
+
+(credit: [alternaivan](https://github.com/prometheus-community/helm-charts/issues/977#issuecomment-888161858)

m06/README.md

@@ -11,14 +11,14 @@ To do this we follow the [instructions](https://docs.aws.amazon.com/eks/latest/u
 ```
 aws eks update-cluster-config \
     --region us-west-2 \
-    --name eks-stage \
+    --name eks-stage-mon \
     --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'
 ```
 
 ## Analyzing metrics in Cloud Watch
 
 
-We visit (CloudWatch)[https://console.aws.amazon.com/cloudwatch/home#logs:prefix=/aws/eks] and search for the prefix `/aws/eks`.  This should show all our clusters. 
+We visit [CloudWatch](https://console.aws.amazon.com/cloudwatch/home#logs:prefix=/aws/eks) and search for the prefix `/aws/eks`.  This should show all our clusters. 
 
 
 There are several different logs we can see: 
@@ -51,5 +51,5 @@ In future classes we may add how to do this explicitly to this section.
 
 aws eks update-cluster-config \
     --region us-west-2 \
-    --name eks-stage \
+    --name eks-stage-mon \
     --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":false}]}'

m07-fek/README.md

@@ -1,4 +1,4 @@
-# The FEK Stack
+# The EFK Stack
 
 Logging information from your applications to search, verify, and index on is a great idea and gives added visibility.  In a production environment developers are constantly looking at logs to analyze user behavior, what went wrong, and find ways to improve the system. 
 
@@ -33,7 +33,7 @@ This lets us login with our standard username/password and now we can visit:
 
 ![open search dashboard](../images/mo/fek01.png)
 
-There's not much to see in here right now because there is
+There's not much to see in here right now because there isn't data being exported to it from fluent!
 
 
 

m07-fek/kibana-proxy.yaml

@@ -8,7 +8,7 @@ metadata:
   namespace: monitoring
 spec:
   type: ExternalName
-  externalName: vpc-opensearch-stage-woesdamvqbli5siagd7guw7ubi.us-west-2.es.amazonaws.com
+  externalName: vpc-opensearch-stage-mon-fhdhhs2ktec5zddpepkno2ereu.us-west-2.es.amazonaws.com
 ---
 apiVersion: networking.k8s.io/v1 
 kind: Ingress

m07-fek/values.yaml

@@ -6,7 +6,7 @@ fileConfigs:
     <label @OUTPUT>
       <match ** >
         @type opensearch
-        host vpc-opensearch-stage-woesdamvqbli5siagd7guw7ubi.us-west-2.es.amazonaws.com
+        host vpc-opensearch-stage-mon-fhdhhs2ktec5zddpepkno2ereu.us-west-2.es.amazonaws.com
         port 443
         ssl_verify false
         logstash_format true

m08-app-logging/README.md

@@ -4,3 +4,14 @@ One way we can ensure we have a handle on what our application is doing is by ha
 
 Since we already have fluent set up, let's log everything in our application and test it out. Then we can filter the logs to only show logs for our application. 
 
+## Edit Fluentd
+
+Let's not get all those logs, but let's only do the logs for our app: 
+
+```
+helm upgrade --install -n fluentd fluentd -f values2.yaml fluent/fluentd
+```
+
+## Logs
+
+![](../images/mo/m08-01.png)

m08-app-logging/app-api.yaml

@@ -0,0 +1,178 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: k8s-sample-db-cm
+data:
+  K8S_DB_HOST: mariadb
+  K8S_DB_PORT: "3306"
+---
+apiVersion: external-secrets.io/v1beta1
+kind: SecretStore
+metadata: 
+  name: my-aws-secrets
+spec:
+  provider: 
+    aws:
+      service: SecretsManager
+      region: us-west-2
+---
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata: 
+  name:  k8s-sample-db-secrets
+spec: 
+  refreshInterval: "0"
+  secretStoreRef: 
+    name: my-aws-secrets
+    kind: SecretStore
+  target:
+    name: k8s-sample-db-secrets
+    creationPolicy: Owner
+  data:
+    - remoteRef:
+        key: sample-app-secret
+        property: SLACK_TOKEN
+      secretKey: SLACK_TOKEN
+    - remoteRef: 
+        key: sample-app-secret
+        property: SLACK_CHANNEL
+      secretKey: SLACK_CHANNEL
+    - remoteRef: 
+        key: sample-app-secret
+        property: K8S_DB_USERNAME
+      secretKey: K8S_DB_USERNAME
+    - remoteRef: 
+        key: sample-app-secret
+        property: K8S_DB_PASSWORD
+      secretKey: K8S_DB_PASSWORD
+    - remoteRef: 
+        key: sample-app-secret
+        property: MYSQL_USER
+      secretKey: MYSQL_USER
+    - remoteRef: 
+        key: sample-app-secret
+        property: MYSQL_ROOT_PASSWORD
+      secretKey: MYSQL_ROOT_PASSWORD
+    - remoteRef: 
+        key: sample-app-secret
+        property: MYSQL_PASSWORD
+      secretKey: MYSQL_PASSWORD
+    - remoteRef:
+        key: sample-app-secret 
+        property: K8S_DB_DATABASE
+      secretKey: K8S_DB_DATABASE
+    - remoteRef: 
+        key: sample-app-secret 
+        property: MYSQL_DATABASE
+      secretKey: MYSQL_DATABASE
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: k8s-sample-api
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      run: k8s-sample-api
+  template:
+    metadata:
+      labels:
+        run: k8s-sample-api
+    spec:
+      #serviceAccountName: dynamo-users
+      containers:
+        - name: k8s-sample-api
+          image: 188966951897.dkr.ecr.us-west-2.amazonaws.com/k8s-sample/api
+          command: ["/code/run.sh"]
+          ports:
+            - containerPort: 80
+          env: 
+            - name: FLUENTD_HOST
+              value: "fluentd.fluentd"
+            - name: FLUENTD_PORT
+              value: "24224"
+          envFrom:
+            - secretRef:
+                name: k8s-sample-db-secrets
+            - configMapRef:
+                name: k8s-sample-db-cm
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mariadb
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      k8s-app: mariadb
+  template:
+    metadata:
+      labels:
+        k8s-app: mariadb
+    spec:
+      containers:
+        - name: mariadb
+          image: mariadb:10.3.8
+          ports:
+            - containerPort: 3306
+          envFrom:
+            - secretRef:
+                name: k8s-sample-db-secrets
+            - configMapRef:
+                name: k8s-sample-db-cm
+---
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    run: k8s-sample-api
+  name: k8s-sample-api
+spec:
+  ports:
+    - port: 80
+      protocol: TCP
+      targetPort: 80
+      name: metrics
+  selector:
+    run: k8s-sample-api
+---
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    k8s-app: mariadb
+  name: mariadb
+spec:
+  ports:
+    - port: 3306
+      protocol: TCP
+      targetPort: 3306
+  selector:
+    k8s-app: mariadb
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    kubernetes.io/ingress.class: nginx
+  name: k8s-sample-api
+spec:
+  tls:
+    - hosts:
+        - api.k8s.castlerock.ai
+      secretName: k8s-sample-api-tls-cert
+  rules:
+    - host: api.k8s.castlerock.ai
+      http:
+        paths:
+          - backend:
+              service:
+                name: k8s-sample-api
+                port:
+                  number: 80
+            pathType: Prefix
+            path: "/"