finished everything up to the FEK stack and started terraform for the…

… FEK stack.

02/terragrunt/live/stage-fek/terragrunt.hcl

@@ -0,0 +1,41 @@
+remote_state {
+  backend = "s3"
+  generate = {
+    path = "backend.tf"
+    if_exists = "overwrite_terragrunt"
+  }
+  config = {
+    bucket = "k8sclass-tf-state"
+    key = "stage/${path_relative_to_include()}/terraform.tfstate"
+    region = "us-west-2"
+    encrypt = true
+  }
+}
+
+generate "provider" {
+  path = "provider.tf"
+  if_exists = "overwrite_terragrunt"
+  contents = <<EOF
+provider "aws" {
+  region  = "us-west-2"
+  assume_role {
+    role_arn = local.iam_state.eks_dude_role.arn
+  }
+}
+EOF
+}
+
+generate "locals" {
+  path = "state.tf"
+  if_exists = "overwrite_terragrunt"
+  contents = <<EOF
+data "terraform_remote_state" "iam" {
+  backend = "s3"
+  config = {
+    region = "us-west-2"
+    bucket = "k8sclass-tf-state"
+    key    = "iam/terraform.tfstate"
+  }
+}
+EOF
+}

README.md

@@ -70,5 +70,11 @@ In this segment we'll go over some Kubernetes primitives.  We move fast, show wh
 
 * [README](m04/README.md)
 
+### M05 - Integrate Grafana/Prometheus in our Application
+
+* [README](m05/README.md)
+
+
+
 
 

app-api/app-api.yaml

@@ -129,6 +129,7 @@ spec:
     - port: 80
       protocol: TCP
       targetPort: 80
+      name: metrics
   selector:
     run: k8s-sample-api
 ---

app-api/app/main.py

@@ -10,14 +10,20 @@
 
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
-from pydantic import BaseModel
+from fastapi_utils.tasks import repeat_every
 from app.routers import base, user, auth
-from app.database import database
+from app.database import database, engine
 
 app = FastAPI()
 app.header = {}
 
 
+# PROMETHEUS (part 5)
+from prometheus_fastapi_instrumentator import Instrumentator
+from prometheus_client import Gauge
+
+# END PROMETHEUS (part 5)
+
 origins = [
     "http://localhost",
     "http://localhost:3000",
@@ -37,6 +43,22 @@
 app.include_router(user.router)
 app.include_router(auth.router)
 
+# PREMETHEUS (part 5)
+@app.on_event("startup")
+def init_instrumentator():
+    """Setup prometheus instrumentation for the API"""
+    Instrumentator().instrument(app).expose(app)
+
+
+@app.on_event("startup")
+@repeat_every(seconds=30, wait_first=True)
+def periodic():
+    count = engine.execute("select count(id) from user").scalar()
+    Gauge("total_users", "Total Users").set(int(count))
+
+
+# END PROMETHEUS (part 5)
+
 
 @app.on_event("startup")
 async def startup():

app-api/app/routers/auth.py

@@ -21,6 +21,15 @@
 
 router = APIRouter(tags=["auth"])
 
+# PROMETHEUS (part 5)
+from prometheus_client import Counter
+
+SUCESSFUL_LOGIN_DETAILS = Counter(
+    "successful_login_details", "Successful login details"
+)
+FAILED_LOGIN_DETAILS = Counter("failed_login_details", "Failed login details")
+# END PROMETHEUS (part 5)
+
 
 @router.post("/auth/signup", response_model=schemas.users.User, status_code=201)  # 1
 def create_user_signup(
@@ -60,8 +69,14 @@ def login(
         db=db,
     )
     if not user:
+        # PROMETHEUS (part 5)
+        FAILED_LOGIN_DETAILS.inc()
+        # END PROMETHEUS (part 5)
         raise HTTPException(status_code=400, detail="Incorrect username or password")
 
+    # PROMETHEUS (part 5)
+    SUCESSFUL_LOGIN_DETAILS.inc()
+    # END PROMETHEUS (part 5)
     return {
         "access_token": create_access_token(sub=user.id),  # 4
         "token_type": "bearer",

app-api/requirements.txt

@@ -1,6 +1,7 @@
 alembic==1.7.6
 bcrypt==3.2.0
 fastapi==0.73.0
+fastapi-utils==0.2.1
 databases[mysql]==0.5.5 
 passlib==1.7.4
 pydantic[email] ~= 1.9.0
@@ -14,3 +15,5 @@ sqlalchemy ~= 1.4.31
 stripe ~= 2.65.0
 urllib3 ~= 1.26.8
 uvicorn ~= 0.17.5
+prometheus-client ~= 0.11.0
+prometheus-fastapi-instrumentator ~= 5.7

m05/README.md

@@ -0,0 +1,191 @@
+# Prometheus to Application
+
+
+## FastAPI and Prometheus
+
+It's worth checking to see what default prometheus libraries already come with whatever framework you are using.  In our case we are using FastAPI as our application framework.  There is a library called [Prometheus FastAPI Instrumentator](https://github.com/trallnag/prometheus-fastapi-instrumentator) that provides us some of this functionality out of the box. 
+
+We can put this in our application in the `main.py`. 
+
+
+```
+from prometheus_fastapi_instrumentator import Instrumentator
+``` 
+
+Then we make a startup function for our application: 
+
+```
+@app.on_event("startup")
+def init_instrumentator():
+    """Setup prometheus instrumentation for the API"""
+    Instrumentator().instrument(app).expose(app)
+```
+
+Restarting our development environment we see that this creates a `/metrics` endpoint for our application.  You can see this on the app: 
+
+![](../images/mo/m05-01.png)
+
+We don't typically like to expose metrics like this outside of Kubernetes to the outside world, but for this application we are ok for now. 
+
+Ways to handle this:
+
+* There may be instead a front end for backend, so keep this entire API not exposed.
+* Others?
+
+## Prometheus Client
+
+There is also the [Official Python Prometheus Client](https://github.com/prometheus/client_python) that we can use to get even more customized with our metrics.
+
+To use this we start defining our own metrics.  For this example we will monitor successful and failed login attempts on our application.  
+
+This can be done by editing the `app/routers/auth.py` file where the login happens.  
+
+We add the following towards the top of the file: 
+
+```
+from prometheus_client import Counter
+
+SUCESSFUL_LOGIN_DETAILS = Counter(
+    "successful_login_details", "Successful login details"
+)
+FAILED_LOGIN_DETAILS = Counter("failed_login_details", "Failed login details")
+```
+
+This defines the Counter we are going to instrument.  Next we can add this to the `login` function at the bottom: 
+
+```
+ if not user:
+        # PROMETHEUS (part 5)
+        FAILED_LOGIN_DETAILS.inc()
+        # END PROMETHEUS (part 5)
+        raise HTTPException(status_code=400, detail="Incorrect username or password")
+
+    # PROMETHEUS (part 5)
+    SUCESSFUL_LOGIN_DETAILS.inc()
+    # END PROMETHEUS (part 5)
+```
+Here we are incrementing the counters.  
+
+Another place we can can have a metric is the total number of users in our application.  For this we query the database periodically to see what our count is. 
+
+In `main.py` we can do: 
+
+```
+from prometheus_client import Gauge
+```
+
+And then a new startup section: 
+
+```
+@app.on_event("startup")
+@repeat_every(seconds=30, wait_first=True)
+def periodic():
+    count = engine.execute("select count(id) from user").scalar()
+    Gauge("total_users", "Total Users").set(int(count))
+```
+We can look at our metics page and search for these metrics. Let's push this up to our kubernetes image: 
+
+
+```
+make push 
+kubectl rollout restart deployment k8s-sample-api
+```
+
+Make sure it restarted by visiting: [https://api.k8s.castlerock.ai/metrics](https://api.k8s.castlerock.ai/metrics).
+
+
+## Scraping
+
+Now that our metrics are up, we need to tell Prometheus to scrape the metrics.  To do this we create a `ServiceMonitor`.  This is a custom resource definition:
+
+
+```
+kubectl apply -f servicemonitor.yaml
+```
+
+Prometheus should pick up this change and you should now see the app in the prometheus targets dashboard.  
+
+![](../images/mo/m05-02.png)
+
+
+Now showing?  There is a [troubleshooting document](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md#troubleshooting-servicemonitor-changes)
+
+## Grafana Integration
+
+Now that we have new metrics from our application, we can add this to our dashboard in Grafana.  This gives us a cool overview of how many users we have as well as failed and successful login attempts!
+
+### Adding the Total Users Dashboard
+
+In Grafana: 
+
+* Add a panel 
+* In the metrics browser just use `total_users`.  Since this is just a gauge and a constant metric we can simply see this and see how many users we have. Since we would rather not have all the labels shown for this simple metric, we can just use the sum.  `sum(total_users)`. If we had multiple containers we may have to change this. 
+* The legend can be changed as well to show something more clean and meaningful.
+
+![](../images/mo/m05-03.png)
+
+We'll see the total users gauge start to form. 
+
+Click `Apply` to see this in our main dashboard. 
+
+### Adding the Login Success/Failures
+
+* Add a Panel
+* Title: Login Success/Failure Rates
+* This time we are using `Counters` and not a `Gauge` so we should use a rate.  (That way when the application restarts we can see it). 
+* The first one is the successful logins: `sum(rate(successful_login_details_total[1m]))`
+* Make it green and update the legend to say `Successful Logins`
+
+* Add a new query with the failed logins metric: `sum(rate(failed_login_details_total[1m]))`
+* Make it red and update the legend to show `Failed Logins`
+
+
+![](../images/mo/m05-04.png)
+
+
+
+Click `Apply` to make this show.  You can arrange the dashboards to get a pretty good overview chart.  What a great application dashboard we have now!
+
+![](../images/mo/m05-05.png)
+
+### Testing it out
+
+Let's make sure this graph works by adding a few users and trying to login a few times. 
+
+![](../images/mo/m05-06.png)
+
+We could make our rates make more sense, but here we see there are some number of successful logins and some failures. 
+
+Similarly for the users:  
+
+![](../images/mo/m05-07.png)
+
+
+
+### Adding persistence
+
+To make this a general part of our stack, save the json file and then put it in the existing `configMap` file we already created.  This will be pretty long!  Again, we usually use helm templates to reference the file so we don't have to get the alignment working. 
+
+There is an export button to view the JSON. Then you can copy it and use it
+
+```
+kubectl apply -f k8s-configMap.yaml
+```
+
+Now we can restart grafana: 
+
+```
+kubectl rollout restart -n monitoring deployment kube-prom-grafana
+```
+
+Now our dashboard persists!
+
+
+
+
+
+
+
+
+
+