A Day in Java Developer’s Life, with a taste of Kubernetes

Table of Contents

• Prerequisite - Create a Kubernetes Cluster on Amazon EKS on AWS
• Application
• Build and Test using Maven
• Build and Test using Docker
  • Build Docker Image using multi-stage Dockerfile
  • Test built container using Docker
• Minimal Docker Image using Custom JRE (Optional)
• Build and Test using Kubernetes
• Debug Docker and Kubernetes
  • Tailing logs ..
  • Debug using Kubernetes using IntelliJ (Optional)
• Migrate from Dev to Prod
• Service Mesh using AWS App Mesh
  • Setup IAM Permissions
  • Configure App Mesh
  • Create App Mesh Components
  • Traffic Shifting
• Deployment Pipeline using CodePipeline
  • Create IAM role
  • Create CloudFormation template
  • View CodePipeline
• Finally, lets cleanup

Deploying your Java application in a Kubernetes cluster could feel like Alice in Wonderland. You keep going down the rabbit hole and don’t know how to make that ride comfortable. This repository explains how a Java application can be deployed, tested, debugged and monitored in Kubernetes. In addition, it also talks about canary deployment and deployment pipeline.

These instructions are a shorter/modified version of the original sample code available at https://github.com/aws-samples/kubernetes-for-java-developers by Arun Gupta and others ..

A comprehensive hands-on course explaining these concepts is available at https://www.linkedin.com/learning/kubernetes-for-java-developers.

Prerequisite - Create a Kubernetes Cluster on Amazon EKS on AWS

This application will be deployed to an Amazon EKS cluster. If you’re looking for a self-paced workshop that provide detailed instructions to get you started with EKS then eksworkshop.com is your place.

Let’s create the cluster first.

1. Install eksctl CLI:

   brew install weaveworks/tap/eksctl

2. Create EKS cluster:

   eksctl create cluster --name myeks --nodes 4 --region us-west-2
   2018-10-25T13:45:38+02:00 [ℹ]  setting availability zones to [us-west-2a us-west-2c us-west-2b]
   2018-10-25T13:45:39+02:00 [ℹ]  using "ami-0a54c984b9f908c81" for nodes
   2018-10-25T13:45:39+02:00 [ℹ]  creating EKS cluster "myeks" in "us-west-2" region
   2018-10-25T13:45:39+02:00 [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
   2018-10-25T13:45:39+02:00 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --name=myeks'
   2018-10-25T13:45:39+02:00 [ℹ]  creating cluster stack "eksctl-myeks-cluster"
   2018-10-25T13:57:33+02:00 [ℹ]  creating nodegroup stack "eksctl-myeks-nodegroup-0"
   2018-10-25T14:01:18+02:00 [✔]  all EKS cluster resource for "myeks" had been created
   2018-10-25T14:01:18+02:00 [✔]  saved kubeconfig as "/Users/argu/.kube/config"
   2018-10-25T14:01:19+02:00 [ℹ]  the cluster has 0 nodes
   2018-10-25T14:01:19+02:00 [ℹ]  waiting for at least 4 nodes to become ready
   2018-10-25T14:01:50+02:00 [ℹ]  the cluster has 4 nodes
   2018-10-25T14:01:50+02:00 [ℹ]  node "ip-192-168-161-180.us-west-2.compute.internal" is ready
   2018-10-25T14:01:50+02:00 [ℹ]  node "ip-192-168-214-48.us-west-2.compute.internal" is ready
   2018-10-25T14:01:50+02:00 [ℹ]  node "ip-192-168-75-44.us-west-2.compute.internal" is ready
   2018-10-25T14:01:50+02:00 [ℹ]  node "ip-192-168-82-236.us-west-2.compute.internal" is ready
   2018-10-25T14:01:52+02:00 [ℹ]  kubectl command should work with "/Users/argu/.kube/config", try 'kubectl get nodes'
   2018-10-25T14:01:52+02:00 [✔]  EKS cluster "myeks" in "us-west-2" region is ready

3. Check the nodes:

   kubectl get nodes
   NAME                                            STATUS   ROLES    AGE   VERSION
   ip-192-168-161-180.us-west-2.compute.internal   Ready    <none>   52s   v1.10.3
   ip-192-168-214-48.us-west-2.compute.internal    Ready    <none>   57s   v1.10.3
   ip-192-168-75-44.us-west-2.compute.internal     Ready    <none>   57s   v1.10.3
   ip-192-168-82-236.us-west-2.compute.internal    Ready    <none>   54s   v1.10.3

4. Get the list of configs:

   kubectl config get-contexts
   CURRENT   NAME                             CLUSTER                      AUTHINFO                         NAMESPACE
   *         [email protected]   myeks.us-west-2.eksctl.io    [email protected]
             docker-for-desktop               docker-for-desktop-cluster   docker-for-desktop

   As indicated by *, kubectl CLI configuration is updated to the recently created cluster.

Application

We will use a simple Java application built using Spring Boot. The application publishes a REST endpoint that can be invoked at http://{host}:{port}/hello.

The source code is in the app directory.

Build and Test using Maven

1. Run application:

   cd app
   mvn spring-boot:run

2. Test application

   curl http://localhost:8080/hello

Build and Test using Docker

Build Docker Image using multi-stage Dockerfile

1. Create m2.tar.gz:

   mvn -Dmaven.repo.local=./m2 clean package
   tar cvf m2.tar.gz ./m2

2. Create Docker image:

   docker image build -t arungupta/greeting .

   Explain multi-stage Dockerfile.

Test built container using Docker

1. Run container:

   docker container run --name greeting -p 8080:8080 -d arungupta/greeting

2. Access application:

   curl http://localhost:8080/hello

3. Remove container:

   docker container rm -f greeting

Minimal Docker Image using Custom JRE (Optional)

1. Download JDK 11 and scp to an Amazon Linux instance

2. Install JDK 11:

   sudo yum install jdk-11.0.1_linux-x64_bin.rpm

3. Create a custom JRE for the Spring Boot application:

   cp target/app.war target/app.jar
   jlink \
   	--output myjre \
   	--add-modules $(jdeps --print-module-deps target/app.jar),\
   	java.xml,jdk.unsupported,java.sql,java.naming,java.desktop,\
   	java.management,java.security.jgss,java.instrument

4. Build Docker image using this custom JRE:

   docker image build --file Dockerfile.jre -t arungupta/greeting:jre-slim .

5. List the Docker images and show the difference in sizes:

   [ec2-user@ip-172-31-21-7 app]$ docker image ls | grep greeting
   arungupta/greeting   jre-slim            9eed25582f36        6 seconds ago       162MB
   arungupta/greeting   latest              1b7c061dad60        10 hours ago        490MB

6. Run the container:

   docker container run -d -p 8080:8080 arungupta/greeting:jre-slim

7. Access the application:

   curl http://localhost:8080/hello

Build and Test using Kubernetes

Kubernetes can be easily enabled on a development machine using Docker for Mac as explained at https://docs.docker.com/docker-for-mac/#kubernetes.

1. Ensure that Kubernetes is enabled in Docker for Mac

2. Show the list of contexts:

   kubectl config get-contexts

3. Configure kubectl CLI for Kubernetes cluster

   kubectl config use-context docker-for-desktop

4. Install the Helm CLI:

   brew install kubernetes-helm

   If Helm CLI is already installed then use brew upgrade kubernetes-helm.

5. Check Helm version:

   helm version

6. Install Helm in Kubernetes cluster:

   helm init

   If Helm has already been initialized on the cluster, then you may have to upgrade Tiller:

   helm init --upgrade

7. Install the Helm chart:

   cd ..
   helm install --name myapp manifests/myapp

8. Check that the pod is running:

   kubectl get pods

9. Check that the service is up:

   kubectl get svc

10. Access the application:

    curl http://$(kubectl get svc/myapp-greeting \
    	-o jsonpath='{.status.loadBalancer.ingress[0].hostname}'):80/hello

Debug Docker and Kubernetes

You can debug a Docker container and a Kubernetes Pod if they’re running locally on your machine.

Tailing logs ..

One of the things that Java developers need to know, is check the application logs, tail the logs and exec into pods. This is well described in this blogpost by Christian Posta - https://blog.christianposta.com/kubernetes/java-remote-debug-for-applications-running-in-kubernetes/

1. List the pods:

   $ kubectl get pod
   NAME                              READY   STATUS    RESTARTS   AGE
   myapp-greeting-67dd4bb9d5-r2qts   1/1     Running   0          19h

2. Tail the logs:

   $  kubectl logs -f myapp-greeting-67dd4bb9d5-r2qts

3. Exec into the shell

   $  kubectl exec -it myapp-greeting-67dd4bb9d5-r2qts bash

Debug using Kubernetes using IntelliJ (Optional)

This was tested using Docker for Mac/Kubernetes. Use the previously deployed Helm chart.

1. Show service:

   kubectl get svc
   NAME               TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
   greeting-service   LoadBalancer   10.101.39.100    <pending>     80:30854/TCP                    8m
   kubernetes         ClusterIP      10.96.0.1        <none>        443/TCP                         90d
   myapp-greeting     LoadBalancer   10.108.104.178   localhost     8080:32189/TCP,5005:31117/TCP   4s

   Highlight the debug port is also forwarded.

2. In IntelliJ, Run, Debug, Remote:

   

3. Click on Debug, setup a breakpoint in the class:

   

4. Access the application:

   curl http://$(kubectl get svc/myapp-greeting \
   	-o jsonpath='{.status.loadBalancer.ingress[0].hostname}'):80/hello

5. Show the breakpoint hit in IntelliJ:

   

6. Delete the Helm chart:

   helm delete --purge myapp

Migrate from Dev to Prod

1. Explicitly set the context:

   kubectl config use-context [email protected]

2. Install Helm:

   kubectl -n kube-system create sa tiller
   kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
   helm init --service-account tiller

3. Check the list of pods:

   kubectl get pods -n kube-system
   NAME                            READY   STATUS    RESTARTS   AGE
   aws-node-774jf                  1/1     Running   1          2m
   aws-node-jrf5r                  1/1     Running   0          2m
   aws-node-n46tw                  1/1     Running   0          2m
   aws-node-slgns                  1/1     Running   0          2m
   kube-dns-7cc87d595-5tskv        3/3     Running   0          8m
   kube-proxy-2ghg6                1/1     Running   0          2m
   kube-proxy-hqxwg                1/1     Running   0          2m
   kube-proxy-lrwrr                1/1     Running   0          2m
   kube-proxy-x77tq                1/1     Running   0          2m
   tiller-deploy-895d57dd9-txqk4   1/1     Running   0          15s

4. Redeploy the application:

   helm install --name myapp manifests/myapp

5. Get the service:

   kubectl get svc
   NAME             TYPE           CLUSTER-IP       EXTERNAL-IP                                                             PORT(S)                         AGE
   kubernetes       ClusterIP      10.100.0.1       <none>                                                                  443/TCP                         17m
   myapp-greeting   LoadBalancer   10.100.241.250   a8713338abef211e8970816cb629d414-71232674.us-east-1.elb.amazonaws.com   8080:32626/TCP,5005:30739/TCP   2m

   It shows the port 8080 and 5005 are published and an Elastic Load Balancer is provisioned. It takes about three minutes for the load balancer to be ready.

6. Access the application:

   curl http://$(kubectl get svc/myapp-greeting \
   	-o jsonpath='{.status.loadBalancer.ingress[0].hostname}'):8080/hello

7. Delete the application:

   helm delete --purge myapp

Service Mesh using AWS App Mesh

AWS App Mesh is a service mesh that provides application-level networking to make it easy for your services to communicate with each other across multiple types of compute infrastructure. App Mesh can be used with Amazon EKS or Kubernetes running on AWS. In addition, it also works with other container services offered by AWS such as AWS Fargate and Amazon ECS. It also works with microservices deployed on Amazon EC2.

A thorough detailed example that shows how to use App Mesh with EKS is available at Service Mesh with App Mesh. This section provides a simplistic setup using the configuration files from there.

All scripts used in this section are in the manifests/appmesh directory.

Setup IAM Permissions

1. Set a variable ROLE_NAME to IAM role for the EKS worker nodes:

   ROLE_NAME=$(aws iam list-roles \
   	--query \
   	'Roles[?contains(RoleName,`eksctl-myeks-nodegroup`)].RoleName' --output text)

2. Setup permissions for the worker nodes:

   aws iam attach-role-policy \
   	--role-name $ROLE_NAME \
   	--policy-arn arn:aws:iam::aws:policy/AWSAppMeshFullAccess

Configure App Mesh

1. Enable side-car injection by running create.sh script from https://github.com/aws/aws-app-mesh-examples/tree/master/examples/apps/djapp/2_create_injector. You need to change ca-bundle.sh and change MESH_NAME to greeting-app.

2. Create prod namespace:

   kubectl create namespace prod

3. Label prod namespace:

   kubectl label namespace prod appmesh.k8s.aws/sidecarInjectorWebhook=enabled

4. Create CRDs:

   kubectl create -f https://raw.githubusercontent.com/aws/aws-app-mesh-examples/master/examples/apps/djapp/3_add_crds/mesh-definition.yaml
   kubectl create -f https://raw.githubusercontent.com/aws/aws-app-mesh-examples/master/examples/apps/djapp/3_add_crds/virtual-node-definition.yaml
   kubectl create -f https://raw.githubusercontent.com/aws/aws-app-mesh-examples/master/examples/apps/djapp/3_add_crds/virtual-service-definition.yaml
   kubectl create -f https://raw.githubusercontent.com/aws/aws-app-mesh-examples/master/examples/apps/djapp/3_add_crds/controller-deployment.yaml

Create App Mesh Components

1. Change directories

   cd manifests/appmesh

2. Create a Mesh:

   kubectl create -f mesh.yaml

3. Create Virtual Nodes:

   kubectl create -f virtualnodes.yaml

4. Create a Virtual Services:

   kubectl create -f virtualservice.yaml

5. Create deployments:

   kubectl create -f app-hello-howdy.yaml

6. Create services:

   kubectl create -f services.yaml

Traffic Shifting

1. Find the name of the talker pod:

   TALKER_POD=$(kubectl get pods \
   	-nprod -lgreeting=talker \
   	-o jsonpath='{.items[0].metadata.name}')

2. Exec into the talker pod:

   kubectl exec -nprod $TALKER_POD -it bash

3. Invoke the mostly-hello service to get back mostly Hello response:

   while [ 1 ]; do curl http://mostly-hello.prod.svc.cluster.local:8080/hello; echo;done

4. CTRL+C to break the loop.

5. Invoke the mostly-howdy service to get back mostly Howdy response:

   while [ 1 ]; do curl http://mostly-howdy.prod.svc.cluster.local:8080/hello; echo;done

6. CTRL+C to break the loop.

Deployment Pipeline using CodePipeline

Complete detailed instructions are available at https://eksworkshop.com/codepipeline/.

Create IAM role

1. Create an IAM role and add an in-line policy that will allow the CodeBuild stage to interact with the EKS cluster:

   ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
   TRUST="{ \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Principal\": { \"AWS\": \"arn:aws:iam::${ACCOUNT_ID}:root\" }, \"Action\": \"sts:AssumeRole\" } ] }"
   echo '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "eks:Describe*", "Resource": "*" } ] }' > /tmp/iam-role-policy
   aws iam create-role --role-name EksWorkshopCodeBuildKubectlRole --assume-role-policy-document "$TRUST" --output text --query 'Role.Arn'
   aws iam put-role-policy --role-name EksWorkshopCodeBuildKubectlRole --policy-name eks-describe --policy-document file:///tmp/iam-role-policy

2. Add this IAM role to aws-auth ConfigMap for the EKS cluster:

   ROLE="    - rolearn: arn:aws:iam::$ACCOUNT_ID:role/EksWorkshopCodeBuildKubectlRole\n      username: build\n      groups:\n        - system:masters"
   kubectl get -n kube-system configmap/aws-auth -o yaml | awk "/mapRoles: \|/{print;print \"$ROLE\";next}1" > /tmp/aws-auth-patch.yml
   kubectl patch configmap/aws-auth -n kube-system --patch "$(cat /tmp/aws-auth-patch.yml)"

Create CloudFormation template

1. Fork the repo https://github.com/aws-samples/kubernetes-for-java-developers

2. Create a new GitHub token https://github.com/settings/tokens/new, select repo as the scope, click on Generate Token to generate the token. Copy the generated token.

3. Launch CodePipeline CloudFormation template.

4. Specify the correct values for GitHubUser, GitHubToken, GitSourceRepo and EKS cluster name. Change the branch if you need to:

   

   Click on Create stack to create the resources.

View CodePipeline

1. Once the stack creation is complete, open CodePipeline in the AWS Console.

2. Select the pipeline and wait for the pipeline status to complete:

   

3. Access the service:

   curl http://$(kubectl get svc/greeting -n default \
   	-o jsonpath='{.status.loadBalancer.ingress[0].hostname}'):8080/hello

Finally, lets cleanup

Like deleting the EKS cluster and other stuff ..

1. Delete the Codepipeline cloudformation template:

2. Delete EKS cluster:

   eksctl delete cluster --name myeks --region us-west-2