-
Notifications
You must be signed in to change notification settings - Fork 471
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Houssem Dellai
committed
Oct 13, 2023
1 parent
2fa8483
commit 8c1b8af
Showing
17 changed files
with
575 additions
and
16 deletions.
There are no files selected for viewing
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,17 +1,16 @@ | ||
# Scan cluster configuration using Kube-Bench | ||
# src: https://github.com/aquasecurity/kube-bench | ||
|
||
kubectl cluster-info | ||
# Deploy job.yaml to scan Kubernetes config and job-aks.yaml to scan specific AKS config | ||
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml | ||
# Pod logs outputs Kubernetes scan results and Job outputs AKS specific results | ||
POD=$(kubectl get pods --selector app=kube-bench -o name) | ||
kubectl logs $POD | ||
kubectl delete -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml | ||
kubectl cluster-info | ||
# Deploy job.yaml to scan Kubernetes config and job-aks.yaml to scan specific AKS config | ||
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml | ||
# Pod logs outputs Kubernetes scan results and Job outputs AKS specific results | ||
POD=$(kubectl get pods --selector app=kube-bench -o name) | ||
kubectl logs $POD | ||
kubectl delete -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml | ||
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-aks.yaml | ||
|
||
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-aks.yaml | ||
|
||
JOB=(kubectl get jobs --selector job-name=kube-bench -o name) | ||
kubectl logs $JOB | ||
# TODO upload the scan results | ||
kubectl delete -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-aks.yaml | ||
JOB=(kubectl get jobs --selector job-name=kube-bench -o name) | ||
kubectl logs $JOB | ||
# TODO upload the scan results | ||
kubectl delete -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-aks.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,194 @@ | ||
## Restrict Egress/Outbound Layer 3 traffic using Calico Network Policy | ||
|
||
## 1. Create demo environment | ||
|
||
```shell | ||
$RG_NAME = "rg-aks-cluster-calico" | ||
$AKS_NAME = "aks-cluster-calico" | ||
|
||
# create an azure rsource group | ||
az group create -n $RG_NAME --location westeurope | ||
|
||
# create AKS cluster with Calico enabled | ||
az aks create -g $RG_NAME -n $AKS_NAME --network-plugin azure --network-policy calico | ||
``` | ||
|
||
Get AKS credentials | ||
|
||
```shell | ||
az aks get-credentials -g $RG_NAME -n $AKS_NAME --overwrite-existing | ||
``` | ||
|
||
Deploy sample online service, just to get public IP | ||
|
||
```shell | ||
$EXTERNAL_IP=(az container create -g $RG_NAME -n aci-app --image nginx:latest --ports 80 --ip-address public --query ipAddress.ip --output tsv) | ||
$EXTERNAL_IP | ||
# 20.76.101.193 | ||
``` | ||
|
||
Deploy nginx pod | ||
|
||
```shell | ||
kubectl run nginx --image=nginx | ||
``` | ||
|
||
Access the external service from nginx pod | ||
|
||
```shell | ||
kubectl exec -it nginx -- curl http://$EXTERNAL_IP | ||
# <title>Welcome to nginx!</title> | ||
# access is allowed | ||
``` | ||
|
||
## 2. Restrict all ingress and egress traffic | ||
|
||
Deny all ingress and egress traffic using the following network policy. | ||
|
||
```yaml | ||
apiVersion: networking.k8s.io/v1 | ||
kind: NetworkPolicy | ||
metadata: | ||
name: default-deny-all | ||
spec: | ||
podSelector: {} | ||
policyTypes: | ||
- Ingress | ||
- Egress | ||
``` | ||
Deploy network policy to deny all traffic. | ||
```shell | ||
kubectl apply -f deny-all.yaml | ||
``` | ||
|
||
Verify egress traffic is denied | ||
|
||
```shell | ||
kubectl exec -it nginx -- curl http://$EXTERNAL_IP --max-time 5 | ||
# timeout | ||
``` | ||
|
||
## 3. Allow egress traffic to external service IP address | ||
|
||
Allow Nginx pod egress traffic to external service IP address by using a network policy like the following. | ||
|
||
```yaml | ||
apiVersion: networking.k8s.io/v1 | ||
kind: NetworkPolicy | ||
metadata: | ||
name: allow-egress-ip | ||
spec: | ||
podSelector: | ||
matchLabels: | ||
run: nginx | ||
policyTypes: | ||
- Egress | ||
egress: | ||
- to: | ||
- ipBlock: | ||
cidr: 20.126.233.217/32 | ||
``` | ||
Make sure to update pod labels and destination IP address or CIDR range. | ||
Get labels of Nginx pod. | ||
```shell | ||
kubectl get pods --show-labels | ||
``` | ||
|
||
Deploy the allow egress policy. | ||
|
||
```shell | ||
kubectl apply -f allow-egress-ip.yaml | ||
``` | ||
|
||
Access the external service from nginx pod | ||
|
||
```shell | ||
kubectl exec -it nginx -- curl http://$EXTERNAL_IP | ||
# <title>Welcome to nginx!</title> | ||
``` | ||
|
||
## 4. Verify that egress traffic to external IP is blocked to other pods | ||
|
||
Verify with another pod with different labels | ||
|
||
```shell | ||
kubectl run nginx1 --image=nginx | ||
kubectl exec -it nginx1 -- curl http://$EXTERNAL_IP --max-time 5 | ||
# access denied | ||
``` | ||
|
||
## 5. Logging denied traffic | ||
|
||
Calico enables logging denied and allowed traffic to `syslog`. | ||
you need to use the `Calico Network Policy` instead of `Kubernetes Network Policy`. | ||
you need to install Calico API Server to deploy Calico Network Policy using `kubectl` instead of `calicoctl`. | ||
Src: [https://docs.tigera.io/calico/latest/operations/install-apiserver](https://docs.tigera.io/calico/latest/operations/install-apiserver) | ||
|
||
```shell | ||
kubectl apply -f calico-apiserver.yaml | ||
# apiserver.operator.tigera.io/default created | ||
|
||
kubectl get tigerastatus apiserver | ||
# NAME AVAILABLE PROGRESSING DEGRADED SINCE | ||
# apiserver True False False 119s | ||
``` | ||
|
||
Deploy Calico Network Policy that denies egress traffic and logs it to syslog. | ||
|
||
Once the `Calico API server` has been installed, you can use `kubectl` to interact with the Calico APIs. | ||
|
||
deploy calico network policy to enable logging. | ||
|
||
```yaml | ||
kind: NetworkPolicy | ||
apiVersion: projectcalico.org/v3 | ||
metadata: | ||
name: logging-traffic | ||
spec: | ||
selector: run=='nginx' | ||
types: | ||
- Egress | ||
egress: | ||
- action: Log | ||
destination: | ||
nets: | ||
- 20.126.233.217/32 | ||
- action: Deny | ||
destination: | ||
nets: | ||
- 20.126.233.217/32 | ||
``` | ||
```shell | ||
kubectl apply -f logging-traffic.yaml | ||
``` | ||
|
||
Enable AKS monitoring and syslog features as logged traffic will be sent to `syslog`. | ||
|
||
```shell | ||
az aks enable-addons -a monitoring --enable-syslog -g $RG_NAME -n $AKS_NAME | ||
``` | ||
|
||
Test logging with nginx pod. | ||
|
||
```shell | ||
kubectl exec -it nginx1 -- curl http://$EXTERNAL_IP --max-time 5 | ||
``` | ||
|
||
Check generated logs in Log Analytics by running this KQL query: | ||
|
||
```kql | ||
Syslog | ||
| project TimeGenerated, SyslogMessage | ||
| where SyslogMessage has "20.126.233.217" | ||
// | where SyslogMessage has "calico-packet" | ||
``` | ||
|
||
You should see the following result. | ||
|
||
[](images/logs.png) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
apiVersion: networking.k8s.io/v1 | ||
kind: NetworkPolicy | ||
metadata: | ||
name: allow-egress-ip | ||
spec: | ||
podSelector: | ||
matchLabels: | ||
run: nginx | ||
policyTypes: | ||
- Egress | ||
egress: | ||
- to: | ||
- ipBlock: | ||
cidr: 20.126.233.217/32 | ||
# --- | ||
# apiVersion: projectcalico.org/v3 | ||
# kind: NetworkPolicy | ||
# metadata: | ||
# name: allow-egress-external | ||
# namespace: production | ||
# spec: | ||
# selector: run == 'nginx' | ||
# types: | ||
# - Egress | ||
# egress: | ||
# - action: Allow | ||
# destination: | ||
# nets: | ||
# - 1.2.3.0/24 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
apiVersion: operator.tigera.io/v1 | ||
kind: APIServer | ||
metadata: | ||
name: default | ||
spec: {} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
## Restrict Egress/Outbound Layer 3 traffic using Calico Network Policy | ||
|
||
## 1. Create demo environment | ||
|
||
$RG_NAME = "rg-aks-cluster-calico" | ||
$AKS_NAME = "aks-cluster-calico" | ||
|
||
# create an azure rsource group | ||
az group create -n $RG_NAME --location westeurope | ||
|
||
# create AKS cluster with Calico enabled | ||
az aks create -g $RG_NAME -n $AKS_NAME --network-plugin azure --network-policy calico | ||
|
||
# get AKS credentials | ||
az aks get-credentials -g $RG_NAME -n $AKS_NAME --overwrite-existing | ||
|
||
# deploy sample online service, just to get public IP | ||
$EXTERNAL_IP=(az container create -g $RG_NAME -n aci-app --image nginx:latest --ports 80 --ip-address public --query ipAddress.ip --output tsv) | ||
$EXTERNAL_IP | ||
# 20.76.101.193 | ||
|
||
# deploy nginx pod | ||
kubectl run nginx --image=nginx | ||
|
||
# access the external service from nginx pod | ||
kubectl exec -it nginx -- curl http://$EXTERNAL_IP | ||
# <title>Welcome to nginx!</title> | ||
# access is allowed | ||
|
||
## 2. Restrict all ingress and egress traffic | ||
|
||
# deny all ingress and egress traffic | ||
kubectl apply -f deny-all.yaml | ||
|
||
kubectl exec -it nginx -- curl http://$EXTERNAL_IP --max-time 5 | ||
# timeout | ||
|
||
## 3. Allow egress traffic to external service IP address | ||
|
||
# get pods with labels | ||
kubectl get pods --show-labels | ||
|
||
# replace IP address in allow-egress-ip.yaml with the IP address of the external service and pod labels | ||
|
||
# allow egress traffic to external service IP address | ||
# replace IP address in allow-egress-ip.yaml with the IP address of the external service and pod labels | ||
kubectl apply -f allow-egress-ip.yaml | ||
|
||
# access the external service from nginx pod | ||
kubectl exec -it nginx -- curl http://$EXTERNAL_IP | ||
# <title>Welcome to nginx!</title> | ||
|
||
## 4. Verify that egress traffic to external IP is blocked to other pods | ||
|
||
# verify with another pod with different labels | ||
kubectl run nginx1 --image=nginx | ||
|
||
kubectl exec -it nginx1 -- curl http://$EXTERNAL_IP --max-time 5 | ||
# access denied | ||
|
||
## 5. Logging denied traffic | ||
|
||
# Calico enables logging denied and allowed traffic to syslog | ||
# you need to use the Calico Network Policy instead of Kubernetes Network Policy | ||
# you need to install Calico API Server to deploy Calico Network Policy using kubectl instead of calicoctl | ||
# Src: https://docs.tigera.io/calico/latest/operations/install-apiserver | ||
|
||
kubectl apply -f calico-apiserver.yaml | ||
# apiserver.operator.tigera.io/default created | ||
|
||
kubectl get tigerastatus apiserver | ||
# NAME AVAILABLE PROGRESSING DEGRADED SINCE | ||
# apiserver True False False 119s | ||
|
||
# deploy Calico Network Policy that denies egress traffic en logs it to syslog | ||
|
||
# Once the API server has been installed, you can use kubectl to interact with the Calico APIs. | ||
|
||
# deploy calico network policy to enable logging | ||
|
||
kubectl apply -f logging-traffic.yaml | ||
|
||
|
||
|
||
az aks enable-addons -a monitoring --enable-syslog -g $RG_NAME -n $AKS_NAME | ||
|
||
kubectl exec -it nginx1 -- curl http://$EXTERNAL_IP --max-time 5 | ||
|
||
# check generated logs in Log Analytics | ||
# run this KQL query: | ||
# Syslog | ||
# | project TimeGenerated, SyslogMessage | ||
# | where SyslogMessage has "20.126.233.217" | ||
# // | where SyslogMessage has "calico-packet" |
Oops, something went wrong.