-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathflux.sh
132 lines (116 loc) · 7.63 KB
/
flux.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
#
# Bootstrapping the cluster with Flux
# The bootstrap process will automatically create a GitRepository custom resource that points to the given repository
# The GitRepository resource is named after the namespace where Flux GitOps ToolKit is installed. In this case, it is 'flux-system'
# The bootstrap process will configure the repository with an SSH key for read-only access
#
export GITHUB_TOKEN=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
export GITHUB_USER=vijayansarathy
export CLUSTER_NAME=k8s-gitops-cluster
kubectl create ns flux-system
flux bootstrap github \
--owner=$GITHUB_USER \
--namespace=flux-system \
--repository=eks-gitops-crossplane-flux \
--ssh-key-algorithm=ecdsa \
--secret-name=flux-bootstrap \
--branch=main \
--path=clusters/$CLUSTER_NAME \
--personal
#
# Alternative method for bootstrapping Flux.
# This approach may be used to install Flux on a remote cluster in scenarios such as when using tools like Crossplane to provision and bootstrap one or more workload clusters from a management cluster.
# In this case, Flux that is running on the management cluster is used to bootstrap Flux on a workload cluster.
# Subsequently, Flux running on the workload cluster will sync the state of that cluster from the Git repo(s) that it has been pointed to.
# The Git repo used as the source of truth for Flux running on the workload cluster can, of course, be separate from the one that is used by Flux running on the management cluster.
# This may be done by installing the following components/tools on the workload cluster:
# 1. Install External Secrets Operator (ESO). This is used for retrieving secrets stored in AWS Secrets Manager and expose them as Kubernetes Secrets
# 2. Deploy SecretStore and ExternalSecret custom resources to make ESO retrieve the Sealed Secrets sealing keys stored in AWS Secrets Manager and create a Kubernetes Secret
# 3. Install Bitnami Sealed Secrets controller which will make use of these sealing keys to unseal SealedSecret custom resources deployed to the cluster
# 4. Deploy a SealedSecret custom resource that provides the SSH keys required to connect to a GitHub repository. This will get unsealed into a Kubernetes Secret
# 5. Deploy the Flux GitOps Toolkit. Manifests needed for this step can be created (one-time process) using the manifests auto-generated by Flux when we bootstrap a cluster with Flux CLI
# There are two files that make up the Flux GitOps Toolkit
# (i) gotk-components.yaml - defines all the CRDs and controllers that make up the toolkit.
# (ii) gotk-sync.yaml - defines a GitRepository and Kustomization resource which together desginate a Git repo and a path within that repo which contain the Kubernetes resources to be synced to the cluster b Flux
# The GitRepository resource references the Kubernetes Secret that was deployed in Step 4 above and uses the SSH keys from that Secret to connect to the Git repo.
# Note that we can make the workload cluster sync from multiple sources by deploying corresponding GitRepository and Kustomization resources as well as Secrets which provide the SSH keys for those repos.
#
#
########################################################################################################################################################
# The next three steps that involve generating a SealedSecret that contains the AWS credentials are not needed when using IRSA for Crossplane Controller
########################################################################################################################################################
#
# In order to authenticate with the external provider API such as AWS, the provider controllers need to have access to credentials.
# It could be an IAM User for AWS
# An AWS user with Administrative privileges is needed to enable Crossplane to create the required resources
# We wil have to first create a configuration file, secrets.conf, with credeantials of an AWS account in the following format.
#
# [default]
# aws_access_key_id = ABCDEFGHIJ0123456789
# aws_secret_access_key = 000111r0H7yT5nGP5OPFcZJ+
#
# Then using this file, a YAML file that defines a Kubernetes Secret is created as follows
#
kubectl -n crossplane-system create secret generic aws-credentials --from-file=credentials=./secrets.conf --dry-run=client -o yaml > aws-credentials.yaml
#
# Next, deploy the Bitnami's Sealed Secrets controller in the 'sealed-secrets' namespace
# Then, generate a SealedSecret corresponding to the 'aws-credentials' Secret created above. This is done using the 'kubeseal' CLI utility as shown below.
# The file 'aws-credentials-sealed.yaml' resulting from the operation below is the one to deploy to the management cluster in the GitOps workflow.
# Push this file to the './deploy/crossplane-composition' directory of the GitHub repo that Flux is pointing to
#
kubeseal --controller-namespace sealed-secrets --format yaml < aws-credentials.yaml > aws-credentials-sealed.yaml
#
# Important! Extract the master sealing key from the controller into a YAML file.
# After extracting the master key, the sealed secrets controller may be termintaed.
# The controller per se will get deployed as part of the GitOps workflow.
# But, you must make sure that the sealing master is deployed using this file before so that all SealedSecrets that were created using this master could be unsealed
#
kubectl get secret -n sealed-secrets -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealing-master.key
#
# Now, you are ready to initiate the GitOps worflow.
# Create a Kustomization resource under 'cluster/$CLUSTER_NAME' that points to the 'crossplane' directory in the config repo.
# Pushing this file to the Git repository will trigger a Flux reconcilliation loop which will install the following:
# 1. Crossplane core components
# 2. Crossplane AWS provider-specific components
# 3. Crossplane Configuration package for creating EKS cluster and other AWS resources
# 4. Composite resource to create an EKS cluster
# When this reconcilliation loop is completed, Crossplane will start provisioning the EKS cluster.
# It will take about 10 minutes for the cluster to be ready and operational
#
mkdir -p ./clusters/${CLUSTER_NAME}
flux create kustomization crossplane \
--source=flux-system \
--namespace=flux-system \
--path=./crossplane \
--prune=true \
--interval=30s \
--export > ./clusters/$CLUSTER_NAME/crossplane.yaml
#
# To deploy workloads to the remote cluster using the credentials of the cluster creator, continue with the following step.
# To deploy using the credentials of a service account with appropriate set of RBAC permissions, refer to ./remote/remote-cluster-setup.sh before proceeding further.
# Create a Kustomization resource under 'cluster/$CLUSTER_NAME' that points to the 'applications' directory
# Pushing this file to the Git repository will trigger a Flux reconcilliation loop which will install the following:
# 1. Sample web application that exposes Prometheus metrics
# 2. Prometheus server which scrapes the metrics from the sample application and sends it to an AMP workspace
#
mkdir -p ./clusters/${CLUSTER_NAME}
flux create kustomization applications \
--source=flux-system \
--namespace=flux-system \
--path=./applications \
--prune=true \
--interval=30s \
--export > ./clusters/$CLUSTER_NAME/applications.yaml
#
# Force a Flux reconcilliation loop
#
flux reconcile kustomization flux-system --with-source
#
# Exteranl Secrets Operator
#
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
-n external-secrets \
--create-namespace \
--set installCRDs=true
kubectl apply -f external-secret.yaml