EBS CSI EKS support #64

nickumia-reisys · 2022-01-05T21:27:28Z

Related to GSA/data.gov#3127

Changes:

Update Fairwinds VPC module to AWS VPC Module
Update AWS Provider from 3.61.0 to 3.73.0
Update Kubernetes Provider from 2.5 to 2.7.1
Update Helm Provider from 2.3 to 2.4

New Additions:

Provision EFS Volume alongside EKS cluster
Setup security group for EFS Volume
Mount EFS Volume to all private subnets in EKS cluster
- No matter where a fargate node gets launched, it will have access to the EFS Volume
Attach EFS Policy to Fargate profile to interact with EFS Volume
EFS Volume Read/Write Test pulled from aws-efs-csi-driver repo

mogul

The code looks fine, but we need to see the tests passing.

nickumia-reisys · 2022-01-06T14:18:00Z

Instructions on setting up EBS CSI Plugin: https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html

… groups from being created, so it was removed

This is the start of using EFS instead of EFS because it will be able to be shared memory between all of the Solr replicas; Plus, EBS is connected to each node individually and cannot connect to more than one node; EBS only works with managed nodes, not fargate; EFS supports static provisioning for Fargate

There's no EKS addon for EFS, so we need to install via helm (https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html)

update VPC module to AWS support; add configuration for efs static provisioning; mount the efs volume on all nodes in VPC private subnets; remove managed nodes

move persistent storage capability to new file; update file name in service definition

make test CI-worthy

For the test, the EFS ID is necessary in order to know where kubernetes should create the persistent volume; the only known way of doing this is through the aws cli

I think I copied this from the VPC example, but we don't need it for our use case; also, only create a single nat gateway instead of one per availability zone

nickumia-reisys · 2022-01-27T05:46:54Z

@mogul We need to enable encryption for the EFS volume. We can't merge this yet!

nickumia-reisys · 2022-01-27T05:52:54Z

Commenting for reference, https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html

nickumia-reisys · 2022-01-27T06:24:38Z

Ughh... I think there are a couple of problems with the tests.. Fargate has the EFS driver installed by default, KiND does not.. Real EKS has the current kubernetes configs applied, KiND does not... Will fix in the morning.

mogul

Please create the PV and storage class during the provision operation too.

test.sh

mogul · 2022-01-27T06:35:50Z

Ughh... I think there are a couple of problems with the tests.. Fargate has the EFS driver installed by default, KiND does not.. Real EKS has the current kubernetes configs applied, KiND does not... Will fix in the morning.

The solr-operator doesn't care how PVs are implemented, only that they are available to make PVCs! So we don't need to set up EFS in KinD, we just need to set up KinD to have a local PV back-end and storageClass of its own available. This is how we do it.

mogul · 2022-01-27T06:55:34Z

A more in-depth example of implementing PVs in KinD, if we need to go any further than the SO answer linked above...

The EFS volume was being created, but kubernetes was not being made aware about it until the 'driver, storageclass, pv' k8s classes were created.

EFS Volumes will conflict without unique creation token.. we'll never have two eks clusters named the same thing, so it's a good reference point for EFS volume creations (+1)

Damn..

It takes about 2-3 minutes for the pod to come up

… volumes

nickumia-reisys · 2022-01-28T16:31:47Z

Yay!! 🎈

new: enable ebs csi addon for eks

438403b

mogul previously approved these changes Jan 6, 2022

View reviewed changes

nickumia-reisys dismissed mogul’s stale review via 41a1358 January 6, 2022 13:40

nickumia-reisys added 3 commits January 6, 2022 08:40

new: add IAM Policy to manage ebs volume lifecycle

41a1358

fix: broker does not like non-terraform file

bf65a8b

fix: terraform needs variable name attribute

ce35f59

new: add managed node group; the old iam_path restricted managed node…

43e36bb

… groups from being created, so it was removed

nickumia-reisys marked this pull request as draft January 8, 2022 01:19

nickumia-reisys added 4 commits January 25, 2022 11:20

new: EBS to EFS provision change

6515dea

There's no EKS addon for EFS, so we need to install via helm (https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html)

new: implement efs volumes in fargate

43096d4

update VPC module to AWS support; add configuration for efs static provisioning; mount the efs volume on all nodes in VPC private subnets; remove managed nodes

new: add tests to verify EFS communication

1cf6f18

nickumia-reisys mentioned this pull request Jan 26, 2022

Persistent check #65

Closed

nickumia-reisys changed the title ~~EBS CSI EKS addon~~ EFS CSI EKS support Jan 26, 2022

nickumia-reisys added 4 commits January 26, 2022 16:45

update: broker service definition

7897bc7

move persistent storage capability to new file; update file name in service definition

docs: update cleanup/README

8ee933e

update: refactor name

80099ba

update: finish test

6073b89

make test CI-worthy

nickumia-reisys marked this pull request as ready for review January 26, 2022 22:15

mogul previously approved these changes Jan 26, 2022

View reviewed changes

update: unpin AWS provider from 3.31

d3074f6

nickumia-reisys dismissed mogul’s stale review via d3074f6 January 26, 2022 22:22

nickumia-reisys added 3 commits January 26, 2022 17:27

update: aws provider binary from 3.61 to 3.73

22f139f

fix: make sure test doesn't run forever

b5ae4d8

fix: grab EFS ID for cluster

1b10014

For the test, the EFS ID is necessary in order to know where kubernetes should create the persistent volume; the only known way of doing this is through the aws cli

nickumia-reisys mentioned this pull request Jan 27, 2022

Provide Persistent Volumes for brokered services in k8s GSA/data.gov#3127

Closed

1 task

new: don't add vpn gateway

280d3df

I think I copied this from the VPC example, but we don't need it for our use case; also, only create a single nat gateway instead of one per availability zone

nickumia-reisys marked this pull request as draft January 27, 2022 05:45

nickumia-reisys mentioned this pull request Jan 27, 2022

Solr Optimizations for CKAN Integrations GSA-TTS/datagov-brokerpak-solr#32

Merged

mogul requested changes Jan 27, 2022

View reviewed changes

test.sh Outdated Show resolved Hide resolved

nickumia-reisys and others added 7 commits January 27, 2022 11:26

new: statically provision efs for k8s cluster

3c7aab4

The EFS volume was being created, but kubernetes was not being made aware about it until the 'driver, storageclass, pv' k8s classes were created.

lint: ran terraform fmt

3f50c99

update: make EFS id unique to each cluster

d8ce2a6

EFS Volumes will conflict without unique creation token.. we'll never have two eks clusters named the same thing, so it's a good reference point for EFS volume creations (+1)

fix: stupid typo..

395b406

Damn..

fix: kubectl wait for pod timeouts too quick

fe29daa

It takes about 2-3 minutes for the pod to come up

Replace null_resource with the resources from the kubernetes provider

2055589

Implement encryption-at-rest and encryption-in-transit for persistent…

6f27e96

… volumes

mogul force-pushed the persistent-volumes branch from 7a36f13 to 9ca65ab Compare January 27, 2022 22:07

mogul added 2 commits January 27, 2022 16:49

Update providers for kubernetes and helm

2bbfb0f

Use a name for the PV that doesn't expose the implementation detail

193f89d

mogul force-pushed the persistent-volumes branch from 24fb89b to 193f89d Compare January 28, 2022 00:50

Skip installing the EFS CSI driver; it's already installed

12142f0

mogul requested a review from a team January 28, 2022 05:46

mogul marked this pull request as ready for review January 28, 2022 05:46

mogul approved these changes Jan 28, 2022

View reviewed changes

mogul merged commit ff467f4 into main Jan 28, 2022

mogul deleted the persistent-volumes branch January 28, 2022 05:47

nickumia-reisys changed the title ~~EFS CSI EKS support~~ EBS CSI EKS support Feb 3, 2022

mogul added the component/ssb label Mar 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EBS CSI EKS support #64

EBS CSI EKS support #64

nickumia-reisys commented Jan 5, 2022 •

edited

Loading

mogul left a comment

nickumia-reisys commented Jan 6, 2022

nickumia-reisys commented Jan 27, 2022

nickumia-reisys commented Jan 27, 2022

nickumia-reisys commented Jan 27, 2022

mogul left a comment

mogul commented Jan 27, 2022 •

edited

Loading

mogul commented Jan 27, 2022

nickumia-reisys commented Jan 28, 2022

EBS CSI EKS support #64

EBS CSI EKS support #64

Conversation

nickumia-reisys commented Jan 5, 2022 • edited Loading

mogul left a comment

Choose a reason for hiding this comment

nickumia-reisys commented Jan 6, 2022

nickumia-reisys commented Jan 27, 2022

nickumia-reisys commented Jan 27, 2022

nickumia-reisys commented Jan 27, 2022

mogul left a comment

Choose a reason for hiding this comment

mogul commented Jan 27, 2022 • edited Loading

mogul commented Jan 27, 2022

nickumia-reisys commented Jan 28, 2022

nickumia-reisys commented Jan 5, 2022 •

edited

Loading

mogul commented Jan 27, 2022 •

edited

Loading