Skip to content
This repository has been archived by the owner on Sep 13, 2022. It is now read-only.

Latest commit

 

History

History
292 lines (240 loc) · 11.9 KB

README.md

File metadata and controls

292 lines (240 loc) · 11.9 KB
Raw

Bootstrap and Operate a Quobyte Cluster in Kubernetes with the Operator

This guide shows you how to bootstrap a Quobyte cluster on a set of kubernetes nodes which have some empty storage devices ready to be used for a distributed storage system. The operator simplifies the cluster bootstrap and management, and the latest Quobyte features let you format and set up the cluster from the web console.

For a quick overview, we have prepared a high level blog post: “Quobernetes” or How to Enable Simple and Efficient Storage Operations for Kubernetes

Prerequisites

  • Kubernetes 1.9 is fully supported by the operator
  • To use Quobyte 2.0 features like automatically mounting Quobyte devices, or the formatting and preparation of unformatted devices requires the mountPropagation kubernetes feature, which is a gated feature in kubernetes 1.9 and comes in beta in kubernetes 1.10. This guide assumes that you have the mountPropagation feature enabled on your cluster.
  • A cluster which consists of at least 4 nodes, with 2 unformatted devices each. In this guide we will refer to the nodes as node1 to node4.

Deploy Operator

Create the quobyte namespace. This is where the operator and the cluster lives.

kubectl create -f quobyte-ns.yaml

Once the namespace is created, deploy required RBAC and the operator.

kubectl -n quobyte create -f operator.yaml

Configure Quobyte Services with the Operator

Quobyte runs best with 3 replicas of the registry, where we require 1 bootstrapped registry. To make the setup as easy as possible, we defined an ephemeral registry, which is used to bootstrap the cluster. The final cluster will have registry devices on nodes 2, 3, and 4, so we will use node1 for bootstrap.

The quobyte-config.yaml file provides a registry.bootstrap_node option and allows to fine tune the memory limits for the services. Edit the file to point to your bootstrap registry.

  registry.bootstrap_node: "node1"

Now create the configurations and services daemonsets.

kubectl -n quobyte create -f quobyte-config.yaml
kubectl -n quobyte create -f services.yaml

To run the Quobyte services in kubernetes, first edit the services-config.yaml file, replace the node1-node4 entries with a list of your nodes and determine which node should run which services. The number of nodes is arbitrary but we recommend using 4+ nodes in order to be able handle outage scenarios, etc.

We chose node1 to be the bootstrap registry, but we need to define 3 other nodes to persist the fully replicated cluster. We also recommend to start at least 3 metadata services and data services on all nodes which contain devices which should store your valuable information. Edit the services-config.yaml to match your cluster:

apiVersion: quobyte.com/v1
kind: QuobyteService
metadata:
name: quobyte-services
spec:
registry:
  daemonSetName: registry
  nodes:
    - node1 # will become the ephemeral bootstrap device
metadata:
  daemonSetName: metadata
  nodes:
    - node1
    - node2
    - node3
    - node4
data:
  daemonSetName: data
  nodes:
    - node1
    - node2
    - node3
    - node4

When the services-config created,the operator will start to deploy the services to the target nodes.

kubectl -n quobyte create -f services-config.yaml
kubectl -n quobyte create -f qmgmt-pod.yaml

kubectl -n quobyte get pods -o wide -w

Now you should see all configured pods running. You will also see a qmgmt-pod and a webconsole running. The qmgmt-pod gives you full cli access to the cluster. Lets check that the primary registry is running:

kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api registry list

Now let's list all unformatted devices which the data and metadata services found and could be formatted now.

kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api device list-unformatted

You can either proceed to set up the devices with qmgmt or jump over to the webconsole to get some visual support.

Unless you already have set up an ingress to access the service, you can acceess the console with a port forward

kubectl -n quobyte port-forward "$(kubectl get po -owide -n quobyte | grep webconsole | awk '{print $1}')" 8080:8080

Then point your browser to http://localhost:8080 and follow the setup wizard.

The Devices tab will show you all unformatted devices. Please note that even if multiple services are running on the same node, only one of the Quobyte pods will be responsible to mount and format devices.

Make the Quobyte Cluster Persistent

The ephemereal registry is great for bootstrapping and trying out a Quobyte cluster. So if you're just interested in a demo, you can safely skip this chapter and just use a single registry to run your cluster. But please note, once the registry pod is terminated, the Quobyte cluster becomes unsuable.

To make the cluster persistent, we first need to create 3 registry devices. There must be only one registry device per registry service, so choose one device from each of the other services and create registry devices on them. A maintenance task will run and format and set up the devices. Give the webconsole some seconds to retrieve the last system state and the devices will show up as unassociated devices.

Now let's spin up our three target registries. Edit the services-config.yaml again and add nodes 2 to 4 as registries.

Edit the `services-config.yaml` to match your cluster

```yaml
  registry:
    daemonSetName: registry
    nodes:
      - node1 # will become the ephemeral bootstrap node
      - node2
      - node3
      - node4

An update to the services-config CRD triggers the operator, which will start the registries then.

kubectl -n quobyte apply -f services-config.yaml

Wait until the pods are running and check that Quobyte found the devices.

kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api device list

Id  Host            Mode    Disk Used  Disk Avail  Services  LED Mode  Tags  
1   registry-vfz59  ONLINE  4 GB       40 GB       REGISTRY  OFF       hdd   
2   registry-f7szc  ONLINE  34 MB      21 GB       REGISTRY  OFF       hdd   
3   registry-zq2vh  ONLINE  34 MB      21 GB       REGISTRY  OFF       hdd   
4   registry-4xxkk  ONLINE  34 MB      21 GB       REGISTRY  OFF       hdd

We see a total of 4 registry devices, but the registry will only use 3 of them.

kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api registry list

Primary  Id  Host            Mode    
-        3   registry-zq2vh  ONLINE  
-        4   registry-4xxkk  ONLINE  
1        1   registry-vfz59  ONLINE

If we see 3 ONLINE registries, the work of the ephemeral bootstrap node is done and it is safe to delete it. So remove it from the services-config.yaml

  registry:
    daemonSetName: registry
    nodes:
      - node2
      - node3
      - node4

and update the service-config. The operator will then terminate the ephemeral registry.

kubectl -n quobyte apply -f services-config.yaml

Wait some seconds and check that all 3 persisted registries are ONLINE.

kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api registry list
P
rimary  Id  Host            Mode    
-        1   registry-vfz59  ONLINE  
-        3   registry-zq2vh  ONLINE  
1        4   registry-4xxkk  ONLINE

As a last step, you should decommission the ephemeral device, since it will never come back.

kubectl -n quobyte exec -it qmgmt-pod -- qmgmt -u api device update status 1 DECOMMISSIONED

Create Data and Metadata Devices

Now we need some data and metadata devices to actually store data. From the webconsole, either format the remaining devices according to your needs, or choose a device and Set devices types to add Data or Metadata contents to the device.

Now you have a fully working Quobyte cluster. For further configuration and creation of volumes, please refer to the Quobyte documentation.

Deploy Quobyte Clients

The operator can deploy and manage Quobyte clients - which serve the volumes to your application pods. Every kubernetes node which should provide access to Quobyte storage, has to run a Quobyte client pod.

If the operator finds a client CRD, it will start to deploy the according pods. First edit the client-config.yaml

spec:
    rollingUpdatesEnabled: true
    daemonSetName: client
    nodes:
      - node1
      - node2
      - node3
      - node4

Nodes are optional. If not given, operator queries nodes with the nodeSelector in the daemonSetName: client and does only updates.

kubectl -n quobyte create -f client.yaml
kubectl -n quobyte create -f client-config.yaml

Once the client-config is created, you should see pods being started on the desired hosts.

If you add or remove clients, edit the client-config.yaml and update it with

kubectl -n quobyte apply -f client-config.yaml

When the clients are ready, you can start using Quobyte volumes in your pods. Please have a look at Volume Access for examples.

Rolling Updates

The operator supports rolling updates only for clients. To trigger rolling update of client, please change the container image of daemonset configured in the client-config.yaml. Set rollingUpdatesEnabled: true , the operator will upgrade one node after the other.

Quobyte service containers are updated with careful timing between pod restarts, to always ensure availability of the Quobyte services.

All pods from all other namespaces can access Quobyte volumes which are managed by the client. Since a client update requires a pod restart, all other pods on the same node, which currently access a Quobyte volume, need to be stopped. It's not a good idea to give an operator full permission to drain a full node, we decided to go for a defensive mode. For every node to upgrade, the operator checks for other pods with Quobyte volumes mounted. If no pods are found, the client is restarted immediately. If pods are found, they are listed on the operator's status page. The operator also supports to retrieve its status as json. The administrator will then need to manually stop or drain the pods.

The operator comes with a service and a status page for clients. With kubectl, you can reach it on http://localhost:7878

kubectl -n quobyte port-forward quobyte-operator-xzy 7878:7878

Services rolling updates should follow standard daemonset updates. One way to trigger rolling update for services is to set the new image for daemonset container as shown below

kubectl set image ds/<daemonset-name> <container-name>=<container-new-image>

Uninstall Quobyte with Operator

If you want to remove all services or clients, remove the config files, before you delete the deployments or the operator. This will terminate the scheduled pods and remove the all labels, which the operator applied to any nodes.

kubectl -n quobyte delete -f services-config.yaml
or
kubectl -n quobyte delete -f client-config.yaml

Build Operator from Source

Requirements

  1. golang 1.10+
  2. glide for package management
  3. docker

Build

  1. Clone the repository.
git clone [email protected]:quobyte/k8s-operator.git github.com/quobyte/k8s-operator
  1. Compile and build binary from source.
cd github.com/quobyte/k8s-operator
export GOPATH=$(pwd)
./build #build the operator binary

If you're building for the first time after clone run glide install --strip-vendor to get the dependencies.

  1. To run operator outside cluster (skip to 4 to run operator inside cluster)
./operator --kubeconfig <kuberenetes-admin-conf>

Follow Deploy clients, and you can skip step 3 of deploy clients.

  1. Build the container and push it to repository ./build <repository-url> # push the built image to the container repository-url
  2. Edit operator.yaml and point quobyte-operator container image to the docker image.