offline-search-index.aca96da804a44f636d8b6a35c211043b556a6bfea119d4e6cc505605911bc3220ea46fc747583577755163c86c4ced89cf26b6c346219e63cb77f13c5476d676.json

[{"body":"To install the operator follow the installation page for your orchestrator.\n Kubernetes Rancher OpenShift  ","excerpt":"To install the operator follow the installation page for your orchestrator.\n Kubernetes Rancher …","ref":"/docs/reference/cluster-operator/install/","title":"Install"},{"body":"Current Version The current StorageOS version is v2.4.0.\nSections Our documentation is arranged into sections, accessible from the navigation bar on the left.\nIntroduction - Quickstart and support information.\nConcepts - Architectural and deep technical information.\nPrerequisites - We require certain prerequisites to be met for the product to function correctly. Please read these carefully and ensure they are implemented.\nPlatforms - Due to differences in the various orchestrators that StorageOS can run under, we list install guides and other platform specific operations here.\nOperations - Platform agnostic operations.\nUse Cases - A set of examples to get up and running with StorageOS quickly.\nReference - Information on our GUI, CLI, and other important information.\n","excerpt":"Current Version The current StorageOS version is v2.4.0.\nSections Our documentation is arranged into …","ref":"/docs/introduction/","title":"Introduction"},{"body":" Make sure the prerequisites for StorageOS are satisfied before proceeding.\n  Any Kubernetes managed service such as EKS, AKS, GKE, DO or DockerEE platform can use the following Kubernetes guide to install StorageOS.\n  StorageOS supports the five most recent Kubernetes releases, at minimum.\n  Make sure to add a StorageOS licence after installing.\n   1.21 1.20 1.19 1.18 1.17     Install StorageOS on Kubernetes 1.21 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.20 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.19 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.18 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.17 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n  First StorageOS volume If this is your first installation you may wish to follow the StorageOS Volume guide for an example of how to mount a StorageOS volume in a Pod.\n","excerpt":"Make sure the prerequisites for StorageOS are satisfied before proceeding.\n  Any Kubernetes managed …","ref":"/docs/install/kubernetes/","title":"Kubernetes"},{"body":"Over the past several months, we\u0026rsquo;ve been hard at work on StorageOS V2, which contains some significant enhancements over our v1 product. We\u0026rsquo;ve built V2 based on our observations of trends in the industry, as well as our own experience.\nMany of our customers want to run big clusters - in the tens or hundreds of nodes. In these sorts of big environments, the challenges multiply. Not only do we need to scale well, but we also need to be more failure tolerant. Bigger environments typically suffer higher failure rates (more nodes = greater chance of something failing), but are also subject to all sorts of transient conditions such as network partitions.\nThe second trend we\u0026rsquo;ve seen become increasingly common is the desire to run multiple clusters, and consume storage between them in some way - sometimes to implement novel topologies such as a centralised storage cluster with satellites consuming the storage, and sometimes to replicate data between those clusters for HA or DR purposes.\nWe\u0026rsquo;ve built V2 with these architectures and design patterns in mind. Not only does it scale well, but it contains the foundations we need to implement a rich set of multi-cluster functionality.\nUpgraded Control Plane At the heart of the V2 release is an upgraded control plane. We\u0026rsquo;ve changed a lot here. Firstly, our usage of etcd is vastly improved. We\u0026rsquo;ve learnt a lot about the subtleties of distributed consensus in the last year, particularly in noisy or unpredictable environments. Not only is StorageOS V2 much lighter on your etcd cluster, but it\u0026rsquo;s a lot more tolerant of transient failure conditions that are often found in cloud environments, or clusters under heavy load.\nWe spent some time describing and testing our internal state machine using the TLA+ formal verification language. This allows us to have a much higher degree of confidence that our algorithms will behave correctly, particularly under hard-to-test edge cases and failure conditions.\nAdditionally, we\u0026rsquo;ve changed the way volumes behave with respect to centralised scheduling. Each volume group (consisting of a master and 0 or more replicas) now behaves as a strongly consistent unit allowing it to take action independent of the activities of the rest of the cluster. Other state can be distributed around the cluster using eventually consistent mechanisms. This approach inherently scales better and allows StorageOS V2 to effectively manage many more nodes and volumes than before.\nWe\u0026rsquo;ve implemented TLS on all endpoints. Not only does this give you encrypted traffic between nodes in your storage cluster, it also protects all endpoints with strong, public key based authentication. Today\u0026rsquo;s IT environments can\u0026rsquo;t rely on firewalls to keep bad actors out - they must implement security at all layers within the stack - defense in depth. While we recognise that this brings a welcome relief to many security conscious administrators, we also know that managing certificate authorities (CAs) can be an unwelcome source of complexity. For this reason, StorageOS V2 implements an internal CA by default, to manage this complexity for you. If you\u0026rsquo;d prefer to integrate your own CA, we support that too - it\u0026rsquo;s up to you.\nFinally - our logging has undergone a complete transformation in this edition. We know that systems engineers and operators don\u0026rsquo;t just value headline features, but that observability and diagnostics are equally important. All logs are now decorated with rich context to help you understand what is happening within your cluster, and we\u0026rsquo;ll output in json by default, for easy ingestion into log aggregators such as Elasticsearch.\nUpgraded Data Plane Not to be outdone, our data plane contains some significant improvements.\nFirstly, we\u0026rsquo;ve completely re-written our sync algorithm (see Delta Sync, used when seeding or catching up replicas that have been offline or partitioned. Our new algorithm uses a Hash List to sync only changed sections of a volume (similar in some ways to what rsync does). StorageOS maintains these hashes during normal operation, meaning that when resyncing a failed replica, for example after a node reboot, we can very quickly and efficiently catch this replica up, rather than needing to promote and build a new one from scratch. This improves resiliency within your cluster, and prevents using excessive network bandwidth during failover conditions - at a time when it might be needed the most.\nSecondly, a new threading model, with dynamic pool sizing, means that StorageOS is faster, a lot faster. In our tests we observed improvements across the board, with improvements in throughput of up to 135% for some scenarios.\n","excerpt":"Over the past several months, we\u0026rsquo;ve been hard at work on StorageOS V2, which contains some …","ref":"/docs/introduction/overview/","title":"What is StorageOS V2"},{"body":"StorageOS implements a MutatingAdmissionWebhook Admission Controller to ensure that Pods using StorageOS Volumes use the storageos-scheduler. An admission controller intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized.\nThe Admission Controller is responsible for mutating the PodSpec at creation time to populate the PodSpec.schedulerName field with the name of the StorageOS Scheduler - storageos-scheduler.\nDuring Pod creation, Kubernetes sends a web request to the StorageOS WebHook with the Pod specification. The PodSpec is only altered to use the StorageOS scheduler if the Pod uses a StorageOS volume.\nWeb Server The Web Server hosting the web hook is executed in the StorageOS Cluster Operator. Since only HTTPS requests are allowed, the Operator generates a self-signed x509 certificate every time it starts. The Cluster Operator will also renew certificates upon expiry (certs are valid for one year).\nThere is no manual intervention required regarding the SSL configuration as the setup is completely transparent between StorageOS and Kubernetes.\nSkipping Mutation To avoid scheduler mutation, the storageos.com/scheduler=false annotation can be added to resources that use StorageOS volumes.\nWhen using StatefulSets the annotation can be set on the spec.template.metadata.annotations field.\napiVersion:apps/v1kind:StatefulSetspec:...template:metadata:annotations:storageos.com/scheduler:\u0026#34;false\u0026#34;# N.B. the value must be a string and not a booleanWhen using Pods the annotation is set on the metadata.annotations field.\napiVersion:v1kind:Podmetadata:...annotations:storageos.com/scheduler:\u0026#34;false\u0026#34;# N.B. the value must be a string and not a boolean...Explicit SchedulerName  It is not necessary to explicitly set the SchedulerName as the Admission Controller automatically populates the PodSpec field. Set the SchedulerName in your manifests, manually, only if you disable or can\u0026rsquo;t execute the StorageOS Admission Controller.\n Kubernetes allows the use of different schedulers by setting the field .spec.schedulerName: storageos-scheduler.\nFor instance a Pod manifest utilising the StorageOS scheduler would appear as follows:\napiVersion: v1 kind: Pod metadata: name: d1 spec: schedulerName: storageos-scheduler # --\u0026gt; StorageOS Scheduler # No need if using Admission Controller # (enabled by default) containers: - name: debian image: debian:9-slim command: [\u0026#34;/bin/sleep\u0026#34;] args: [ \u0026#34;3600\u0026#34; ] volumeMounts: - mountPath: /mnt name: v1 volumes: - name: v1 persistentVolumeClaim: claimName: persistent-volume # ----\u0026gt; StorageOS PVC Compatibility The Admission Controller doesn\u0026rsquo;t need to be enabled at Kubernetes cluster bootstrap time because it is a Dynamic Admission Controller. Hence, any cluster that has the MutatingAdmissionWebhook enabled is supported. Most Kubernetes cluster enable the Webhook admission controller by default.\nThe MutatingAdmissionWebhook is available from Kubernetes v1.13.\nYou can check your Kubernetes cluster compatibility by checking if the following object exists.\nkubectl api-versions | grep admissionregistration.k8s.io ","excerpt":"StorageOS implements a MutatingAdmissionWebhook Admission Controller to ensure that Pods using …","ref":"/docs/reference/scheduler/admission-controller/","title":"Admission Controller"},{"body":"StorageOSCluster Resource Configuration The following table lists the configurable spec parameters of the StorageOSCluster custom resource and their default values.\n   Parameter Description Default     csi.deploymentStrategy CSI helper deployment strategy (statefulset or deployment) statefulset   csi.enable Enable CSI setup false   csi.enableControllerPublishCreds Enable CSI controller publish credentials false   csi.enableNodePublishCreds Enable CSI node publish credentials false   csi.enableProvisionCreds Enable CSI provision credentials false   debug Enable debug mode for all the cluster nodes false   disableFencing Disable Pod fencing false   disableScheduler Disable StorageOS scheduler false   disableTCMU Disable TCMU to allow co-existence with other TCMU users. Disabling TCMU degrades performance false   disableTelemetry Disable telemetry reports false   forceTCMU Forces TCMU to be enabled or causes StorageOS to abort startup false   images.apiManagerContainer StorageOS API Manager container image storageos/api-manager:v1.0.0   images.csiClusterDriverRegistrarContainer CSI Cluster Driver Registrar Container image quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1   images.csiExternalAttacherContainer CSI External Attacher Container image quay.io/k8scsi/csi-attacher:v1.0.1   images.csiExternalProvisionerContainer CSI External Provisioner Container image storageos/csi-provisioner:v1.0.1   ìmages.csiLivenessProbeContainer CSI Liveness Probe Container Image quay.io/k8scsi/livenessprobe:v1.0.1   images.csiNodeDriverRegistrarContainer CSI Node Driver Registrar Container image quay.io/k8scsi/csi-node-driver-registrar:v1.0.1   images.hyperkubeContainer Deprecated field - HyperKube Container image Default dependent on Scheduler version   images.initContainer StorageOS init container image storageos/init:2.1.0   images.kubeSchedulerContainer Kube scheduler container image Default dependent on Scheduler version   images.nfsContainer StorageOS nfs container image storageos/nfs:1.0.0   images.nodeContainer StorageOS node container image storageos/node:v2.4.0   ingress.annotations Annotations of the ingress used by the cluster    ingress.enable Enable ingress for the cluster false   ingress.hostname Hostname to be used in cluster ingress storageos.local   ingress.tls Enable TLS for the ingress false   k8sDistro The name of the Kubernetes distribution is use, e.g. rancher or eks    kvBackend.address Comma-separated list of addresses of external key-value store. (1.2.3.4:2379,2.3.4.5:2379)    kvBackend.backend (v2 deprecated) Name of the key-value store to use. Set to etcd for external key-value store. embedded   namespace Namespace where storageos cluster resources are created kube-system   nodeSelectorTerms Set node selector for storageos pod placement    pause Pause the operator for cluster maintenance false   resources Set resource requirements for the containers    secretRefName Reference name of storageos secret    secretRefNamespace Namespace of storageos secret    service.annotations Annotations of the Service used by the cluster    service.externalPort External port of the Service used by the cluster 5705   service.internalPort Internal port of the Service used by the cluster 5705   service.name Name of the Service used by the cluster storageos   service.type Type of the Service used by the cluster ClusterIP   sharedDir Path to be shared with kubelet container when deployed as a pod /var/lib/kubelet/plugins/kubernetes.io~storageos   storageClassName The name of the default StorageClass created for StorageOS volumes fast   tlsEtcdSecretRefName Secret containing etcd client certificates    tlsEtcdSecretRefNamespace Namespace of the tlsEtcdSecretRefName    tolerations Set pod tolerations for storageos pod placement     ","excerpt":"StorageOSCluster Resource Configuration The following table lists the configurable spec parameters …","ref":"/docs/reference/cluster-operator/configuration/","title":"Configuration"},{"body":" Make sure the prerequisites for StorageOS are satisfied before proceeding. Including the deployment of an etcd cluster and configuration of CRI-O PID limits.\n  For OpenShift upgrades, refer to the OpenShift platform page.\n  If you have installed OpenShift in AWS ensure that the requisite ports are opened for the worker nodes\u0026rsquo; security group.\n  Make sure to add a StorageOS licence after installing.\n StorageOS v2 supports OpenShift v4. For more information check the OpenShift platform page.\n OperatorHub RH Marketplace Manual    OperatorHub   Select the OperatorHub from the Catalog sub menu and search for StorageOS\n Choose between using the RedHat Market Place or the Community Operators installation.\n   Select StorageOS and click install\n  Select the install options\n Make sure the Approval Strategy is set to Manual. So the StorageOS Operator doesn\u0026rsquo;t upgrade versions without explicit approval.\n   Start the approval procedure\n  Follow the approval link\n  Approve the installation\n  The StorageOS Cluster Operator is installed along the required CRDs\n  Create a Secret in the openshift-operators project\n  Use the YAML options to create a secret containing the apiUsername and an apiPassword key. The username and password defined in the secret will be used to authenticate when using the StorageOS CLI and GUI. For the communication between StorageOS and OpenShift, the CSI credentials csiProvisionUsername, csiProvisionPassword, csiControllerPublishUsername, csiControllerPublishPassword, csiNodePublishUsername, csiNodePublishPassword are needed. Take note of which project you created the secret in.\nInput the Secret as YAML for simplicity.\napiVersion:v1kind:Secretmetadata:name:storageos-apinamespace:openshift-operatorstype:\u0026#34;kubernetes.io/storageos\u0026#34;data:# echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64apiUsername:c3RvcmFnZW9zapiPassword:c3RvcmFnZW9z# CSI CredentialscsiProvisionUsername:c3RvcmFnZW9zcsiProvisionPassword:c3RvcmFnZW9zcsiControllerPublishUsername:c3RvcmFnZW9zcsiControllerPublishPassword:c3RvcmFnZW9zcsiNodePublishUsername:c3RvcmFnZW9zcsiNodePublishPassword:c3RvcmFnZW9zcsiControllerExpandUsername:c3RvcmFnZW9zcsiControllerExpandPassword:c3RvcmFnZW9z  Go to the \u0026ldquo;Installed Operators\u0026rdquo;\n Verify that the StorageOS Cluster Operator is installed\n   Go to the \u0026ldquo;StorageOS Cluster\u0026rdquo; section\n  Create the StorageOS Cluster\n A StorageOS Cluster is defined using a Custom Resource Definition\n   Create the Custom Resource\nThe StorageOS cluster resource describes the StorageOS cluster that will be created. Parameters such as the secretRefName, the secretRefNamespace and the kvBackend.address are mandatory.\nAdditional spec parameters are available on the Cluster Operator configuration page.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: storageos namespace: openshift-operators spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference the Secret created in the previous step secretRefNamespace: \u0026#34;openshift-operators\u0026#34; # Namespace of the Secret created in the previous step k8sDistro: \u0026#34;openshift\u0026#34; kvBackend: address: \u0026#39;storageos-etcd-client.etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Verify that the StorageOS Cluster Resource enters a running state.\n It can take up to a minute to report the StorageOS Pods ready\n   Check the StorageOS Pods in the kube-system project\n A Status of 3/3 for the Daemonset Pods indicates that StorageOS is bootstrapped successfully.\n   License cluster\n A newly installed StorageOS cluster does not include a licence. A cluster must be licensed within 24 hours of the installation. For more information, check the reference licence page.\n You can apply a Free Developer licence following the operations licensing page, or purchase a licence contacting sales@storageos.com.\n   Red Hat Marketplace  The installation of StorageOS using the Red Hat Marketplace requires the Openshift cluster to be registered to the Marketplace Portal, including the roll out of the PullSecret in your cluster. Failure to do so will result in a image pull authentication failure with the Red Hat registry.\n   Select the OperatorHub from the Catalog sub menu and search for StorageOS.\n Choose the RedHat Marketplace option.\n   Select StorageOS and click purchase. Note that Openshift needs to be registered with the Red Hat Marketplace portal.\n  Select the most suitable install option.\n Project Edition is suitable for production workloads, Developer Edition for personal experimentation and evaluation.\n   Specify the product configuration to fit your needs.\n  Navigate to your software within Red Hat Marketplace and install the StorageOS software as specified in the image.\n  Install the Operator. Set the update approval strategy to Automatic to ensure that you always have the latest version of StorageOS installed.\n  The StorageOS Cluster Operator is installed into your specified cluster.\n  Create a Secret in the openshift-operators project\n  Use the YAML options to create a secret containing the apiUsername and an apiPassword key. The username and password defined in the secret will be used to authenticate when using the StorageOS CLI and GUI. For the communication between StorageOS and OpenShift, the CSI credentials csiProvisionUsername, csiProvisionPassword, csiControllerPublishUsername, csiControllerPublishPassword, csiNodePublishUsername, csiNodePublishPassword are needed. Take note of which project you created the secret in.\nInput the Secret as YAML for simplicity.\napiVersion:v1kind:Secretmetadata:name:storageos-apinamespace:openshift-operatorstype:\u0026#34;kubernetes.io/storageos\u0026#34;data:# echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64apiUsername:c3RvcmFnZW9zapiPassword:c3RvcmFnZW9z# CSI CredentialscsiProvisionUsername:c3RvcmFnZW9zcsiProvisionPassword:c3RvcmFnZW9zcsiControllerPublishUsername:c3RvcmFnZW9zcsiControllerPublishPassword:c3RvcmFnZW9zcsiNodePublishUsername:c3RvcmFnZW9zcsiNodePublishPassword:c3RvcmFnZW9zcsiControllerExpandUsername:c3RvcmFnZW9zcsiControllerExpandPassword:c3RvcmFnZW9z  Go to StorageOS in your \u0026ldquo;Installed Operators\u0026rdquo;\n Verify that the StorageOS Cluster Operator is installed\n   Go to the \u0026ldquo;StorageOS Cluster\u0026rdquo; section\n  Create the StorageOS Cluster\n A StorageOS Cluster is defined using a Custom Resource Definition\n   Create the Custom Resource\nThe StorageOS cluster resource describes the StorageOS cluster that will be created. Parameters such as the secretRefName, the secretRefNamespace and the kvBackend.address are mandatory.\nAdditional spec parameters are available on the Cluster Operator configuration page.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: storageos namespace: openshift-operators spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference the Secret created in the previous step secretRefNamespace: \u0026#34;openshift-operators\u0026#34; # Namespace of the Secret created in the previous step k8sDistro: \u0026#34;openshift\u0026#34; kvBackend: address: \u0026#39;storageos-etcd-client.etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Verify that the StorageOS Cluster Resource enters a running state.\n It can take up to a minute to report the StorageOS Pods ready\n   Check the StorageOS Pods in the kube-system project\n A Status of 3/3 for the Daemonset Pods indicates that StorageOS is bootstrapped successfully.\n   License cluster\n A newly installed StorageOS cluster does not include a licence. A cluster must be licensed within 24 hours of the installation. For more information, check the reference licence page.\n You can apply a Free Developer licence following the operations licensing page, or purchase a licence contacting sales@storageos.com.\n   Manual install The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\noc create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# oc -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;openshift\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;  Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# oc -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n  First StorageOS volume If this is your first installation you may wish to follow the StorageOS Volume guide for an example of how to mount a StorageOS volume in a Pod.\n","excerpt":"Make sure the prerequisites for StorageOS are satisfied before proceeding. Including the deployment …","ref":"/docs/install/openshift/","title":"OpenShift"},{"body":"  Start by understanding the architecture.\n  Follow our installation guides for Kubernetes, OpenShift or Rancher.\n  Dive into our Use Case documentation for Workloads deployed in Kubernetes.\n  ","excerpt":"  Start by understanding the architecture.\n  Follow our installation guides for Kubernetes, …","ref":"/docs/introduction/quickstart/","title":"Quickstart"},{"body":"Before deploying a StorageOS cluster, create a Secret to define the StorageOS API Username and Password in base64 encoding.\nkubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;default\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z END This example contains a default password, for production installations, use a unique, strong password.\n Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;.\n  You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n Create a cluster-config.yaml according to your needs from the examples below.\nkubectl create -f cluster-config.yaml Note that StorageOS will be deployed in spec.namespace (storageos by default), irrespective of what NameSpace the CR is defined in.\n Examples  You can checkout all the parameters configurable in the configuration page.\n All examples must reference the storageos-api Secret.\nspec:secretRefName:\u0026#34;storageos-api\u0026#34;# Reference to the Secret created in the previous stepsecretRefNamespace:\u0026#34;storageos-operator\u0026#34;# Namespace of the SecretCheck out Cluster Definition examples for full CR files.\nInstalling with an external etcd spec:kvBackend:address:\u0026#39;10.43.93.95:2379\u0026#39;# IP of the SVC that exposes ETCD# address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can specify individual IPs of the etcd serversIf using Etcd with mTLS, you need to specify the secret that hold the certificates with the following parameters:\nspec:# External mTLS secured etcd cluster specific propertiestlsEtcdSecretRefName:\u0026#34;etcd-client-tls\u0026#34;# Secret containing etcd client certificatestlsEtcdSecretRefNamespace:\u0026#34;storageos-etcd\u0026#34;# Namespace of the client certificates secretFollow the etcd operations page to setup the secret with the Etcd client certificate, client key and CA.\nInstalling to a subset of nodes In this case we select nodes that are workers. To make sure that StorageOS doesn\u0026rsquo;t start in Master nodes.\nYou can see the labels in the nodes by kubectl get node --show-labels.\nspec:nodeSelectorTerms:- matchExpressions:- key:\u0026#34;node-role.kubernetes.io/worker\u0026#34;operator:Invalues:- \u0026#34;true\u0026#34;# OpenShift uses \u0026#34;node-role.kubernetes.io/compute=true\u0026#34;# Rancher uses \u0026#34;node-role.kubernetes.io/worker=true\u0026#34;# Kops uses \u0026#34;node-role.kubernetes.io/node=\u0026#34; Different provisioners and Kubernetes distributions use node labels differently to specify master vs workers. Node Taints are not enough to make sure StorageOS doesn\u0026rsquo;t start in a node. The JOIN variable is defined by the operator by selecting all the nodes that match the nodeSelectorTerms.\n Enabling CSI spec:csi:enable:truedeploymentStrategy:deploymentenableProvisionCreds:trueenableControllerPublishCreds:trueenableNodePublishCreds:trueenableControllerExpandCreds:trueThe credentials must be defined in the storageos-api Secret\napiVersion:v1kind:Secretmetadata:name:\u0026#34;storageos-api\u0026#34;namespace:\u0026#34;storageos-operator\u0026#34;labels:app:\u0026#34;storageos\u0026#34;type:\u0026#34;kubernetes.io/storageos\u0026#34;data:# echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64apiUsername:c3RvcmFnZW9zapiPassword:c3RvcmFnZW9z# Add base64 encoded creds below for CSI credentials.csiProvisionUsername:c3RvcmFnZW9zcsiProvisionPassword:c3RvcmFnZW9zcsiControllerPublishUsername:c3RvcmFnZW9zcsiControllerPublishPassword:c3RvcmFnZW9zcsiNodePublishUsername:c3RvcmFnZW9zcsiNodePublishPassword:c3RvcmFnZW9zcsiControllerExpandUsername:c3RvcmFnZW9zcsiControllerExpandPassword:c3RvcmFnZW9zSpecifying a shared directory for use with kubelet as a container spec:sharedDir:\u0026#39;/var/lib/kubelet/plugins/kubernetes.io~storageos\u0026#39;Defining pod resource requests and reservations spec:resources:requests:memory:\u0026#34;512Mi\u0026#34;# cpu: \u0026#34;1\u0026#34;# limits:# memory: \u0026#34;4Gi\u0026#34;Limiting StorageOS can cause malfunction for IO to StorageOS volumes, therefore we do not currently recommend applying upper limits to resources for StorageOS pods.\nSpecifying custom Tolerations spec:tolerations:- key:\u0026#34;key1\u0026#34;operator:\u0026#34;Equal\u0026#34;value:\u0026#34;value1\u0026#34;effect:\u0026#34;EffectToTolerate\u0026#34;- key:\u0026#34;key2\u0026#34;operator:\u0026#34;Exists\u0026#34;Custom tolerations specified in the StorageOSCluster definition are added to all StorageOS components; the StorageOS daemonset, CSI helper and scheduler.\nIn the above example a toleration key1=value1:EffectToTolerate would be tolerated and key2 would be tolerated regardless of the value and effect. For more information about tolerations please see the Kubernetes documentation.\n","excerpt":"Before deploying a StorageOS cluster, create a Secret to define the StorageOS API Username and …","ref":"/docs/reference/cluster-operator/examples/","title":"Cluster Operator examples"},{"body":"StorageOS is a certified Rancher application. We offer two installation methods:\n Rancher Catalogue - this is the easiest and requires just a few clicks Manual - allowing more control and visibility  Before proceeding, ensure that you have followed our prerequisites. On Rancher, pay particular attention to the OS version and image used - some platforms require extra mainline kernel modules to be enabled.\n Catalog Manual    Catalog Install StorageOS is a Certified application in the Rancher Catalog. You can install StorageOS using the Rancher application install.\nBefore completing the steps below, you will need an etcd cluster. For evaluations it is sufficient to use our simple test recipe. For production installations, please follow our production recipe. Make a note of the etcd endpoint URL in either case.\n  Select the System project of your cluster\n  Select the Apps tab and click Launch\n  Search for StorageOS and click on the App\nThis will install the StorageOS operator, which manages the StorageOS DaemonSet.\n  Check and ammend installation options\nA generic configuration for StorageOS is preset using the default values in the form. Be sure to check the etcd address and ensure it matches the value you noted at the beginning of this guide.\nThe catalog form exposes several useful parameters - documented below.\nFor further customization, you can opt to set the option to \u0026lsquo;Install StorageOS Cluster\u0026rsquo; to false and install a custom CR. See below for this.\n  Launch the StorageOS cluster\n  Verify the cluster bootstrap has successfully completed\n  License the newly installed cluster\nNewly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nYou will need access to the StorageOS API on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):\n$ kubectl port-forward -n kube-system svc/storageos 5705 Now follow the instructions on our licensing operations page to obtain and apply a license.\nInstallation of StorageOS is now complete.\n  Simple Customization - Modify Catalog Form The following options are exposed by the catalog form to allow some simple customization of the StorageOS installation.\n Cluster Operator namespace : The Kubernetes namespace where the StorageOS Cluster Operator and other resources will be created. Container Images : By default images are pulled from DockerHub, you can specify the image URLs when using private registries. Install StorageOS cluster : Controls the automatic deployment of StorageOS after installing the Cluster Operator. If set to false, the Operator will be created, but a Custom Resource will not be applied to the cluster. Launch the operator and proceed to the section Advanced Customization below. Namespace : The Kubernetes namespace where StorageOS will be installed. By default, StorageOS installs into the kube-system namespace, which will add a priority class to ensure high priority resource allocation. Installing StorageOS with the priority class prevents StorageOS from being evicted during periods of resource contention. It is inadvisable to modify this under normal circumstances. Username/Password : Default Username and Password for the admin account to be created at StorageOS bootstrap. A random password will be generated by leaving the field empty or clicking the Generate button. External etcd address(es) : Connection and configuration details for an external Etcd cluster.See our documentation here. Node Selectors and Tolerations : Control placement of StorageOS DaemonSet Pods. StorageOS will only be installed on the selected nodes. Tolerations : Define any tolerations you wish the DaemonSet to observe.  Advanced Customization - Apply Custom CR If Install StorageOS Cluster was set to false, StorageOS will not be bootstrapped automatically. After the StorageOS Operator is installed, you can now create a Custom Resource that describes the StorageOS cluster.\n  Select the System Workloads and Import YAML   Create the Secret and CustomResource This is an example.\n--- apiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z --- apiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;rancher\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips sharedDir: \u0026#39;/var/lib/kubelet/plugins/kubernetes.io~storageos\u0026#39; # Needed when Kubelet as a container resources: requests: memory: \u0026#34;512Mi\u0026#34; nodeSelectorTerms: - matchExpressions: - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation operator: In values: - \u0026#34;true\u0026#34;  Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n    Manual Installation The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;rancher\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.4.0\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;  Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n  First StorageOS volume If this is your first installation you may wish to follow the StorageOS Volume guide for an example of how to mount a StorageOS volume in a Pod.\n","excerpt":"StorageOS is a certified Rancher application. We offer two installation methods:\n Rancher Catalogue …","ref":"/docs/install/rancher/","title":"Rancher"},{"body":"Upgrade StorageOS operator from yaml manifest Upgrade the StorageOS operator using the following yaml manifest.\nkubectl apply -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml  When you run the above command StorageOS Operator resources will be updated. Since, the Update Strategy of the StorageOS Operator Deployment is set to rolling update, a new StorageOS Operator Pod will be created. Only when the new Pod enters the Running Phase will the old Pod be deleted. Your StorageOS Cluster will not be affected while the StorageOS Operator is upgrading.\n Upgrade StorageOS Operator using Helm If you have installed the StorageOS Operator using the Helm Chart, then you can upgrade the operator using the following commands.\n$ helm list NAME REVISION STATUS CHART APP VERSION NAMESPACE storageos-v1 4 DEPLOYED storageos-operator-0.2.11 1.3.0 storageos-operator $ helm repo update $ helm upgrade $NAME storageos/storageos-operator  When you run the above command StorageOS Operator resources will be updated. Since, the Update Strategy of the StorageOS Operator Deployment is set to rolling update, a new StorageOS Operator Pod will be created. Only when the new Pod enters the Running Phase will the old Pod be deleted. Your StorageOS Cluster will not be affected while the StorageOS Operator is upgrading.\n ","excerpt":"Upgrade StorageOS operator from yaml manifest Upgrade the StorageOS operator using the following …","ref":"/docs/reference/cluster-operator/upgrade/","title":"Cluster Operator Upgrade"},{"body":"OS  Linux X86_64 Kernels satisfying our module prerequisites 3.x kernels have a limitation of 256 active volumes per node 4.x kernels have a limitation of 4096 active volumes per node We are distribution agnostic as long as our prerequisites are met  Orchestrators  Kubernetes 1.17 to 1.21 OpenShift 4.0+  ","excerpt":"OS  Linux X86_64 Kernels satisfying our module prerequisites 3.x kernels have a limitation of 256 …","ref":"/docs/introduction/platforms/","title":"Supported Platforms and Orchestrators"},{"body":"StorageOS requires certain kernel modules to function. In particular it requires Linux-IO, an open-source implementation of the SCSI target, on all nodes that will execute StorageOS (usually the workers).\nWe require the following modules to be loaded:\n target_core_mod tcm_loop target_core_file configfs target_core_user uio   N.B. Other applications utilising TCMU cannot be run concurrently with StorageOS. Doing so may result in corruption of data. On startup, StorageOS will detect if other applications are using TCMU and fall back to FUSE. However if StorageOS is started first there is no mechanism for StorageOS to fallback to FUSE if another application begins to use TCMU. TCMU can be disabled using the DISABLE_TCMU StorageOSCluster spec parameter.\n Depending on the distribution, the modules are shipped as part of the base kernel package or as part of a kernel extras package which needs to be installed.\nDistribution Specifics The following distributions are supported by default:\n RHEL 7.5 CentOS 7 Debian 9 Ubuntu Azure RancherOS - Note CSI is not supported on RancherOS  Ubuntu 16.04/18.04 requires the installation of additional packages.\n N.B. Ubuntu 16.04/18.04 AWS and Ubuntu 18.04 GCE do not provide the necessary linux-image-extra package - see below for more information\n Ubuntu Package Installation Ubuntu 16.04/18.04 Generic and Ubuntu 16.04 GCE require extra packages:\nUbuntu 16.04:\nsudo apt -y update sudo apt -y install linux-image-extra-$(uname -r) Ubuntu 18.04+:\nsudo apt -y update sudo apt -y install linux-modules-extra-$(uname -r) Ubuntu With AWS Or GCE Kernels Ubuntu 16.04/18.04 AWS and Ubuntu 18.04 GCE do not yet provide the linux-image-extra package. As such you should either use Debian, CentOS or RHEL, or install the non-cloud-provider optimised Ubuntu kernel.\nInstalling the non-cloud-provider optimised Ubuntu kernel is something that should only be done with full understanding of potential ramifications.\nsudo apt -y update sudo apt install -y linux-virtual linux-image-extra-virtual sudo apt purge -y linux*aws # Reboot the machine sudo shutdown -r now Automatic Configuration Once required kernel modules are installed on the system, for convenience we provide a container which will ensure the appropriate modules are loaded and ready for use at runtime. On Docker installations, you will need to run the init container prior to starting StorageOS. Our installation guides for Kubernetes and OpenShift include this step.\n# Load the required kernel modules. The Kubernetes and OpenShift installations include this step. docker run --name enable_lio \\  --privileged \\  --rm \\  --cap-add=SYS_ADMIN \\  -v /lib/modules:/lib/modules \\  -v /sys:/sys:rshared \\  storageos/init:0.2 Manual Configuration For those wishing to manage their own kernel configuration, rather than using the init container, perform the following steps:\n Ensure kernel modules are all loaded per list above Ensure configfs is loaded and mounted at /sys/kernel/config  ","excerpt":"StorageOS requires certain kernel modules to function. In particular it requires Linux-IO, an …","ref":"/docs/prerequisites/systemconfiguration/","title":"System Configuration"},{"body":"Availability of IPv6 Address Family Certain StorageOS components need to be able to listen on a standard dual-stack socket of type AF_INET6. The IPv6 address family must be supported on the server so that this socket can be allocated. StorageOS does not require IPv6 to be configured on the server - no addressing or routing needs to be in place, however StorageOS does need this functionality to be enabled in the kernel.\n","excerpt":"Availability of IPv6 Address Family Certain StorageOS components need to be able to listen on a …","ref":"/docs/prerequisites/ipv6/","title":"Availability of IPv6"},{"body":"","excerpt":"","ref":"/docs/concepts/","title":"Concepts"},{"body":"Port list StorageOS daemons listen on specific ports, which we require to be accessible between all nodes in the cluster:\n   Port Number TCP/UDP Use     5701 TCP gRPC   5703 TCP DirectFS   5704 TCP Dataplane Supervisor   5705 TCP REST API   5711 TCP \u0026amp; UDP Gossip service   25705-25960 TCP RWX Volume Endpoints    StorageOS also uses ephemeral ports to dial-out to these ports on other StorageOS nodes. For this reason, outgoing traffic should be enabled.\nFirewalls and VPS providers Some VPS providers (such as Digital Ocean) ship default firewall rulesets which must be updated to allow StorageOS to run. Some example rules are shown below - modify to taste.\nUFW For distributions using UFW, such as RHEL and derivatives:\nufw default allow outgoing ufw allow 5701:5711/tcp ufw allow 5711/udp ufw allow 25705:25960/tcp Firewalld For distributions that enable firewalld to control iptables such as some installations of OpenShift.\nfirewall-cmd --permanent --new-service=storageos firewall-cmd --permanent --service=storageos --add-port=5700-5800/tcp --add-port=25705-25960/tcp firewall-cmd --add-service=storageos --zone=public --permanent firewall-cmd --reload Iptables For those using plain iptables:\n# Inbound traffic iptables -I INPUT -i lo -m comment --comment \u0026#39;Permit loopback traffic\u0026#39; -j ACCEPT iptables -I INPUT -m state --state ESTABLISHED,RELATED -m comment --comment \u0026#39;Permit established traffic\u0026#39; -j ACCEPT iptables -I INPUT -p tcp --dport 5701:5711 -m comment --comment \u0026#39;StorageOS\u0026#39; -j ACCEPT iptables -I INPUT -p udp --dport 5711 -m comment --comment \u0026#39;StorageOS\u0026#39; -j ACCEPT iptables -I INPUT -p tcp --dport 25705:25960 -m comment --comment \u0026#39;StorageOS\u0026#39; -j ACCEPT # Outbound traffic iptables -I OUTPUT -o lo -m comment --comment \u0026#39;Permit loopback traffic\u0026#39; -j ACCEPT iptables -I OUTPUT -d 0.0.0.0/0 -m comment --comment \u0026#39;Permit outbound traffic\u0026#39; -j ACCEPT Please ensure that the iptables rules you have added above come before any default DROP or REJECT rules.\n","excerpt":"Port list StorageOS daemons listen on specific ports, which we require to be accessible between all …","ref":"/docs/prerequisites/firewalls/","title":"Firewalls"},{"body":" Modern versions of Kubernetes, Docker or other Container Runtimes enable mount propagation by default.\n StorageOS requires mount propagation enabled to present devices as volumes for containers (see linux kernel documentation here).\nCertain versions of docker ship with a systemd manifest with MountFlags set to \u0026lsquo;slave\u0026rsquo;, thus preventing StorageOS from working. This can be removed or set to \u0026lsquo;shared\u0026rsquo; with a systemd drop in:\nmkdir -p /etc/systemd/system/docker.service.d/ cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/systemd/system/docker.service.d/mount_propagation_flags.conf [Service] MountFlags=shared EOF # systemctl daemon-reload # systemctl restart docker.service To confirm behaviour, the following command should run without error.\ndocker run -it --rm -v /mnt:/mnt:shared busybox sh -c /bin/date Orchestrators such as Kubernetes or OpenShift have their own ways of exposing this setting. Kubernetes 1.10 and OpenShift 3.10 have mount propagation enabled by default. Previous versions require that feature gates are enabled on the Kubernetes master\u0026rsquo;s controller-manager and apiserver services and in the kubelet service on each node.\nInstallations of orchestrators using Docker require that mount propagation is enabled for both.\nRefer to our installation pages for the orchestrators to see details on how to check and enable mount propagation where appropriate.\n","excerpt":"Modern versions of Kubernetes, Docker or other Container Runtimes enable mount propagation by …","ref":"/docs/prerequisites/mountpropagation/","title":"Mount Propagation"},{"body":"Minimum requirements: One machine with the following:\n Minimum two core with 4GB RAM. Linux with a 64-bit architecture. Kubernetes 1.17 to 1.21 Container Runtime Engine: CRI-O, Containerd or Docker 1.10+ with mount propagation enabled. The necessary ports should be open. See the ports and firewall settings. Etcd cluster for StorageOS A mechanism for device presentation.  Recommended:  At least three nodes for replication and high availability. Kubernetes 1.17 to 1.21 Install the storageos CLI. If using Helm2, make sure the tiller ServiceAccount has enough privileges to create resources such as Namespaces, ClusterRoles, etc. For instance, following this installation procedure. System clocks synchronized using NTP or similar methods. While our distributed consensus algorithm does not require synchronised clocks, it does help to more easily correlate logs across multiple nodes. A PID cgroup limit of 32768. Some aspects of product operation require kernel support for IPv6. See the IPv6 prequisites page.  ","excerpt":"Minimum requirements: One machine with the following:\n Minimum two core with 4GB RAM. Linux with a …","ref":"/docs/prerequisites/","title":"Prerequisites"},{"body":"Kubernetes with StorageOS StorageOS integrates transparently with Kubernetes and different distributions such as OpenShift, Rancher, EKS, AKS, GKE, etc. The user can provide standard PVC definitions and StorageOS will dynamically provision matching volumes. StorageOS presents volumes to containers with standard POSIX mount targets. This enables the Kubelet to mount StorageOS volumes using standard linux device files. Checkout device presentation for more details.\nKubernetes and StorageOS communicate with each other to perform actions such as creation, deletion or mounting of volumes. The CSI (Container Storage Interface) driver is the standard method of communication. Using CSI, Kubernetes and StorageOS communicate over a Unix domain socket.\nCSI (Container Storage Interface) Note CSI is the standard method of communication that enables storage drivers for Kubernetes to release on their own schedule. The CSI standard allows storage vendors to upgrade, update, and enhance their drivers without the need to update Kubernetes source code, or follow Kubernetes release cycles.\nCSI was released GA from Kubernetes 1.13. StorageOS v2 only supports the use of CSI as a storage driver. In addition, the StorageOS Cluster Operator handles the configuration of the CSI driver and its complexity by detecting the version of the Kubernetes installed.\nCheck out the status of the CSI release cycle in relation with Kubernetes in the CSI project page.\nCSI communication is fully supported by StorageOS if the cluster is deployed with any supported Linux Distribution.\nKubernetes Upgrades on Managed Services Managed services that support in place upgrades are fully supported. However, upgrading Kubernetes using green/blue deployments is not supported. This is because nodes are replaced rather than being upgraded. Any data stored on the nodes is lost when new nodes replace the previous ones.\nDocker Some managed Kubernetes platforms such as Azure AKS, enable the \u0026lsquo;Live-Restore\u0026rsquo; Docker feature, enabling containers to continue running while Docker is stopped or upgraded. This feature can cause nodes to hang while shutting-down or rebooting, as rather than going through an orderly shutdown, StorageOS (and other processes) are killed before the disks are synced and unmounted. Devices in this inaccessible state will log a warning similar to:\nTransport endpoint not connected  To prevent this behaviour, we advise disabling this feature by setting\n{ ... \u0026quot;live-restore\u0026quot;: false }  in /etc/docker/daemon.json.\nHere\u0026rsquo;s an example Ansible snippet that might be used to achieve this\n... - name: configure /etc/docker/daemon.json lineinfile: path: /etc/docker/daemon.json regexp: '^.*\u0026quot;live-restore\u0026quot;: true,$' line: ' \u0026quot;live-restore\u0026quot;: false,' backrefs: yes notify: restart docker ...   Note: Use at own risk; you may need to adapt the example to work in your environment\n ","excerpt":"Kubernetes with StorageOS StorageOS integrates transparently with Kubernetes and different …","ref":"/docs/install/","title":"Install"},{"body":"StorageOS recommends that a PID cgroup limit of 32768 be used for StorageOS pods.\n Most environments fulfill this prerequisite by default. Check the StorageOS init container logs as shown below to ensure this is the case.\n StorageOS pods running in Kubernetes are part of a PID cgroup that may limit the maximum number of PIDs that all containers in the PID cgroup slice can spawn. As the Linux kernel assigns a PID to processes and Light Weight Processes (LWP) a low limit can be easily reached under certain circumstances. The PID limit can be set by the Kubernetes distribution or by the container runtime. Generally the limit is set to the machine wide default limit of 32768 but some environments can set this as low as 1024. A low PID limit may prevent StorageOS from spawning the required threads.\nThe StorageOS init container runs a script that checks for the PID limit of the PID cGroup slice that the StorageOS pod runs in. If the script finds that the limit is less than 32768 it will log a warning. This warning can be viewed using kubectl to check the init container logs.\n$ kubectl -n kube-system logs -l app=storageos,kind=daemonset -c storageos-init WARNING: Effective max.pids limit (1024) less than RECOMMENDED_MAX_PIDS_LIMIT (32768) Setting a Kubernetes PID limit Kubernetes defaults to an unlimited PodPidsLimit, which results in the usage of the machine wide limit; typically 32768.\nFor information on how to configure the Kubernetes PID limit see the Kubernetes documentation here.\nSetting a CRI-O PID limit Certain orchestrators or setups use CRI-O as the container runtime. Openshift 4.x currently has CRI-O set a PID limit of 1024 by default. To configure the default CRI-O limit in Openshift 4.x see the RedHat documentation here. To configure CRI-O more generally see the CRI-O documentation here.\n","excerpt":"StorageOS recommends that a PID cgroup limit of 32768 be used for StorageOS pods.\n Most environments …","ref":"/docs/prerequisites/pidlimits/","title":"PID Limits"},{"body":"While StorageOS is platform agnostic, there are certain configurations or operations that relate to a specific platform.\nStorageOS can be used in any Kubernetes distribution when the prerequisites are met. That includes Kubernetes manged services.\n","excerpt":"While StorageOS is platform agnostic, there are certain configurations or operations that relate to …","ref":"/docs/platforms/","title":"Platforms"},{"body":"Use an external Etcd cluster StorageOS uses the etcd distributed key-value store to store essential cluster metadata and manage distributed configuration state. For production environments and testing of production workloads, we recommend deploying an external etcd cluster. For more details about, and an example of, how to run etcd, see the External etcd Operations page.\nIt is highly recommended to use external etcd for cloud environments and place the etcd cluster on stable nodes. Placing the etcd on nodes that are recycled often might affect the normal operations of StorageOS.\nEtcd low latency IO It is recommended to run etcd on low-latency disks and keep other IO-intensive applications separate from the etcd nodes. Etcd is very sensitive to IO latency. Thus, the effect of disk contention can cause etcd downtime.\nBatch jobs such as backups, builds or application bundling can easily cause a high usage of disks making etcd unstable. It is recommended to run such workloads apart from the etcd servers.\nSetup of storage on the hosts We recommend creating a separate filesystem for StorageOS to mitigate the risk of filling the root filesystem on nodes. This has to be done for each node in the cluster.\nFollow the managing host storage best practices page for more details.\nResource reservations StorageOS resource consumption depends on the workloads and the StorageOS features in use.\nThe recommended minimum memory reservation for the StorageOS Pods is 512MB for non-production environments. However it is recommended to prepare nodes so StorageOS can operate with at least with 1-2GB of memory. StorageOS frees memory when possible.\nFor production environments, we recommend 4GB of Memory and 1 CPU as a minimum and to test StorageOS using realistic workloads and tune resources accordingly.\nStorageOS Pods resource allocation will impact directly on the availability of volumes in case of eviction or resource limit triggered restart. It is recommended to not limit StorageOS Pods.\nStorageOS implements a storage engine, therefore limiting CPU consumption might affect the I/O throughput of your volumes.\nSetting a Kubernetes PID limit StorageOS recommends that a PID cgroup limit of 32768 be set. StorageOS is a multi-threaded application and while most Kubernetes distributions set the PID cgroup limit to 32768, some environments can set a limit as low as 1024. The StorageOS init container will print a log message warning if the PID cgroup limit is too low. See our prerequisites for more information.\nMaintain a sufficient number of nodes for replicas to be created To ensure that a new replica can always be created, an additional node should be available. To guarantee high availability, clusters using Volumes with 1 replica must have at least 3 storage nodes. When using Volumes with 2 replicas, at least 4 storage nodes, 3 replicas, 5 nodes, etc.\nMinimum number of storage nodes = 1 (primary) + N (replicas) + 1\nFor more information, see the section on replication.\nStorageOS API username/password The API grants full access to StorageOS functionality, therefore we recommend that the default administrative password of \u0026lsquo;storageos\u0026rsquo; is reset to something unique and strong.\nYou can change the default parameters by encoding the apiUsername and apiPassword values (in base64) into the storageos-api secret.\nTo generate a unique password, a technique such as the following, which generates a pseudo-random 24 character string, may be used:\n# Generate strong password PASSWORD=$(cat -e /dev/urandom | tr -dc \u0026#39;a-zA-Z0-9-!@#$%^\u0026amp;*()_+~\u0026#39; | fold -w 24 | head -n 1) # Convert password to base64 representation for embedding in a K8S secret BASE64PASSWORD=$(echo -n $PASSWORD | base64) Note that the Kubernetes secret containing a strong password must be created before bootstrapping the cluster. Multiple installation procedures use this Secret to create a StorageOS account when the cluster first starts.\nStorageOS Pod placement StorageOS must run on all nodes that will contribute storage capacity to the cluster or that will host Pods which use StorageOS volumes. For production environments, it is recommended to avoid placing StorageOS Pods on Master nodes.\nStorageOS is deployed with a DaemonSet controller, and therefore tolerates the standard unschedulable (:NoSchedule) action. If that is the only taint placed on master or cordoned nodes StorageOS pods might start on them (see the Kubernetes docs for more details). To avoid scheduling StorageOS pods on master nodes, you can add an arbitrary taint to them for which the StorageOS DaemonSet won\u0026rsquo;t have a toleration.\nDedicated instance groups Cloud environments give users the ability to quickly scale the number of nodes in a cluster in response to their needs. Because of the ephemeral nature of the cloud, StorageOS recommends setting conservative downscaling policies.\nFor production clusters, it recommended to use dedicated instance groups for Stateful applications that allow the user to set different scaling policies and define StorageOS pools based on node selectors to collocate volumes.\nLosing a few nodes at the same time could cause the loss of data even when volume replicas are being used.\nPort blocking StorageOS exposes ports to operate. It is recommended that the ports are not accessible from outside the scope of your cluster.\nStorageOS in Docker EE StorageOS does not support running on Swarm nodes nor on mixed (Kubernetes and Swarm) nodes. StorageOS volumes have to be provisioned and used from Kubernetes nodes.\n","excerpt":"Use an external Etcd cluster StorageOS uses the etcd distributed key-value store to store essential …","ref":"/docs/best-practices/","title":"StorageOS Best Practices"},{"body":"As part of the dataplane operation, StorageOS uses Linux AIO (Asynchronous Input Output) contexts to serve I/O requests without blocking. StorageOS requires 4 AIO contexts per deployment (i.e. a StorageOS volume deployment, whether master or replica).\nMax AIO prerequisite. By default there is a maximum number of AIO contexts that can be allocated at once.\nThe current and maximum number of AIO requests is visible in the virtual files /proc/sys/fs/aio-nr and /proc/sys/fs/aio-max-nr.\nThe default context limit has been set at 2^16 or 65536. This figure may vary so please check your /proc/sys/fs/aio-max-nr\nWhen aio-nr reaches aio-max-nr the io_setup syscall will fail with EAGAIN. For more information please see the Linux kernel docs here.\nWhy is this relevant? As StorageOS requires 4 AIO contexts per deployed volume, there is a limit to the number of volumes that can be deployed per node. Trying to provision additional deployments once the aio-max-nr has been reached will fail as the kernel will be unable to create enough new AIO contexts.\nIncreasing your AIO context cap. If your nodes aio-max-nr is set too low you can either provision additional nodes to reduce the number of deployments per node, or increase the aio-max-nr kernel parameter.\nYou can do this by editing your /etc/sysctl.conf file with the following example line:\nfs.aio-max-nr = 1048576 To activate the new settings, run the following command:\n$ sysctl -p /etc/sysctl.conf ","excerpt":"As part of the dataplane operation, StorageOS uses Linux AIO (Asynchronous Input Output) contexts to …","ref":"/docs/prerequisites/max-aio/","title":"Max AIO"},{"body":"","excerpt":"","ref":"/docs/operations/","title":"Operations"},{"body":"StorageOS can be used to provide permanent storage for other applications running in Kubernetes or other Orchestrators that are derived from Kubernetes such as OpenShift or Rancher. This is useful for running stateful applications, such as databases or CI/CD applications, under the control of Kubernetes as Kubernetes can make scheduling decisions without the application data being lost.\nWhat we have outlined in the cookbooks below are some quick deployments of stateful applications into a Kubernetes cluster. These examples are not production ready but have been provided to give you some insight into how to use StorageOS with stateful applications.\nStatefulSets The examples we have provided use StatefulSets as a way to deploy applications into Kubernetes. The reason for this is that the StatefulSet controller is designed to manage stateful applications and it \u0026ldquo;provides guarantees about the ordering and uniqueness\u0026rdquo; of sets of pods.\nPractically this means that when a StatefulSet scales, pods are created in order from 0-(N-1), if a StatefulSet scales down then pods are deleted in reverse order from (N-1)-0.\nSecondly, it means that the behaviour of the StatefulSet controller differs from that of Deployment and ReplicaSet controllers. For Deployment and ReplicaSet controllers \u0026ldquo;\u0026hellip; at many points in the lifetime of a replica set there will be 2 copies of a pod\u0026rsquo;s processes running concurrently\u0026rdquo;. Having two different pods mount a volume at the same time can cause corruption of data. Currently Kubernetes accessModes only apply restrictions to nodes mounting volumes rather than pods, so it is important that StatefulSets are used with StorageOS volumes so the necessary pod uniqueness guarantees are maintained.\nStatefulSet Manifests The StorageOS specific part of the Kubernetes manifests for these examples lies in the VolumeClaimTemplate that\u0026rsquo;s part of the statefulset definition.\nVolumeClaimTemplate\napiVersion:apps/v1kind:StatefulSetmetadata:name:mssqlspec:selector:matchLabels:app:mssqlenv:prodserviceName:mssqlreplicas:1template:metadata:labels:app:mssqlenv:prodspec:serviceAccountName:mssqlcontainers:- name:fooimage:barvolumeMounts:- name:bazmountPath:/var/opt/barenvFrom:- configMapRef:name:mssqlvolumeClaimTemplates:- metadata:name:bazlabels:env:prodspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClass resources:requests:storage:5GiIn the StatefulSet definition above the container has a volume mount defined called baz. The definition for this volume is found in the VolumeClaimTemplate where the fast storageClass will be used to dynamically provision storage if the persistent volume does not already exist.\n","excerpt":"StorageOS can be used to provide permanent storage for other applications running in Kubernetes or …","ref":"/docs/usecases/","title":"Use Cases: Install applications on Kubernetes with StorageOS"},{"body":"StorageOS requires an etcd cluster in order to function. For more information on why etcd is required please see our etcd concepts page.\nWe do not support using the Kubernetes etcd for StorageOS installations.\nWe provide two methods for installing etcd. For those looking for a quick route to evaluating StorageOS, our first method installs etcd into your Kubernetes cluster using the CoreOS operator. Due to limitations in the CoreOS operator, this installation is not backed by any persistent storage, and therefore is unsuitable for production installations.\nFor production installations, there is currently no satisfactory way of installing a production-grade etcd cluster inside Kubernetes (although the landscape is changing rapidly, watch this space), and our production guidance remains to install etcd on separate machines outside of your Kubernetes cluster. This method is the best way to ensure a stable StorageOS cluster. Please see our etcd operations page for additional information on deployment best practices and concerns.\n Ephemeral pods within Kubernetes (Testing) External Virtual Machines (Production)  Click the tabs below to select the installation method of your choice.\n Testing Production    Testing - Installing Etcd Into Your Kubernetes Cluster This fast and convenient method is useful for quickly creating an etcd cluster in order to evaluate StorageOS. Do not use it for production installations.\nThis method uses the CoreOS etcd-operator to install a 3 node etcd cluster within your Kubernetes cluster, in the storageos-etcd namespace. We then install a Kubernetes service in that same namespace.\nThe official etcd-operator repository also has a backup deployment operator that can help backup etcd data. A restore of the etcd keyspace from a backup might cause issues due to the disparity between the cluster state and its metadata in a different point in time. If you need to restore from a backup after a failure of etcd, contact the StorageOS support team.\nQuick Install For a one command install, the following script uses kubectl to create an etcd cluster in the storageos-etcd namespace. It requires kubectl in the system path, and the context set to the appropriate cluster.\ncurl -s https://docs.storageos.com/sh/deploy-etcd.sh | bash Installation Step by Step For those who would prefer to execute the steps by themselves, they are as follows:\n  Configure Namespace\nexport NAMESPACE=storageos-etcd   Create Namespace\nkubectl create namespace $NAMESPACE   If running in Openshift, an SCC is needed to start Pods\noc adm policy add-scc-to-user anyuid system:serviceaccount:$NAMESPACE:default   Create ClusterRole and ClusterRoleBinding\n$ kubectl -n $NAMESPACE create -f-\u0026lt;\u0026lt;END apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: etcd-operator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: etcd-operator subjects: - kind: ServiceAccount name: default namespace: $NAMESPACE END $ kubectl -n $NAMESPACE create -f-\u0026lt;\u0026lt;END apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: etcd-operator rules: - apiGroups: - etcd.database.coreos.com resources: - etcdclusters - etcdbackups - etcdrestores verbs: - \u0026#34;*\u0026#34; - apiGroups: - apiextensions.k8s.io resources: - customresourcedefinitions verbs: - \u0026#34;*\u0026#34; - apiGroups: - \u0026#34;\u0026#34; resources: - pods - services - endpoints - persistentvolumeclaims - events verbs: - \u0026#34;*\u0026#34; - apiGroups: - apps resources: - deployments verbs: - \u0026#34;*\u0026#34; # The following permissions can be removed if not using S3 backup and TLS - apiGroups: - \u0026#34;\u0026#34; resources: - secrets verbs: - get END   Deploy Etcd Operator\n$ kubectl -n $NAMESPACE create -f - \u0026lt;\u0026lt;END apiVersion: apps/v1 kind: Deployment metadata: name: etcd-operator spec: selector: matchLabels: app: etcd-operator replicas: 1 template: metadata: labels: app: etcd-operator spec: containers: - name: etcd-operator image: quay.io/coreos/etcd-operator:v0.9.4 command: - etcd-operator env: - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name END Wait for the Etcd Operator Pod to start\nkubectl -n $NAMESPACE get pod -lapp=etcd-operator   Create the EtcdCluster resource\n$kubectl-n$NAMESPACEcreate-f- \u0026lt;\u0026lt;ENDapiVersion:\u0026#34;etcd.database.coreos.com/v1beta2\u0026#34;kind:\u0026#34;EtcdCluster\u0026#34;metadata:name:\u0026#34;storageos-etcd\u0026#34;spec:size:3version:\u0026#34;3.4.7\u0026#34;pod:etcdEnv:- name:ETCD_QUOTA_BACKEND_BYTESvalue:\u0026#34;2147483648\u0026#34;# 2 GB- name:ETCD_AUTO_COMPACTION_RETENTIONvalue:\u0026#34;1000\u0026#34;# Keep 1000 revisions (default)- name:ETCD_AUTO_COMPACTION_MODEvalue:\u0026#34;revision\u0026#34;# Set the revision moderesources:requests:cpu:200mmemory:300MisecurityContext:runAsNonRoot:truerunAsUser:9000fsGroup:9000tolerations:- operator:\u0026#34;Exists\u0026#34;affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight:100podAffinityTerm:labelSelector:matchExpressions:- key:etcd_clusteroperator:Invalues:- storageos-etcdtopologyKey:kubernetes.io/hostnameEND  Installation Verification $ kubectl -n storageos-etcd get pod,svc NAME READY STATUS RESTARTS AGE pod/etcd-operator-55978c4587-8kx7b 1/1 Running 0 2h pod/storageos-etcd-qm9tmrpnlm 1/1 Running 0 2h pod/storageos-etcd-rzhjdz74hp 1/1 Running 0 2h pod/storageos-etcd-wvvv2d9g98 1/1 Running 0 2h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/storageos-etcd ClusterIP None \u0026lt;none\u0026gt; 2379/TCP,2380/TCP 22h service/storageos-etcd-client ClusterIP 172.30.132.255 \u0026lt;none\u0026gt; 2379/TCP 22h  The URL from the Service storageos-etcd-client.storageos-etcd.svc:2379 will be used later in the StorageOS Cluster CustomResource the kvBackend.address.\n Known etcd-operator issues Etcd is a distributed key-value store database focused on strong consistency. That means that etcd nodes perform operations across the cluster to ensure quorum. If quorum is lost, etcd nodes stop and etcd marks its contents as read-only. This is because it cannot guarantee that new data will be valid. Quorum is fundamental for etcd operations. When running etcd using the CoreOS Operator, it is important to consider that a loss of quorum could arise from etcd pods being evicted from nodes.\nOperations such as Kubernetes Upgrades with rolling node pools could cause a total failure of the etcd cluster as nodes are discarded in favor of new ones.\nA 3 etcd node cluster can survive losing one node and recover, a 5 node cluster can survive the loss of two nodes. Loss of further nodes will result in quorum being lost.\nThe etcd-operator doesn\u0026rsquo;t support a full stop of the cluster. Stopping the etcd cluster causes the loss of all the etcd keystore and make StorageOS unable to perform metadata changes.\n Production - Etcd on External Virtual Machines For production installations, StorageOS strongly recommends running etcd outside of Kubernetes on a minimum of 3 dedicated virtual machines. This topology offers strong guarantees of resilience and uptime. We recommend this architecture in all environments, including those where Kubernetes is being deployed as a managed service.\nStorageOS doesn\u0026rsquo;t require a high performance etcd cluster, as the throughput of metadata to the cluster is low. However, we recommend a careful assessment of IOPS capacity best practices to ensure that etcd operates normally.\nDepending on the level of redundancy you feel comfortable with you can install etcd on the Kubernetes Master nodes. Take extreme care to avoid collisions of the StorageOS etcd installation with the Kubernetes etcd when using the Kubernetes Master nodes. Precautions such as changing the default configuration for the client and peer ports, and ensuring the etcd data directory is modified. The ansible playbook below will default the etcd installation directory to /var/lib/storageos-etcd.\nYou can choose between two installation options.\n Manual Installation Ansible Installation  Installation - Manual This section documents the steps required for manual installation of etcd using standard package management commands and systemd manifests.\n Repeat the following steps on all the nodes that will run etcd as a systemd service.\n   Configure Etcd version and ports\nexport ETCD_VERSION=\u0026#34;3.4.9\u0026#34; export CLIENT_PORT=\u0026#34;2379\u0026#34; export PEERS_PORT=\u0026#34;2380\u0026#34;  If targeting Kubernetes Master nodes, you must change CLIENT_PORT, PEERS_PORT\n   Download Etcd from CoreOS official site\ncurl -L https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz mkdir -p /tmp/etcd-v${ETCD_VERSION}-linux-amd64 tar -xzvf /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz -C /tmp/etcd-v${ETCD_VERSION}-linux-amd64 --strip-components=1 rm /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz   Install Etcd binaries\ncd /tmp/etcd-v${ETCD_VERSION}-linux-amd64 mv etcd /usr/local/sbin/etcd3 mv etcdctl /usr/local/sbin/etcdctl chmod 0755 /usr/local/sbin/etcd3 /usr/local/sbin/etcdctl   Set up persistent Etcd data directory\nmkdir /var/lib/storageos-etcd   Create the systemd environment file\nOn all nodes that will run etcd create a systemd environemnt file /etc/etcd.conf which has the IPs of all the nodes. The NODE_IP will need to change to correspond to the node IP where the environment file resides. NODE1_IP, NODE2_IP and NODE3_IP will remain the same across all three files.\n$ cat \u0026lt;\u0026lt;END \u0026gt; /etc/etcd.conf # NODE_IP is the IP of the node where this file resides. NODE_IP=10.64.10.228 # Node 1 IP NODE1_IP=10.64.10.228 # Node 2 IP NODE2_IP=10.64.14.233 # Node 3 IP NODE3_IP=10.64.12.111 CLIENT_PORT=${CLIENT_PORT} PEERS_PORT=${PEERS_PORT} END # Verify that variables are expanded in the file $ cat /etc/etcd.conf   Create the systemd unit file for etcd3 service\nCreate a systemd unit file /etc/systemd/system/etcd3.service with the following information:\n[Unit] Description=etcd3 Documentation=https://github.com/coreos/etcd Conflicts=etcd2.service [Service] Type=notify Restart=always RestartSec=5s LimitNOFILE=40000 TimeoutStartSec=0 EnvironmentFile=/etc/etcd.conf ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \\  --data-dir /var/lib/storageos-etcd \\  --quota-backend-bytes 8589934592 \\  --auto-compaction-retention 1000 \\  --auto-compaction-mode revision \\  --initial-cluster-state new \\  --initial-cluster-token etcd-token \\  --listen-client-urls http://${NODE_IP}:${CLIENT_PORT},http://127.0.0.1:${CLIENT_PORT} \\  --advertise-client-urls http://${NODE_IP}:${CLIENT_PORT} \\  --listen-peer-urls http://${NODE_IP}:${PEERS_PORT} \\  --initial-advertise-peer-urls http://${NODE_IP}:${PEERS_PORT} \\  --initial-cluster etcd-${NODE1_IP}=http://${NODE1_IP}:${PEERS_PORT},etcd-${NODE2_IP}=http://${NODE2_IP}:${PEERS_PORT},etcd-${NODE3_IP}=http://${NODE3_IP}:${PEERS_PORT} [Install] WantedBy=multi-user.target  $NODE_IP is the IP address of the machine you are installing etcd on.`\n  Note that setting the advertise-client-urls incorrectly will cause any client connection to fail. StorageOS will fail to communicate to Etcd.\n  If enabling TLS, it is recomended to generate your own CA certificate and key. You will need to distribute the keys and certificates for the client auth on all etcd nodes. Moreover, the ExecStart value should look as below:\n ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \\  --data-dir /var/lib/storageos-etcd \\  --quota-backend-bytes 8589934592 \\  --auto-compaction-retention 1000 \\  --auto-compaction-mode revision \\  --peer-auto-tls \\  --client-cert-auth --trusted-ca-file=/path/to/client-cert.pem \\  --cert-file=/path/to/ca.pem \\  --key-file=/path/to/client-key.pem \\  --initial-cluster-state new \\  --initial-cluster-token etcd-token \\  --listen-client-urls https://${NODE_IP}:${CLIENT_PORT} \\  --advertise-client-urls https://${NODE_IP}:${CLIENT_PORT} \\  --listen-peer-urls https://${NODE_IP}:${PEERS_PORT} \\  --initial-advertise-peer-urls https://${NODE_IP}:${PEERS_PORT} \\  --initial-cluster etcd-${NODE1_IP}=https://${NODE1_IP}:${PEERS_PORT},etcd-${NODE2_IP}=https://${NODE2_IP}:${PEERS_PORT},etcd-${NODE3_IP}=https://${NODE3_IP}:${PEERS_PORT}   Reload and start the etc3 systemd service\n$ systemctl daemon-reload $ systemctl enable etcd3.service $ systemctl start etcd3.service   Installation Verification\n The etcdctl binary is installed at /usr/local/bin on the nodes.\n $ ssh $NODE # Any node running the new etcd $ ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:${CLIENT_PORT} member list # $NODE_IP - the IP of the node 66946cff1224bb5, started, etcd-b94bqkb9rf, http://172.28.0.1:2380, http://172.28.0.1:2379 17e7256953f9319b, started, etcd-gjr25s4sdr, http://172.28.0.2:2380, http://172.28.0.2:2379 8b698843a4658823, started, etcd-rqdf9thx5p, http://172.28.0.3:2380, http://172.28.0.3:2379  Read the etcd operations page for our etcd recommendations.\n   Installation - Ansible For a repeatable and automated installation, use of a configuration management tool such as ansible is recommended. StorageOS provides an ansible playbook to help you deploy etcd on standalone virtual machines.\n  Clone StorageOS deployment repository\ngit clone https://github.com/storageos/deploy.git cd k8s/deploy-storageos/etcd-helpers/etcd-ansible-systemd   Edit the inventory file\n The inventory file targets the nodes that will run etcd. The file hosts is an example of such an inventory file.\n $ cat hosts [nodes] centos-1 ip=\u0026#34;10.64.10.228\u0026#34; fqdn=\u0026#34;ip-10-64-10-228.eu-west-2.compute.internal\u0026#34; centos-2 ip=\u0026#34;10.64.14.233\u0026#34; fqdn=\u0026#34;ip-10-64-14-233.eu-west-2.compute.internal\u0026#34; centos-3 ip=\u0026#34;10.64.12.111\u0026#34; fqdn=\u0026#34;ip-10-64-12-111.eu-west-2.compute.internal\u0026#34; # Edit the inventory file $ vi hosts # Or your own inventory file  The ip or fqdn are used to expose the advertise-client-urls of Etcd. Failing to provide valid ip/fqdn will cause any client connection to fail. StorageOS will fail to communicate to Etcd.\n   Edit the etcd configuration\n If targeting Kubernetes Master nodes, you must change etcd_port_client, etcd_port_peers\n $ cat group_vars/all etcd_version: \u0026#34;3.4.9\u0026#34; etcd_port_client: \u0026#34;2379\u0026#34; etcd_port_peers: \u0026#34;2380\u0026#34; etcd_quota_bytes: 8589934592 # 8 GB etcd_auto_compaction_mode: \u0026#34;revision\u0026#34; etcd_auto_compaction_retention: \u0026#34;1000\u0026#34; members: \u0026#34;{{ groups[\u0026#39;nodes\u0026#39;] }}\u0026#34; installation_dir: \u0026#34;/var/lib/storageos-etcd\u0026#34; advertise_format: \u0026#39;fqdn\u0026#39; # fqdn || ip backup_file: \u0026#34;/tmp/backup.db\u0026#34; tls: enabled: false ca_common_name: \u0026#34;eu-west-2.compute.internal\u0026#34; etcd_common_name: \u0026#34;*.eu-west-2.compute.internal\u0026#34; cert_dir: \u0026#34;/etc/etcdtls\u0026#34; ca_cert_file: \u0026#34;etcd-ca.pem\u0026#34; etcd_server_cert_file: \u0026#34;server.pem\u0026#34; etcd_server_key_file: \u0026#34;server-key.pem\u0026#34; etcd_client_cert_file: \u0026#34;etcd-client.crt\u0026#34; etcd_client_key_file: \u0026#34;etcd-client.key\u0026#34; $ vi group_vars/all  Choose between using IP addressing or FQDN in the advertise_format parameter. It allows you to decide how Etcd advertises its address to clients. This is particularly relevant when using TLS.\n  If enabling TLS, it is recomended to generate your own CA certificate and key. You can do it by generating the CA from the machine running Ansible by: ansible-playbook create_ca.yaml.\n   Install\nansible-playbook -i hosts install.yaml   Installation Verification\n The playbook installs the etcdctl binary on the nodes, at /usr/local/bin.\n $ ssh $NODE # Any node running the new etcd $ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 member list 66946cff1224bb5, started, etcd-b94bqkb9rf, http://172.28.0.1:2380, http://172.28.0.1:2379 17e7256953f9319b, started, etcd-gjr25s4sdr, http://172.28.0.2:2380, http://172.28.0.2:2379 8b698843a4658823, started, etcd-rqdf9thx5p, http://172.28.0.3:2380, http://172.28.0.3:2379   Benefits of Running External to Kubernetes Etcd is a distributed key-value store database focused on strong consistency. That means that etcd nodes perform operations across the cluster to ensure quorum. In the case that quorum is lost, an etcd node stops and marks its contents as read-only. Another peer might have a newer version that has not been committed to the database. Quorum is fundamental for etcd operations.\nIn a Kubernetes environment, applications are scheduled across and in some scenarios such as \u0026ldquo;DiskPressure\u0026rdquo; they may need to be evicted from a node, and be scheduled onto a different node. With an application such as etcd, the scenario described can result in quorum being lost, making the cluster unable to recover automatically. Usually a 3 node etcd cluster can survive losing one node and recover. However, losing a second node at the same time or even having a network partition between them will result in quorum lost.\nBind Etcd IPs to Kubernetes Service Kubernetes external services use a DNS name to reference external endpoints, making them easy to reference from inside the cluster. You can use the example from the helper github repository to deploy the external Service. Using an external service can make monitoring of etcd from Prometheus easier.\n  Using Etcd with StorageOS During installation of StorageOS the kvBackend.address parameter of the StorageOS operator is used to specify the address of the etcd cluster. See the StorageOS cluster operator configuration examples for more information.\n","excerpt":"StorageOS requires an etcd cluster in order to function. For more information on why etcd is …","ref":"/docs/prerequisites/etcd/","title":"Etcd"},{"body":"","excerpt":"","ref":"/docs/reference/bundles/","title":"Bundles"},{"body":"","excerpt":"","ref":"/docs/reference/","title":"Reference"},{"body":"When you need support, please raise a ticket via our Help Desk portal. It is important to select the priority of your ticket in accordance with the severity:\n Severity 1 - Urgent Severity 2 - High Severity 3 - Medium Severity 4 - Low  This helps us route and prioritise the ticket accordingly. You can find definitions of our support severity levels on our website.\nResponses to tickets will be cc\u0026rsquo;d via email.\nFor community support and general enquiries, please join our public Slack channel. \nInformation to include in tickets To help us provide effective support, we request that you provide as much information as possible when contacting us. The list below is a suggested starting point. Additionally, please include anything specific, such as log entries, that may help us debug your issue.\nPlatform  Cloud provider/Bare metal OS distribution and version Kernel version docker version and installation procedure (distro packages or docker install)  StorageOS  Version of StorageOS storageos get nodes storageos get volumes storageos describe volume VOL_ID # in case of issues with a specific volume  Orchestrator related (Kubernetes, OpenShift, etc)  Version and installation method Managed or self managed? kubectl -n kube-system get pod kubectl -n kube-system logs -lapp=storageos -c storageos kubectl -n kube-system get storageclass Specific for your namespaces: kubectl describe pvc PVC_NAME Specific for your namespaces: kubectl describe pod POD_NAME  Environment Changes  Details of any recent changes to your environment such as planned maintenance, node reboots, network failures, etcd outage, etc.. This can help speed up ticket triage and resolution considerably  StorageOS Support Bundle StorageOS provides the ability to generate a support bundle that aggregates cluster information. See Support Bundle for a list of what is included.\nStorageOS engineers might ask for a support bundle to be generated during support cases.\nThe information in the bundle is used only for support purposes, and will be removed once it is no longer needed. If the information is sensitive and can\u0026rsquo;t be given to StorageOS, please make sure that the support engineers have as much information about your environment as possible.\nRefer to the Support Bundle documentation page for details of how to generate a bundle.\n","excerpt":"When you need support, please raise a ticket via our Help Desk portal. It is important to select the …","ref":"/docs/support/","title":"Support"},{"body":"Our self-evaluation guide is a step by step recipe for installing and testing StorageOS. This guide is divided into three sections:\n Installation - install StorageOS with a single command Feature Testing - short walkthrough of some of our features Benchmarking - a recipe to benchmark StorageOS on your infrastructure  For more comprehensive documentation including installation advice for complex setups, operational guides, and use-cases, please see our main documentation site.\nSupport for Self Evaluations Should you have questions or require support, there are several ways to get in touch with us. The fastest way to get in touch is to join our public Slack channel. You can also get in touch via email to info@storageos.com.\nFurthermore you can fill out the form below and we will get in touch.\nGet Started Now   hbspt.forms.create({ portalId: \"3402546\", formId: \"a07fecd3-ce5b-4835-b136-51a94a35632b\", sfdcCampaignId: \"70158000000BAZzAAO\" });   Installation In this document we detail a simple installation suitable for evaluation purposes. The etcd we install uses a 3 node cluster with ephemeral storage, and as such is not suitable for production workloads. However, for evaluation purposes it should be sufficient. For production deployments, please see our main documentation pages.\nA standard StorageOS installation uses the StorageOS operator, which performs most platform-specific configuration for you. The StorageOS operator has been certified by Red Hat and is open source.\nThe basic installation steps are:\n Check prerequisites Install an etcd cluster Install the StorageOS Operator Install StorageOS by applying a CR for the Operator to action  Prerequisites While we do not require custom kernel modules or additional userspace tooling, StorageOS does have a few basic prerequisites that are met by default by most modern distributions:\n At least 1 CPU core, 2GB RAM free. A Kubernetes release within the four most recent versions. TCP ports 5701-5710 and TCP \u0026amp; UDP 5711 open between all nodes in the cluster. A 64bit supported operating system - StorageOS can run without additional packages in Debian 9, RancherOS, RHEL7.5,8 and CentOS7,8 and need the package linux-image-extra for Ubuntu. Mainline kernel modules target_core_mod, tcp_loop, target_core_file, target_core_user, configfs, and ui. These are present by default on most modern linux distributions, and can be installed with standard package managers. See our system configuration page for instructions.  Quick Installation For those feeling brave and/or lucky, we have encapsulated the installation instructions in this guide into a short bash script. As with similar instructions elsewhere on the internet, caution is advised here. The script is liberally commented and we recommend that you inspect it in your text editor of choice before running.\n Run the following command where kubectl is installed and with the context set for your Kubernetes cluster\n curl -sL https://storageos.run | bash Note that this script deploys a cluster with etcd on transient storage, suitable for evaluation purposes only. Do not use this cluster for production workloads.\nYour new cluster can be used straight away. You can now jump to the Provision a Volume section below to provision your first volume and license the cluster.\nInstalling StorageOS Step By Step This section describes the installation steps in some more detail.\nInstall etcd Here we install a 3 node etcd cluster using the CoreOS operator to install on ephemeral storage within your Kubernetes cluster. This is suitable for evaluation purposes. For production deployment instructions, please see our production etcd page.\n  Download repo\n$ git clone https://github.com/coreos/etcd-operator.git   Configure NS, Role and RoleBinding\n$ export ROLE_NAME=etcd-operator $ export ROLE_BINDING_NAME=etcd-operator $ export NAMESPACE=etcd   Create Namespace\n$ kubectl create namespace $NAMESPACE   Deploy Operator\n$ ./etcd-operator/example/rbac/create_role.sh $ kubectl -n $NAMESPACE create -f - \u0026lt;\u0026lt;END apiVersion: apps/v1 kind: Deployment metadata: name: etcd-operator spec: selector: matchLabels: app: etcd-operator replicas: 1 template: metadata: labels: app: etcd-operator spec: containers: - name: etcd-operator image: quay.io/coreos/etcd-operator:v0.9.4 command: - etcd-operator env: - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name END   Create the EtcdCluster resource\n$ kubectl -n $NAMESPACE create -f - \u0026lt;\u0026lt;YAML apiVersion: \u0026#34;etcd.database.coreos.com/v1beta2\u0026#34; kind: \u0026#34;EtcdCluster\u0026#34; metadata: name: \u0026#34;storageos-etcd\u0026#34; spec: size: 3 # Set to 1 if insufficient nodes version: \u0026#34;3.4.7\u0026#34; pod: etcdEnv: - name: ETCD_QUOTA_BACKEND_BYTES value: \u0026#34;2147483648\u0026#34; # 2 GB - name: ETCD_AUTO_COMPACTION_RETENTION value: \u0026#34;1000\u0026#34; # Keep 1000 revisions - name: ETCD_AUTO_COMPACTION_MODE value: \u0026#34;revision\u0026#34; # Set the revision mode resources: requests: cpu: 200m memory: 300Mi securityContext: runAsNonRoot: true runAsUser: 9000 fsGroup: 9000 tolerations: - operator: \u0026#34;Exists\u0026#34; # add tolerations to match all taints on the master nodes where etcd will be deployed affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: etcd_cluster operator: In values: - storageos-etcd topologyKey: kubernetes.io/hostname YAML   This creates a 3 node etcd cluster and associated service in Kubernetes. We can use the service endpoint when creating the StorageOS cluster.\nInstall StorageOS Operator In order to install the StorageOS operator download the requisite yaml manifests or apply them with kubectl.\n$ kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml You can verify the operator is running using the following command\n$ kubectl get pods -n storageos-operator  Install StorageOS Once the StorageOS operator has been installed a StorageOS cluster can be generated by creating a StorageOSCluster resource.\nA StorageOSCluster resource describes the state of the StorageOS cluster that is desired and the StorageOS operator will create the desired StorageOS cluster. For examples of StorageOSCluster resources please see our examples page here. For a full list of the configurable spec parameters of the StorageOSCluster resource please see here.\n  Create a secret defining the API username and password\n$kubectlcreate-f- \u0026lt;\u0026lt;ENDapiVersion:v1kind:Secretmetadata:name:\u0026#34;storageos-api\u0026#34;namespace:\u0026#34;storageos-operator\u0026#34;labels:app:\u0026#34;storageos\u0026#34;type:\u0026#34;kubernetes.io/storageos\u0026#34;data:# echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64apiUsername:c3RvcmFnZW9zapiPassword:c3RvcmFnZW9z# CSI CredentialscsiProvisionUsername:c3RvcmFnZW9zcsiProvisionPassword:c3RvcmFnZW9zcsiControllerPublishUsername:c3RvcmFnZW9zcsiControllerPublishPassword:c3RvcmFnZW9zcsiNodePublishUsername:c3RvcmFnZW9zcsiNodePublishPassword:c3RvcmFnZW9zcsiControllerExpandUsername:c3RvcmFnZW9zcsiControllerExpandPassword:c3RvcmFnZW9zEND  Create a StorageOSCluster resource\n$kubectlcreate-f- \u0026lt;\u0026lt;ENDapiVersion:storageos.com/v1kind:StorageOSClustermetadata:name:example-storageosclusternamespace:\u0026#34;storageos-operator\u0026#34;spec:# StorageOS Pods are in kube-system by defaultsecretRefName:\u0026#34;storageos-api\u0026#34;secretRefNamespace:\u0026#34;storageos-operator\u0026#34;k8sDistro:\u0026#34;upstream\u0026#34;# Set the Kubernetes distribution for your cluster (upstream, eks, aks, gke, rancher, dockeree)# storageClassName: fast # The storage class creates by the StorageOS operator is configurablekvBackend:address:\u0026#34;storageos-etcd-client.etcd.svc.cluster.local:2379\u0026#34;END  Confirm that the cluster has been created and that StorageOS pods are running\n$ kubectl -n kube-system get pods StorageOS pods enter a ready state after a minimum of 45s has passed.\n  Deploy the StorageOS CLI as a container\n$ kubectl -n kube-system run \\ --image storageos/cli:v2.4.0 \\ --restart=Never \\ --env STORAGEOS_ENDPOINTS=storageos:5705 \\ --env STORAGEOS_USERNAME=storageos \\ --env STORAGEOS_PASSWORD=storageos \\ --command cli \\ -- /bin/sh -c \u0026#34;while true; do sleep 999999; done\u0026#34;   Provision a StorageOS Volume Now that we have a working StorageOS cluster, we can provision a volume to test everything is working as expected.\n  Create a PVC\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-1 spec: storageClassName: \u0026#34;fast\u0026#34; accessModes: - ReadWriteOnce resources: requests: storage: 5Gi END   Create 2 replicas by labeling your PVC\n$ kubectl label pvc pvc-1 storageos.com/replicas=2   Verify that the volume and replicas were created with the CLI\n pvc-1 should be listed in the CLI output\n $ kubectl -n kube-system exec -it cli -- storageos get volumes   Create a pod that consumes the PVC\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: Pod metadata: name: d1 spec: containers: - name: debian image: debian:9-slim command: [\u0026#34;/bin/sleep\u0026#34;] args: [ \u0026#34;3600\u0026#34; ] volumeMounts: - mountPath: /mnt name: v1 volumes: - name: v1 persistentVolumeClaim: claimName: pvc-1 END   Check that the pod starts successfully. If the pod starts successfully then the StorageOS cluster is working correctly\n$ kubectl get pod d1 -w The pod mounts a StorageOS volume under /mnt so any files written there will persist beyond the lifetime of the pod. This can be demonstrated using the following commands.\n  Execute a shell inside the pod and write some data to a file\n$ kubectl exec -it d1 -- bash root@d1:/# echo Hello World! \u0026gt; /mnt/hello root@d1:/# cat /mnt/hello  Hello World! should be printed out.\n   Delete the pod\n$ kubectl delete pod d1   Recreate the pod\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: Pod metadata: name: d1 spec: containers: - name: debian image: debian:9-slim command: [\u0026#34;/bin/sleep\u0026#34;] args: [ \u0026#34;3600\u0026#34; ] volumeMounts: - mountPath: /mnt name: v1 volumes: - name: v1 persistentVolumeClaim: claimName: pvc-1 END   Open a shell inside the pod and check the contents of /mnt/hello\n$ kubectl exec -it d1 -- cat /mnt/hello  Hello World! should be printed out.\n   License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nYou will need access to the StorageOS API on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):\n$ kubectl port-forward -n kube-system svc/storageos 5705 Now follow the instructions on our operations licensing page to obtain and apply a license.\nInstallation of StorageOS is now complete.\nStorageOS Features Now that you have a fully functional StorageOS cluster we will explain some of our features that may be of use to you as you complete application and synthetic benchmarks.\nStorageOS features are all enabled/disabled by applying labels to volumes. These labels can be passed to StorageOS via persistent volume claims (PVCs) or can be applied to volumes using the StorageOS CLI or GUI.\nThe following is not an exhaustive feature list but outlines features which are commonly of use during a self-evaluation.\nVolume Replication StorageOS enables synchronous replication of volumes using the storageos.com/replicas label.\nThe volume that is active is referred to as the master volume. The master volume and its replicas are always placed on separate nodes. In fact if a replica cannot be placed on a node without a replica of the same volume, the volume will fail to be created. For example, in a three node StorageOS cluster a volume with 3 replicas cannot be created as the third replica cannot be placed on a node that doesn\u0026rsquo;t already contain a replica of the same volume.\nSee our replication documentation for more information on volume replication.\n  To test volume replication create the following PersistentVolumeClaim\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-replicated labels: storageos.com/replicas: \u0026#34;1\u0026#34; spec: storageClassName: \u0026#34;fast\u0026#34; accessModes: - ReadWriteOnce resources: requests: storage: 5Gi END  Note that volume replication is enabled by setting the storageos.com/replicas label on the volume.\n   Confirm that a replicated volume has been created by using the StorageOS CLI or UI\n$ kubectl -n kube-system exec -it cli -- storageos get volumes   Create a pod that uses the PVC\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: Pod metadata: name: replicated-pod spec: containers: - name: debian image: debian:9-slim command: [\u0026#34;/bin/sleep\u0026#34;] args: [ \u0026#34;3600\u0026#34; ] volumeMounts: - mountPath: /mnt name: v1 volumes: - name: v1 persistentVolumeClaim: claimName: pvc-replicated END   Write data to the volume\n$ kubectl exec -it replicated-pod -- bash root@replicated-pod:/# echo Hello World! \u0026gt; /mnt/hello root@replicated-pod:/# cat /mnt/hello  Hello World! should be printed out.\n   Find the location of the master volume and shutdown the node\nShutting down a node causes all volumes, with online replicas, on the node to be evicted. For replicated volumes this immediately promotes a replica to become the new master.\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-replicated Bound pvc-29e2ad6e-8c4e-11e9-8356-027bfbbece86 5Gi RWO fast 1m $ kubectl exec -it -n kube-system cli -- storageos get volumes NAMESPACE NAME SIZE LOCATION ATTACHED ON REPLICAS AGE default pvc-4e796a62-0271-45f9-9908-21d58789a3fe 5.0 GiB kind-worker (online) kind-worker2 1/1 26 seconds ago   Check the location of the master volume and notice that it is on a new node. If the pod that mounted the volume was located on the same node that was shutdown then the pod will need to be recreated.\n$ kubectl exec -it -n kube-system cli -- storageos get volumes NAMESPACE NAME SIZE LOCATION ATTACHED ON REPLICAS AGE default pvc-4e796a62-0271-45f9-9908-21d58789a3fe 5.0 GiB kind-worker2 (online) kind-worker2 1/1 46 seconds ago   Check that the data is still accessible to the pod\n$ kubectl exec -it replicated-pod -- bash root@replicated-pod:/# cat /mnt/hello  Hello World! should be printed out.\n    Benchmarking Benchmarking storage is a complex topic. Considering the many books and papers that have been written about benchmarking, we could write many paragraphs here and only begin to scratch the surface.\nTaking this complexity into account we present recipes for both synthetic benchmarks using FIO, and a sample application benchmark to test PostgreSQL using pgbench. The approaches are complementary - synthetic benchmarks allow us to strictly control the parameters of the IO we put through the system in order to stress various aspects of it. Application benchmarks allow us to get a sense of the performance of the system when running an actual representative workload - which of course is the ultimate arbiter of performance.\nDespite the inherent complexity of benchmarking storage there are a few general considerations to keep in mind.\nConsiderations Local vs. Remote Volumes When a workload is placed on the same node as a volume, there is no network round trip for IO, and performance is consequently improved. When considering the performance of StorageOS it is important to evaluate both local and remote volumes; since for certain workloads we want the high performance of a local attachment, but we also desire the flexibility of knowing that our less performance-sensitive workloads can run from any node in the cluster.\nThe Effect of Synchronous Replication Synchronous replication does not impact the read performance of a volume, but it can have a significant impact on the write performance of the volume, since all writes must be propagated to replicas before being acked to the application. The impact varies in proportion to the inter-node latency. For an inter-node latency of 1ms, we have a max ceiling of 1000 write IOPS, and in reality a little less than that to allow for processing time on the nodes. This is less concerning then it may first appear, since many applications will issue multiple writes in parallel (known as increasing the queue depth).\nSynthetic Benchmarks Prerequisites  Kubernetes cluster with a minimum of 3 nodes and a minimum of 30 Gb available capacity StorageOS CLI running as a pod in the cluster  Synthetic benchmarks using tools such as FIO are a useful way to begin measuring StorageOS performance. While not fully representative of application performance, they allow us to reason about the performance of storage devices without the added complexity of simulating real world workloads, and provide results easily comparable across platforms.\nFIO has a number of parameters that can be adjusted to simulate a variety of workloads and configurations. Parameters that are particularly important are:\n Block Size - the size of the IO units performed. Queue Depth - the amount of concurrent IOs in flight IO Pattern - access can be random across the disk, or sequentially. IO subsystems typically perform better with sequential IO, because of the effect of read ahead caches, and other factors  For this self-evaluation we will run a set of tests based on the excellent DBench, which distills the numerous FIO options into a series of well-crafted scenarios:\n Random Read/Write IOPS and BW - these tests measure the IOPS ceiling (with a 4k block size) and bandwidth ceiling (with a 128K block size) of the volume using a random IO pattern and high queue depth Average Latency - these tests measure the IO latency of the volume under favourable conditions using a random access pattern, low queue depth and low block size Sequential Read/Write - these tests measure the sequential read/write throughput of the volume with a high queue depth and high block size Mixed Random Read/Write IOPS - these tests measure the performance of the volume under a 60/40 read/write workload using a random access pattern and low blocksize  For convenience we present a single script to run the scenarios using local and remote volumes both with and without a replica. Deploy the FIO tests for the four scenarios using the following command:\ncurl -sL https://docs.storageos.com/sh/deploy-synthetic-benchmarks.sh | bash The script will take approximately 20 minutes to complete, and will print the results to STDOUT.\nThe exact results observed will depend on the particular platform and environment, but the following trends should be observable:\n local volumes perform faster than remote volumes read performance is independent of the number of replicas write performance is dependent on the number of replicas  Application Benchmarks While synthetic benchmarks are useful for examining the behaviour of StorageOS with very specific workloads, in order to get a realistic picture of StorageOS performance actual applications should be tested.\nMany applications come with test suites which provide standard workloads. For best results, test using your application of choice with a representative configuration and real world data.\nAs an example of benchmarking an application the following steps lay out how to benchmark a Postgres database backed by a StorageOS volume.\n  Start by cloning the StorageOS use cases repository. Note this is the same repository that we cloned earlier so if you already have a copy just cd storageos-usecases/pgbench.\n$ git clone https://github.com/storageos/use-cases.git storageos-usecases   Move into the Postgres examples folder\n$ cd storageos-usecases/pgbench   Decide which node you want the pgbench pod and volume to be located on. The node needs to be labelled app=postgres\n$ kubectl label node \u0026lt;NODE\u0026gt; app=postgres   Then set the storageos.com/hint.master label in 20-postgres-statefulset.yaml file to match the StorageOS nodeID for the node you have chosen before creating all the files. The StorageOS nodeID can be obtained using the cli and doing a describe node\n$ kubectl create -f .   Confirm that Postgres is up and running\n$ kubectl get pods -w -l app=postgres   Use the StorageOS CLI or the GUI to check the master volume location and the mount location. They should match\n$ kubectl -n kube-system exec -it cli -- storageos get volumes   Exec into the pgbench container and run pgbench\n$ kubectl exec -it pgbench -- bash -c \u0026#39;/opt/cpm/bin/start.sh\u0026#39;    Conclusion After completing these steps you will have benchmark scores for StorageOS. Please keep in mind that benchmarks are only part of the story and that there is no replacement for testing actual production or production like workloads.\nStorageOS invites you to provide feedback on your self-evaluation to the slack channel or by directly emailing us at info@storageos.com]\n","excerpt":"Our self-evaluation guide is a step by step recipe for installing and testing StorageOS. This guide …","ref":"/docs/self-eval/","title":"Self Evaluation Guide"},{"body":"","excerpt":"","ref":"/index.json","title":""},{"body":" \t window.onload = function() { const ui = SwaggerUIBundle({ url: \"/api-swagger-v2.yaml\", dom_id: '#ohpen_swagger_ui', presets: [ SwaggerUIBundle.presets.apis, SwaggerUIStandalonePreset ] }) window.ui = ui } \t","excerpt":" \t window.onload = function() { const ui = SwaggerUIBundle({ url: \"/api-swagger-v2.yaml\", dom_id: …","ref":"/docs/reference/api/","title":"API Reference"},{"body":"Apply changes to resources $ storageos apply --help Make changes to existing resources Usage: storageos apply [command] Available Commands: licence Apply a product licence to the cluster Flags: -h, --help help for apply Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Use \u0026#34;storageos apply [command] --help\u0026#34; for more information about a command. Apply License $ storageos apply licence --help Apply a product licence to the cluster Usage: storageos apply licence [flags] Aliases: licence, license Examples: $ storageos apply licence --from-file \u0026lt;path-to-licence-file\u0026gt; $ echo \u0026#34;\u0026lt;licence file contents\u0026gt;\u0026#34; | storageos apply licence --from-stdin Flags: --cas string make changes to a resource conditional upon matching the provided version --from-file string reads a StorageOS product licence from a specified file path --from-stdin reads a StorageOS product licence from the standard input -h, --help help for licence Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"Apply changes to resources $ storageos apply --help Make changes to existing resources Usage: …","ref":"/docs/reference/cli/apply/","title":"Apply"},{"body":"$ storageos attach --help Attach a volume to a node Usage: storageos attach [flags] Examples: $ storageos attach --namespace my-namespace-name my-volume my-node Flags: -h, --help help for attach Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos attach --help Attach a volume to a node Usage: storageos attach [flags] Examples: $ …","ref":"/docs/reference/cli/attach/","title":"Attach"},{"body":"StorageOS is fully compatible with AWS EKS. To install StorageOS on EKS, please follow our Kubernetes installation instructions page.\nAn EKS deployment of Kubernetes uses AWS Linux by default with an optimized kernel. As the requisite kernel modules are not available for StorageOS to use TCMU, FUSE will be used as a fallback. Using FUSE instead of TCMU has performance implications.\nFor more details about the OS Distributions check the System Configuration page.\n","excerpt":"StorageOS is fully compatible with AWS EKS. To install StorageOS on EKS, please follow our …","ref":"/docs/platforms/aws-eks/","title":"AWS EKS"},{"body":"StorageOS is fully compatible with Azure AKS. To install StorageOS on AKS please follow our Kubernetes installation instructions page.\nAn AKS deployment of Kubernetes uses Ubuntu by default with an optimized kernel. All versions of Ubuntu with a kernel version later than4.15.0-1029-azure meet the StorageOS prerequisites.\n","excerpt":"StorageOS is fully compatible with Azure AKS. To install StorageOS on AKS please follow our …","ref":"/docs/platforms/azure-aks/","title":"Azure AKS"},{"body":"Cassandra is a popular distributed NoSQL open source database.\nUsing StorageOS persistent volumes with Cassandra means that if a Cassandra pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up the pod data is immediately available. Should Kubernetes schedule the Cassandra pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not a StorageOS master is located on the same node.\nAs Cassandra has features to allow it to handle replication careful consideration of whether to allow StorageOS or Cassandra to handle replication is required.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Cassandra on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion\napiVersion:apps/v1kind:StatefulSetmetadata:name:cassandraspec:selector:matchLabels:app:cassandraserviceName:cassandrareplicas:3...spec:...volumeMounts:- name:cassandra-datamountPath:/var/lib/cassandra...volumeClaimTemplates:- metadata:name:cassandra-dataspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClassresources:requests:storage:5GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.\n  Move into the Cassandra examples folder and create the objects\ncd storageos-usecases kubectl create -f ./cassandra   Confirm Cassandra is up and running.\n$ kubectl get pods -w -l app=cassandra NAME READY STATUS RESTARTS AGE cassandra-0 1/1 Running 0 8m32s cassandra-1 1/1 Running 0 7m51s cassandra-2 1/1 Running 0 6m36s   Connect to the Cassandra client pod and connect to the Cassandra server through the service\n$ kubectl exec -it cassandra-0 -- cqlsh cassandra-0.cassandra Connected to K8Demo at cassandra-0.cassandra:9042. [cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4] Use HELP for help. cqlsh\u0026gt; SELECT cluster_name, listen_address FROM system.local; cluster_name | listen_address --------------+---------------- K8Demo | 100.96.7.124 (1 rows)   ","excerpt":"Cassandra is a popular distributed NoSQL open source database.\nUsing StorageOS persistent volumes …","ref":"/docs/usecases/cassandra/","title":"Cassandra"},{"body":"The storageos command line interface (CLI) is used to manage cluster-wide configuration.\nInstallation # linux/amd64 curl -sSLo storageos \\  https://github.com/storageos/go-cli/releases/download/v2.4.0/storageos_linux_amd64 \\  \u0026amp;\u0026amp; chmod +x storageos \\  \u0026amp;\u0026amp; sudo mv storageos /usr/local/bin/ # MacOS curl -sSLo storageos \\  https://github.com/storageos/go-cli/releases/download/v2.4.0/storageos_darwin_amd64 \\  \u0026amp;\u0026amp; chmod +x storageos \\  \u0026amp;\u0026amp; sudo mv storageos /usr/local/bin/ You will need to provide the correct credentials to connect to the API. The default installation uses the storageos-api Secret to generate the first admin user. By default, it creates a single user with username storageos and password storageos:\nexport STORAGEOS_USERNAME=storageos export STORAGEOS_PASSWORD=storageos export STORAGEOS_ENDPOINTS=10.1.5.249:5705 Run CLI as a container You can run the cli as a container in your Kubernetes cluster. Then exec into it to run commands.\nkubectl -n kube-system run \\  --image storageos/cli:v2.4.0 \\  --restart=Never \\  --env STORAGEOS_ENDPOINTS=storageos:5705 \\  --env STORAGEOS_USERNAME=storageos \\  --env STORAGEOS_PASSWORD=storageos \\  --command cli \\  -- /bin/sh -c \u0026#34;while true; do sleep 100000; done\u0026#34; Usage $ storageos Storage for Cloud Native Applications. By using this product, you are agreeing to the terms of the the StorageOS Ltd. End User Subscription Agreement (EUSA) found at: https://storageos.com/legal/#eusa To be notified about stable releases and latest features, sign up at https://my.storageos.com. Usage: storageos [command] Available Commands: apply Make changes to existing resources attach Attach a volume to a node create Create new resources delete Delete resources in the cluster describe Fetch extended details for resources detach Detach a volume from its current location get Fetch basic details for resources help Help about any command nfs Make changes and attach nfs volumes update Make changes to existing resources version View version information for the StorageOS CLI Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/home/user/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/home/user/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -h, --help help for storageos -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Additional help topics: storageos config-file View help information for using a configuration file storageos env View documentation for configuration settings which can be set in the environment storageos exitcodes View documentation for the exit codes used by the StorageOS CLI Use \u0026#34;storageos [command] --help\u0026#34; for more information about a command. Formatting CLI Output StorageOS CLI output can be formatted using the --output option. The strings that are passed to --output are \u0026lsquo;json\u0026rsquo;, \u0026lsquo;yaml\u0026rsquo; or \u0026lsquo;text\u0026rsquo;.\nCheatsheet    Command Subcommand Description     apply licence Make changes to existing resources.   attach  Attach Volume to a node.   create namespace, policy-group, user, volume Create resources.   delete namespace, node, policy-group, user, volume Delete resources.   describe cluster, licence, namespace, node, policy-group, user, volume Show detailed view of resources.   detach  Detach volume from a node.   get cluster, diagnostics, namespace, node, policy-group, user, volume List resources.   help  Help   nfs attach, endpoint, exports Make changes to and attach nfs volumes   update volume Make changes to existing resources   version  Show CLI version.    Source is available on Github.\n","excerpt":"The storageos command line interface (CLI) is used to manage cluster-wide configuration. …","ref":"/docs/reference/cli/","title":"Command line interface"},{"body":"Our cluster operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator acts as a Kubernetes controller that watches the StorageOSCluster CR (Custom Resource). Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\nYou can find the source code in the cluster-operator repository.\nInstall the operator following orchestrator specific procedure.\nTo deploy a StorageOS cluster you will need to fulfil the following steps.\n Install Etcd Install operator Create storageos-api Secret Create a CR (Custom Resource) to deploy StorageOS  ","excerpt":"Our cluster operator is a Kubernetes native application developed to deploy and configure StorageOS …","ref":"/docs/reference/cluster-operator/","title":"Cluster Operator"},{"body":"StorageOS makes it possible for you to organise your cluster in multiple ways, depending on your priorities and needs. The approaches below are idealised representations of possible StorageOS clusters and can be mixed, modified and changed at execution time.\nStorageOS performs file I/O over the network, which is how we ensure that your data is always available throughout your cluster. This also affords the user certain possibilities for organising your cluster, as we suggest below.\nHyperconverged Model In this topology all nodes can store data and present data. This gives maximum flexibility to the StorageOS and Kubernetes schedulers, and maximum choice for pod placement. No matter how you deploy your workloads they will be able to store and access data on every node.\nBy default we place workloads locally where possible - this means that we default to a hybrid of the hyperconverged/high-performance models, maximising performance without extra effort.\nCentralised Storage Model In this cluster topology volume data is placed on a particular subset of nodes, while the remaining nodes in your cluster are set to computeonly, allowing access to data hosted on other nodes, while consuming no storage themselves. This model can be advantageous if, for example, you want to take advantage of the hardware characteristics of particular nodes. A centralised storage model can also help avoid problems with naive resource allocation, since storage nodes and compute workloads can be kept apart.\nThis mode is also very suitable for elastic fleets with burstable workloads. A fleet can be expanded with many new machines for computing, while maintaining a central data store not impacted by rapid and repeated cluster scaling.\nHigh Performance Mode In this mode we co-locate pods with the volumes they are using in order to take advantage of the performance gains from running on the same node, while retaining the utility of orchestrators for managing app lifecycle.\n","excerpt":"StorageOS makes it possible for you to organise your cluster in multiple ways, depending on your …","ref":"/docs/concepts/cluster-topologies/","title":"Cluster Topologies"},{"body":"Every StorageOS cluster has a unique ClusterID generated at bootstrap. A StorageOS Licence is specific for a ClusterID.\nHow to obtain the ClusterID You can obtain the ClusterID using either the CLI or GUI.\nGUI You will need access to the StorageOS GUI on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):\n$ kubectl port-forward -n kube-system svc/storageos 5705 As an alternative, an Ingress controller may be preferred.\nOnce you have obtained access to the GUI, login using whatever credentials you used to create the cluster and go to the \u0026ldquo;Licence\u0026rdquo; section on the left navigation menu.\nCLI This CLI command can print the cluster ID:\n$ storageos get cluster ID: 704dd165-9580-4da4-a554-0acb96d328cb Licence: expiration: 2021-03-25T13:48:46Z (1 year from now) capacity: 5.0 TiB kind: professional customer name: storageos Created at: 2020-03-25T13:48:33Z (1 hour ago) Updated at: 2020-03-25T13:48:46Z (1 hour ago) Given the Cluster ID, the StorageOS team can generate a licence.\n","excerpt":"Every StorageOS cluster has a unique ClusterID generated at bootstrap. A StorageOS Licence is …","ref":"/docs/operations/cluster-id/","title":"Find ClusterID"},{"body":"StorageOS clusters represent groups of nodes which run a common distributed control plane.\nTypically, a StorageOS cluster maps one-to-one to a Kubernetes (or similar orchestrator) cluster, and we expect our daemonset to run on all worker nodes within the cluster that will consume or present storage.\nClusters use etcd to maintain state and manage distributed consensus between nodes.\n","excerpt":"StorageOS clusters represent groups of nodes which run a common distributed control plane. …","ref":"/docs/concepts/clusters/","title":"Clusters"},{"body":"StorageOS compression is handled on a per volume basis and is disabled by default in v2.2+, as performance is generally increased when compression is disabled due to block alignment. This means that there is a trade off between volume performance and the space the volume occupies on the backend device.\nCompression can be enabled by setting the label storageos.com/nocompress=false on a volume at volume creation time.\nStorageOS utilises the lz4 compression algorithm when writing to the backend store and when compressing replication traffic before it is sent across the network.\nStorageOS detects whether a block can be compressed or not by creating a heuristic that predicts the size of a compressed block. If the heuristic indicates that the compressed block is likely to be larger than the original block then the uncompressed block is stored. Block size increases post compression if the compression dictionary is added to a block that cannot be compressed. By verifying whether blocks can be compressed, disk efficiency is increased and CPU resources are not wasted on attempts to compress uncompressible blocks. StorageOS\u0026rsquo; patented on disk format is used to tell whether individual blocks are compressed without overhead. As such volume compression can be dynamically enabled/disabled even while a volume is in use.\nWhen compression and encryption are both enabled for a volume, blocks are compressed then encrypted.\n","excerpt":"StorageOS compression is handled on a per volume basis and is disabled by default in v2.2+, as …","ref":"/docs/concepts/compression/","title":"Compression"},{"body":"We are always looking to improve our documentation. If you like to help people and can write, read on for the process for submitting your contributions. If your guide is published, you\u0026rsquo;ll receive $250 per article by PayPal.\nContent guidelines A guide contains step by step instructions for how to accomplish a specific task using StorageOS. To be accepted, guides must be:\n Written in English. Relevant, accurate and complete. Technically correct and thoroughly tested. Follow the writing style guide.  Guides should avoid:\n Duplicating an existing guide or other sources, such as blogs or forum posts. Including irrelevant material.  Submission and review You should submit your guide as a pull request to the GitHub repo.\nYour submission will be left open for community review for two weeks. After this, your submission will be reviewed internally for about another week.\nIf accepted, your pull request will be approved and you will have 36 hours to send your submission title and PayPal account information. Non-response will be taken as a go-ahead to publish.\nLegal COPYRIGHT OWNERSHIP. The StorageOS Guides \u0026amp; Tutorials repository is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license.\nCREDIT. Nothing contained in this Agreement shall be deeded to require StorageOS to use the Work, or any part thereof, in connection with StorageOS Guides \u0026amp; Tutorials or otherwise. Credit for the Work shall read, “Contributed by writer’s name.”\nPAYMENT. Upon publication of a submission to the StorageOS Guides \u0026amp; Tutorials Repository, the writer will be paid the sum of USD $250.00 as an electronic payment.\n","excerpt":"We are always looking to improve our documentation. If you like to help people and can write, read …","ref":"/docs/reference/contributing/","title":"Contributing to the docs"},{"body":"$ storageos create --help Create new resources Usage: storageos create [command] Available Commands: namespace Provision a new namespace policy-group Provision a new policy group user Create a new user account volume Provision a new volume Flags: -h, --help help for create Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Use \u0026#34;storageos create [command] --help\u0026#34; for more information about a command. create volume $ storageos create volume --help Provision a new volume Usage: storageos create volume [flags] Examples: $ storageos create volume --description \u0026#34;This volume contains the data for my app\u0026#34; --fs-type \u0026#34;ext4\u0026#34; --labels env=prod,rack=db-1 --size 10GiB --namespace my-namespace-name my-app $ storageos create volume --replicas 1 --namespace my-namespace-name my-replicated-app Flags: --async perform the operation asynchronously, using the configured timeout duration --cache caches volume data (default true) --compress compress data stored by the volume at rest and during transit -d, --description string a human-friendly description to give the volume -f, --fs-type string the filesystem to format the new volume with once provisioned (default \u0026#34;ext4\u0026#34;) -h, --help help for volume -l, --labels strings an optional set of labels to assign to the new volume, provided as a comma-separated list of key=value pairs -r, --replicas uint the number of replicated copies of the volume to maintain -s, --size string the capacity to provision the volume with (default \u0026#34;5GiB\u0026#34;) --throttle deprioritises the volumes traffic by reducing the rate of disk I/O Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) create user $ storageos create user --help Create a new user account Usage: storageos create user [flags] Examples: $ storageos create user --with-username=alice --with-admin=true Flags: -h, --help help for user --with-admin control whether the user is given administrative privileges --with-groups stringArray the list of policy groups to assign to the user --with-password string the password to assign to the user. If not specified, this will be prompted for. --with-username string the username to assign Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) create namespace $ storageos create namespace --help Provision a new namespace Usage: storageos create namespace [flags] Examples: $ storageos create namespace --labels env=prod,rack=db-1 my-namespace-name Flags: -h, --help help for namespace -l, --labels strings an optional set of labels to assign to the new namespace, provided as a comma-separated list of key=value pairs Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) create policy-group $ storageos create policy-group --help Provision a new policy group Usage: storageos create policy-group [flags] Examples: $ storageos create policy-group -r \u0026#39;namespace-name:*:r\u0026#39; -r \u0026#39;namespace-name-2:volume:w\u0026#39; my-policy-group-name $ storageos create policy-group -r \u0026#39;namespace-name:*:r,namespace-name-2:volume:w\u0026#39; my-policy-group-name Flags: -h, --help help for policy-group -r, --rules strings set of rules to assign to the new policy group, provided as a comma-separated list of namespace:resource:rw triples. Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos create --help Create new resources Usage: storageos create [command] Available Commands: …","ref":"/docs/reference/cli/create/","title":"Create"},{"body":"$ storageos delete -h Delete resources in the cluster Usage: storageos delete [command] Available Commands: namespace Delete a namespace node Delete a node policy-group Delete a policy group user Delete a user volume Delete a volume. By default the target volume must be online. If the volume is offline then the request must specify that an offline delete is desired. Flags: -h, --help help for delete Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Use \u0026#34;storageos delete [command] --help\u0026#34; for more information about a command. delete volume $ storageos delete volume --help Delete a volume. By default the target volume must be online. If the volume is offline then the request must specify that an offline delete is desired. Usage: storageos delete volume [volume name] [flags] Examples: $ storageos delete volume my-test-volume my-unneeded-volume $ storageos delete volume --namespace my-namespace my-old-volume Flags: --async perform the operation asynchronously, using the configured timeout duration --cas string make changes to a resource conditional upon matching the provided version -h, --help help for volume --offline-delete request deletion of an offline volume. Volume data is not removed until the node reboots Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) delete user $ storageos delete user --help Delete a user Usage: storageos delete user [user name] [flags] Examples: $ storageos delete user my-unneeded-user $ storageos delete user --use-ids my-user-id Flags: --cas string make changes to a resource conditional upon matching the provided version -h, --help help for user Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) delete policy-group $ storageos delete policy-group --help Delete a policy group Usage: storageos delete policy-group [policy group name] [flags] Examples: $ storageos delete policy-group my-unneeded-policy-group $ storageos delete policy-group --use-ids my-policy-group-id Flags: --cas string make changes to a resource conditional upon matching the provided version -h, --help help for policy-group Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) delete node $ storageos delete node --help Delete a node Usage: storageos delete node [node name] [flags] Examples: $ storagoes delete node my-old-node Flags: --async perform the operation asynchronously, using the configured timeout duration --cas string make changes to a resource conditional upon matching the provided version -h, --help help for node Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) delete namespace $ storageos delete namespace --help Delete a namespace Usage: storageos delete namespace [namespace name] [flags] Examples: $ storageos delete namespace my-unneeded-namespace $ storageos delete namespace --use-ids my-namespace-id Flags: --cas string make changes to a resource conditional upon matching the provided version -h, --help help for namespace Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos delete -h Delete resources in the cluster Usage: storageos delete [command] Available …","ref":"/docs/reference/cli/delete/","title":"Delete"},{"body":"When a Kubernetes object is deleted, StorageOS controllers automatically sync this deletion to StorageOS. For example, when a Kubernets node is deleted, this will automatically be mirrored in StorageOS, likewise with Namespaces.\n  Here is a StorageOS node, running on Kubernetes node worker1. A StorageOS node is any machine that is running the StorageOS daemonset pod. The node is visible below in kubectl.\n$ kubectl describe -n kube-system pod storageos-daemonset-6q4g8 Name: storageos-daemonset-6q4g8 Namespace: kube-system Priority: 2000001000 Priority Class Name: system-node-critical Node: worker1/192.168.152.238 Start Time: Thu, 06 May 2021 15:53:34 +0100 Labels: app=storageos app.kubernetes.io/component=storageos-daemonset app.kubernetes.io/instance=example-storageos app.kubernetes.io/managed-by=storageos-operator app.kubernetes.io/name=storageos app.kubernetes.io/part-of=storageos controller-revision-hash=f5dcf577d kind=daemonset pod-template-generation=1 storageos_cr=example-storageos Annotations: kubectl.kubernetes.io/default-logs-container: storageos Status: Running IP: 192.168.152.238 ... The nodes in your cluster can be seen with storageos get nodes.\n$ storageos get nodes NAME HEALTH AGE LABELS worker1 online 1 day ago beta.kubernetes.io/arch=amd64, beta.kubernetes.io/os=linux, cattle.io/creator=norman, kubernetes.io/arch=amd64, kubernetes.io/hostname=worker1, kubernetes.io/os=linux, node-role.kubernetes.io/worker=true storageos.com/computeonly=true worker2 online 1 day ago beta.kubernetes.io/arch=amd64, beta.kubernetes.io/os=linux, cattle.io/creator=norman, kubernetes.io/arch=amd64, kubernetes.io/hostname=worker2, kubernetes.io/os=linux, node-role.kubernetes.io/worker=true worker3 online 1 day ago beta.kubernetes.io/arch=amd64, beta.kubernetes.io/os=linux, cattle.io/creator=norman, kubernetes.io/arch=amd64, kubernetes.io/hostname=worker3, kubernetes.io/os=linux, node-role.kubernetes.io/worker=true ...   Delete the node.\n$ kubectl delete node worker1   Verify that the node has been deleted with kubectl get nodes or storageos get nodes. The node has now disappeared from StorageOS.\n$ storageos get nodes NAME HEALTH AGE LABELS worker2 online 1 day ago beta.kubernetes.io/arch=amd64, beta.kubernetes.io/os=linux, cattle.io/creator=norman, kubernetes.io/arch=amd64, kubernetes.io/hostname=worker2, kubernetes.io/os=linux, node-role.kubernetes.io/worker=true worker3 online 1 day ago beta.kubernetes.io/arch=amd64, beta.kubernetes.io/os=linux, cattle.io/creator=norman, kubernetes.io/arch=amd64, kubernetes.io/hostname=worker3, kubernetes.io/os=linux, node-role.kubernetes.io/worker=true ...   ","excerpt":"When a Kubernetes object is deleted, StorageOS controllers automatically sync this deletion to …","ref":"/docs/operations/delete-stos-objects/","title":"Deleting StorageOS Objects"},{"body":"$ storageos describe --help Fetch extended details for resources Usage: storageos describe [command] Available Commands: cluster Retrieve detailed information for the current cluster licence Fetch current licence configuration details namespace Retrieve detailed information for one or many namespaces node Retrieve detailed information for nodes in the cluster policy-group Show detailed information for policy groups user Show detailed information for users volume Show detailed information for volumes Flags: -h, --help help for describe Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Use \u0026#34;storageos describe [command] --help\u0026#34; for more information about a command. describe cluster $ storageos describe cluster --help Retrieve detailed information for the current cluster Usage: storageos describe cluster [flags] Aliases: cluster, nodes Examples: $ storageos describe cluster Flags: -h, --help help for cluster Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) describe license $ storageos describe license --help Fetch current licence configuration details Usage: storageos describe licence [flags] Aliases: licence, license Examples: $ storageos describe licence Flags: -h, --help help for licence Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) describe namespace $ storageos describe namespace --help Retrieve detailed information for one or many namespaces Usage: storageos describe namespace [flags] Aliases: namespace, namespaces Examples: $ storageos describe namespace my-namespace-name $ storageos describe namespace --use-ids my-namespace-id $ storageos describe namespaces Flags: -h, --help help for namespace Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) describe node $ storageos describe node --help Retrieve detailed information for nodes in the cluster Usage: storageos describe node [node names...] [flags] Aliases: node, nodes Examples: $ storageos describe node my-node-name Flags: -h, --help help for node -l, --selector stringArray filter returned results by a set of comma-separated label selectors Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) describe policy-group $ storageos describe user --help Show detailed information for policy groups Usage: storageos describe policy-group [policy group names...] [flags] Aliases: policy-group, policy-groups Examples: $ storageos describe policy-groups $ storageos describe policy-group my-policy-group-name Flags: -h, --help help for policy-group Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) describe user $ storageos describe user --help Show detailed information for users Usage: storageos describe user [user names...] [flags] Aliases: user, users Examples: $ storageos describe users $ storageos describe user my-username $ storageos describe user my-username-1 my-username-2 $ storageos describe user --use-ids my-userid $ storageos describe user --use-ids my-userid-1 my-userid-2 Flags: -h, --help help for user Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) describe volume $ storageos describe volume --help Show detailed information for volumes Usage: storageos describe volume [volume names...] [flags] Aliases: volume, volumes Examples: $ storageos describe volumes $ storageos describe volume --namespace my-namespace-name my-volume-name Flags: -A, --all-namespaces retrieves volumes from all accessible namespaces. This option overrides the namespace configuration -h, --help help for volume -l, --selector stringArray filter returned results by a set of comma-separated label selectors Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos describe --help Fetch extended details for resources Usage: storageos describe [command] …","ref":"/docs/reference/cli/describe/","title":"Describe"},{"body":"$ storageos detach --help Detach a volume from its current location Usage: storageos detach [flags] Examples: $ storageos detach --namespace my-namespace-name my-volume Flags: --async perform the operation asynchronously, using the configured timeout duration --cas string make changes to a resource conditional upon matching the provided version -h, --help help for detach Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos detach --help Detach a volume from its current location Usage: storageos detach [flags] …","ref":"/docs/reference/cli/detach/","title":"Detach Volume"},{"body":"StorageOS can generate a cluster diagnostic bundle from the GUI or CLI under user request. The bundle packages the information needed for the engineering team to understand the context of your cluster and begin troubleshooting any issues you may be experiencing.\nGenerate Bundle CLI The StorageOS CLI can generate the bundle.\n$ storageos get diagnostics  The file generated is in the form of diagnostics-${TIME_STAMP}.gz\n GUI Or you can use the StoregeOS GUI.\n Go to section \u0026ldquo;Cluster\u0026rdquo; Press the button \u0026ldquo;DOWNLOAD DIAGNOSTICS\u0026rdquo;.  Data collected in the bundle Most of the data collected in the bundle is regarding the state of the StorageOS cluster, however some other information regarding the infrastructure is also gathered. The information is used to have a clear view of the cluster where StorageOS is running.\nThe bundle incorporates for each node:\n StorageOS Daemonset Pod logs lshw dmesg (kernel logs) StorageOS metadata for the ControlPlane and DataPlane  StorageOS metadata collected  cluster metadata namespaces metadata nodes metadata volumes metadata capacity stats environment variables health  Privacy StorageOS can only obtain the bundle if it is downloaded by the user and given to our engineering team, or uploaded for analysis. The data received by StorageOS is private and never leaves nor will leave StorageOS Inc.\nThe data contained in the cluster diagnostic bundle has the sole purpose of helping customers troubleshoot their issues.\n","excerpt":"StorageOS can generate a cluster diagnostic bundle from the GUI or CLI under user request. The …","ref":"/docs/reference/bundles/diagnostic-bundle/","title":"Diagnostic bundle"},{"body":"To install StorageOS on DockerEE, please follow our Kubernetes installation instructions page.\nDocker EE and the Universal Control Plane can run on different Linux distributions. StorageOS supports RHEL, CentOS, Debian, and selected Ubuntu images. For more details, check out the supported OSs in the prerequisites page.\nStorageOS only supports Kubernetes nodes managed by Docker Enterprise Edition, not those nodes running Swarm. Mixed nodes (those running Kubernetes and Swarm workloads) are not supported. As a consequence, StorageOS volumes can only be provisioned on Kubernetes nodes, and only these nodes should be used for stateful workloads.\n","excerpt":"To install StorageOS on DockerEE, please follow our Kubernetes installation instructions page. …","ref":"/docs/platforms/dockeree/","title":"Docker Enterprise Edition"},{"body":"","excerpt":"","ref":"/docs/","title":"Documentation"},{"body":"Elasticsearch is a distributed, RESTful search and analytics engine, most popularly used to aggregate logs, but also to serve as a search backend to a number of different applications.\nUsing StorageOS persistent volumes with ElasticSearch (ES) means that if a pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up, the pod data is immediately available. Should Kubernetes schedule the Elasticsearch pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not the original StorageOS master volume is located on the same node.\nElasticsearch has features to allow it to handle data replication, and as such careful consideration of whether to allow StorageOS or Elasticsearch to handle replication is required.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Elasticsearch on Kubernetes Prerequisites Some OS tuning is required, which is done automatically when using our example from the use cases repository.\nElasticsearch requires vm.max_map_count to be increased to a minimum of 262144, which is a system wide setting. One way to achieve this is to run sysctl -w vm.max_map_count=262144 and update /etc/sysctl.conf to ensure it persists over a reboot. See ElasicSearch reference here.\nAdministrators should be aware that this impacts the behaviour of nodes and that there may be collisions with other application settings. Administrators are advised to centrally collate sysctl settings using the tooling of their choice.\nDeployment of the application StatefulSet defintion ---apiVersion:apps/v1kind:StatefulSetmetadata:name:esdata[...]spec:serviceAccountName:elasticsearchcontainers:- name:dataimage:elasticsearch:6.7.0imagePullPolicy:IfNotPresent[...]volumeMounts:- name:datamountPath:/usr/share/elasticsearch/data/data[...]volumeClaimTemplates:- metadata:name:\u0026#34;data\u0026#34;spec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# \u0026lt;--- default StorageOS storage class nameresources:requests:storage:10Gi# \u0026lt;--- change this to the appropriate valueThis excerpt is from the StatefulSet definition (/elasticsearch/10-es-data.yaml). The file contains the PersistentVolumeClaim template that will dynamically provision the necessary storage, using the StorageOS storage class.\nDynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.\nInstallation Clone the use cases repo You can find the latest files in the StorageOS use cases repostiory in /elasticsearch/\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases   Create the kubernetes objects\nThis will install an ES cluster with 3 master, 3 data and 3 coordinator nodes. Combined they will require ~ 14 GiB of available memory in your cluster, however, more may be used as the application is being used\nkubectl apply -f ./elasticsearch/ Once completed, an internal service object will have been created making the cluster available as http://elasticsearch:9200/ which is the default Kibana (when installed via Helm) will be using.\n  Confirm Elasticsearch is up and running\nkubectl get pods -l component=elasticsearch NAME READY STATUS RESTARTS AGE elasticsearch-exporter-d86ffd94-zw45l 1/1 Running 0 5m44s es-coordinator-b7b984dd4-7wlz5 1/1 Running 0 5m44s es-coordinator-b7b984dd4-89w26 1/1 Running 0 5m44s es-coordinator-b7b984dd4-b4t6j 1/1 Running 0 5m44s es-master-78dfd5b49f-9gf5c 1/1 Running 0 5m44s es-master-78dfd5b49f-smsbw 1/1 Running 0 5m44s es-master-78dfd5b49f-z4qpj 1/1 Running 0 5m44s esdata-0 1/1 Running 0 5m44s esdata-1 1/1 Running 0 4m34s esdata-2 1/1 Running 0 3m22s   Connect to ElasticSearch\nTo connect to ES directly, you can use the following port-forward command\nkubectl port-forward svc/elasticsearch 9200 and then access it via http://localhost:9200\n  Kibana (optional) One of the most popular uses of ES is to use it for log aggregation and indexing, Kibana helps us visualize the data in these indices and can be easily used when installed via its Helm chart\n  Install the helm chart.\nhelm install stable/kibana   Once installed, use a port-foward to Kibana instead of directly to ES\nkubectl port-forward --namespace default $(kubectl get pods --namespace default -l \u0026#34;app=kibana\u0026#34; -o jsonpath=\u0026#34;{.items[0].metadata.name}\u0026#34;) 5601 and then access it via http://localhost:5601\n  Monitoring (optional) As part of the example deployment, ES metrics are exposed and can be scraped by Prometheus on port 9108 (see 77-es-exporter.yaml). This is enabled by default, and should work with the default Prometheus install via Helm. If you\u0026rsquo;re using the Prometheus service monitors, you can monitor this installation by creating a monitor for the es-exporter service. For an example of how this is done to monitor StorageOS, please see prometheus-setup.\n","excerpt":"Elasticsearch is a distributed, RESTful search and analytics engine, most popularly used to …","ref":"/docs/usecases/elasticsearch/","title":"Elasticsearch"},{"body":"For more detail on encryption at rest in StorageOS, please see the reference page.\nEnabling encryption on a volume Encrypting a volume is done by simply creating a volume with the storageos.com/encryption=true label set on the volume\u0026rsquo;s PVC. This label is not applicable to be set as a StorageClass parameter\nThis label is all that is needed. If it is present, the mutating admission webhook that runs as part of the StorageOS API Manager will create the encryption key, link it to the PVC and store it in a secret.\nEncryption is enabled when a volume is provisioned, and it can not be removed during during the volume\u0026rsquo;s lifetime.\nAn example encrypted volume   Add the label in the PVC definition, for instance:\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:encrypted-vollabels:storageos.com/encryption:\u0026#34;true\u0026#34;# Label \u0026lt;-----spec:storageClassName:\u0026#34;fast\u0026#34;accessModes:- ReadWriteOnceresources:requests:storage:1G  ","excerpt":"For more detail on encryption at rest in StorageOS, please see the reference page.\nEnabling …","ref":"/docs/operations/encryption/","title":"Encryption"},{"body":"StorageOS supports encryption for data-at-rest and data-in-transit.\nData-in-transit is data as it is travelling between nodes. It is encrypted by default with mTLS. Data-at-rest is the data stored in your volumes as blob files. Encryption of these blob files is optional and can be enabled by adding a label to your volume definitions before they\u0026rsquo;re provisioned.\nFor information on how to enable encryption on your volumes, please see our Encryption Operations page.\nHow volumes are encrypted Volumes are encrypted using AES-256 in the XTS-AES mode with 512-bit keys, as specified by IEEE Standard 1619-2007. There is a non-zero performance impact of using encrypted volumes. A 10-25% cost in read/write throughput can be expected from XTS-AES, dependent on workload. Thin provisioning still applies to encrypted volumes.\nEncryption Key Generation On PVC creation, if encryption is enabled, StorageOS will automatically generate up to two keys as Kubernetes secrets. Both keys are stored in the same namespace as the PVC.\nFirstly, if it doesn\u0026rsquo;t already exist, a namespace key is generated. It is always named storageos-namespace-key and only one exists per namespace.\nSecondly a volume key is created for each encrypted volume. It has a name in the format storageos-volume-key-\u0026lt;random-id\u0026gt;, with no connection to the name of the volume. The volume it is associated with can be determined by looking at the storageos.com/pvc label on the secret. The storageos.com/encryption-secret-name and storageos.com/encryption-secret-namespace annotations are added to the PVC by an admission controller to map the PVC back to its secret.\nThe encryption key is passed to StorageOS as part of the CSI volume creation request and is used to encrypt the volume.\nEncryption Key Use The volume specific secret is needed whenever a volume is attached to a node for use by a pod. When this happens, the StorageOS node container\u0026rsquo;s Service Account reads the secret and passes it to the StorageOS controlplane.\nA volume missing its key or with a malformed key will be unable to attach.\nThe key is stored in memory by StorageOS only on the node that the volume is being used on. As a result, encryption and decryption are performed where the data is consumed, rather than where it is stored.\nBecause of this, the use of encrypted volumes is transparent to the user. There is a complete integration between Kubernetes applications and StorageOS encryption.\nKey Management Best Practices StorageOS does not store encryption keys outside of Kubernetes secrets. Backups are therefore imperative in case the Kubernetes etcd is lost. StorageOS has no ability to decrypt a volume whose encryption keys have been lost.\nSecrets in Kubernetes are not encrypted by default, they are stored in the Kubernetes etcd in simple Base64 encoding. As StorageOS encryption keys are stored as Kubernetes Secrets, this means that anyone with access to a Kubernetes etcd installation can read encryption keys and decrypt encrypted volumes, unless the cluster has an external secrets store.\nFor better security secret encryption or an access control system such as Vault should be in operation on production-grade or sensitive clusters.\nSecrets are not garbage-collected by StorageOS. To clean up completely upon deletion of a volume it is necessary to also delete that volume\u0026rsquo;s secret. There is no benefit to doing this, however.\n","excerpt":"StorageOS supports encryption for data-at-rest and data-in-transit.\nData-in-transit is data as it is …","ref":"/docs/reference/encryption/","title":"Encryption"},{"body":"Etcd is an open-source distributed, strongly consistent key value store that is used by StorageOS to durably persist the StorageOS cluster state. As the backing store for Kubernetes, StorageOS uses etcd for many of the same reasons.\nStorageOS uses etcd as the single source of truth for all StorageOS objects. Whenever a request is made to create, update or delete an object the result is written to etcd before the request is completed. Using etcd as a configuration store allows nodes to retrieve the current cluster state after being offlined, allowing offlined nodes to rejoin the cluster.\n N.B. StorageOS v2.0 does not provide an embedded etcd server as previous versions did. You will need to setup an etcd server for StorageOS to use prior to installation of StorageOS. Please see our etcd prerequisites page for more information on how to install and configure etcd.\n ","excerpt":"Etcd is an open-source distributed, strongly consistent key value store that is used by StorageOS to …","ref":"/docs/concepts/etcd/","title":"Etcd"},{"body":"Check the etcd prerequisites page for a step by step installation of etcd.\nBest practices StorageOS uses etcd as a service, whether it is deployed following the step by step instructions or as a custom installation. It is expected that the user maintains the availability and integrity of the etcd cluster.\nIt is highly recommended to keep the cluster backed up and ensure high availability of its data.\nNetwork low latency It is important to keep the latency between StorageOS nodes and the etcd replicas low. Deploying an etcd cluster in a different data center or region can make StorageOS detect etcd nodes as unavailable due to latency. 10ms latency between StorageOS and etcd would be the maximum threshold for proper functioning of the system.\nDisk low latency Etcd is very sensitive to disk latency. Because of that, it is recommended to run etcd away from other IO-intensive workloads. Operations such as backups, builds or application bundling cause a heavy usage of disks. If such operations run alongside the etcd nodes, they will cause etcd to become unstable and suffer downtime. It is best to run etcd nodes isolated from other IO workloads.\nIOPS requirements As a general rule, for etcd to operate normally on production clusters, we recommend using the size of machine offered by your cloud provider that guarantees a minimum of 500 IOPS. For example, 750 baseline IOPS are guaranteed on a 250GB AWS gp2 EBS instance at time of writing, and block instances on other cloud providers will also specify baseline IOPS figures.\nCloud providers usually provide \u0026ldquo;bursts\u0026rdquo; of IOPS - temporarily higher rates, limited by credits - with larger volumes providing higher burst capacity. If you are relying on burst capacity for etcd, which requires sustained high performance, careful assessment is necessary to ensure sufficient capacity.\nThe rate of etcd operations is affected by the number of nodes, volumes and replicas in the cluster, therefore the figure of 500 is provided as a guideline only. A development cluster with 5 nodes will not have the same etcd traffic as a production cluster with 100 nodes. Adding monitoring to etcd will help to characterise the traffic, and therefore to assess the individual requirements of a cluster and adjust its resources accordingly.\nEtcd advertise urls The etcd startup parameters advertise-client-urls and initial-advertise-peer-urls specify the addresses etcd clients or other etcd members should use to contact the etcd server. The advertised addresses must be reachable from the remote machines - i.e. where StorageOS is running - so it can connect successfully. Do not advertise addresses like localhost or 0.0.0.0 for a production setup since these addresses are unreachable from remote machines.\nMonitoring It is highly recommended to add monitoring to the etcd cluster. Etcd serves Prometheus metrics on the client port http://etcd-url:2379/metrics.\nYou can use StorageOS developed Grafana Dashboards for etcd. When using etcd for production, you can use the etcd-cluster-as-service, while the etcd-cluster-as-pod can be used when using etcd from the operator.\nDefragmentation Etcd uses revisions to store multiple versions of keys. Compaction removes all key revision prior to a certain revision from etcd. Typically the etcd configuration enables the automatic compaction of keys to prevent performance degradation and limit the storage required. Compaction of revisions can create fragmentation that means space on disk is available for use by etcd but is unavailable for use by the file system. In order to reclaim this space, etcd can be defragmented.\nReclaiming space is important because when the etcd database file grows over the \u0026ldquo;DB_BACKEND_BYTES\u0026rdquo; parameter, the cluster triggers an alarm and sets itself read only and only allows reads and deletes. To avoid hitting the db backend bytes limit, compaction and defragmentation are required. How often defragmentation is required depends on the churn of key revisions in etcd.\nThe Grafana Dashboards mentioned above indicate when nodes require defragmentation. Be aware that defragmentation is a blocking operation that is performed per node, hence the etcd node will be locked for the duration of the defragmentation. Defragmentation usually takes a few milliseconds to complete.\nYou can also set cronjobs that execute the following defragmentation script. It will run a defrag when the DB is at 80% full. A defragmentation operation has to be executed per etcd node and it is a blocking operation. It is recommended to not execute the defragmentation on all etcd members at the same time. If using a cronjob, set them up for different times.\nhttps://raw.githubusercontent.com/storageos/deploy/master/k8s/deploy-storageos/etcd-helpers/etcd-ansible-systemd/roles/install_etcd/templates/defrag-etcd.sh.j2 chmod +x defrag-etcd.sh Known CoreOS Etcd Operator issues This topology is only recommended for deployments where isolated nodes cannot be used.\nEtcd is a distributed key-value store database focused on strong consistency. That means that etcd nodes perform operations across the cluster to ensure quorum. If quorum is lost, etcd nodes stop and etcd marks its contents as read-only. This is because it cannot guarantee that new data will be valid. Quorum is fundamental for etcd operations. When running etcd in pods it is therefore important to consider that a loss of quorum could arise from etcd pods being evicted from nodes.\nOperations such as Kubernetes Upgrades with rolling node pools could cause a total failure of the etcd cluster as nodes are discarded in favor of new ones.\nA 3 etcd node cluster can survive losing one node and recover, a 5 node cluster can survive the loss of two nodes. Loss of further nodes will result in quorum being lost.\nThe etcd-operator doesn\u0026rsquo;t support a full stop of the cluster. Stopping the etcd cluster causes the loss of all the etcd keystore and make StorageOS unable to perform metadata changes.\nThe official etcd-operator repository also has a backup deployment operator that can help backup etcd data. A restore of the etcd keyspace from a backup might cause issues due to the disparity between the cluster state and its metadata in a different point in time. If you need to restore from a backup after a failure of etcd, contact the StorageOS support team.\n","excerpt":"Check the etcd prerequisites page for a step by step installation of etcd.\nBest practices StorageOS …","ref":"/docs/operations/etcd/","title":"Etcd"},{"body":"This procedure explains how to add a new etcd member for your StorageOS etcd cluster while removing one of the current members. This is useful when the nodes hosting etcd must be recycled.\n It is assumed that the StorageOS etcd cluster is installed following the production etcd installation page, where etcd nodes are installed on their own machines.\n  It is also assumed that etcd members are referenced from Kubernetes using a External Service. Example available in the etcd external Service example. This service should be referred to in the spec.kvbackend.address section of your StorageOS CustomResource. If that Service is not used, a full restart of the StorageOS cluster will be required. The StorageOS CustomResource would need to be removed, and amended to reflect the new etcd urls created.\n Preparation Prepare the installation of etcd on a new node, making sure that etcd is not starting on that new node.\nThe steps for preparing an etcd node can be found in the etcd prerequisites page.\n  Back up etcd\n$ export ETCDCTL_API=3 $ # Set all your endpoints $ export endpoints=\u0026quot;192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379\u0026quot; $ etcdctl --endpoints $endpoints snapshot save /var/tmp/etcd-snapshot.db   Verify etcd health\n$ export ETCDCTL_API=3 $ # Set all your endpoints $ export endpoints=\u0026quot;192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379\u0026quot; $ etcdctl member list --endpoints $endpoints -wtable +------------------+---------+-----------------------+------------------------------+-----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+-----------------------+------------------------------+-----------------------------+------------+ | 7817aa073b059aab | started | etcd-192.168.195.168 | http://192.168.195.168:2380 | http://192.168.195.168:2379 | false | | e22cdd20a03e5e73 | started | etcd-192.168.202.40 | http://192.168.202.40:2380 | http://192.168.202.40:2379 | false | | e5d0f0e242014d3d | started | etcd-192.168.174.117 | http://192.168.174.117:2380 | http://192.168.174.117:2379 | false | +------------------+---------+-----------------------+------------------------------+-----------------------------+------------+ $ etcdctl endpoint health --endpoints $endpoints -wtable +---------------------+--------+------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +---------------------+--------+------------+-------+ |192.168.174.117:2379 | true | 5.048177ms | | |192.168.195.168:2379 | true | 5.926681ms | | |192.168.202.40:2379 | true | 5.526928ms | | +---------------------+--------+------------+-------+ $ etcdctl endpoint status --endpoints $endpoints -wtable +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ |192.168.174.117:2379 | e5d0f0e242014d3d | 3.4.9 | 311 kB | false | false | 2 | 4281 | 4281 | | |192.168.195.168:2379 | 7817aa073b059aab | 3.4.9 | 315 kB | false | false | 2 | 4281 | 4281 | | |192.168.202.40:2379 | e22cdd20a03e5e73 | 3.4.9 | 352 kB | true | false | 2 | 4281 | 4281 | | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+   Migration  In the following procedure NODE4 is a new member to add to the cluster, while NODE1 ought to be removed.\n   Amend etcd configuration to reference the new node (NODE4)\n Make the following change on all running etcd members and NODE4\n $ # NODE4_IP is the NEW_NODE_ADDRESS $ echo \u0026quot;NODE4_IP=192.168.152.142\u0026quot; \u0026gt;\u0026gt; /etc/etcd.conf $ # Check the contents of /etc/etcd.conf $ cat /etc/etcd.conf CLIENT_PORT=2379 PEERS_PORT=2380 # NODE_IP is the IP of the node where this file resides. NODE_IP=192.168.202.40 NODE1_IP=192.168.195.168 NODE2_IP=192.168.202.40 NODE3_IP=192.168.174.117 NODE4_IP=192.168.152.142   Amend SystemD service file on the new etcd node (NODE4)\n The SystemD service file is expected in /etc/systemd/system/etcd3.service\n Change the --initial-cluster-state to existing and add the reference to NODE4 in the --initial-cluster variable.\nvi /etc/systemd/system/etcd3.service The resulting changes would appear as follows:\n... ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \\ ... --initial-cluster-state existing \\ ... --initial-cluster \\ etcd-${NODE1_IP}=http://${NODE1_IP}:${PEERS_PORT},\\ etcd-${NODE2_IP}=http://${NODE2_IP}:${PEERS_PORT},\\ etcd-${NODE3_IP}=http://${NODE3_IP}:${PEERS_PORT},\\ etcd-${NODE4_IP}=http://${NODE4_IP}:${PEERS_PORT} ...  Note the reference to NODE4 at the end of the --initial-cluster variable\n Make sure etcd is not started on the new member NODE4\n  Add etcd member as a learner\n# Set environment variable for the the new etcd member (NODE4) $ NODE4_IP=192.168.152.142 $ ETCD_NEW_MEMBER=\u0026#34;etcd-${NODE4_IP}\u0026#34; $ ETCD_NEW_MEMBER_PEER=\u0026#34;http://$NODE4_IP:2380\u0026#34; # Add the new member to the cluster $ export ETCDCTL_API=3 $ etcdctl member add \\  --learner $ETCD_NEW_MEMBER \\  --peer-urls=\u0026#34;$ETCD_NEW_MEMBER_PEER\u0026#34; Member 52e5c9ac117b3df2 added to cluster b4f4ed717ea44b8d ETCD_NAME=\u0026#34;etcd-192.168.152.142\u0026#34; ETCD_INITIAL_CLUSTER=\u0026#34;etcd-192.168.152.142=http://192.168.152.142:2380,etcd-192.168.195.168=http://192.168.195.168:2380,etcd-192.168.202.40=http://192.168.202.40:2380,etcd-192.168.174.117=http://192.168.174.117:2380\u0026#34; ETCD_INITIAL_ADVERTISE_PEER_URLS=\u0026#34;http://192.168.152.142:2380\u0026#34; ETCD_INITIAL_CLUSTER_STATE=\u0026#34;existing\u0026#34;   Check the etcd members\n$ export endpoints=\u0026quot;192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379\u0026quot; $ etcdctl member list --endpoints $endpoints -wtable +------------------+-----------+----------------------+-----------------------------+-----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+-----------+----------------------+-----------------------------+-----------------------------+------------+ | 52e5c9ac117b3df2 | unstarted | | http://192.168.152.142:2380 | | true | | 7817aa073b059aab | started | etcd-192.168.195.168 | http://192.168.195.168:2380 | http://192.168.195.168:2379 | false | | e22cdd20a03e5e73 | started | etcd-192.168.202.40 | http://192.168.202.40:2380 | http://192.168.202.40:2379 | false | | e5d0f0e242014d3d | started | etcd-192.168.174.117 | http://192.168.174.117:2380 | http://192.168.174.117:2379 | false | +------------------+-----------+----------------------+-----------------------------+-----------------------------+------------+  Note that the learner is not started yet\n   Start etcd on the new node (NODE4)\n Make sure that /etc/systemd/system/etcd.service only have currently active nodes specified in the --initial-cluster flag.\n # On the new node (NODE4) systemctl daemon-reload systemctl enable etcd3.service systemctl start etcd3.service   Check the etcd members\n$ export endpoints=\u0026quot;192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379\u0026quot; $ etcdctl member list --endpoints $endpoints -wtable +------------------+---------+----------------------+-----------------------------+-----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+----------------------+-----------------------------+-----------------------------+------------+ | 52e5c9ac117b3df2 | started | etcd-192.168.152.142 | http://192.168.152.142:2380 | http://192.168.152.142:2379 | true | | 7817aa073b059aab | started | etcd-192.168.195.168 | http://192.168.195.168:2380 | http://192.168.195.168:2379 | false | | e22cdd20a03e5e73 | started | etcd-192.168.202.40 | http://192.168.202.40:2380 | http://192.168.202.40:2379 | false | | e5d0f0e242014d3d | started | etcd-192.168.174.117 | http://192.168.174.117:2380 | http://192.168.174.117:2379 | false | +------------------+---------+----------------------+-----------------------------+-----------------------------+------------+  Note that the learner is started\n   Check that the new learner has the same revision applied as the current members\n$ export ETCDCTL_API=3 # Added NODE4 in the endpoints variable $ export endpoints=\u0026quot;192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379,192.168.152.142:2379\u0026quot; etcdctl endpoint status --endpoints $endpoints -wtable +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | 192.168.174.117:2379| e5d0f0e242014d3d | 3.4.9 | 352 kB | false | false | 2 | 24570 | 24570 | | | 192.168.195.168:2379| 7817aa073b059aab | 3.4.9 | 352 kB | false | false | 2 | 24570 | 24570 | | | 192.168.202.40:2379 | e22cdd20a03e5e73 | 3.4.9 | 352 kB | true | false | 2 | 24570 | 24570 | | | 192.168.152.142:2379| 52e5c9ac117b3df2 | 3.4.9 | 467 kB | false | true | 2 | 24570 | 24570 | | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+  Wait until the cluster has the learner ready, by ensuring that the RAFT TERM and RAFT INDEX of the learner node match the rest of the cluster.\n   Remove the node that needs to be evicted (NODE1)\n Before promoting the learner to a full member, it is best to remove the node from the cluster that initially was selected to be decommissioned to avoid breaking quorum while having 4 nodes being part of the cluster. For more details, check the official etcd documentation regarding this topic.\n $ export ETCDCTL_API=3 $ # Select member of id of the node to remove (NODE1) $ NODE1_MEMBER_ID=e22cdd20a03e5e73 $ etcdctl member remove $NODE1_MEMBER_ID Member e22cdd20a03e5e73 removed from cluster b4f4ed717ea44b8d   Promote the learner to a member\n$ export ETCDCTL_API=3 $ # Select member of id of the node to remove (NODE1) $ NODE4_MEMBER_ID=52e5c9ac117b3df2 $ etcdctl member promote $NODE4_MEMBER_ID Member 52e5c9ac117b3df2 promoted in cluster b4f4ed717ea44b8d  The promotion will fail if the learner is not in sync with the leader member.\n   Check the etcd health\n$ export endpoints=192.168.174.117:2379,192.168.195.168:2379,192.168.152.142:2379 $ etcdctl member list --endpoints $endpoints -wtable +------------------+---------+---------------------+----------------------------+----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+---------------------+----------------------------+----------------------------+------------+ | 52e5c9ac117b3df2 | started |etcd-192.168.152.142 |http://192.168.152.142:2380 |http://192.168.152.142:2379 | false | | 7817aa073b059aab | started |etcd-192.168.195.168 |http://192.168.195.168:2380 |http://192.168.195.168:2379 | false | | e5d0f0e242014d3d | started |etcd-192.168.174.117 |http://192.168.174.117:2380 |http://192.168.174.117:2379 | false | +------------------+---------+---------------------+----------------------------+----------------------------+------------+ $ etcdctl endpoint status --endpoints $endpoints -wtable +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ |192.168.174.117:2379 | e5d0f0e242014d3d | 3.4.9 | 352 kB | false | false | 3 | 35939 | 35939 | | |192.168.195.168:2379 | 7817aa073b059aab | 3.4.9 | 352 kB | true | false | 3 | 35939 | 35939 | | |192.168.152.142:2379 | 52e5c9ac117b3df2 | 3.4.9 | 467 kB | false | false | 3 | 35939 | 35939 | | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+  Note that NODE4 is now a full quorum member, while NODE1 is no longer part of the cluster\n   Edit Endpoints referencing the Kubernetes Service\n Remove the reference to NODE1 and add the IP for NODE4\n $ kubectl edit -n storageos-etcd endpoints/storageos-etcd   Make amendments in the SystemD configuration files removing any reference to NODE1\n It is not required to restart the etcd service, but to keep the service file up to date.\n   ","excerpt":"This procedure explains how to add a new etcd member for your StorageOS etcd cluster while removing …","ref":"/docs/operations/etcd/migrate-etcd-cluster/","title":"Etcd node migration"},{"body":"For more information about replication and failure modes please see our Replication concepts page.\nThe failure mode for a specific volume can be set using a label on a PVC or it can be set as a parameter on a StorageClass. The PVC definition takes precedence over the StorageClass.\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:my-vol-1labels:storageos.com/replicas:\u0026#34;2\u0026#34;storageos.com/failure-mode:\u0026#34;soft\u0026#34;spec:storageClassName:\u0026#34;fast\u0026#34;accessModes:- ReadWriteOnceresources:requests:storage:5GiFailure Modes StorageOS failure modes offer different guarantees with regards to a volume\u0026rsquo;s mode of operation in the face of replica failure. If the failure mode is not specified it defaults to Hard. Volume failure modes can be dynamically updated at run time.\nHard Hard failure mode requires that the number of declared replicas matches the available number of replicas at all times. If a replica fails StorageOS will attempt creation of a new replica for 90 seconds. After 90s if the old replica is not available and a new replica cannot be provisioned, StorageOS cannot guarantee that the data is stored on the number of multiple nodes requested by the user. StorageOS will therefore set the volume to be read-only.\nIf a volume has gone read-only there are two stages to making it read-write again. Firstly, sufficient replicas must be provisioned to match the desired replica count. Depending on your environment, additional nodes and/or disk capacity may be required for this. Secondly, the volume must be remounted - necessitating pod deletion/recreation in Kubernetes.\nstorageos.com/failure-mode: hard Number of nodes required for hard failure mode\nWhen a node fails, a new replica is provisioned and synced as described above. To ensure that a new replica can always be created, an additional node should be available. To guarantee high availability using storageos.com/failure-mode: hard, clusters using volumes with 1 replica must have at least 3 storage nodes. When using volumes with 2 replicas, at least 4 storage nodes, 3 replicas, 5 nodes, etc.\nMinimum number of storage nodes = 1 (primary) + N (replicas) + 1\nSoft Soft failure mode allows a volume to continue serving I/O even when a replica goes offline and a new replica fails to provision. So long as there are not less than max(1, n-1) available replicas where n is the number of replicas for the volume.\nFor example, if a volume with 2 replicas loses 1 replica, then I/O would continue to be served since 1 replica remaining \u0026gt;= max(1, 1). If a volume with 1 replica loses 1 replica, then I/O would halt after 90 seconds since 0 replicas remaining \u0026lt; max(1, 0).\nstorageos.com/failure-mode: soft Number of nodes required for soft failure mode\nTo ensure that a storageos.com/failure-mode: soft volume is highly available, clusters using volumes with 1 replica must have at least 2 storage nodes. When using volumes with 2 replicas, at least 3 storage nodes, 3 replicas, 3 nodes, etc.\nMinimum number of storage nodes = 1 (primary) + N (replicas)\nThreshold Threshold failure mode allows the user to set the minimum required number of online replicas for a volume. For example for a volume with 2 replicas, setting the threshold to 1 would allow a single replica to be offline, whereas setting threshold to 0 would allow 2 replicas to be offline.\nstorageos.com/failure-mode: (0-5) Number of nodes required for threshold failure mode\nThe minimum number of nodes for a threshold volume is determined by the threshold that is set.\nMinimum number of storage nodes = 1 (primary) + T (threshold)\nAlwaysOn AlwaysOn failure mode allows all replicas for a volume to be offline and keeps the volume writeable. A volume with failure mode AlwaysOn will continue to serve I/O regardless of how many replicas it currently has. This mode should be used with caution as it effectively allows for only a single copy of the data to be available.\nstorageos.com/failure-mode: alwayson Number of nodes required for AlwaysOn failure mode\nA storageos.com/failure-mode: alwayson volume is highly available albeit at the cost of reliability. The minimum node count here is 1 as the loss of all replicas will be tolerated.\nMinimum number of storage nodes = 1 (primary)\n","excerpt":"For more information about replication and failure modes please see our Replication concepts page. …","ref":"/docs/operations/failure-modes/","title":"Failure Modes"},{"body":"StatefulSet behaviour In order to understand what StorageOS Fencing for Kubernetes is and when it is needed, it is required to first understand the behaviour of StatefulSets.\nStatefulSets are the de facto Kubernetes controller to use for stateful applications. The StatefulSet controller offers guarantees around pod uniqueness, sticky identities and the persistence of PVCs beyond the lifetime of their pods. As such, StatefulSets have different characteristics and provide different guarantees than Deployments.\nDeployments guarantee the amount of healthy replicas by reconciling towards the deployment desired state. Attempts to align the number of healthy pods with the deployment\u0026rsquo;s desired state happen as fast as possible by aggressively initializing and terminating pods. If one pod is terminating, another will be automatically scheduled to start even if the first pod is not yet completely terminated. Stateless applications benefit from this behaviour as one pod executes the same work as any other in the deployment.\nStatefulSets, on the other hand, guarantee that every pod scheduled has a unique identity, which is to say that only a single copy of a pod is running in the cluster at any one time. Whenever scheduling decisions are made, the StatefulSet controller ensures that only one copy of this pod is running at any time. If a pod is deleted, a new pod will not be scheduled until the first pod is fully terminated. This is an important guarantee as FileSystems need to be unmounted before they can be remounted in a new pod. Any ReadWriteOnce PVC defining a device requires this behaviour to ensure the consistency of the data and thus the PVC.\nTo protect data integrity, Kubernetes guarantees that there will never be more than one instance of a StatefulSet Pod running at a time. It assumes that when a node is determined to be offline it may still be running the workload but partitioned from the network. Since Kubernetes is unable to verify that the Pod has been stopped it errs on the side of caution and does not allow a replacement to start on another node.\nKubernetes does reschedule pods from some controllers when nodes become unavailable. The default behaviour is that when a node becomes unavailable its status becomes \u0026ldquo;Unknown\u0026rdquo; and after the pod-eviction-timeout has passed pods are scheduled for deletion. By default, the pod-eviction-timeout is 300 seconds.\nFor this reason, Kubernetes requires manual intervention to initiate timely failover of a StatefulSet Pod. The StorageOS Fencing Controller gives the capability to enable fast failover of workloads when a node goes offline.\nFor more information on the rationale behind the design of StatefulSets please see the Kubernetes design proposal for Pod Safety.\nStorageOS Fencing Controller  The StorageOS Fencing Controller is part of the StorageOS API Manager which is deployed in high availability when StorageOS is installed.\n HA for StatefulSet applications can be achieved with the StorageOS Fencing feature.\nSince StorageOS is able to determine when a node is no longer able to access a volume and has protections in place to ensure that a partitioned or formerly partitioned node can not continue to write data, it can work with Kubernetes to perform safe, fast failovers of Pods, including those running in StatefulSets.\nWhen StorageOS detects that a node has gone offline or become partitioned, it marks the node offline and performs volume failover operations.\nThe StorageOS Fencing Controller watches for these node failures and determines if there are any pods assigned to the failed node with the label storageos.com/fenced=true, and if the pods have any PVCs backed by StorageOS volumes.\nWhen a Pod has StorageOS volumes and if they are all healthy, the StorageOS fencing controller deletes the Pod to allow it to be rescheduled on another node. It also deletes the VolumeAtachments for the corresponding volumes so that they can be immediately attached to the new node.\nNo changes are made to Pods that have StorageOS volumes that are unhealthy. This is usually because a volume was configured to not have any replicas, and the node with the single copy of the data is offline. In this case it is better to wait for the node to recover.\nFencing works with both dynamically provisioned PVCs and PVCs referencing pre-provisioned volumes.\nThe fencing feature is opt-in and Pods must have the storageos.com/fenced=true label set, and be using at least one StorageOS volume, to enable fast failover.\nFor more information about how to enable pod fencing please see our Fencing Operations page.\n","excerpt":"StatefulSet behaviour In order to understand what StorageOS Fencing for Kubernetes is and when it is …","ref":"/docs/concepts/fencing/","title":"Fencing"},{"body":"For information regarding the StorageOS Fencing feature please see our fencing concepts page.\nLabeling a pod for fencing When StorageOS detects that a node has gone offline or become partitioned, it marks the node offline and performs volume failover operations.\nThe StorageOS Fencing Controller watches for node failures and determines if there are any pods targeted for fencing. In order for a pod to be fenced, the following criteria is required.\n The pod must have the label storageos.com/fenced=true The pod to be fenced must claim a StorageOS volume The StorageOS volume claimed by the pod needs to be online  If the node becomes offline and these criteria are met, the pod is deleted and rescheduled on another node.\n No changes are made to Pods that have StorageOS volumes that are unhealthy. This is typically the case when a volume was configured to not have any replicas, and the node with the single copy of the data is offline. In this case it is better to wait for the Node to recover.\n StatefulSet example with fencing enabled The following example shows how a Statefulset can enable fencing to all its pods.\n Note that the fenced label is applied only in the .spec.template.metadata.label section, as the label must only be present on the Pod, but not on the PVC.\n apiVersion:apps/v1kind:StatefulSetmetadata:name:my-statefulsetspec:selector:matchLabels:app:prodserviceName:\u0026#34;default\u0026#34;replicas:1template:metadata:labels:app:prodstorageos.com/fenced:\u0026#34;true\u0026#34;# Fencing labelspec:containers:- name:debianimage:debian:10-slimcommand:[\u0026#34;/bin/sleep\u0026#34;]args:[\u0026#34;3600\u0026#34;]volumeMounts:- name:storageos-volumemountPath:/mntvolumeClaimTemplates:- metadata:name:storageos-volumelabels:env:prodstorageos.com/replicas:\u0026#34;1\u0026#34;spec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;resources:requests:storage:10GiFencing trigger The StorageOS Fencing Controller checks the StorageOS node health every 5 seconds. This is how quickly the fencing controller can react to node failures.\nPods assigned to unhealthy nodes will be evaluated immediately on state change, and then re-evaluated every hour, though this is configurable. This retry allows pods that had unhealthy volumes which have now recovered to eventually failover, or pods that were rescheduled on an unhealthy node to be re-evaluated for fencing.\nFencing example The following example shows how the StorageOS API manager fences a Pod.\n  Ensure the storageos.com/fenced label is present\n$ kubectl -n mysql get pod --show-labels -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS mysql-0 1/1 Running 0 6m33s 10.42.3.7 worker1 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; app=mysql,controller-revision-hash=mysql-799fd74b87,env=prod,statefulset.kubernetes.io/pod-name=mysql-0,storageos.com/fenced=true $ kubectl -n mysql get pvc --show-labels -owide NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE LABELS data-mysql-0 Bound pvc-5d7b23a6-e754-4998-98fd-318b3f9382bb 5Gi RWO fast 19m Filesystem app=mysql,env=prod,storageos.com/replicas=1  Note that the mysql-0 pod has the storageos.com/fenced=true label.\n   Stop node hosting the mysql-0 pod\nroot@worker1:~# shutdown -h now   Check logs from the StorageOS API Manager\n$ kubectl -n kube-system logs storageos-api-manager-68759bbc78-7l5fw ... {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.413811357Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;nginx-ingress-controller-xbqjf\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;ingress-nginx\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.417605039Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;storageos-api-manager-68759bbc78-7l5fw\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;kube-system\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.417748651Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;coredns-7c5566588d-8g5xq\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;kube-system\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.417792281Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;metrics-server-6b55c64f86-cnwtk\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;kube-system\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.417883383Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;default-http-backend-67cf578fc4-w8sm4\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;ingress-nginx\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.417975663Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;coredns-autoscaler-65bfc8d47d-ph6pn\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;kube-system\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.418024204Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;canal-stzdv\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;kube-system\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.418065315Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;storageos-etcd-2hkff82fq2\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;storageos-etcd\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.418092165Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;storageos-daemonset-6zrkk\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;kube-system\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.418182036Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;skipping pod without storageos.com/fenced=true label set\u0026quot;,\u0026quot;name\u0026quot;:\u0026quot;cattle-node-agent-sjspk\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;cattle-system\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.439513312Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;pod has fenced label set and volume(s) still healthy after node failure, proceeding with fencing\u0026quot;,\u0026quot;pod\u0026quot;:\u0026quot;mysql-0\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;mysql\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.495807296Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;pod deleted\u0026quot;,\u0026quot;pod\u0026quot;:\u0026quot;mysql-0\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;mysql\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.505411162Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;volume attachment deleted\u0026quot;,\u0026quot;pod\u0026quot;:\u0026quot;mysql-0\u0026quot;,\u0026quot;namespace\u0026quot;:\u0026quot;mysql\u0026quot;,\u0026quot;pvc\u0026quot;:\u0026quot;data-mysql-0\u0026quot;,\u0026quot;va\u0026quot;:\u0026quot;csi-c2b44cee5a647e20d77e0e217dfaec07afd592eae57bcccc09b3447de653ae8c\u0026quot;,\u0026quot;node\u0026quot;:\u0026quot;worker1\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.505439792Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;fenced pod\u0026quot;} {\u0026quot;level\u0026quot;:\u0026quot;info\u0026quot;,\u0026quot;timestamp\u0026quot;:\u0026quot;2021-04-28T13:06:49.573478266Z\u0026quot;,\u0026quot;msg\u0026quot;:\u0026quot;set scheduler\u0026quot;,\u0026quot;scheduler\u0026quot;:\u0026quot;storageos-scheduler\u0026quot;,\u0026quot;pod\u0026quot;:\u0026quot;mysql/mysql-0\u0026quot;} ...  The StorageOS API Manager detects all the pods that are on the failed node, and selects only the ones that meet the fencing criteria as described above. In this case only mysql-0 is selected for fencing.\n   Check the pod\u0026rsquo;s new node\n$ kubectl -n mysql get pod --show-labels -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS mysql-0 1/1 Running 0 6m33s 10.42.3.7 worker2 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; app=mysql,controller-revision-hash=mysql-799fd74b87,env=prod,statefulset.kubernetes.io/pod-name=mysql-0,storageos.com/fenced=true  The pod mysql-0 started on a different node successfully.\n   ","excerpt":"For information regarding the StorageOS Fencing feature please see our fencing concepts page. …","ref":"/docs/operations/fencing/","title":"Fencing"},{"body":"Follow the recipes on this page to create your first PVC (Persistent Volume Claim) using StorageOS. StorageOS implements dynamic provisioning, so the creation of a PVC will automatically provision a PV (PersistentVolume) that can be used to persist data written by a Pod.\nCreate the PersistentVolumeClaim   You can find the basic examples in the StorageOS use-cases repository, in the 00-basic directory.\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases/00-basic PVC definition\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:my-vol-1spec:storageClassName:\u0026#34;fast\u0026#34;# StorageOS StorageClassaccessModes:- ReadWriteOnceresources:requests:storage:5GiThe above PVC will dynamically provision a 5GB volume using the fast StorageClass. This StorageClass was created during the StorageOS install and triggers creation of a PersistentVolume by StorageOS.\nFor installations with CSI, you can create multiple StorageClasses in order to specify default labels.\napiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:storageos-replicatedprovisioner:csi.storageos.com# Provisioner when using CSIparameters:csi.storage.k8s.io/fstype:ext4storageos.com/replicas:\u0026#34;1\u0026#34;# Enforces 1 replica for the Volume# Change the Namespace below if StorageOS doesn\u0026#39;t run in kube-systemcsi.storage.k8s.io/node-publish-secret-namespace:kube-system# Namespace that runs StorageOS Daemonsetcsi.storage.k8s.io/provisioner-secret-namespace:kube-system# Namespace that runs StorageOS Daemonsetcsi.storage.k8s.io/controller-publish-secret-namespace:kube-system# Namespace that runs StorageOS Daemonsetcsi.storage.k8s.io/node-publish-secret-name:csi-node-publish-secretcsi.storage.k8s.io/provisioner-secret-name:csi-provisioner-secretcsi.storage.k8s.io/controller-publish-secret-name:csi-controller-publish-secretThe above StorageClass has the storageos.com/replicas label set. This label tells StorageOS to create a volume with a replica. Adding StorageOS feature labels to the StorageClass ensures all volumes created with the StorageClass have the same labels. For simplicity\u0026rsquo;s sake this example will use unreplicated volumes.\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:my-vol-1spec:storageClassName:\u0026#34;storageos-replicated\u0026#34;# Reference to the StorageClassaccessModes:- ReadWriteOnceresources:requests:storage:5GiYou can also choose to add the label in the PVC definition rather than the StorageClass. The PVC definition takes precedence over the SC.\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:my-vol-1labels:storageos.com/replicas:\u0026#34;1\u0026#34;spec:storageClassName:\u0026#34;fast\u0026#34;accessModes:- ReadWriteOnceresources:requests:storage:5GiThe above PVC has the storageos.com/replicas label set. This label tells StorageOS to add a replica for the volume that is created. For the sake of keeping this example simple an unreplicated volume will be used.\n  Move into the examples folder and create a PVC using the PVC definition above.\n$ # from storageos-usecases/00-basic $ kubectl create -f ./pvc-basic.yaml You can view the PVC that you have created with the command below\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-1 Bound pvc-f8ffa027-e821-11e8-bc0b-0ac77ccc61fa 5Gi RWO fast 1m   Create a pod that mounts the PVC created in step 2.\n$ kubectl create -f ./pod.yaml The command above creates a Pod that uses the PVC that was created in step 1.\napiVersion:v1kind:Podmetadata:name:d1spec:containers:- name:debianimage:debian:9-slimcommand:[\u0026#34;/bin/sleep\u0026#34;]args:[\u0026#34;3600\u0026#34;]volumeMounts:- mountPath:/mntname:v1volumes:- name:v1persistentVolumeClaim:claimName:pvc-1In the Pod definition above volume v1 references the PVC created in step 2, and is mounted in the pod at /mnt. In this example a debian image is used for the container but any container image with a shell would work for this example.\n  Confirm that the pod is up and running\n$ kubectl get pods NAME READY STATUS RESTARTS AGE d1 1/1 Running 0 1m   Execute a shell inside the container and write some contents to a file\n$ kubectl exec -it d1 -- bash root@d1:/# echo \u0026#34;Hello World!\u0026#34; \u0026gt; /mnt/helloworld root@d1:/# cat /mnt/helloworld Hello World! By writing to /mnt inside the container, the StorageOS volume created by the PVC is being written to. If you were to kill the pod and start it again on a new node, the helloworld file would still be avaliable.\nIf you wish to see more use cases with actual applications please see our Use Cases documentation.\n  ","excerpt":"Follow the recipes on this page to create your first PVC (Persistent Volume Claim) using StorageOS. …","ref":"/docs/operations/firstpvc/","title":"StorageOS Volume Guide"},{"body":"$ storageos get --help Fetch basic details for resources Usage: storageos get [command] Available Commands: cluster Fetch cluster-wide configuration details diagnostics Fetch a cluster diagnostic bundle licence Fetch current licence configuration details namespace Retrieve basic details of cluster namespaces node Retrieve basic details of nodes in the cluster policy-group Retrieve basic details of policy groups user Fetch user details volume Retrieve basic details of volumes Flags: -h, --help help for get Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Use \u0026#34;storageos get [command] --help\u0026#34; for more information about a command. get cluster $ storageos get cluster --help Fetch cluster-wide configuration details Usage: storageos get cluster [flags] Examples: $ storageos get cluster Flags: -h, --help help for cluster Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) get diagnostics $ storageos get diagnostics --help Fetch a cluster diagnostic bundle from the target node. Due to the work involved this command will run with a minimum command timeout duration of 1h, although accepts longer durations Usage: storageos get diagnostics [flags] Examples: $ storageos get diagnostics $ storageos get diagnostics --output-file ~/my-diagnostics.gz Flags: -h, --help help for diagnostics --output-file string writes the generated diagnostic bundle to a specified file path Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) get license $ storageos get license --help Fetch current licence configuration details Usage: storageos get licence [flags] Aliases: licence, license Examples: $ storageos get licence Flags: -h, --help help for licence Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) get namespace $ storageos get namespace --help Retrieve basic details of cluster namespaces Usage: storageos get namespace [namespace names...] [flags] Aliases: namespace, namespaces Examples: $ storageos get namespaces $ storageos get namespace my-namespace-name Flags: -h, --help help for namespace -l, --selector stringArray filter returned results by a set of comma-separated label selectors Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) get node storageos get node --help Retrieve basic details of nodes in the cluster Usage: storageos get node [node names...] [flags] Aliases: node, nodes Examples: $ storageos get node my-node-name Flags: -h, --help help for node -l, --selector stringArray filter returned results by a set of comma-separated label selectors Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) get policy-group $ storageos get policy-group --help Retrieve basic details of policy groups Usage: storageos get policy-group [policy-group names...] [flags] Aliases: policy-group, policy-groups Examples: $ storageos get policy-group $ storageos get policy-group my-policy-group-name $ storageos get policy-group --use-ids my-policy-group-id Flags: -h, --help help for policy-group Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) get user $ storageos get user --help Fetch user details Usage: storageos get user [user names...] [flags] Aliases: user, users Examples: $ storageos get user my-username $ storageos get user my-username-1 my-username-2 $ storageos get user --use-ids my-userid $ storageos get user --use-ids my-userid-1 my-userid-2 Flags: -h, --help help for user Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) get volume $ storageos get volume --help Retrieve basic details of volumes Usage: storageos get volume [volume names...] [flags] Aliases: volume, volumes Examples: $ storageos get volumes --all-namespaces $ storageos get volume --namespace my-namespace-name my-volume-name Flags: -A, --all-namespaces retrieves volumes from all accessible namespaces. This option overrides the namespace configuration -h, --help help for volume -l, --selector stringArray filter returned results by a set of comma-separated label selectors Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos get --help Fetch basic details for resources Usage: storageos get [command] Available …","ref":"/docs/reference/cli/get/","title":"Get"},{"body":"StorageOS provides a GUI for cluster and volume management.\nThe GUI is available at port 5705 on any of the nodes in the cluster. Initially you can log in as the default administrator, with the username and password from the storageos-api Secret. By default storageos, storageos.\n You can access the GUI by either port-forwarding with kubectl or using an Ingress rule. i.e kubectl -n kube-system port-forward svc/storageos 5705.\n Nodes Volumes You can create volumes, including replicated volumes, and view volume details:\nLicensing Cluster info Namespaces Volumes can be namespaced across different projects or teams, and you can switch namespace using the left hand panel:\n","excerpt":"StorageOS provides a GUI for cluster and volume management.\nThe GUI is available at port 5705 on any …","ref":"/docs/reference/gui/","title":"Graphical user interface (GUI)"},{"body":"Various tools are available for checking on the status of a cluster.\nThe StorageOS CLI displays the status of nodes in the cluster.\n$ storageos get nodes NAME HEALTH AGE LABELS node1 online 44 minutes ago node2 online 44 minutes ago node3 online 44 minutes ago node4 online 44 minutes ago node5 online 44 minutes ago ","excerpt":"Various tools are available for checking on the status of a cluster.\nThe StorageOS CLI displays the …","ref":"/docs/operations/health/","title":"Cluster health"},{"body":"$ storageos help Storage for Cloud Native Applications. By using this product, you are agreeing to the terms of the the StorageOS Ltd. End User Subscription Agreement (EUSA) found at: https://storageos.com/legal/#eusa To be notified about stable releases and latest features, sign up at https://my.storageos.com. Usage: storageos [command] Available Commands: apply Make changes to existing resources attach Attach a volume to a node create Create new resources delete Delete resources in the cluster describe Fetch extended details for resources detach Detach a volume from its current location get Fetch basic details for resources help Help about any command nfs Make changes and attach nfs volumes update Make changes to existing resources version View version information for the StorageOS CLI Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -h, --help help for storageos -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Additional help topics: storageos config-file View help information for using a configuration file storageos env View documentation for configuration settings which can be set in the environment storageos exitcodes View documentation for the exit codes used by the StorageOS CLI Use \u0026#34;storageos [command] --help\u0026#34; for more information about a command. ","excerpt":"$ storageos help Storage for Cloud Native Applications. By using this product, you are agreeing to …","ref":"/docs/reference/cli/help/","title":"Help"},{"body":"StorageOS uses the storage available on the nodes where it is installed to present as available for volumes.\nIn order to mitigate against problems caused by filling the host root disk, we recommend mounting a separate device into the /var/lib/storageos directory. StorageOS is agnostic to the type of filesystem mounted in /var/lib/storageos.\nExtending Available Storage StorageOS uses subdirectories of /var/lib/storageos/data to hold user data. By default, the directory /var/lib/storageos/data/dev1 will be created when a node is bootstrapped, and used for pool data. It is possible to shard the data by creating more directories into this structure. StorageOS will save data in any directory that conforms to the pattern /var/lib/storageos/data/dev[0-9]+, such as /var/lib/storageos/data/dev2 or /var/lib/storageos/data/dev5. This functionality enables operators to mount different devices into devX directories and StorageOS will recognise them as available storage automatically.\nThere are two possible options to expand the available disk space for StorageOS to allocate:\n Mount filesystem in /var/lib/storageos/data/devX Use LVM to expand the logical volume available to StorageOS  Option 1: Mount Additional Devices This option enables operators to expand the cluster\u0026rsquo;s available space at any time without having to stop applications or forcing operational downtime. The expansion of disk is transparent for applications and StorageOS Volumes. StorageOS will use the new available space to create new data files.\n  Context\nWe assume that there is a disk available in our Linux system without formatting in addition to the root filesystem. StorageOS data dir dev1 (/var/lib/storageos/data/dev1) is using /dev/xvda1. We will use the device /dev/xvdf to expand StorageOS available space.\nList available block devices in the host.\nroot@node0:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 128G 0 disk `-xvda1 202:1 0 128G 0 part / xvdf 202:80 0 100G 0 disk Check StorageOS cluster\u0026rsquo;s available capacity.\n$ storageos get node -ojson | jq -r '.[] | { name: .name, capacity: .capacity.total }' \u0026quot;node0\u0026quot; 137,438,953,472 \u0026quot;node1\u0026quot; 137,438,953,472 \u0026quot;node2\u0026quot; 137,438,953,472   Format device\nroot@node0:/var/lib/storageos/data# mkfs -t ext4 /dev/xvdf mke2fs 1.42.12 (29-Aug-2014) Creating filesystem with 26214400 4k blocks and 6553600 inodes Filesystem UUID: 380712fa-6f82-477a-81a5-d7466d4c6b7f Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done   Mount filesystem\nroot@node0:~# mkdir -p /var/lib/storageos/data/dev2 root@node0:~# mount /dev/xvdf /var/lib/storageos/data/dev2   Verify available storage\nIn less than 30 seconds, StorageOS will see the new available capacity.\n$ storageos get node -ojson | jq -r '.[] | { name: .name, capacity: .capacity.total }' \u0026quot;node0\u0026quot; 244,491,013,324 \u0026quot;node1\u0026quot; 137,438,953,472 \u0026quot;node2\u0026quot; 137,438,953,472 Note that the node node0 has increased the TOTAL capacity in 100Gi.\n   Persist the mount at boot by adding the mount endpoint to /etc/fstab\n Option 2: Expand Existing Devices Backed by LVM This option enables operators to take advantage of LVM to manage disks.\n  Context\nWe assume that /var/lib/storageos is mounted onto an LVM volume. We are using a volumegroup named storageos and logical volume called data. There is a second physical disk /dev/xvdg unused.\nList available block devices in the host.\nroot@node2:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 128G 0 disk `-xvda1 202:1 0 128G 0 part / xvdf 202:80 0 100G 0 disk `-storageos-data 254:0 0 99G 0 lvm /var/lib/storageos xvdg 202:96 0 100G 0 disk Check StorageOS cluster\u0026rsquo;s available capacity.\n$ storageos get node -ojson | jq -r '.[] | { name: .name, capacity: .capacity.total }' \u0026quot;node0\u0026quot; 137,438,953,472 \u0026quot;node1\u0026quot; 137,438,953,472 \u0026quot;node2\u0026quot; 107,696,304,947 # --\u0026gt; LVM storageos/data volume   Add physical disk to LVM\nroot@node2:~# vgextend storageos /dev/xvdg Volume group \u0026quot;storageos\u0026quot; successfully extended The volume group storageos must have 2 physical volumes (#PV)\nroot@node2:~# vgs VG #PV #LV #SN Attr VSize VFree storageos 2 1 0 wz--n- 199.99g 104.99g   Extend logical volume data\nroot@node2:~# lvextend -L+100G /dev/storageos/data Size of logical volume storageos/data changed from 95.00 GiB (24320 extents) to 195.00 GiB (49920 extents). Logical volume data successfully resized   Resize the FileSystem\n Your filesystem must support the option to be expanded, and to do so while in use. Otherwise, you need to unmount first.\n root@node2:~# resize2fs /dev/storageos/data resize2fs 1.42.12 (29-Aug-2014) Filesystem at /dev/storageos/data is mounted on /var/lib/storageos; on-line resizing required old_desc_blocks = 6, new_desc_blocks = 13 The filesystem on /dev/storageos/data is now 51118080 (4k) blocks long.   Check new available space\nThe mounted file system to /var/lib/storageos has increased its size.\nroot@node2:~# df -h /dev/mapper/storageos-data Filesystem Size Used Avail Use% Mounted on /dev/mapper/storageos-data 192G 60M 183G 1% /var/lib/storageos StorageOS available storage has increased too.\n$ storageos get node -ojson | jq -r '.[] | { name: .name, capacity: .capacity.total }' \u0026quot;node0\u0026quot; 137,438,953,472 \u0026quot;node1\u0026quot; 137,438,953,472 \u0026quot;node2\u0026quot; 206,158,430,208 # --\u0026gt; 100G more available    Persist the mount at boot by adding the mount point to /etc/fstab\n ","excerpt":"StorageOS uses the storage available on the nodes where it is installed to present as available for …","ref":"/docs/operations/managing-host-storage/","title":"Managing Host Storage"},{"body":"InfluxDB is a popular open source time series database application optimised for managing datasets consisting of many small measurements. Its advantages include the ability to handle very high write and query loads. Its uses include monitoring, analytics and the recording and analysis of data from sensors.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying InfluxDB on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases StatefulSet definition:\napiVersion:apps/v1kind:StatefulSetmetadata:name:influxdbspec:replicas:1selector:matchLabels:app:influxdbserviceName:influxdb...spec:serviceAccountName:influxdb...volumeMounts:- mountPath:/var/lib/influxdbname:data...volumeClaimTemplates:- metadata:name:dataspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClassresources:requests:storage:20GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaimTemplate that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaim.\n  Create the InfluxDB objects\nkubectl create -f ./influxdb   Confirm InfluxDB is up and running.\n$ kubectl get pods NAME READY STATUS RESTARTS AGE influxdb-client 1/1 Running 0 1m influxdb-0 1/1 Running 0 1m   Connect to the InfluxDB client pod, then to the InfluxDB server through the service (this reflects the common kubernetes pattern of maintaining a client pod to conveniently inspect a resource interactively). The default user (admin) and password (admin) are defined in the StatefulSet.\n$ kubectl exec -it influxdb-client -- bash root@influxdb-client:/# influx -host influxdb-0.influxdb Connected to http://influxdb-0.influxdb:8086 version 1.8.2 InfluxDB shell version: 1.8.2 \u0026gt; auth username: admin password: \u0026gt; show databases name: databases name _internal \u0026gt; CREATE DATABASE weather; \u0026gt; USE weather Using database weather \u0026gt; INSERT temperature,location=London value=26.4 \u0026gt; INSERT temperature,location=London value=24.9 \u0026gt; INSERT temperature,location=London value=22.2 \u0026gt; INSERT temperature,location=London value=14.7 \u0026gt; INSERT temperature,location=London value=19.5 \u0026gt; INSERT temperature,location=Paris value=27.1 \u0026gt; INSERT temperature,location=Paris value=27.5 \u0026gt; INSERT temperature,location=Paris value=21.3 \u0026gt; INSERT temperature,location=Paris value=26.7 \u0026gt; INSERT temperature,location=Paris value=30.0 \u0026gt; SELECT MEAN(*) FROM \u0026#34;temperature\u0026#34; GROUP BY \u0026#34;location\u0026#34; name: temperature tags: location=London time mean_value 0 25.65 name: temperature tags: location=Paris time mean_value 0 26.90   In the above steps we have inserted some time series data on the temperature at two locations, and calculated the mean for both. InfluxDB offers a variety of such aggregations (see the docs here), allowing convenient analysis of time series data.\nConfiguration If you need custom startup options, you can edit or add to the environment variables within the 20-statefulset.yaml file.\nBackups In this example of how to perform backups of an InfluxDB database on a Kubernetes cluster, we write the output backup file to an Amazon Web Services (AWS) S3 bucket. Other approaches, such as backing up to internal servers or other StorageOS volumes, are possible. For this example to run successfully, Base64-encoded AWS credentials and an S3 bucket name should be inserted into the data field of the backup/50-secret-config.yaml file.\n$ echo -n \u0026#39;\u0026lt;your-aws-access-key-id\u0026gt;\u0026#39; | base64 XXXXXXXXXXXX $ echo -n \u0026#39;\u0026lt;your-aws-secret-access-key\u0026gt;\u0026#39; | base64 XXXXXXXXXXXX $ echo -n \u0026#39;\u0026lt;your-aws-default-region\u0026gt;\u0026#39; | base64 XXXXXXXXXXXX $ echo -n \u0026#39;\u0026lt;your-S3-bucket-name\u0026gt;\u0026#39; | base64 XXXXXXXXXXXX Secret definition:\napiVersion:v1kind:Secretmetadata:name:backup-pod-environmenttype:Opaquedata:AWS_ACCESS_KEY_ID:XXXXXXXXXXXXAWS_SECRET_ACCESS_KEY:XXXXXXXXXXXXAWS_DEFAULT_REGION:XXXXXXXXXXXXBUCKET_NAME:XXXXXXXXXXXXDB_NAME:d2VhdGhlcg==DB_HOST:aW5mbHV4ZGItMC5pbmZsdXhkYjo4MDg4To perform the backup, create the secret and job from the manifest files in the backup directory.\nkubectl create -f ./influxdb/backup/ Confirm that the backup pod has been created, and the backup performed successfully.\nNAME READY STATUS RESTARTS AGE backup-ks976 0/1 Completed 0 1m client 1/1 Running 0 10m influxdb-0 1/1 Running 0 10m mysql-0 1/1 Running 0 10m The files generated by the backup operation should now be available in you S3 bucket. Re-perform the operation at any time by deleting and re-creating the backup job.\n$ kubectl delete -f \u0026#34;influxdb/40-backup-job.yaml\u0026#34; job.batch \u0026#34;backup\u0026#34; deleted $ kubectl create -f \u0026#34;influxdb/40-backup-job.yaml\u0026#34; job.batch/backup created ","excerpt":"InfluxDB is a popular open source time series database application optimised for managing datasets …","ref":"/docs/usecases/influxdb/","title":"InfluxDB"},{"body":"StorageOS has requirements for the configuration of host systems. As such, StorageOS starts an init container that sets the system configuration for StorageOS. The container also manages configuration changes required when upgrading StorageOS versions.\nThe container belongs to the DaemonSet that the StorageOS Cluster Operator starts when a StorageOSCluster resource is created. The storageos-init container is executed as an initContainer as part of a Kubernetes Pod. Therefore, only successful execution of the storageos-init container processes will result in the main container starting.\nScript Framework The code responsible for fulfilling the requirements is based on a Script Framework.\nThe script framework executes a set of scripts, performing checks, verifications and other procedures needed for StorageOS to be able to start. The scripts stdout and stderr are written to the stdout and stderr of the init app. The container logs contain all the logs of the individual scripts that run. The exit statuses of the scripts are used to determine initialization failure or success. Any non-zero exit status is logged as an event in the Kubernetes Pod events.\nIf any of the scripts fail, the storageos-init container will propagate the failure to Kubernetes, showing the status of the Pod as Init:Err.\nTo view the output of all storageos-init containers the following command can be used:\nkubectl -n kube-system logs -l app=storageos,kind=daemonset -c storageos-init For more details, check the StorageOS init container project.\nScripts executed The storageos-init container executes the following scripts.\n enable-lio dbupgrade  ","excerpt":"StorageOS has requirements for the configuration of host systems. As such, StorageOS starts an init …","ref":"/docs/reference/init-container/","title":"Init container"},{"body":"This example shows an example of how to deploy Jenkins on Kubernetes with a StorageOS persistent volume being mounted on /var/jenkins_home. Deploying Jenkins using StorageOS offers multiple benefits. Firstly Jenkins can spin up multiple build pods at once to allow concurrent builds of different projects. Secondly Jenkins configuration is on a PersistentVolume so even if the Jenkins pod is rescheduled the configuration will persist.\nUsing StorageOS volume replicas  allows for failure of nodes holding the PersistentVolume without interrupting Jenkins. Lastly by enabling StorageOS fencing  Jenkins time to recover, in case of node failures, is greatly reduced.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information\nDeploying Jenkins on Kubernetes   You can find the latest files in the StorageOS example deployment repository.\n$ git clone https://github.com/storageos/use-cases.git storageos-usecases $ cd storageos-usecases $ kubectl create -f ./jenkins   Confirm that Jenkins is up and running\n$ kubectl get pods -w -l app=jenkins NAME READY STATUS RESTARTS AGE jenkins-0 1/1 Running 0 1m   Connect to the Jenkins UI through the Jenkins service.\nYou can do this by port forwarding the Jenkins Kubernetes service to your localhost and accessing the UI via your browser. Alternatively if you have network access to your Kubernetes nodes then you can create a NodePort service and access Jenkins like that. A NodePort service has been left in 10-service.yaml commented out.\nTo port-foward the Jenkins service use the following command.\n$ kubectl port-foward svc/jenkins 8080 To login to the Jenkins UI use the credentials specified in 07-config.yaml, unless these have been changed from the defaults the username/password is admin/password.\n  Create a Jenkins job.\nOnce you are logged into the UI you can create a job that will be farmed out to a Kubernetes plugin build agent. Click New Item, enter a name for the project and select Freestyle project. Next add an Execute shell build step. As a proof of concept you can use the bash below to have the pod execute a sleep.\n#!/bin/bash sleep 1000 Save the project and select Schedule a build of your project. You can watch for the appearance of a build pod using kubectl get pods -l jenkins=agent -w. Once the pod is created you should see the Build Executor status in the Jenkins UI display the pod.\nTo see multiple projects being built at once create another project and try scheduling a build of both projects at the same time.\n  ","excerpt":"This example shows an example of how to deploy Jenkins on Kubernetes with a StorageOS persistent …","ref":"/docs/usecases/jenkins/","title":"Jenkins"},{"body":"Kafka is a popular stream processing platform combining features from pub/sub and traditional queues.\nUsing StorageOS persistent volumes with Apache Kafka means that if a pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up, the pod data is immediately available. Should Kubernetes schedule the kafka pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not the original StorageOS master volume is located on the same node.\nKafka has features to allow it to handle replication, and as such careful consideration of whether to allow StorageOS or Kafka to handle replication is required.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nPrerequisites  Apache Zookeeper is required by Kafka to function; we assume it to already exist and be accessible within the Kubernetes cluster as zookeeper, see how to run Zookeeper with StorageOS here StorageOS is assumed to have been installed; please check for the latest available version here Kafka pods require 1536 MB of memory for successful scheduling  Helm To simplify the deployment of kafka, we\u0026rsquo;ve used this Kafka helm chart (incubator) (version 0.13.8, app version 5.0.1) and rendered it into the example deployment files you can find in our GitHub repo.\nClone the use cases repo You can find the latest files in the StorageOS use cases repository in /kafka/\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases StatefulSet definition\n---apiVersion:apps/v1beta1kind:StatefulSetmetadata:name:kafkalabels:app:kafka...spec:serviceName:kafka-headlesspodManagementPolicy:OrderedReadyupdateStrategy:type:OnDeletereplicas:3# \u0026lt;--- number of kafa pods to runtemplate:...spec:serviceAccountName:kafkacontainers:...- name:kafka-brokerimage:\u0026#34;confluentinc/cp-kafka:5.0.1\u0026#34;imagePullPolicy:\u0026#34;IfNotPresent\u0026#34;...volumeMounts:- name:datadirmountPath:\u0026#34;/var/data\u0026#34;volumes:- name:jmx-configconfigMap:name:kafka-metricsterminationGracePeriodSeconds:60volumeClaimTemplates:- metadata:name:datadirspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]resources:requests:storage:10Gi# \u0026lt;--- storage requested for each podstorageClassName:\u0026#34;fast\u0026#34;# \u0026lt;--- the StorageClass to useThis excerpt is from the StatefulSet definition (10-statefulset.yaml). The file contains the PersistentVolumeClaim template that will dynamically provision the necessary storage, using the StorageOS storage class.\nDynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.\n  Create the kubernetes objects\nkubectl apply -f ./kafka/   Confirm kafka is up and running\n$ kubectl get pods -l app=kafka NAME READY STATUS RESTARTS AGE kafka-0 2/2 Running 0 10m kafka-1 2/2 Running 0 9m26s kafka-2 2/2 Running 0 7m59s   Connect to kafka\nConnect to the kafka test client pod and send some test data to kafka through its service endpoint\n  Connect to the pod\nkubectl exec -it kafka-test-client /bin/bash   Create a topic\n/usr/bin/kafka-topics --zookeeper zookeeper:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1   Send some test data\n/usr/bin/kafka-run-class org.apache.kafka.tools.ProducerPerformance --topic test-rep-one --num-records 5000 --record-size 100 --throughput -1 --print-metrics --producer-props acks=1 bootstrap.servers=kafka:9092 buffer.memory=67108864 batch.size=8196   ","excerpt":"Kafka is a popular stream processing platform combining features from pub/sub and traditional …","ref":"/docs/usecases/kafka/","title":"Kafka"},{"body":"The below controllers are part of the StorageOS API manager, and handle a variety of cases where information about Kubernetes objects from your cluster needs to be synced to your StorageOS cluster.\nThe CSI Driver annotation mentioned below is added to your PVC or Node automatically by StorageOS and is not removed. It is set by default in the case of a PVC StorageClass or PVC\u0026rsquo;s StorageClassName parameter, or in the case of a node, by the node driver registrar.\nPVC Label Sync The PVC Label Sync Controller applies labels that have been added to your PVCs to your StorageOS Volume objects. The PVC must have the StorageOS CSI Driver annotation. If there is a label with the same key on the PVC and on the StorageClass the PVC label will take precedence.\nStorageOS dynamically provisions StorageOS Volumes when you create a PVC object. Labels are initially applied to your StorageOS Volume object when it is created. These come from the labels specified in the PVC manifest as well as any default labels specified in the StorageClass.\nThis controller is triggered on any subsequent PVC label update event, so long as the CSI Driver annotation is present.\nIf a label sync fails the change will be requeued and retried. A periodic resync runs every hour (this is configurable via -pvc-label-resync-interval flag for the API Manager).\nNode Label Sync The Node Label Sync controller ensures that labels applied to your Kubernetes nodes are synced through to StorageOS. It is necessary for your Node to have the StorageOS CSI Driver annotation.\nWhen labels are applied to your Kubernetes nodes, they do not automatically sync to StorageOS, hence this controller is required to automatically apply the expected behaviour to your StorageOS cluster.\nIt is triggered on any Kubernetes label update event, so long as the CSI Drive annotation is present.\nIf a label sync fails the change will be requeued and retried. A periodic resync runs every hour (this is configurable via -node-label-resync-interval flag for the API Manager).\nNode Delete The Node Delete Controller syncs deletions from your Kubernetes cluster to StorageOS.\nThis controller dynamically removes nodes from your StorageOS cluster, being triggered when the Kubernetes node is removed.\nWhenever a node delete event occurs the Node Delete Controller will trigger if the node has the StorageOS CSI driver annotation.\nIf the node holds a StorageOS Volume without a replica then it cannot be deleted by this controller. The Volume must be deleted first and then the node. This is to prevent data loss by accidental deletion of a master volume.\nA periodic garbage collection runs every hour (this is configurable via -node-delete-gc-interval flag for the API Manager.\nNamespace Delete The Namespace Delete Controller is responsible for removing StorageOS namespaces from the StorageOS cluster when the corresponding Kubernetes namespace has been removed from Kubernetes.\nThe StorageOS controlplane automatically creates a new StorageOS namespace when a PVC is created in a Kubernetes namespace. StorageOS does not automatically remove namespaces when there are no volumes in them. Instead this controller triggers on any Kubernetes namespace deletion event, syncing Kubernetes namespace deletion to StorageOS namespace deletion.\nA periodic garbage collection runs every hour (this is configurable via -namespace-delete-gc-interval flag for the API Manager.\n","excerpt":"The below controllers are part of the StorageOS API manager, and handle a variety of cases where …","ref":"/docs/reference/kubernetes-object-sync/","title":"Kubernetes Object Sync"},{"body":" Kubevirt is a CNCF sandbox project that allows the running of virtual machines (VMs) in Kubernetes pods.\nDeploying Kubevirt using StorageOS offers multiple benefits. Kubevirt can spin up VMs as Kubernetes pods, using images on StorageOS persistent volumes. Doing this allows the VM data to persist through restarts and rescheduling. Using StorageOS volume replicas also allows for failure of nodes holding the PersistentVolume without interrupting the VM running off the PersistentVolume. Containerized Data Importer (CDI) can also be used to prepare StorageOS volumes with disk images in an automated fashion. Simply by declaring that a VirtualMachine will use a DataVolume and providing the disk image URL, a StorageOS volume can be dynamically provisioned and automatically prepared with the disk image.\nThis usecase will guide you through installing KubeVirt and CDI on your Kubernetes cluster, and create a VM. By the end of the guide you\u0026rsquo;ll be able to launch a shell inside the KubeVirt VM that\u0026rsquo;s running as a Kubernetes pod.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nPrerequisites Please ensure you have met the Kubevirt prerequisites, please see the Kubevirt installation instructions  for more information.\nAs part of this installation it is assumed that you are running a Kubernetes cluster on VMs. As such nested virtualization or hardware emulation need to be enabled.\nDeploying KubeVirt on Kubernetes  For ease of installation we have enabled hardware emulation. If your VMs support nested virtualization then edit the Kubevirt ConfigMap ./kubevirt-install/10-cm.yaml, removing the line debug.useEmulation: \u0026quot;true\u0026quot;.\n   In order to deploy Kubevirt you just need to clone this repository and use kubectl to create the Kubernetes objects.\n$ git clone https://github.com/storageos/use-cases.git storageos-usecases $ cd storageos-usecases/kubevirt $ kubectl create -f ./kubevirt-install   Check that the Kubevirt pods are running.\n$ kubectl get pods -w -n kubevirt NAME READY STATUS RESTARTS AGE virt-api-57546d479b-p26d4 1/1 Running 0 1m virt-api-57546d479b-zs5dw 1/1 Running 0 1m virt-controller-56b5498854-7xsfz 1/1 Running 1 1m virt-controller-56b5498854-bz559 1/1 Running 1 1m virt-handler-6z4kq 1/1 Running 0 1m virt-handler-7szhl 1/1 Running 0 1m virt-handler-jmm6w 1/1 Running 0 1m virt-operator-79c9bdd859-8xq98 1/1 Running 0 1m virt-operator-79c9bdd859-kfjz6 1/1 Running 0 1m   Once Kubevirt is running install CDI.\n$ kubectl create -f ./cdi   Check that the CDI pods are running correctly.\n$ kubectl get pods -n cdi NAME READY STATUS RESTARTS AGE cdi-apiserver-8668f888df-s6pp4 1/1 Running 0 1m cdi-deployment-5cf794896b-whh4j 1/1 Running 0 1m cdi-operator-5887f96c-dz2hg 1/1 Running 0 1m cdi-uploadproxy-97fbbfcbf-6f9xs 1/1 Running 0 1m   Now that CDI and Kubevirt are running, VMs can be created. In this example VMs running Cirros, a small and lightweight OS, will be created. The vm-cirros.yaml manifest creates a VirtualMachine that uses a DataVolume. This means that CDI will create a StorageOS backed PVC and download the image that the VirtualMachineInstance (VMI) will boot from onto the PVC.\n$ kubectl create -f ./vm-cirros.yaml   Check that the VMI is running. Note that the VMI will only be created after CDI has downloaded the Cirros disk image onto a StorageOS persistent volume so depending on your connection speed this may take some time.\n$ kubectl get vmi NAME AGE PHASE IP NODENAME cirros 1m Running 10.244.2.12 ip-10-1-10-154.storageos.net $ kubectl get pods NAME READY STATUS RESTARTS AGE virt-launcher-cirros-drqhr 1/1 Running 0 1m   Connect to the VM console.\nThis example uses the virtctl kubectl plugin in order to connect to the VMs console. The escape sequence ^] is ctrl + ]\n$ kubectl virt console cirros Successfully connected to cirros console. The escape sequence is ^] login as \u0026#39;cirros\u0026#39; user. default password: \u0026#39;gocubsgo\u0026#39;. use \u0026#39;sudo\u0026#39; for root. cirros login: cirros Password: $   Cloning Volumes CDI allows for images to be cloned using a DataVolume manifest. Verify that the cirros pvc, created as part of the vm-cirros.yaml file, exists before attempting to clone the volume.\n N.B. Ensure that the VMI is stopped before continuing!\n   Verify that the VMI is stopped before continuing, and that the cirros pvc, created as part of the vm-cirros.yaml file, exists before attempting to clone the volume.\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cirros Bound pvc-f4833060-5a77-420c-927e-6bc518d9df3c 12Gi RWO fast 1m   Once the PVC\u0026rsquo;s existence is confirmed then create a new DataVolume that uses the cirros PVC as its source.\n$ kubectl create -f ./cloned.yaml   Watch as the CDI pods are created.\n$ kubectl get pods -w You\u0026rsquo;ll see that a cdi-upload-cloned-datavolume pod is created and then a cdi-clone-source pod is created. The cdi-source pod mounts the original cirros volume and sends the contents of the volume to the cdi-upload pod. The cdi-upload pod creates and mounts a new PVC and writes the contents of the original volume to it.\n  ","excerpt":"Kubevirt is a CNCF sandbox project that allows the running of virtual machines (VMs) in Kubernetes …","ref":"/docs/usecases/kubevirt/","title":"Kubevirt"},{"body":"For more information regarding the StorageOS Label Sync feature, please see our Kubernetes Object Sync reference page.\nLabel Syncing The StorageOS API Manager handles cases where information from objects in your Kubernetes Cluster needs to be synced to your StorageOS cluster.\nLabel Syncing on PVCs The below guide shows how to apply a label to your PVCs, and how these labels sync through to your StorageOS Volumes. This operation is used often - for example it is used here to add replicas to a StorageOS Volume.\n  Create a PVC, following the instructions here. When you create a PVC, StorageOS automatically provisions a StorageOS volume for it. An example PVC and StorageOS volume can be seen below. Note the labels app=mysql and env=prod under Labelsin the PVC description and the VolumeAttributesof the StorageOS Volume.\n$ kubectl describe pvc/data-mysql-0 Name: data-mysql-0 Namespace: default StorageClass: fast Status: Bound Volume: pvc-2e6339f0-96f9-4098-a388-149fd0daa14f Labels: app=mysql env=prod Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes storageos.com/storageclass: 572794ab-2d02-4fec-9aaf-43cd725f498e volume.beta.kubernetes.io/storage-provisioner: csi.storageos.com ... $ storageos describe volume pvc-2e6339f0-96f9-4098-a388-149fd0daa14f -oyaml id: 286fd3a6-c8f8-480a-b5e1-d16896db0c72 name: pvc-2e6339f0-96f9-4098-a388-149fd0daa14f ... labels: app: mysql csi.storage.k8s.io/pv/name: pvc-2e6339f0-96f9-4098-a388-149fd0daa14f csi.storage.k8s.io/pvc/name: data-mysql-0 csi.storage.k8s.io/pvc/namespace: default env: prod storageos.com/nocompress: \u0026quot;true\u0026quot; storageos.com/replicas: \u0026quot;0\u0026quot; ...   Now, apply a label to the PVC.\n$ kubectl label pvc data-mysql-0 storageos.com/replicas=3   By using the StorageOS CLI, it can be seen that the label applied has been synced through to our StorageOS volume and the replicas are all present.\n$ storageos describe volume pvc-2e6339f0-96f9-4098-a388-149fd0daa14f -oyaml id: 286fd3a6-c8f8-480a-b5e1-d16896db0c72 name: pvc-2e6339f0-96f9-4098-a388-149fd0daa14f description: \u0026quot;\u0026quot; attachedOn: 72b50fa0-d870-4d57-95fc-980cc41ab951 attachedOnName: worker5 attachmentType: host nfs: exports: [] serviceendpoint: \u0026quot;\u0026quot; namespaceID: d4eb1a29-39e1-477f-b57c-1c264b797575 namespaceName: default labels: app: mysql csi.storage.k8s.io/pv/name: pvc-2e6339f0-96f9-4098-a388-149fd0daa14f csi.storage.k8s.io/pvc/name: data-mysql-0 csi.storage.k8s.io/pvc/namespace: default env: prod storageos.com/nocompress: \u0026quot;true\u0026quot; storageos.com/replicas: \u0026quot;3\u0026quot; filesystem: ext4 sizeBytes: 5368709120 master: id: 836d8bbc-d356-4ad1-89d2-b1da7f6a4e47 nodeID: 72b50fa0-d870-4d57-95fc-980cc41ab951 nodeName: worker5 health: online promotable: true replicas: - id: 46d9ef46-7572-4b23-80c9-097b77c4f7a0 nodeID: b812eb26-f59e-4867-824f-152acfa70968 nodeName: worker4 health: ready promotable: true - id: 7876164b-b2f0-4148-9688-6b154dfa073a nodeID: 52a98f1a-4d33-41e6-891a-6931052c4ba3 nodeName: worker1 health: ready promotable: true - id: 3987e698-68c9-4e8b-adee-16d0f424a106 nodeID: d2b8ca25-4ffb-43ae-90cc-e1bc68be12ee nodeName: worker6 health: ready promotable: true createdAt: 2021-05-06T14:57:58Z updatedAt: 2021-05-06T16:47:26Z version: Mzg   Label Syncing on Nodes Some StorageOS functionality is set by labeling nodes - for example setting a node to \u0026ldquo;compute-only\u0026rdquo; mode, as demonstrated here.\n  Note labels on the node that will be labeled and on the StorageOS node corresponding to that Kubernetes node.\n$ kubectl describe node worker1 Name: worker1 Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux cattle.io/creator=norman kubernetes.io/arch=amd64 kubernetes.io/hostname=worker1 kubernetes.io/os=linux node-role.kubernetes.io/worker=true Annotations: csi.volume.kubernetes.io/nodeid: {\u0026quot;csi.storageos.com\u0026quot;:\u0026quot;52a98f1a-4d33-41e6-891a-6931052c4ba3\u0026quot;} flannel.alpha.coreos.com/backend-data: {\u0026quot;VtepMAC\u0026quot;:\u0026quot;06:7b:af:b9:a5:2b\u0026quot;} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 212.71.244.105 node.alpha.kubernetes.io/ttl: 0 projectcalico.org/IPv4IPIPTunnelAddr: 10.42.2.1 rke.cattle.io/external-ip: 212.71.244.105 rke.cattle.io/internal-ip: 192.168.152.238 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Thu, 06 May 2021 13:28:30 +0100 ... $ kubectl describe -n kube-system pod storageos-daemonset-6q4g8 Name: storageos-daemonset-6q4g8 Namespace: kube-system Priority: 2000001000 Priority Class Name: system-node-critical Node: worker1/192.168.152.238 Start Time: Thu, 06 May 2021 15:53:34 +0100 Labels: app=storageos app.kubernetes.io/component=storageos-daemonset app.kubernetes.io/instance=example-storageos app.kubernetes.io/managed-by=storageos-operator app.kubernetes.io/name=storageos app.kubernetes.io/part-of=storageos controller-revision-hash=f5dcf577d kind=daemonset pod-template-generation=1 storageos_cr=example-storageos Annotations: kubectl.kubernetes.io/default-logs-container: storageos Status: Running IP: 192.168.152.238 IPs: IP: 192.168.152.238   Now, apply the label to the node.\n$ kubectl label node worker1 storageos.com/computeonly=true   The label has synced to the node and it has been set to compute-only mode.\n$ storageos describe node worker1 ID 52a98f1a-4d33-41e6-891a-6931052c4ba3 Name worker1 Health online Addresses: Data Transfer address 192.168.152.238:5703 Gossip address 192.168.152.238:5711 Supervisor address 192.168.152.238:5704 Clustering address 192.168.152.238:5710 Labels beta.kubernetes.io/arch=amd64, beta.kubernetes.io/os=linux, cattle.io/creator=norman, kubernetes.io/arch=amd64, kubernetes.io/hostname=worker1, kubernetes.io/os=linux, node-role.kubernetes.io/worker=true, storageos.com/computeonly=true   ","excerpt":"For more information regarding the StorageOS Label Sync feature, please see our Kubernetes Object …","ref":"/docs/operations/label-stos-objects/","title":"Labeling StorageOS Objects"},{"body":"Feature labels are a powerful and flexible way to control storage features.\nApplying specific feature labels triggers compression, replication and other storage features. No feature labels are present by default.\nStorageOS Node labels Nodes do not have any feature labels present by default. When StorageOS is run within Kubernetes, the StorageOS API Manager syncs any Kubernetes node labels to the corresponding StorageOS node. The Kubernetes node labels act as the \u0026ldquo;source of truth\u0026rdquo;, so labels should be applied to the Kubernetes nodes rather than to StorageOS nodes. This is because the Kubernetes node labels overwrite the StorageOS node labels on sync.\n   Feature Label Values Description     Compute only storageos.com/computeonly true / false Specifies whether a node should be computeonly where it only acts as a client and does not host volume data locally, otherwise the node is hyperconverged (the default), where the node can operate in both client and server modes.    You can set the computeonly label on the Kubernetes node and the label will be sync\u0026rsquo;d to the StorageOS node (labels take an eventual consistency reconciliation time of ~1min).\nkubectl label node $NODE storageos.com/computeonly=true StorageOS Volume labels Volumes do not have any feature labels present by default.\n WARNING: The encryption, caching and compression labels can only apply at provisioning time, they can\u0026rsquo;t be changed during execution.\n    Feature Label Values Description     Caching storageos.com/nocache true / false Switches off caching.   Compression storageos.com/nocompress true / false Switches off compression of data at rest and in transit (compression is not enabled by default to maximise performance).   Encryption storageos.com/encryption true / false Encrypts the contents of the volume. For each volume, a key is automatically generated, stored, and linked with the PVC.   Failure Mode storageos.com/failure-mode hard, soft, alwayson or integers [0, 5] Sets the failure mode for a volume, either explicitly using a failure mode or implicitly using a replica threshold.   Replication storageos.com/replicas integers [0, 5] Sets the number of replicas i.e full copies of the data across nodes. Typically 1 or 2 replicas is sufficient (2 or 3 instances of the data); latency implications need to be assesed when using more than 2 replicas.    To create a volume with a feature label:\n  Option 1: PVC Label\nAdd the label in the PVC definition, for instance:\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:pvc-3labels:storageos.com/replicas:\u0026#34;1\u0026#34;# Label \u0026lt;-----spec:storageClassName:\u0026#34;fast\u0026#34;accessModes:- ReadWriteOnceresources:requests:storage:1G  Option 2: Set label in the StorageClass\nAny PVC using the StorageClass inherits the label. The PVC label takes precedence over the StorageClass parameters.\nThe encryption label is not applicable to StorageClasses\napiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:storageos-replicatedparameters:csi.storage.k8s.io/fstype:ext4storageos.com/replicas:\u0026#34;1\u0026#34;# Label \u0026lt;--------provisioner:storageos# CSI driver (recommended)# Change the NameSpace below if StorageOS doesn\u0026#39;t run in kube-systemcsi.storage.k8s.io/controller-expand-secret-name:csi-controller-expand-secretcsi.storage.k8s.io/controller-publish-secret-name:csi-controller-publish-secretcsi.storage.k8s.io/node-publish-secret-name:csi-node-publish-secretcsi.storage.k8s.io/provisioner-secret-name:csi-provisioner-secretcsi.storage.k8s.io/controller-expand-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/controller-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/node-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/provisioner-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonset   N.B. The StorageOS API manager periodically syncs labels from Kubernetes PVCs to the corresponding StorageOS volume. Therefore changes to StorageOS volume labels should be made to the corresponding Kubernetes PVC rather than to the StorageOS volume directly.\n StorageOS Pod labels    Feature Label Values Description     Pod fencing storageos.com/fenced true / false Targets a pod to be fenced in case of node failure. (default: false)     For a pod to be fenced by StorageOS, a few requirements described in the Fencing Operations page need to be fulfilled.\n kubectl label pod $POD storageos.com/fenced=true It is recommended to define the fenced label in the pod\u0026rsquo;s manifest, i.e in the Statefulset definitions. Statefulsets pass labels to their VolumeClaimTemplates. You must set the label only at the spec.template.metadata.labels. Otherwise, the StorageOS volumes will fail to provision as only special accepted labels can be passed to volumes.\napiVersion: apps/v1 kind: StatefulSet metadata: name: my-statefulset spec: selector: matchLabels: # \u0026lt;----- Note that the matchLabels don\u0026#39;t have the fenced label env: prod serviceName: my-statefulset-svc replicas: 1 template: metadata: labels: # \u0026lt;----- Note that the fenced label IS PRESENT env: prod storageos.com/fenced: \u0026#34;true\u0026#34; spec: containers: ... ","excerpt":"Feature labels are a powerful and flexible way to control storage features.\nApplying specific …","ref":"/docs/reference/labels/","title":"StorageOS Feature Labels"},{"body":"A newly installed StorageOS cluster does not include a licence. A cluster can run unlicensed for 24 hours. After that, new operations such as volume provisioning or adding nodes are not permitted. Normal functioning of the cluster can be unlocked by applying a Free Developer licence.\nTo learn how to apply a licence to your cluster, check the operations licensing page.\nFree Developer licence The developer licence only requires the user to register and issue a licence through StorageOS.\nThe developer licence is free and grants a licence for a 5 node StorageOS cluster with 5TB of provisioned capacity. It is designed to enable basic cloud native workflows in Kubernetes that require the persistence of stateful application data. Dynamic provisioning, distributed access to data and high availability of volumes through synchronous replication and automatic failover are some of the features that are available under the developer licence.\nThe free developer licence includes community support through our public Slack channel..\nCommercial licences For information on our commercial offerings, including support, please contact sales@storageos.com.\nNote about capacity limits Some StorageOS licences have limits on capacity. StorageOS allows provisioning volumes until the limit of the licence is reached. Only the size of the volume requested by the Persistent Volume Claim counts for the licence limit, regardless whether of whether the volume has replication enabled.\nOnce the licence limit is reached, new volumes are not able to provision unless provisioned capacity is released, i.e deleting volumes. That behaviour is not tight to the capacity of the backend disks on your nodes.\n","excerpt":"A newly installed StorageOS cluster does not include a licence. A cluster can run unlicensed for 24 …","ref":"/docs/reference/licence/","title":"Licence"},{"body":"A newly installed StorageOS cluster does not include a licence. A cluster can run unlicensed for 24 hours. After that, new operations such as volume provisioning or adding nodes are not permitted. Normal functioning of the cluster can be unlocked by applying for a Free Developer licence.\nObtaining a Developer licence via the GUI You will need access to the StorageOS GUI on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):\n$ kubectl port-forward -n kube-system svc/storageos 5705 As an alternative, an Ingress controller may be preferred.\nOnce you have obtained access to the GUI, login using whatever credentials you used to create the cluster.\nYou can obtain and apply a free Developer licence in the StorageOS web GUI by creating or logging in with a StorageOS account on the StorageOS portal via the licence page of the GUI:\nWait a few seconds for the licence generation process to complete, at which point your licence will be visible.\nApplying a previously obtained licence via the GUI Commercial licences are delivered through contact with the StorageOS team. To apply such keys, via the web GUI, visit the licence section of the GUI and click on the tab \u0026ldquo;Upgrade\u0026rdquo;, for the specific licence level you purchased. Then paste the licence key and click on \u0026ldquo;UPLOAD KEY TO CLUSTER\u0026rdquo;. Note that you can also view your cluster ID on the same page.\nApplying a licence via the CLI Before getting a licence, you need to know the ID of your StorageOS cluster.\nThis CLI command can print the cluster ID:\n$ storageos get cluster ID: 704dd165-9580-4da4-a554-0acb96d328cb Licence: expiration: 2021-03-25T13:48:46Z (1 year from now) capacity: 5.0 TiB kind: professional customer name: storageos Created at: 2020-03-25T13:48:33Z (1 hour ago) Updated at: 2020-03-25T13:48:46Z (1 hour ago) Given the Cluster ID, the StorageOS team can generate a licence. Once, given the key, you can apply the licence by using the following command.\n$ echo PASTE-THE-LICENCE-KEY-HERE | storageos apply licence --from-stdin Read the licence CLI command reference for further information.\nObtaining an Enterprise licence Please contact sales@storageos.com to discuss pricing for commercial licences.\n","excerpt":"A newly installed StorageOS cluster does not include a licence. A cluster can run unlicensed for 24 …","ref":"/docs/operations/licensing/","title":"Licensing"},{"body":"Beginning with Microsoft SQL Server 2017, Microsoft has supported MSSQL on linux.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying MS SQL on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion\nkind:StatefulSetmetadata:name:mssqlspec:selector:matchLabels:app:mssqlenv:prodserviceName:mssqlreplicas:1...spec:serviceAccountName:mssql...volumeMounts:- name:datamountPath:/var/opt/mssql...volumeClaimTemplates:- metadata:name:datalabels:env:prodspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClass resources:requests:storage:5GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaim.\n  Move into the MS SQL examples folder and create the objects\ncd storageos-usecases kubectl create -f ./mssql   Confirm MS SQL is up and running.\n$ kubectl get pods -w -l app=mssql NAME READY STATUS RESTARTS AGE mssql-0 1/1 Running 0 1m   Connect to the MS SQL client pod and connect to the MS SQL server through the service\n$ kubectl exec -it mssql-0 -- /opt/mssql-tools/bin/sqlcmd -S mssql-0.mssql -U SA -P \u0026#39;Password15\u0026#39; 1\u0026gt; USE master; 2\u0026gt; GO Changed database context to \u0026#39;master\u0026#39;. 1\u0026gt; SELECT name, database_id, create_date FROM sys.databases ; 2\u0026gt; GO name database_id create_date --------------------------- ----------- ----------------------- master 1 2003-04-08 09:13:36.390 tempdb 2 2018-11-02 16:30:37.907 model 3 2003-04-08 09:13:36.390 msdb 4 2018-10-19 01:18:57.300 (4 rows affected)   Configuration If you need custom startup options, you can edit the ConfigMap file 15-mssql-configmap.yaml with your desired MS SQL configuration settings.\n","excerpt":"Beginning with Microsoft SQL Server 2017, Microsoft has supported MSSQL on linux.\nBefore you start, …","ref":"/docs/usecases/mssql/","title":"MS SQL"},{"body":"StorageOS supports secure communication with an external etcd cluster using mutual TLS (mTLS). With mTLS both StorageOS and etcd authenticate each other ensuring that communication only happens between mutually authenticated end points, and that all communication is encrypted.\nStorageOS uses the certificates and keys from a Secret to cypher and authenticate Etcd traffic.\nHow to create the certificates Secret The client auth certificates need the following file names, in the Secret.\n etcd-client-ca.crt - containing the etcd Certificate Authority certificate etcd-client.crt - containing the etcd Client certificate etcd-client.key - cotaining the etcd Client key  kubectl create secret -n storageos-etcd generic \\  etcd-client-tls \\  --from-file=\u0026#34;etcd-client-ca.crt\u0026#34; \\  --from-file=\u0026#34;etcd-client.crt\u0026#34; \\  --from-file=\u0026#34;etcd-client.key\u0026#34; How to use the mTLS certificates Secret with StorageOS Below is an example StorageOSCluster resource that can be used to setup StorageOS with etcd using mTLS.\napiVersion:storageos.com/v1kind:StorageOSClustermetadata:name:storageos-clusternamespace:\u0026#34;storageos-operator\u0026#34;spec:# StorageOS Pods are in kube-system by defaultsecretRefName:\u0026#34;storageos-api\u0026#34;secretRefNamespace:\u0026#34;default\u0026#34;images:nodeContainer:\u0026#34;storageos/node:v2.4.0\u0026#34;namespace:\u0026#34;storageos\u0026#34;# External mTLS secured etcd cluster specific propertiestlsEtcdSecretRefName:\u0026#34;etcd-client-tls\u0026#34;# Secret containing etcd client certificatestlsEtcdSecretRefNamespace:\u0026#34;etcd\u0026#34;# Namespace of the client certificates secretkvBackend:address:\u0026#34;https://storageos-etcd-cluster-client.storagos-etcd.svc:2379\u0026#34;# Etcd client service address.backend:\u0026#34;etcd\u0026#34;# Backend typetlsEtcdSecretRefName and tlsEtcdSecretRefNamespace are used to pass a reference to the Secret.\nThe StorageOS operator uses the etcd secret that contains the client certificates, to build a secret in the StorageOS installation namespace. This secret contains the certificate filenames and certificate file contents. The StorageOS daemonset that is created by the operator mounts the secret as a volume so that the certificate files are available inside the pod. Environment variables containing the file paths are passed to the StorageOS process in order to use the files from the mounted path.\nA worked example of setting up StorageOS with external etcd using mTLS is avaliable here. For ease of use the example uses the CoreOS etcd operator and the CoreOS guide The example uses the CoreOS etcd operator and follows the CoreOS guide for Cluster TLS.\n","excerpt":"StorageOS supports secure communication with an external etcd cluster using mutual TLS (mTLS). With …","ref":"/docs/operations/etcd/storageos-secret-info/","title":"Encrypting communication with Etcd"},{"body":" MySQL is a popular SQL open source database for a wide range of popular web-based applications including WordPress.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying MySQL on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion\napiversion:apps/v1kind:statefulsetmetadata:name:mysqlspec:selector:matchlabels:app:mysqlenv:prodservicename:mysqlreplicas:1...spec:serviceaccountname:mysql...volumemounts:- name:datamountpath:/var/lib/mysqlsubpath:mysql- name:confmountpath:/etc/mysql/mysql.conf.d...volumeclaimtemplates:- metadata:name:datalabels:env:prodspec:accessmodes:[\u0026#34;readwriteonce\u0026#34;]storageclassname:\u0026#34;fast\u0026#34;# storageos storageclass resources:requests:storage:5giThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaim.\n  Move into the MySQL examples folder and create the objects\ncd storageos-usecases kubectl create -f ./mysql   Confirm MySQL is up and running.\n$ kubectl get pods -w -l app=mysql NAME READY STATUS RESTARTS AGE mysql-0 1/1 Running 0 1m   Connect to the MySQL client pod and connect to the MySQL server through the service\n$ kubectl exec client -- mysql -h mysql-0.mysql -e \u0026#34;show databases;\u0026#34; Database information_schema mysql performance_schema   Configuration If you need custom startup options, you can edit the ConfigMap file 15-mysqld-configmap.yaml with your desired MySQL configuration settings.\n","excerpt":"MySQL is a popular SQL open source database for a wide range of popular web-based applications …","ref":"/docs/usecases/mysql/","title":"MySQL"},{"body":"StorageOS namespaces are an identical concept to Kubernetes namespaces. They are intended to allow a StorageOS cluster to be used by multiple teams across multiple projects.\nIt is not necessary to create StorageOS namespaces manually, as StorageOS maps Kubernetes namespaces on a one-to-one basis when PersistentVolumeClaims using the StorageOS StorageClass are created.\nAccess to Namespaces is controlled through user or group level policies.\n","excerpt":"StorageOS namespaces are an identical concept to Kubernetes namespaces. They are intended to allow a …","ref":"/docs/concepts/namespaces/","title":"Namespaces"},{"body":"Namespaces help different projects or teams share a StorageOS cluster. Only the default namespace is created by default.\nNamespaces apply to volumes.\nManaging Namespaces In order to create a new namespace navigate to \u0026ldquo;Namespaces\u0026rdquo; in the GUI, and select \u0026ldquo;Create Namespace\u0026rdquo;.\nWhen a Kubernetes PVC is created in a namespace, StorageOS automatically maps the Volume in the same namespace. Namespaces are created by StorageOS to fulfil the RBAC rules enforced by Kubernetes roles.\nIn order to delete a namespace, all volumes must be deleted from the namespace before the namespace can be deleted.\n","excerpt":"Namespaces help different projects or teams share a StorageOS cluster. Only the default namespace is …","ref":"/docs/operations/namespaces/","title":"Namespaces"},{"body":"Nginx is a popular web server that can be used as a reverse proxy, load balancer or even as a Kubernetes ingress controller.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Nginx on Kubernetes   You can find the latest files in the StorageOS use cases repostiory\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet definition\napiVersion:apps/v1kind:StatefulSetmetadata:name:nginxspec:serviceName:nginxspec:serviceAccountName:nginxcontainers:- name:nginximage:nginxports:- containerPort:80volumeMounts:- name:nginx-datamountPath:/usr/share/nginx/htmlsubPath:html- name:nginx-configmountPath:/etc/nginx/conf.dvolumes:- name:nginx-configconfigMap:name:nginxvolumeClaimTemplates:- metadata:name:nginx-dataspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClass resources:requests:storage:5GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaim.\n  Move into the Nginx examples folder and create the objects\ncd storageos-usecases kubectl create -f ./nginx   Confirm Nginx is up and running.\n$ kubectl get pods -w -l app=nginx NAME READY STATUS RESTARTS AGE nginx-0 1/1 Running 0 1m   Connect to the nginx pod and write a file to /usr/share/nginx/html that Nginx will serve.\n$ kubectl exec nginx-0 -it -- bash root@nginx-0:/# echo Hello world! \u0026gt; /usr/share/nginx/html/greetings.txt   Connect to the Busybox pod and connect to the Nginx server through the service and retrieve the directory index from Nginx.\n$ kubectl exec -it busybox -- /bin/sh / # wget -q -O- nginx \u0026lt;html\u0026gt; \u0026lt;head\u0026gt;\u0026lt;title\u0026gt;Index of /\u0026lt;/title\u0026gt;\u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;h1\u0026gt;Index of /\u0026lt;/h1\u0026gt;\u0026lt;hr\u0026gt;\u0026lt;pre\u0026gt;\u0026lt;a href=\u0026#34;../\u0026#34;\u0026gt;../\u0026lt;/a\u0026gt; \u0026lt;a href=\u0026#34;greetings.txt\u0026#34;\u0026gt;greetings.txt\u0026lt;/a\u0026gt; 27-Feb-2019 12:04 13 \u0026lt;/pre\u0026gt;\u0026lt;hr\u0026gt;\u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt;   Retrieve and display the contents of the greetings.txt file\n/ # wget -q -O- nginx/greetings.txt Hello world!   Configuration If you need custom startup options, you can edit the ConfigMap file 15-nginx-configmap.yaml with your desired Nginx configuration settings.\n","excerpt":"Nginx is a popular web server that can be used as a reverse proxy, load balancer or even as a …","ref":"/docs/usecases/nginx/","title":"Nginx"},{"body":"A StorageOS node is any machine (virtual or physical) that is running the StorageOS daemonset pod. A node must be running a daemonset pod in order to consume and/or present storage.\nNodes can be run in several modes.\nHyperconverged Mode By default StorageOS nodes run in hyperconverged mode. This means that the node hosts data from StorageOS volumes and can present volumes to applications.\nA hyperconverged node can store data from a volume and present volumes to applications regardless of whether the data for the volume consumed is placed on that node or is being served remotely. Remote volumes like this are handled by an internal protocol to present block device access to applications running on different nodes from the one to which their backing data store is attached.\nStorageOS implements an extension of a Kubernetes Scheduler object that influences the placement of Pods on the same nodes as their data.\nCompute-only Mode Alternatively, a node can run in computeonly mode, which means no storage is consumed on the node itself and the node only presents volumes hosted by other nodes. Volumes presented to applications running on compute only nodes are therefore all remote. Compute only nodes can be very useful for topologies where nodes are ephemeral and should not host data, but the ephemeral nodes host applications that require StorageOS volumes. The nodes that are not intended to hold data, but just to present StorageOS volumes, can be set as computeonly.\nA node can be marked as compute only at any point in time by adding the label storageos.com/computeonly=true, following the labels reference.\nStorage Mode Finally, nodes can be set to storage mode. Nodes set to storage mode don\u0026rsquo;t present data locally - instead all data is accessed through the network. This topology is enforced by tainting the relevant nodes to ensure that application workloads cannot be scheduled there.\nThis mode is ideal for ensuring maximum stability of data access as the node is isolated from resource drains that may occur due to applications running alongside. For redundancy purposes, in high load clusters it is ideal to have several nodes running in this mode.\n","excerpt":"A StorageOS node is any machine (virtual or physical) that is running the StorageOS daemonset pod. A …","ref":"/docs/concepts/nodes/","title":"Nodes"},{"body":"The following document lists the open source software attributions in the StorageOS Control Plane, Data Plane and CLI.\nCNats The MIT License (MIT) Copyright (c) 2015 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. CppUTest  Copyright (c) 2007, Michael Feathers, James Grenning and Bas Vodde All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the \u0026lt;organization\u0026gt; nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE EARLIER MENTIONED AUTHORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL \u0026lt;copyright holder\u0026gt; BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Hayai Copyright (c) 2011 - Nick Bruun. This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this software. Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely, subject to the following restrictions: The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment in the product documentation would be appreciated but is not required. Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software. If you meet (any of) the author(s), you're encouraged to buy them a beer, a drink or whatever is suited to the situation, given that you like the software. This notice may not be removed or altered from any source distribution. LZ4 LZ4 Library Copyright (c) 2011-2016, Yann Collet All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. LibreSSL  LICENSE ISSUES ============== The OpenSSL toolkit stays under a dual license, i.e. both the conditions of the OpenSSL License and the original SSLeay license apply to the toolkit. See below for the actual license texts. Actually both licenses are BSD-style Open Source licenses. In case of any license issues related to OpenSSL please contact openssl-core@openssl.org. OpenSSL License --------------- ==================================================================== Copyright (c) 1998-2011 The OpenSSL Project. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. All advertising materials mentioning features or use of this software must display the following acknowledgment: \u0026quot;This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/)\u0026quot; 4. The names \u0026quot;OpenSSL Toolkit\u0026quot; and \u0026quot;OpenSSL Project\u0026quot; must not be used to endorse or promote products derived from this software without prior written permission. For written permission, please contact openssl-core@openssl.org. 5. Products derived from this software may not be called \u0026quot;OpenSSL\u0026quot; nor may \u0026quot;OpenSSL\u0026quot; appear in their names without prior written permission of the OpenSSL Project. 6. Redistributions of any form whatsoever must retain the following acknowledgment: \u0026quot;This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/)\u0026quot; THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ==================================================================== This product includes cryptographic software written by Eric Young (eay@cryptsoft.com). This product includes software written by Tim Hudson (tjh@cryptsoft.com). Original SSLeay License ----------------------- Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com) All rights reserved. This package is an SSL implementation written by Eric Young (eay@cryptsoft.com). The implementation was written so as to conform with Netscapes SSL. This library is free for commercial and non-commercial use as long as the following conditions are aheared to. The following conditions apply to all code found in this distribution, be it the RC4, RSA, lhash, DES, etc., code; not just the SSL code. The SSL documentation included with this distribution is covered by the same copyright terms except that the holder is Tim Hudson (tjh@cryptsoft.com). Copyright remains Eric Young's, and as such any Copyright notices in the code are not to be removed. If this package is used in a product, Eric Young should be given attribution as the author of the parts of the library used. This can be in the form of a textual message at program startup or in documentation (online or textual) provided with the package. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. All advertising materials mentioning features or use of this software must display the following acknowledgement: \u0026quot;This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)\u0026quot; The word 'cryptographic' can be left out if the rouines from the library being used are not cryptographic related :-). 4. If you include any Windows specific code (or a derivative thereof) from the apps directory (application code) you must include an acknowledgement: \u0026quot;This product includes software written by Tim Hudson (tjh@cryptsoft.com)\u0026quot; THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The licence and distribution terms for any publically available version or derivative of this code cannot be changed. i.e. this code cannot simply be copied and put under another distribution licence [including the GNU Public Licence.] protobuf This license applies to all parts of Protocol Buffers except the following: - Atomicops support for generic gcc, located in src/google/protobuf/stubs/atomicops_internals_generic_gcc.h. This file is copyrighted by Red Hat Inc. - Atomicops support for AIX/POWER, located in src/google/protobuf/stubs/atomicops_internals_power.h. This file is copyrighted by Bloomberg Finance LP. Copyright 2014, Google Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Code generated by the Protocol Buffer compiler is owned by the owner of the input file used when generating it. This code is not standalone and requires a support library to be linked with it. This support library is itself covered by the above license. rocksdb BSD License For rocksdb software Copyright (c) 2011-present, Facebook, Inc. All rights reserved. --------------------------------------------------------------------- Copyright (c) 2011 The LevelDB Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/armon/go-metrics The MIT License (MIT) Copyright (c) 2013 Armon Dadgar Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/armon/go-radix The MIT License (MIT) Copyright (c) 2014 Armon Dadgar Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/bgentry/speakeasy  Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. \u0026quot;License\u0026quot; shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. \u0026quot;Licensor\u0026quot; shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. \u0026quot;Legal Entity\u0026quot; shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \u0026quot;control\u0026quot; means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) shall mean an individual or Legal Entity exercising permissions granted by this License. \u0026quot;Source\u0026quot; form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. \u0026quot;Object\u0026quot; form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. \u0026quot;Work\u0026quot; shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). \u0026quot;Derivative Works\u0026quot; shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. \u0026quot;Contribution\u0026quot; shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \u0026quot;submitted\u0026quot; means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \u0026quot;Not a Contribution.\u0026quot; \u0026quot;Contributor\u0026quot; shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a \u0026quot;NOTICE\u0026quot; text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets \u0026quot;[]\u0026quot; replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same \u0026quot;printed page\u0026quot; as the copyright notice for easier identification within third-party archives. Copyright [2013] [the CloudFoundry Authors] Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. github.com/dgrijalva/jwt-go Copyright (c) 2012 Dave Grijalva Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/dgrijalva/jwt-go/request Copyright (c) 2012 Dave Grijalva Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/dgryski/go-metro MetroHash This package is a mechanical translation of the reference C++ code for MetroHash, available at https://github.com/jandrewrogers/MetroHash I claim no additional copyright over the original implementation. The MIT License (MIT) Copyright (c) 2015 J. Andrew Rogers Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/docker/docker/pkg/mount  Apache License Version 2.0, January 2004 https://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. \u0026quot;License\u0026quot; shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. \u0026quot;Licensor\u0026quot; shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. \u0026quot;Legal Entity\u0026quot; shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \u0026quot;control\u0026quot; means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) shall mean an individual or Legal Entity exercising permissions granted by this License. \u0026quot;Source\u0026quot; form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. \u0026quot;Object\u0026quot; form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. \u0026quot;Work\u0026quot; shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). \u0026quot;Derivative Works\u0026quot; shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. \u0026quot;Contribution\u0026quot; shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \u0026quot;submitted\u0026quot; means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \u0026quot;Not a Contribution.\u0026quot; \u0026quot;Contributor\u0026quot; shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a \u0026quot;NOTICE\u0026quot; text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS Copyright 2013-2016 Docker, Inc. Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. github.com/docker/libkv  Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. \u0026quot;License\u0026quot; shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. \u0026quot;Licensor\u0026quot; shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. \u0026quot;Legal Entity\u0026quot; shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \u0026quot;control\u0026quot; means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) shall mean an individual or Legal Entity exercising permissions granted by this License. \u0026quot;Source\u0026quot; form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. \u0026quot;Object\u0026quot; form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. \u0026quot;Work\u0026quot; shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). \u0026quot;Derivative Works\u0026quot; shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. \u0026quot;Contribution\u0026quot; shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \u0026quot;submitted\u0026quot; means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \u0026quot;Not a Contribution.\u0026quot; \u0026quot;Contributor\u0026quot; shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a \u0026quot;NOTICE\u0026quot; text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS Copyright 2014-2016 Docker, Inc. Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. github.com/docker/libkv/store  Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. \u0026quot;License\u0026quot; shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. \u0026quot;Licensor\u0026quot; shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. \u0026quot;Legal Entity\u0026quot; shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \u0026quot;control\u0026quot; means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) shall mean an individual or Legal Entity exercising permissions granted by this License. \u0026quot;Source\u0026quot; form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. \u0026quot;Object\u0026quot; form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. \u0026quot;Work\u0026quot; shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). \u0026quot;Derivative Works\u0026quot; shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. \u0026quot;Contribution\u0026quot; shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \u0026quot;submitted\u0026quot; means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \u0026quot;Not a Contribution.\u0026quot; \u0026quot;Contributor\u0026quot; shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a \u0026quot;NOTICE\u0026quot; text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS Copyright 2014-2016 Docker, Inc. Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. github.com/fsnotify/fsnotify Copyright (c) 2012 The Go Authors. All rights reserved. Copyright (c) 2012 fsnotify Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/golang/protobuf/proto Go support for Protocol Buffers - Google's data interchange format Copyright 2010 The Go Authors. All rights reserved. https://github.com/golang/protobuf Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/gorilla/context Copyright (c) 2012 Rodrigo Moraes. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/gorilla/handlers Copyright (c) 2013 The Gorilla Handlers Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/gorilla/mux Copyright (c) 2012 Rodrigo Moraes. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/gorilla/websocket Copyright (c) 2013 The Gorilla WebSocket Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/hashicorp/consul/api Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/errwrap Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/go-cleanhttp Mozilla Public License, version 2.0 1. Definitions 1.1. \u0026quot;Contributor\u0026quot; means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. \u0026quot;Contributor Version\u0026quot; means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor's Contribution. 1.3. \u0026quot;Contribution\u0026quot; means Covered Software of a particular Contributor. 1.4. \u0026quot;Covered Software\u0026quot; means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. \u0026quot;Incompatible With Secondary Licenses\u0026quot; means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. \u0026quot;Executable Form\u0026quot; means any form of the work other than Source Code Form. 1.7. \u0026quot;Larger Work\u0026quot; means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. \u0026quot;License\u0026quot; means this document. 1.9. \u0026quot;Licensable\u0026quot; means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. \u0026quot;Modifications\u0026quot; means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. \u0026quot;Patent Claims\u0026quot; of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. \u0026quot;Secondary License\u0026quot; means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. \u0026quot;Source Code Form\u0026quot; means the form of the work preferred for making modifications. 1.14. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) means an individual or a legal entity exercising rights under this License. For legal entities, \u0026quot;You\u0026quot; includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, \u0026quot;control\u0026quot; means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party's modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients' rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients' rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an \u0026quot;as is\u0026quot; basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party's negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party's ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - \u0026quot;Incompatible With Secondary Licenses\u0026quot; Notice This Source Code Form is \u0026quot;Incompatible With Secondary Licenses\u0026quot;, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/go-msgpack/codec Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/hashicorp/go-multierror Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/memberlist Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/serf/coordinate Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/serf/serf Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/kelseyhightower/envconfig Copyright (c) 2013 Kelsey Hightower Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/mattn/go-isatty Copyright (c) Yasuhiro MATSUMOTO \u0026lt;mattn.jp@gmail.com\u0026gt; MIT License (Expat) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/miekg/dns Extensions of the original work are copyright (c) 2011 Miek Gieben As this is fork of the official Go code the same license applies: Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/mitchellh/cli Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/mitchellh/mapstructure The MIT License (MIT) Copyright (c) 2013 Mitchell Hashimoto Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/conf The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/logger The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/server The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/server/pse The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/util The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/nats The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/nats/encoders/builtin The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/nuid The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/pborman/uuid Copyright (c) 2009,2014 Google Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/Sirupsen/logrus The MIT License (MIT) Copyright (c) 2014 Simon Eskildsen Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/thejerf/suture Copyright (c) 2014-2015 Barracuda Networks, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/urfave/negroni The MIT License (MIT) Copyright (c) 2014 Jeremy Saenz Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. golang.org/x/crypto/bcrypt Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/crypto/blowfish Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/net/context Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/sys/unix Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/sys/windows Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/sys/windows/registry Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/sys/windows/svc/eventlog Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/time/rate Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. gopkg.in/yaml.v2 Copyright 2011-2016 Canonical Ltd. Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ","excerpt":"The following document lists the open source software attributions in the StorageOS Control Plane, …","ref":"/docs/reference/open_source_attribution/","title":"StorageOS Open Source Software Attribution Notice"},{"body":"StorageOS V2 supports Openshift v4.\nOpenShift and StorageOS communicate with each other to perform actions such as creation, deletion and mounting of volumes through CSI. The CSI container running in the StorageOS Daemonset creates a Linux socket that allows the communication between OpenShift and StorageOS\nInstallation StorageOS v2 supports OpenShift 4.0, 4.1, 4.2, 4.3, 4.4 and 4.5.\nTo install StorageOS on OpenShift, please follow our installation instructions page.\n N.B. Openshift 4 uses the CRI-O container runtime that sets a default PID limit of 1024. StorageOS recommends that the limit be raised to 32768. Please see our prerequisites for more details.\n OpenShift Upgrades OpenShift provides an upgrade operator that automates the process of orchestrator version changes.\nThis procedure can cause StorageOS to malfunction due to sequential node restarts not taking the presence of stateful application data into consideration. To avoid this issue, make sure all stateful workloads using StorageOS Volumes are stopped - usually by scaling StatefulSets to 0. Please contact StorageOS support for further advice if required.\n OpenShift requires the internal registry to be available during the upgrade, however StorageOS volumes may not be available. Therefore using StorageOS for the internal registry is not recommended.\n CSI (Container Storage Interface) Note CSI is the standard that enables storage drivers to release on their own schedule. This allows storage vendors to upgrade, update, and enhance their drivers without the need to update Kubernetes source code, or follow Kubernetes release cycles. StorageOS v2 uses CSI to implement communication with the OpenShift controlplane.\nStorageOS PersistentVolumeClaims The user can provide standard PVC definitions and StorageOS will dynamically provision them. StorageOS presents volumes to containers with standard POSIX mount targets. This enables the Kubelet to mount StorageOS volumes using standard linux device files. Checkout device presentation for more details.\n","excerpt":"StorageOS V2 supports Openshift v4.\nOpenShift and StorageOS communicate with each other to perform …","ref":"/docs/platforms/openshift/","title":"OpenShift"},{"body":"StorageOS policies are a way to control user and group access to StorageOS Namespaces. To grant a user or group access to a namespace, a policy needs to be created mapping the user or group to the namespace.\n Note: Users always have access to the default namespace\n For more information on how to use policies, see the Policies operations page.\n","excerpt":"StorageOS policies are a way to control user and group access to StorageOS Namespaces. To grant a …","ref":"/docs/concepts/policies/","title":"Policies"},{"body":"Policies control access to StorageOS namespaces. Policies can be configured at the group or user level so access can be controlled granularly.\nUsers can belong to one or more groups to control their namespace permissions. Additionally user specific policies can be created to grant a user access to a namespace. Users can belong to any number of groups and have any number of user level policies configured.\n Note: Users are created with access to the default namespace. Policies cannot be applied to the default namespace.\n Managing Policies To start creating policies, at least one custom namespace and user are required. For more information on how to create namespaces see our Namespace guide, for users see our Users CLI reference.\nIn order to create a policy navigate to \u0026ldquo;Policies\u0026rdquo; in the GUI and select \u0026ldquo;Create Policy\u0026rdquo;. A policy controls access to a variety of StorageOS resources and is applied to a user, by placing the user in the policies group.\nIn order to delete a policy, all users must be removed from the policy group before deletion of the policy can be completed.\n","excerpt":"Policies control access to StorageOS namespaces. Policies can be configured at the group or user …","ref":"/docs/operations/policies/","title":"Policies"},{"body":"PostgreSQL or \u0026ldquo;Postgres\u0026rdquo; is an open source object-relational database management system (ORDBMS).\nPostgres is deployed across a wide variety of platforms with a mix of workloads ranging from small, single-node use cases to large Internet-facing clusters with many concurrent users.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. If you need to setup StorageOS on Kubernetes then please see our guide to [installing StorageOS on Kubernetes]({% link \u0026ldquo;docs/install/kubernetes.md\u0026rdquo; \u0026gt;}}).\nDeploying PostgreSQL on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases PersistentVolumeClaim and Pod definition excerpts\nkind:PersistentVolumeClaimmetadata:name:pg-dataannotations:volume.beta.kubernetes.io/storage-class:fast...kind:Podmetadata:name:postgressspec:securityContext:fsGroup:26containers:- name:pgimage:crunchydata/crunchy-postgres:centos7-10.4-1.8.3volumeMounts:- mountPath:/pgdataname:data...volumes:- name:datapersistentVolumeClaim:claimName:pg-dataThis excerpt is from the PersistentVolumeClaim and Pod definition. The pod definition references the pg-data VolumeClaim so storage is dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a Volume Claim.\n  Move into the PostgreSQL examples folder and create the objects\ncd storageos-usecases kubectl create -f ./postgres   Confirm PostgreSQL is up and running.\n$ kubectl get pod postgres-0 -w NAME READY STATUS RESTARTS AGE postgres-0 1/1 Running 0 1m   Connect to the PostgreSQL client pod and connect to the PostgreSQL server through the service.\n$ kubectl exec -it postgres-0 -- psql -h postgres-0.postgres -U primaryuser postgres -c \u0026#34;\\l\u0026#34; Password for user primaryuser: password List of Databases Name | Owner | Encoding | Collate | Ctype | Access privileges +=========================================================================+ postgres | postgres | SQL_ASCII | C | C | template0 | postgres | SQL_ASCII | C | C | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | SQL_ASCII | C | C | =c/postgres + | | | | | postgres=CTc/postgres userdb | postgres | SQL_ASCII | C | C | =Tc/postgres + | | | | | postgres=CTc/postgres+ | | | | | testuser=CTc/postgres (4 rows) The password for the primary user is password. You can see this is set in the ConfigMap file.\n  Configuration If you need custom startup options, you can edit the ConfigMap file 15-postgresd-configmap.yaml with your desired PostgreSQL configuration settings.\n","excerpt":"PostgreSQL or \u0026ldquo;Postgres\u0026rdquo; is an open source object-relational database management system …","ref":"/docs/usecases/postgres/","title":"PostgreSQL"},{"body":"Prometheus is a popular application used for event monitoring and alerting in Kubernetes.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Prometheus on Kubernetes This is the Prometheus use case for StorageOS. Following are the steps for creating a Prometheus StatefulSet and using StorageOS to provide persistent storage.\n  You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases Prometheus Custom Resource definition\napiVersion:monitoring.coreos.com/v1kind:Prometheusmetadata:name:prometheus-storageoslabels:app:prometheus-operatorspec:...storage:volumeClaimTemplate:metadata:name:datalabels:env:prodspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:storageos-replicatedresources:requests:storage:1GiThis excerpt is from the Prometheus Custom Resource definition. This file contains the VolumeClaimTemplate that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs due to the VolumeClaimTemplate in the Prometheus StatefulSet. The Prometheus StatefulSet is created by the Prometheus Operator, triggered by the creation of the Prometheus resource.\n  Move into the Prometheus examples folder and install the Prometheus Operator and create a Prometheus resource.\n$ cd storageos-usecases/prometheus $ ./install-prometheus.sh   Confirm Prometheus is up and running.\n$ kubectl get pods -w -l app=prometheus NAME READY STATUS RESTARTS AGE prometheus-prometheus-storageos-0 3/3 READY 0 1m   You can see the created PVC using.\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-prometheus-prometheus-storageos-0 Bound pvc-b6c17c0a-e76b-4a0b-8fc6-46c0e1629210 1Gi RWO storageos-replicated 65m   In the Prometheus deployment script, a service monitor is created. A Service Monitor is a special object that the Prometheus operator uses to create configuration for Endpoints for Prometheus to scrape. Although the name implies that a Service Monitor defines Kubernetes services that will be scraped, Prometheus actually targets the Endpoints that the services point to. The new Prometheus instance will use the storageos-etcd service monitor to start scraping metrics from the storageos-etcd pods. Assuming the storageos cluster was setup using ETCD as pods. For more information about service monitors, have a look at the upstream documentation.\n$ kubectl get servicemonitor NAME AGE storageos-etcd 5d1h   The Prometheus web ui can be accessed by port-forwarding the Prometheus pods port to localhost.\n$ kubectl port-forward prometheus-prometheus-storageos-0 9090 Then launch a web browser and go to localhost:9090 to access the Prometheus web ui. You can confirm that the storageos-etcd target is configured there.\n  Configuration In the storageos-usecases/prometheus/manifests/prometheus directory there are other example Service Monitors. For more information about Prometheus, check out the prometheus documentation.\n","excerpt":"Prometheus is a popular application used for event monitoring and alerting in Kubernetes.\nBefore you …","ref":"/docs/usecases/prometheus/","title":"Prometheus"},{"body":"To install StorageOS on Rancher, please follow our installation instructions page.\nStorageOS transparently supports Rancher deployments using CentOS, RHEL or Debian and can support other Linux distributions detailed in the systems supported page if the appropriate kernel modules are present.\n","excerpt":"To install StorageOS on Rancher, please follow our installation instructions page.\nStorageOS …","ref":"/docs/platforms/rancher/","title":"Rancher"},{"body":" Please note: StorageOS Project edition is required to create RWX Volumes.\n StorageOS supports ReadWriteMany (RWX) access mode Persistent Volumes. A RWX PVC can be used simultaneously by many Pods in the same Kubernetes namespace for read and write operations.\nStorageOS RWX Volumes are based on a shared filesystem - in the case of our implementation, this is NFS.\nArchitecture For each RWX Volume, the following components are involved:\nStorageOS ReadWriteOnly (RWO) Volume\nStorageOS provisions a standard Volume that provides a block device for the file system of the NFS server. This means that every RWX Volume has its own RWO Volume. This allows RWX Volumes to leverage the synchronous replication and automatic failover functionality of StorageOS, providing the NFS server with high availability.\nNFS-Ganesha server\nFor each RWX Volume, an NFS-Ganesha server is spawned by StorageOS. The NFS server runs in user space on the Node containing the primary Volume. Each NFS server uses its own RWO Volume to store data so the data of each Volume is isolated.\nStorageOS binds an ephemeral port to the host network interface for each NFS-Ganesha server. The NFS export is presented using NFS v4.2. Check the prerequisites page to see the range of ports needed for StorageOS RWX Volumes.\nStorageOS API Manager\nStorageOS fully integrates with Kubernetes. The StorageOS API Manager Pod monitors StorageOS RWX Volumes to create and maintain a Kubernetes Service that points towards each RWX Volume\u0026rsquo;s NFS export endpoint. The API Manager is responsible for updating the Service endpoint when a RWX Volume failover occurs.\nProvisioning and using RWX PVCs The sequence in which a RWX PVC is provisioned and used is as follows:\n A PersistentVolumeClaim (PVC) is created with RWX access mode using any StorageOS StorageClass. StorageOS dynamically provisions the PV. A new StorageOS RWO Volume is provisioned internally (not visible in Kubernetes). When the RWX PVC is consumed by a pod, an NFS-Ganesha server is instantiated on the same Node as the primary Volume. The NFS-Ganesha server thus uses the RWO StorageOS Volume as its back end disk. The StorageOS API Manager publishes the host IP and port for the NFS service endpoint, by creating a Kubernetes Service that points to the NFS-Ganesha server export endpoint. StorageOS issues a NFS mount on the Node where the Pod using the PVC is scheduled.  High availability RWX Volumes failover in the same way as standard RWO StorageOS Volumes. The replica Volume is promoted upon detection of Node failure and the NFS-Ganesha server is started on the Node containing the promoted replica. The StorageOS API Manager updates the endpoint of the Volume\u0026rsquo;s NFS service, causing traffic to be routed to the URL of the new NFS-Ganesha server. The NFS client in the application Node (where the user\u0026rsquo;s Pod is running) automatically reconnects.\nNotes  All feature labels that work on RWO Volumes will also work on RWX Volumes. A StorageOS RWX Volume is matched one-to-one with a PVC. Therefore the StorageOS RWX Volume can only be accessed by Pods in the same Kubernetes namespace. StorageOS RWX Volumes support volume resize. Refer to the resize documentation for more details.  ","excerpt":"Please note: StorageOS Project edition is required to create RWX Volumes.\n StorageOS supports …","ref":"/docs/concepts/rwx/","title":"ReadWriteMany"},{"body":" Please note: StorageOS Project edition is required to create RWX Volumes.\n StorageOS supports ReadWriteMany (RWX) access mode Persistent Volumes. A RWX PVC can be used simultaneously by many Pods in the same Kubernetes namespace for read and write operations.\nStorageOS RWX Volumes are based on a shared filesystem.\nTo create a ReadWriteMany (RWX) volume with StorageOS, create a Persistent Volume Claim (PVC) with an access mode of ReadWriteMany (see the First PVC documentation for examples of creating standard PVCs with StorageOS).\nThe following YAML manifest files provide an example:\nA 5Gi PVC with the StorageOS storageClassName of fast, with an accessMode of ReadWriteMany:\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:pvc-rwxspec:storageClassName:fastaccessModes:- ReadWriteManyresources:requests:storage:5GiA Deployment of 3 Pods, each of which consume this PVC:\napiVersion:apps/v1kind:Deploymentmetadata:name:sharedlabels:app:sharedspec:replicas:3selector:matchLabels:app:sharedtemplate:metadata:labels:app:sharedspec:containers:- name:debianimage:debian:9-slimcommand:[\u0026#34;/bin/sleep\u0026#34;]args:[\u0026#34;3600\u0026#34;]volumeMounts:- mountPath:/mnt/name:v1volumes:- name:v1persistentVolumeClaim:claimName:pvc-rwxAfter creating the above resources, the PVC (here named pvc-rwx) should be bound and show an access mode of RWX:\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-rwx Bound pvc-59f7a152-8342-415c-a6ca-1cbb463410ab 5Gi RWX fast 60s Ensure that the Deployment Pods are running:\n$ kubectl get pods NAME READY STATUS RESTARTS AGE shared-1771418926-7o5ns 1/1 Running 0 1m shared-1771418926-r18az 1/1 Running 0 1m shared-1771418926-ds8f7 1/1 Running 0 1m The NFS-Ganesha service that exposes the StorageOS volume as RWX can now be viewed, showing the cluster IP of the service, and the default NFS port (2049).\n$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 \u0026lt;none\u0026gt; 443/TCP 120m pvc-59f7a152-8342-415c-a6ca-1cbb463410ab ClusterIP 10.107.10.0 \u0026lt;none\u0026gt; 2049/TCP 1m Features of the NFS volume can also be examined in the StorageOS UI. Under Volumes, the Attachment column shows an nfs tag. The Volume Details section provides information about the NFS Volume, such as the service endpoint and the node on which the underlying Volume is attached.\n","excerpt":"Please note: StorageOS Project edition is required to create RWX Volumes.\n StorageOS supports …","ref":"/docs/operations/rwx/","title":"ReadWriteMany"},{"body":"Redis is a popular networked, in-memory, key-value data store with optional durability to disk.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Redis on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion\nkind:StatefulSetmetadata:name:redisspec:selector:matchLabels:app:redisenv:prodserviceName:redisreplicas:1...spec:serviceAccountName:redis...volumeMounts:- name:datamountPath:/bitnami/redis/data...volumeClaimTemplates:- metadata:name:datalabels:env:prodspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClass resources:requests:storage:5GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a Volume Claim.\n  Move into the Redis examples folder and create the objects\ncd storageos-usecases kubectl create -f ./redis   Confirm Redis is up and running.\n$ kubectl get pods -w -l app=redis NAME READY STATUS RESTARTS AGE redis-0 1/1 Running 0 1m   Connect to the Redis client pod and connect to the Redis server through the service\n$ kubectl exec -it redis-0 -- redis-cli -a password Warning: Using a password with \u0026#39;-a\u0026#39; option on the command line interface may not be safe. 127.0.0.1:6379\u0026gt; CONFIG GET maxmemory 1) \u0026#34;maxmemory\u0026#34; 2) \u0026#34;0\u0026#34;   Configuration If you need custom startup options, you can edit the ConfigMap file 15-redis-configmap.yaml with your desired Redis configuration settings.\n","excerpt":"Redis is a popular networked, in-memory, key-value data store with optional durability to disk. …","ref":"/docs/usecases/redis/","title":"Redis"},{"body":"We recommend always using \u0026ldquo;tagged\u0026rdquo; versions of StorageOS rather than \u0026ldquo;latest\u0026rdquo;, and to perform upgrades only after reading the release notes.\nThe latest tagged release is v2.4.0. For installation instructions please see our Install page.\nThe latest CLI release is v2.4.0, available from Github.\nUpgrading To upgrade from version 1.x to 2.x, please contact support for assistance.\nv2.4.0 - Released 2021-05-27 This release adds production-grade encryption at rest  for StorageOS volumes, as well as:\n Fencing TRIM Failure modes Kubernetes object sync  Note: v2.4.0 requires Kubernetes 1.17 or newer.\nNew  Volume encryption-at-rest. Fencing support. Block trim support. Kubernetes label sync. Kubernetes node and namespace delete sync. Failure tolerance threshold support.  Fixed   controlplane/api: Compression is not disabled by default when provisioning volumes via the API.\n  controlplane/api: Spec has incorrect response body for partial bundle.\n  controlplane/csi: Error incorrectly returned when concurrent namespace creation requests occur.\n  controlplane/diagnostics: GetDiagnostics RPC response does not indicate if node timed out collecting some data.\n  controlplane/diagnostics: Invalid character \u0026lsquo;\\u0080\u0026rsquo; looking for beginning of value via CLI when a node is down.\n  controlplane/diagnosticutil: Include attachment type in unpacking local volumes.\n  controlplane/diagnotics: Node timing out during local diagnostics is missing logs.\n  controlplane/healthcheck: Combined sources fires callback in initialisation.\n  controlplane/volumerpc: \u0026ldquo;Got unknown replica state 0\u0026rdquo; discards results.\n  dataplane/fix: Check blob writes don\u0026rsquo;t exceed internal limit.\n  dataplane/fix: Checking the return code of InitiatorAddConnection().\n  dataplane/fix: Director signal hander thread is not joined.\n  dataplane/fix: Don\u0026rsquo;t block I/O when many retries are in progress.\n  dataplane/fix: gRPC API robustness improvements.\n  dataplane/fix: Initiator needs to include the node UUID in Endpoint.\n  dataplane/fix: Low-level I/O engines don\u0026rsquo;t propagate IO failures via Wait().\n  dataplane/fix: Log available contextual information where possible.\n  dataplane/fix: Ensure BackingStore is not deleted twice.\n  dataplane/fix: Serialise LIO create/delete operations to avoid kernel bug.\n  dataplane/fix: Dataplane shutdown time can exceed 10 seconds.\n  dataplane/fix: Fix non-threadsafe access on TCMU device object.\n  dataplane/fix: Don\u0026rsquo;t hold lock unecessarily in Rdb::Reap.\n  k8s/api-manager: First ip octet should not be 0, 127, 169 or 224.\n  k8s/api-manager: Keygen should only operate on StorageOS PVCs.\n  k8s/cluster-operator: Add perm to allow VolumeAttachment finalizer removal.\n  k8s/cluster-operator: Fix apiManagerContainer tag in v1 deploy CRD.\n  k8s/cluster-operator: Fix docker credentials check.\n  k8s/cluster-operator: Fix ServiceAccountName in the OLM bundle.\n  k8s/cluster-operator: Set webhook service-for label to be unique.\n  Improved   controlplane/api: Make version provided consistent for NFS/Host attach handler.\n  controlplane/attachtracker: Cleanup NFS mounts at shutdown.\n  controlplane/build: Migrate to go modules for dependency management.\n  controlplane/build: Use sentry prod-url if build branch has \u0026ldquo;release\u0026rdquo; prefix.\n  controlplane/cli: Colour for significant feedback.\n  controlplane/cli: Update node must set compute only separately to other labels.\n  controlplane/cli: Warn user that updating labels action can be reverted.\n  controlplane/csi: Bound request handlers with timeout similar to HTTP API.\n  controlplane/csi: Remove error obfuscation and clarify log messages.\n  controlplane/csi: Stop logging not found.\n  controlplane/dataplane: Remove UUID mappings during failed presentation creation rollback.\n  controlplane/dataplaneevents: Decorate logs with extra event details.\n  controlplane/diagnostics: Asymmetrically encrypt bundles.\n  controlplane/diagnostics: Collect FSM state.\n  controlplane/diagnostics: Support single node bundle collection.\n  controlplane/diagnosticutil: Decorate log entries with well-known field corresponding to node id/name.\n  controlplane/diagnosticutil: Parallelise unpacking of disjoint data.\n  controlplane/diagnosticutil: Unpack gathered NFS config data.\n  controlplane/fsm: Perform state match check before version check.\n  controlplane/k8s: Use secret store.\n  controlplane/log: Fix race condition writing logs.\n  controlplane/log: Handle originator timestamps from dataplane logs.\n  controlplane/meta: Error checking code uses Go 1.13 error features.\n  controlplane/rpc: Make CP gRPC calls to the DP configuration endpoints idempotent.\n  controlplane/sentry: Prevent some unnecessary alerts.\n  controlplane/slog: Clean up error logging in RPC provision stack.\n  controlplane/states: Add the \u0026ldquo;from\u0026rdquo; state as a log field for state transition msgs.\n  controlplane/store/etcd: Decorate lock logs with associated ID fields.\n  controlplane/ui: Warn user that updating labels action will be reverted.\n  controlplane/vendor: Bump service repo.\n  controlplane/volume: Encryption support in kubernetes.\n  dataplane/fs: Don\u0026rsquo;t return from PresentationCreate RPC until the device is fully created.\n  dataplane/fs: Each LUN should have it\u0026rsquo;s own HBA.\n  dataplane/fs: Improve device ready check.\n  dataplane/internals: Improve DP stats implementation.\n  dataplane/internals: Major director refactor.\n  dataplane/log: Logs should output originating timestamps.\n  dataplane/log: Move to log3 API exclusively.\n  dataplane/log: Remove log2.\n  dataplane/log: Set log_level and log_filter via the supctl tool.\n  dataplane/rdb: Handle unaligned I/O in RdbPlugin.\n  dataplane/rdb: Implement low-level \u0026ldquo;delete block\u0026rdquo; functionality.\n  dataplane/rdb: rocksdb Get() should use an iterator.\n  dataplane/story: Support for block unmapping.\n  dataplane/story: Add backuptool binary to export volume data.\n  dataplane/story: Volume encryption-at-rest.\n  dataplane/sync: Add retries for failed sync IOs.\n  dataplane/sync: VolumeHash performance improvements.\n  dataplane/sys: Find and check OS pids.max on startup.\n  k8s/api-manager: Don\u0026rsquo;t attempt service creation if the owning PVC doesn\u0026rsquo;t exist.\n  k8s/api-manager: Compare SC and PVC creation time during label sync.\n  k8s/api-manager: Add action to ensure modules tidy \u0026amp; vendored.\n  k8s/api-manager: Add defaults from StorageClass.\n  k8s/api-manager: Add fencing controller.\n  k8s/api-manager: Add flag and support for cert validity.\n  k8s/api-manager: Add flags to disable label sync controllers.\n  k8s/api-manager: Add namespace delete controller.\n  k8s/api-manager: Add node delete controller.\n  k8s/api-manager: Add OpenTelemetry tracing with Jaeger backend.\n  k8s/api-manager: Add PVC label sync controller.\n  k8s/api-manager: Add PVC mutating controller.\n  k8s/api-manager: Add support for failure-mode label.\n  k8s/api-manager: Add support for volume encryption.\n  k8s/api-manager: Allow multiple mutators.\n  k8s/api-manager: Build and tests should use vendored deps.\n  k8s/api-manager: Bump controller-runtime to v0.6.4.\n  k8s/api-manager: Encrypt only provisioned PVCs.\n  k8s/api-manager: Fix tracing example.\n  k8s/api-manager: Introduce StorageClass to PVC annotation mutator.\n  k8s/api-manager: Log api reason.\n  k8s/api-manager: Migrate namespace delete to operator toolkit.\n  k8s/api-manager: Migrate node delete to operator toolkit.\n  k8s/api-manager: Migrate to kubebuilder v3.\n  k8s/api-manager: Node label sync.\n  k8s/api-manager: Node label update must include current reserved labels.\n  k8s/api-manager: Pass context to api consistently.\n  k8s/api-manager: Rename leader election config map.\n  k8s/api-manager: RFC 3339 and flags to configure level \u0026amp; format.\n  k8s/api-manager: Run shared volume controller with manager.\n  k8s/api-manager: Set initial sync delay.\n  k8s/api-manager: Set Pod scheduler.\n  k8s/api-manager: Standardise on ObjectKeys for all API function signatures.\n  k8s/api-manager: StorageOS API interface and mocks.\n  k8s/api-manager: Update dependencies and go version 1.16.\n  k8s/api-manager: Update to new external object sync.\n  k8s/api-manager: Use composite client in admission controllers.\n  k8s/api-manager: Use Object interface.\n  k8s/cluster-operator: Changes to the StorageOSCluster CR get applied to StorageOS.\n  k8s/cluster-operator: Increase provisioner timeout from 15 to 30s.\n  k8s/cluster-operator: Reduce CSI provisioner worker pool.\n  k8s/cluster-operator: Set priority class for helper pods.\n  k8s/cluster-operator: Add pod anti-affinity to api-manager.\n  k8s/cluster-operator: Add pvc mutator config.\n  k8s/cluster-operator: Add rbac for api-manager fencing.\n  k8s/cluster-operator: Add RBAC for encryption key management.\n  k8s/cluster-operator: Add RBAC needed for csi-resizer v1.0.0.\n  k8s/cluster-operator: Add webhook resource migration.\n  k8s/cluster-operator: Add workflow to push image to RedHat registry.\n  k8s/cluster-operator: Bump csi-provisioner to v2.1.1.\n  k8s/cluster-operator: Call APIManagerWebhookServiceTest test.\n  k8s/cluster-operator: Delete CSI expand secret when cluster is deleted.\n  k8s/cluster-operator: Docker login to avoid toomanyrequests error.\n  k8s/cluster-operator: Move pod scheduler webhook to api-manager.\n  k8s/cluster-operator: RBAC to allow sync functions move to api-manager.\n  k8s/cluster-operator: Remove pool from StorageClass, not used in v2.\n  k8s/cluster-operator: Remove some other v1.14 specific logic.\n  k8s/cluster-operator: Set the default container for kubectl logs.\n  k8s/cluster-operator: Update code owners.\n  k8s/cluster-operator: Update CSI sidecar images.\n  k8s/cluster-operator: Validate minimum Kubernetes version.\n  v2.3.4 - Released 2021-03-24  controlplane/build: Use Sentry prod-url for release branches (CP-4600). controlplane/csi: Improve CSI handler timeout (CP-4585). controlplane/dataplane: UUID mapping cleanup on failed volume creation (CP-4588). controlplane/slog: Improve RPC error logging (CP-4616). dataplane: Allocate fewer aio contexts per volume (DP-305) dataplane: Defer fallocate(2) until first write (DP-312). dataplane: Don\u0026rsquo;t fail replica sync if inter-node connection establishment is slow (DP-319, DP-280). dataplane: Improve logging around gRPC context cancellations (DP-315). dataplane: Improve rollback for failed volume creation (DP-308). dataplane: New support tool to cleanup orphaned volume storage (DP-307). dataplane: supctl can reap named volumes (DP-309). k8s: API token reset failures should trigger re-authentication directly (#38). k8s: Increase lint timeout to reduce CI errors (#305). k8s: Remove PriorityClass from helper pods (#312). k8s: Toleration defaults for helper pods (#311). k8s: Use ubi-minimal base image directly (#307).  v2.3.3 - Released 2021-02-12  Support CSI ListVolumes() API, addressing volume attach problems seen by some customers. Quality-of-life fixes.  New  operator: Use CSI attacher v3 for k8s 1.17+. controlplane/csi: ListVolumes support.  Fixed  api-manager: Reset api after token refresh error. operator: Set scheduler when PVCs use default StorageClass. operator: Update base container image. controlplane/volumerpc: \u0026ldquo;Got unknown replica state 0\u0026rdquo; discards results. controlplane/healthcheck: Combined sources fires callback in initialisation. controlplane/fsm: Perform state match check before version check.  v2.3.2 - Released 2020-11-25 Fixed  controlplane/rejoin: Failure to delete data causes re-advertise loop. controlplane/rejoin: Handle timeout waiting for progress report. dataplane/log: Change buffering of symmetra output to prevent stalls.  v2.3.1 - Released 2020-11-16  Allows access to ReadWriteMany shared volumes when running containers as a non-root user.  Fixed  nfs: root squash to uid=0 is now configured on all shared volumes.  v2.3.0 - Released 2020-10-31 This release adds production-grade shared file support to v2, previously a technology preview in v1.\nBreaking  The v2.3.0 operator is no longer able to run StorageOS v1.  New  Adds support for ReadWriteMany shared volumes. See ReadWriteMany. Adds api-manager deployment to support shared volumes. See the api manager Github repository for more information. Kubernetes 1.19 support.  Improved  dataplane: Reduce replication thread usage by having the replication processes share the main thread pool. This helps ensure that there isn\u0026rsquo;t a spike in thread usage when a node recovers and begins re-syncing its volumes. This is particularly relevant on CRIO-based orchestrators such as Openshift where the default maximum allowed PID limit (which also governs the thread limit) is low. dataplane: Detect and log the effective maximum PID limit on startup. dataplane: Internal device presentation mappings are now ephemeral and are not persisted across reboots. dataplane: Disabled default verbose logging for fdatasync/flushWAL timers. dataplane: Log both volume inode and UUID in replication error messages for easier correlation. dataplane: On startup, ensure any remnant devices that may have been left after an unclean shutdown have been properly cleared. dataplane: Signal when all startup tasks complete. This ensures no IO can be initiated before this time. ha: Implement a backoff when attempting to repoint an attached volume after the master has failed. ha: Replicas can now rejoin after an asymmetric partition. This can occur when the master has not lost communication to the replica, but the replica can\u0026rsquo;t communicate with the master. Previously the replica would not be able to rejoin until the master determined it had failed. ha: A master that was partitioned can now re-join to the new master as a replica. api: node label changes update target node prior to committing new state. api: Validation errors now include more information on the failure and how to resolve. csi: Volume resize error messages (e.g. capacity exceeded) now passed through in CSI response. csi: Volume attachment is now verified prior to mount for more instructive error message. csi: Returns RESOURCE EXHAUSTED error when attempting to exceed maximum of 250 StorageOS volume attachments per node. diagnostics: Multiple improvements to bundle collection and collected data. ui: Allow collection of partial diagnostics bundles. ui: Tolerate clock skew when authenticating via the UI. licensing: Read-through cache added. Licence updates will take up to 60s to propagate to all nodes. cli: Set replicas output formatting. init: Checks the effective maximum PID limit and warns if less than the StorageOS recommended PID limit (32,768). CRIO-based distributions such as Openshift have a much lower default value (1024). Consult prerequisites for more information.  Fixed  dataplane: Fixes transport endpoint is not connected on startup after an unclean shutdown. csi: Volume unmount requests now succeed when the mountpoint has already been removed by the orchestrator. csi: Volume detach requests now succeed when the volume has already been deleted. Previously the volume would be stuck in Terminating status.  v2.2.0 - Released 2020-08-18 This release focuses on performance. We analysed existing performance characteristics across a variety of real-world use cases and ended up with improvements across the board. Of particular note:\n Sequential reads have improved by up to 130% Sequential writes have improved by up to 737% Random reads have improved by up to 45% Random writes have improved by up to 135% I/O for large block sizes (128K) has improved by up to 353%  We are extremely proud of our performance and we love to talk about it. Have a look at the Benchmarking section of the self-evaluation guide and consider sharing your results. Our PRE engineers are available to discuss in our slack channel.\nNew   Data engine revamp focused on provable consistency and performance. Key characteristics:\n Metadata is stored in an optimised index, lowering I/O latency and improving performance for all workloads. Large block reads/writes are now be handled in a single operation. Applications like Kafka will go much faster.    On-disk compression is now disabled by default as in most scenarios this offers better performance. To enable on-disk compression for a specific workload, see compression.\n  Improved  dataplane: The number of I/O threads are now determined by the number of processing cores available. This improves scalability and performance on larger servers. ha: Improve partition tolerance behaviour when a volume master that has lost its connection to etcd rejoins. ha: Allow replicas in unhealthy states to be remediated and re-used while maintaining partition tolerance. ha: When a master fails and the new master is not yet available, introduce a back-off to the redirection logic to avoid spamming the logs with connection failure errors. ha: Ignore health advertisements for local node. Local nodes are handled directly. node delete: Only refuse to delete a node if the node health can be authoritatively verified to be in use. api: Increase HTTP server write timeout. cli/ui: Allow partial diagnostic bundle downloads. ui: Namespace dropdown can now be scrolled. ui: Add \u0026ldquo;Job title\u0026rdquo; to UI licence form. logging: Log version at startup at INFO level. logging: Lower verbosity of SCSI warnings that do not apply to StorageOS. diagnostics: Include logs that have been rotated. diagnostics: Bundle collection across providers is now done in parallel. build: Update base image to RHEL 8.2. operator: Removed DB migration utility required for v1.3 -\u0026gt; v1.4 upgrades. operator: Automatically refreshes StorageOS API token without failing requests when the token expires. operator: Updated CSI attacher and provisioner to latest upstream. operator: Remove cluster.local suffix on Pod scheduler service address. This allows the scheduler to work in clusters with custom DNS configuration. operator: Defaults are now set for most CSI configuration options in the StorageOSCluster custom resource.  Fixed  csi: When unmount request is received for a volume that has already been unmounted, return success. csi: Verify volume is attached on the node before mounting it. xfs: Support older RHEL kernels which have an XFS library that does not allow reflinks/dedupe. dataplane: Reserve 1GiB of capacity on the target disk to allow manual cleanup operations, rather than filling target disk to capacity. operator: In some cases /var/lib/storageos could fail to unmount cleanly after a restart. This resulted in multiple entries in /proc/mounts.  v2.1.0 - Released 2020-06-26 New  csi: Volume expansion now supported in offline mode. To expand a volume, stop any workloads accessing the volume, then edit the PVC to increase the capacity. For more information, see our Volume Resize operations page and the CSI Volume Expansion page. api: Volume configuration including replica count can now be updated while the volume is in use. Other updateable fields include labels and description. failover: Before determining that a node is offline and performing recovery operations, the I/O path is also verified. This provides more robust failure detection and ensures that nodes that are still responding to I/O do not get replaced. This I/O path verification is in addition to the gossip-based failure detection. operator: Default tolerations are now set for the StorageOS node container. This helps ensure that the StorageOS node container does not get evicted when the node is running low on resources.  Improved  api: Added checks to prevent deletion of a node with active volumes, or if it is the master of at least one volume. This helps prevent orphaned volumes. cli: Add an --offline-delete flag to allow removal of volumes whose master and replica nodes are offline. This allows cleanup of orphaned volumes. ui: Add an offline volume delete option. ui: Volumes can now be detached from the UI. cli: Labels are no longer truncated. api: When a new node is added to the cluster, its capacity is available to use immediately.  Fixed  ui: Favicon was missing. ui: Duplicate volumes could be shown on the node details page. operator: During uninstall a ClusterRoleBinding was not removed.  v2.0.0 - Released 2020-05-05 New  operator: StorageOS containers now run in the kube-system namespace by default to allow the system-node-critical priority class to be set. This instructs Kubernetes to start StorageOS before application Pods, and to evict StorageOS only after application Pods have finished. This setting was previously recommended in documentation; it is now the default. operator: StorageOS CSI helper containers now run as privileged. This ensures that the CSI endpoint can be seen on systems with SELinux enabled. ui: replication progress for new or re-joining replicas is now displayed. ui: show warning for unlicensed clusters. cli: new commands:  licence management get policy create namespace create policy describe user describe namespace describe policy delete user delete namespace delete policy   licence: removed the default licence expiry date added for v2.0.0-rc.1.  Improved  dataplane: improved retry behaviour for network I/O. cli: \u0026ldquo;get volumes\u0026rdquo; for all namespaces should be done in parallel. cli: help text document config file ui: link node name and get to node details on the volume details page. ui: node details add available capacity spinner. ui: node list remove capacity values / address port. ui: node list show master/replica counts. ui: node list remove edit action. ui: format entity labels. ui: node details link volumes. ui: align buttons for licences. ui: k8s warning in \u0026ldquo;create volume\u0026rdquo; modal. ui: node list remove \u0026ldquo;API\u0026rdquo; from \u0026ldquo;API Address\u0026rdquo; ui: add some details about the Licence on the licence page. api: include valid for duration in login response. licence: restrict nodes which are unregistered after 24 hours. scheduler: return error for namespace/volume not found dataplane: start gRPC threads separately from rest of the supervisor.  Fixed  ui: centre licence types. ui: capacity in ui is per namespace. cli: fail gracefully if missing some output details (i.e. no node exists for id).  v2.0.0-rc.1 - Released 2020-03-31 Initial release of version 2.x. See StorageOS v2.0 Release Blog for details.\n","excerpt":"We recommend always using \u0026ldquo;tagged\u0026rdquo; versions of StorageOS rather than …","ref":"/docs/reference/release_notes/","title":"Release notes"},{"body":"StorageOS replicates volumes across nodes for data protection and high availability. Synchronous replication ensures strong consistency for applications such as databases and Elasticsearch, incurring one network round trip on writes.\nThe basic model for StorageOS replication is of a master volume with distributed replicas. Each volume can be replicated between 0 and 5 times, which are provisioned to 0 to 5 nodes, up to the number of remaining nodes in the cluster.\nIn this diagram, the master volume D was created on node 1, and two replicas, D2 and D3 on nodes 3 and 5.\nWrites that come into D (step 1) are written in parallel to D2 and D3 (step 2). When both replicas and the master acknowledge that the data has been written (step 3), the write operation return successfully to the application (step 4).\nFor most applications, one replica is sufficient (storageos.com/replicas=1).\nAll replication traffic on the wire is compressed using the lz4 algorithm, then streamed over tcp/ip to target port tcp/5703.\nIf the master volume is lost, a replica is promoted to master (D2 or D3 above) and a new replica is created and synced on an available node (Node 2 or 4). This is transparent to the application and does not cause downtime.\nIf a replica volume is lost and there are enough remaining nodes, a new replica is created and synced on an available node. While a new replica is created and being synced, the volume\u0026rsquo;s health will be marked as degraded.\nIf the lost replica comes back online before the new replica has finished synchronizing, then StorageOS will calculate which of the two synchronizing replicas has the smallest difference compared to the master volume and keep that replica. The same holds true if a master volume is lost and a replica is promoted to be the new master. If possible, a new replica will be created and begin to sync. Should the former master come back online it will be demoted to a replica and the replica will the smallest difference to the current master will be kept.\nWhile the replica count is controllable on a per-volume basis, some environments may prefer to set default labels on the StorageClass.\nDelta Sync StorageOS implements a delta sync between a volume master and its replicas. This means that if a replica for a volume goes offline, that when the replica comes back online only the regions with changed blocks need to be synchronized. This optimization reduces the time it takes for replicas to catch up, improving volume resilience. Additionally, it reduces network and IO bandwidth which can reduce costs when running in public clouds.\nFailure Modes StorageOS failure modes offer different guarantees with regards to a volume\u0026rsquo;s mode of operation in the face of replica failure. If the failure mode is not specified it defaults to Hard. Volume failure modes can be dynamically updated at run time.\nHard Hard failure mode requires that the number of declared replicas matches the available number of replicas at all times. If a replica fails StorageOS will attempt creation of a new replica for 90 seconds. After 90s if the old replica is not available and a new replica cannot be provisioned, StorageOS cannot guarantee that the data is stored on the number of multiple nodes requested by the user. StorageOS will therefore set the volume to be read-only.\nIf a volume has gone read-only there are two stages to making it read-write again. Firstly, sufficient replicas must be provisioned to match the desired replica count. Depending on your environment, additional nodes and/or disk capacity may be required for this. Secondly, the volume must be remounted - necessitating pod deletion/recreation in Kubernetes.\nstorageos.com/failure-mode: hard Number of nodes required for hard failure mode\nWhen a node fails, a new replica is provisioned and synced as described above. To ensure that a new replica can always be created, an additional node should be available. To guarantee high availability using storageos.com/failure-mode: hard, clusters using volumes with 1 replica must have at least 3 storage nodes. When using volumes with 2 replicas, at least 4 storage nodes, 3 replicas, 5 nodes, etc.\nMinimum number of storage nodes = 1 (primary) + N (replicas) + 1\nSoft Soft failure mode allows a volume to continue serving I/O even when a replica goes offline and a new replica fails to provision. So long as there are not less than max(1, n-1) available replicas where n is the number of replicas for the volume.\nFor example, if a volume with 2 replicas loses 1 replica, then I/O would continue to be served since 1 replica remaining \u0026gt;= max(1, 1). If a volume with 1 replica loses 1 replica, then I/O would halt after 90 seconds since 0 replicas remaining \u0026lt; max(1, 0).\nstorageos.com/failure-mode: soft Number of nodes required for soft failure mode\nTo ensure that a storageos.com/failure-mode: soft volume is highly available, clusters using volumes with 1 replica must have at least 2 storage nodes. When using volumes with 2 replicas, at least 3 storage nodes, 3 replicas, 3 nodes, etc.\nMinimum number of storage nodes = 1 (primary) + N (replicas)\nThreshold Threshold failure mode allows the user to set the minimum required number of online replicas for a volume. For example for a volume with 2 replicas, setting the threshold to 1 would allow a single replica to be offline, whereas setting threshold to 0 would allow 2 replicas to be offline.\nstorageos.com/failure-mode: (0-5) Number of nodes required for threshold failure mode\nThe minimum number of nodes for a threshold volume is determined by the threshold that is set.\nMinimum number of storage nodes = 1 (primary) + T (threshold)\nAlwaysOn AlwaysOn failure mode allows all replicas for a volume to be offline and keeps the volume writeable. A volume with failure mode AlwaysOn will continue to serve I/O regardless of how many replicas it currently has. This mode should be used with caution as it effectively allows for only a single copy of the data to be available.\nstorageos.com/failure-mode: alwayson Number of nodes required for AlwaysOn failure mode\nA storageos.com/failure-mode: alwayson volume is highly available albeit at the cost of reliability. The minimum node count here is 1 as the loss of all replicas will be tolerated.\nMinimum number of storage nodes = 1 (primary)\nCheck the failure modes operations page to find details about how to use the labels on the Volumes.\n","excerpt":"StorageOS replicates volumes across nodes for data protection and high availability. Synchronous …","ref":"/docs/concepts/replication/","title":"Replication"},{"body":"Managing Resources for StorageOS containers Kubernetes resource requests and limits are two optional Pod properties that allow you to specify how much of a resource a container in a Pod needs or can use. They are two main resources that you can specify requests and limits for, CPU and Memory.\nAs StorageOS is an infrastructure component, the health of other applications depends on being able to write to the StorageOS volumes. As such it is of paramount importance to avoid restarts of the StorageOS DaemonSet Pods. Restarting a StorageOS Pod results in the volumes of the node the StorageOS Pod is running on being marked as Read Only, and causes the failover of primary volumes on that node to their replicas. After a StorageOS Pod restart, once the StorageOS DaemonSet Pod is \u0026ldquo;READY\u0026rdquo;, the application Pods running on the node need to be restarted in order to trigger a mount of the filesystem hosted on the StorageOS volume and resume normal operations. To avoid restarts of the StorageOS main container by Kubernetes due to resource limits being reached, it is recommended to not set resource limits on the StorageOS DaemonSet. In addition to avoiding resource limits, StorageOS uses a high priority class when the DaemonSet is installed in the \u0026lsquo;kube-system\u0026rsquo; namespace. That avoids the DaemonSet Pods of being evicted.\nFor more information about managing resources for containers please see the Kubernetes documentation\nDefining Pod resource requests and reservations To add resource requests and reservations to the StorageOS DaemonSet configure them in the StorageOSCluster resource.\n","excerpt":"Managing Resources for StorageOS containers Kubernetes resource requests and limits are two optional …","ref":"/docs/reference/resource_requests_and_limits/","title":"Resource Requests and Limits"},{"body":"StorageOS has the capacity to influence Kubernetes Pod placement decisions to ensure that Pods are scheduled on the same nodes as their data. This functionality is known as Pod Locality.\nStorageOS grants access to data by presenting, on local or remote nodes, the devices used in a Pod\u0026rsquo;s VolumeMounts. However, it is often the case that it is required or preferred to place the Pod on the node where the StorageOS Primary Volume is located, because IO operations are fastest as a result of minimized network traffic and associated latency. Read operations are served locally and writes require fewer round trips to the replicas of the volume.\nStorageOS automatically enables the use of a custom scheduler for any Pod using StorageOS Volumes. Checkout the Admission Controller reference for more information.\nStorageos Kubernetes Scheduler StorageOS achieves Pod locality by implementing a Kubernetes scheduler extender. The Kubernetes standard scheduler interacts with the StorageOS scheduler when placement decisions need to be made.\nThe Kubernetes standard scheduler selects a set of nodes for a placement decision based on nodeSelectors, affinity rules, etc. This list of nodes is sent to the StorageOS scheduler which sends back the target node where the Pod shall be placed.\nThe StorageOS scheduler logic is provided by a Pod in the Namespace where StorageOS Pods are running.\nScheduling process When a Pod needs to be scheduled, the scheduler collects information about all available nodes and the requirements of the Pod. The collected data is then passed through the Filter phase, during which the scheduler predicates are applied to the node data to decide if the given nodes are compatible with the Pod requirements. The result of the filter consists of a list of nodes that are compatible for the given Pod and a list of nodes that aren\u0026rsquo;t compatible.\nThe list of compatible nodes is then passed to the Prioritize phase, in which the nodes are scored based on attributes such as the state. The result of the Prioritize phase is a list of nodes with their respective scores. The more favorable nodes get higher scores than less favorable nodes. The list is then used by the scheduler to decide the final node to schedule the Pod on.\nOnce a node has been selected, the third phase, Bind, handles the binding of the Pod to the Kubernetes apiserver. Once bound, the kubelet on the node provisions the Pod.\nThe StorageOS scheduler implement Filter and Prioritization phases and leaves binding to the default Kubernetes scheduler.\nAvailable +------------------+ +------------------+ NodeList \u0026amp; Pod | | Filtered NodeList | | Scored Information | | \u0026amp; Pod Information | | NodeList +--------------------\u0026gt;+ Filter +--------------------\u0026gt;+ Prioritize |---------------\u0026gt; | (Predicates) | | (Priorities) | | | | | +------------------+ +------------------+ Scheduling Rules The StorageOS scheduler filters nodes ensuring that the remaining subset fulfill the following prerequisites:\n The node is running StorageOS The node is healthy  ","excerpt":"StorageOS has the capacity to influence Kubernetes Pod placement decisions to ensure that Pods are …","ref":"/docs/reference/scheduler/","title":"StorageOS Scheduler"},{"body":"","excerpt":"","ref":"/search/","title":"Search Results"},{"body":"In this example use case we provide three different strategies for accessing files that have been written to a StorageOS persistent volume.\nIn the following examples the \u0026ldquo;application\u0026rdquo; container is the container main, which has a rsync, Nginx or sftp sidecar container. The StorageOS volume that the application is writing to will be mounted into the sidecar container so files written by the application are available for export. Files can be exported using Nginx as a web file server, transferred using rsync or accessed via SFTP.\nThe files create a stateful set that can be used AFTER a StorageOS cluster has been created. See our guide on how to install StorageOS on Kubernetes for more information.\nClone Repository In order to deploy the examples, clone this repository and use kubectl to create the Kubernetes objects.\n$ git clone https://github.com/storageos/use-cases.git storageos-usecases $ cd storageos-usecases/backup  Before deploying the backup-example stateful set we recommend looking through the examples to understand how the different containers are configured\n Exfiltrating files through HTTP   Deploy the Nginx example\n$ kubectl create -f nginx/ service/backup-example created configmap/nginx-config created statefulset.apps/backup-example created pod/busybox created   Check that a backup-example pod is running\n$ kubectl get pods -w -l app=backup-example-nginx NAME READY STATUS RESTARTS AGE backup-example-0 1/1 Running 0 1m   Exec into the main container and write some data to a file\n$ kubectl exec -it backup-example-nginx-0 -c main bash root@backup-example-0:/# echo $(date) \u0026gt; /data/date.txt   Check that the service exists\n$ kubectl get svc backup-example-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE backup-example-nginx ClusterIP 100.65.18.199 \u0026lt;none\u0026gt; 80/TCP 46s   Use wget to access the files served by Nginx. Nginx is sharing files from the same volume that the main application container is writing to. The connection to the Nginx container is made via the backup-example service.\n$ kubectl exec -it busybox -- /bin/wget -q -O- http://backup-example-nginx \u0026lt;html\u0026gt; \u0026lt;head\u0026gt;\u0026lt;title\u0026gt;Index of /\u0026lt;/title\u0026gt;\u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;h1\u0026gt;Index of /\u0026lt;/h1\u0026gt;\u0026lt;hr\u0026gt;\u0026lt;pre\u0026gt;\u0026lt;a href=\u0026#34;../\u0026#34;\u0026gt;../\u0026lt;/a\u0026gt; \u0026lt;a href=\u0026#34;lost%2Bfound/\u0026#34;\u0026gt;lost+found/\u0026lt;/a\u0026gt; 12-Feb-2019 12:32 - \u0026lt;a href=\u0026#34;date.txt\u0026#34;\u0026gt;date.txt\u0026lt;/a\u0026gt; 12-Feb-2019 12:49 29 \u0026lt;/pre\u0026gt;\u0026lt;hr\u0026gt;\u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; $ kubectl exec -it busybox -- /bin/wget -q -O- http://backup-example-nginx/date.txt Tue Feb 12 12:49:15 UTC 2019   Depending on what files have been written to the StorageOS volume the output of the index file will be different. In the example the date.txt file we created in Step 2 is present on the volume.\nExfiltrating files through Rsync   Deploy the rsync example\n$ kubectl create -f rsync/ service/backup-example created configmap/rsync-config created secret/rsync-credentials created statefulset.apps/backup-example created pod/rsync created   Check that a backup-example pod is running\n$ kubectl get pods -w -l app=backup-example-rsync NAME READY STATUS RESTARTS AGE backup-example-0 1/1 Running 0 1m   Exec into the main container and write some data to a file\n$ kubectl exec -it backup-example-rsync-0 -c main bash root@backup-example-0:/# echo $(date) \u0026gt; /data/date.txt   Check that the service exists\n$ kubectl get svc backup-example-rsync NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE backup-example-rsync ClusterIP 100.65.18.199 \u0026lt;none\u0026gt; 873/TCP 46s   Use rsync to access the files shared by the rsync daemon. rsync is sharing files from the same volume that the main container is writing to. A username and password that are set in the rsync-credentials secret. The secret supplied in the example has the username and password set to username and password.\n$ kubectl exec -it rsync sh / # rsync --list-only rsync://username@backup-example-rsync/share Password: drwxr-xr-x 4,096 2019/02/12 12:49:15 . -rw-r--r-- 29 2019/02/12 12:49:15 date.txt drwx------ 16,384 2019/02/12 12:32:40 lost+found / # rsync -chavzP rsync://username@backup-example-rsync/share/date.txt . Password: receiving incremental file list date.txt 29 100% 28.32kB/s 0:00:00 (xfr#1, to-chk=0/1) sent 43 bytes received 135 bytes 50.86 bytes/sec total size is 29 speedup is 0.16 / # cat date.txt Tue Feb 12 12:49:15 UTC 2019   In the example above the list of available files was shown and a file called date.txt was synchronized to the rsync container.\nExfiltrating files through SFTP   Deploy the sftp example\n$ kubectl create -f sftp/   Exec into the main container and write some data to a file\n$ kubectl exec -it backup-example-sftp-0 -c main bash root@backup-example-0:/# echo $(date) \u0026gt; /data/date.txt   Check that the service exists\n$ kubectl get svc backup-example-sftp NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE backup-example-sftp ClusterIP 100.70.50.56 \u0026lt;none\u0026gt; 22/TCP 2h   Use SFTP to access the files shared by the SFTP container. If you have made no changes to the sftp-config secret the password is password.\n$ kubectl exec -it sftp -- bash root@sftp:/# sftp alex@backup-example-sftp alex@backup-example-sftp\u0026#39;s password: Connected to backup-example-sftp. sftp\u0026gt; ls date.txt lost+found sftp\u0026gt; get date.txt Fetching /date.txt to date.txt /date.txt 100% 29 15.9KB/s 00:00 sftp\u0026gt; bye root@sftp:/# cat date.txt Tue Feb 12 17:51:32 UTC 2019   In order to do this a SFTP user needs to be configured. The details for the user are stored in the sftp-config secret (see sftp/17-secret.yaml). The secret consists of base64 encoded username:password:uid:guid and the user is chroot\u0026rsquo;ed inside their home directory so the mount point for the StorageOS volume in the SFTP container in sftp/20-backup-pod.yaml needs to be configured.\nUsing custom SSH Keys The ConfigMap ssh-key-pub (see sftp/15-configmap.yaml) needs to be populated with a public key. The corresponding private key needs to be base64 encoded and put into the ssh-key-private secret (see sftp/17-secret.yaml). The user to connect as is determined by the user that is configured in the sftp-config configMap. To restrict logins to the SSH key edit the sftp-config secret so it contains no password (user::uid:guid).\n  Connect to the sftp pod and connect through the service to the SFTP container running inside the backup-example pod.\n$ kubectl exec -it sftp -- bash root@sftp:/# sftp -i /home/alex/.ssh/id_rsa alex@backup-example-sftp Connected to backup-example-sftp. sftp\u0026gt; ls date.txt lost+found   ","excerpt":"In this example use case we provide three different strategies for accessing files that have been …","ref":"/docs/usecases/sidecar-backups/","title":"Sidecar Backups"},{"body":"StorageClasses in Kubernetes are used to link PVCs with a backend storage provisioner - for instance, StorageOS. A StorageClass defines parameters to pass to the provisioner, which in the case of StorageOS can be translated into behaviour applied to the Volumes. Many StorageClasses can be provisioned to apply different feature labels to the StorageOS Volumes.\nBy default the StorageOS Cluster Operator installs the fast StorageClass at bootstrap of StorageOS. You can define its name in the StorageOS Cluster Resource.\napiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:fastlabels:app:storageosprovisioner:csi.storageos.com# CSI DriverallowVolumeExpansion:trueparameters:csi.storage.k8s.io/fstype:ext4# Change the NameSpace below if StorageOS doesn\u0026#39;t run in kube-systemcsi.storage.k8s.io/controller-expand-secret-name:csi-controller-expand-secretcsi.storage.k8s.io/controller-publish-secret-name:csi-controller-publish-secretcsi.storage.k8s.io/node-publish-secret-name:csi-node-publish-secretcsi.storage.k8s.io/provisioner-secret-name:csi-provisioner-secretcsi.storage.k8s.io/controller-expand-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/controller-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/node-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/provisioner-secret-namespace:kube-system# NameSpace that runs StorageOS DaemonsetStorageClasses can be created to define default labels for StorageOS volumes, but also to map to any semantic aggregation of volumes that suits your use case, whether there are different roles (dev, staging, prod), or a StorageClass maps to a team or customer using the cluster.\nExamples You can find the basic examples in the StorageOS use-cases repository, in the 00-basic directory.\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases/00-basic StorageClass definition in v2-storageclass-replicated.yaml.\napiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:storageos-replicatedprovisioner:csi.storageos.com# CSI DriverallowVolumeExpansion:trueparameters:csi.storage.k8s.io/fstype:ext4storageos.com/replicas:\u0026#34;1\u0026#34;# Enforces 1 replica for the Volume# Change the NameSpace below if StorageOS doesn\u0026#39;t run in kube-systemcsi.storage.k8s.io/controller-expand-secret-name:csi-controller-expand-secretcsi.storage.k8s.io/controller-publish-secret-name:csi-controller-publish-secretcsi.storage.k8s.io/node-publish-secret-name:csi-node-publish-secretcsi.storage.k8s.io/provisioner-secret-name:csi-provisioner-secretcsi.storage.k8s.io/controller-expand-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/controller-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/node-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/provisioner-secret-namespace:kube-system# NameSpace that runs StorageOS DaemonsetThat StorageClass can be used by a PVC:\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:my-vol-1spec:storageClassName:\u0026#34;storageos-replicated\u0026#34;# StorageOS StorageClassaccessModes:- ReadWriteOnceresources:requests:storage:5GiThe above StorageClass has the storageos.com/replicas label set. This label tells StorageOS to create a volume with a replica. Adding StorageOS feature labels to the StorageClass ensures all volumes created with the StorageClass have the same labels.\nYou can also choose to add the label in the PVC definition rather than the StorageClass. The PVC definition takes precedence over the SC.\n","excerpt":"StorageClasses in Kubernetes are used to link PVCs with a backend storage provisioner - for …","ref":"/docs/operations/storageclasses/","title":"Kubernetes StorageClasses"},{"body":"","excerpt":"","ref":"/","title":"StorageOS"},{"body":"StorageOS is a software-defined storage platform for running stateful applications in Kubernetes.\nFundamentally, StorageOS uses the storage attached to the nodes in the StorageOS cluster to create and present virtual volumes into containers. Space on the host is consumed from the mount point /var/lib/storageos/data, so it is therefore recommended that disk devices are used exclusively for StorageOS, as described in Managing Host Storage \nStorageOS is agnostic to the underlying storage and runs equally well on bare metal, in virtual machines or on cloud providers.\nRead about the cloud native storage principles behind StorageOS.\nStorageOS on Kubernetes StorageOS is architected as a series of containers that fulfil separate, discrete functions.\nLinks where appropriate have been given to our open-source GitHub repo.\n StorageOS Cluster Operator  Responsible for the creation and maintenance of the StorageOS cluster. This operator is primarily responsible for ensuring that all the relevant applications are running in your cluster.\n StorageOS Controlplane  Responsible for monitoring and maintaining the state of volumes and nodes in the cluster. The Controlplane and the Dataplane run together in a single container, managed by a daemonset. The Controlplane works with etcd to maintain state consensus in your cluster.\n StorageOS Dataplane  Responsible for all I/O path related tasks; reading, writing, compression and caching.\n StorageOS Scheduler  Responsible for scheduling applications on the same node as an application\u0026rsquo;s volumes. StorageOS uses a custom Kubernetes scheduler to handle pod placement, ensuring that volumes are deployed on the same nodes as the relevant workloads as often as possible.\n CSI helper  Responsible for registering StorageOS with Kubernetes as a CSI driver. It is necessary because the internal persistent volume controller running in Kubernetes controller-manager does not have any direct interfaces to CSI drivers. It monitors PVC objects created by users and creates/deletes volumes for them.\n StorageOS API manager  Acts as a middle-man between various APIs. It has all the capabilities of a Kubernetes Operator and is also able to communicate with the StorageOS control plane API. This application handles typical operator tasks like labelling or removing nodes from StorageOS when removed from the Kubernetes. It is continually monitoring the state of the cluster and moving it towards the desired state when necessary.\nStorageOS is deployed by the StorageOS Cluster Operator. In Kubernetes, the StorageOS Controlplane and Dataplane are deployed in a single pod managed by a daemonset. This daemonset runs on every node in the cluster that will consume or present storage. The Scheduler, CSI helper, Cluster Operator and API Manager run as separate pods and are controlled as deployments.\nStorageOS is designed to feel familiar to Kubernetes and Docker users. Storage is managed through standard StorageClasses and PersistentVolumeClaims, and features are controlled by Kubernetes-style labels and selectors, prefixed with storageos.com/. By default, volumes are cached to improve read performance and compressed to reduce network traffic.\nAny pod may mount a StorageOS virtual volume from any node that is also running StorageOS, regardless of whether the pod and volume are collocated on the same node. Therefore, applications may be started or restarted on any node and access volumes transparently.\n","excerpt":"StorageOS is a software-defined storage platform for running stateful applications in Kubernetes. …","ref":"/docs/concepts/components/","title":"StorageOS Components"},{"body":"StorageOS collects telemetry and error reports from StorageOS clusters via two different methods for two different purposes.\nTelemetry  Telemetry is made up a DNS version check query and a once per day report. Error reporting is the sentry.io crash dump reporting.  sentry.io StorageOS sends crash reports to sentry.io. This information helps our developers monitor and fix crashes. Information is sent to sentry.io when a process inside the StorageOS container crashes.\n The crash report contains the signal that triggered the shutdown (e.g. SIGSEGV), the exit code and whether or not the crash generated a core dump.  All StorageOS clusters with a routable connection to the internet will send crash reports to sentry.io over tcp/443. StorageOS respects environment variables that ProxyFromEnvironment uses.\nAn exhaustive list of information included in the crash report is below:\n StorageOS version Crash description string Anonymized Cluster ID Anonymized Node ID  DNS Query StorageOS will perform a \u0026ldquo;latest version check\u0026rdquo; using a DNS query in order to inform administrators that a new version is available. StorageOS will also send anonymized node ids, cluster id and StorageOS version information to StorageOS using a DNS query. The information that we send in the query is encoded as well as being anonymized. This query allows us to inform Cluster admins when StorageOS upgrades are available in the StorageOS GUI and in the logs.\nThe DNS query includes:\n Anonymized StorageOS Cluster ID Anonymized StorageOS node ID StorageOS version number  Once Per Day Report The once per day report contains information about the StorageOS cluster and Kubernetes versions to help StorageOS focus our development efforts on the most popular platforms. The once per day data is encrypted and sent to a StorageOS telemetry server so it is never processed outside of StorageOS assets.\nAn exhaustive list of information included in the once per day report is below:\n api_call_metrics cluster_disable_crash_reporting cluster_disable_version_check cluster_log_format cluster_log_level cluster_tls_provided k8s_distribution k8s_in_k8s k8s_scheduler_extender_enabled k8s_version node_available_bytes node_capacity node_crash_files_on_disk node_created_at_time node_etcd_config node_etcd_namespacing_enabled node_etcd_tls_enabled node_free_bytes node_health node_http_tls_enabled node_id node_labels node_status node_storageos_version node_system_clock_time node_total_bytes node_version volume_fs_type volume_id volume_labels volume_master_attach volume_master_delete_deployment volume_master_detach volume_master_failover_deployment volume_master_promote volume_master_provision volume_master_recover_replica volume_master_trigger_rejoin volume_metrics volume_placement_strategy volume_provision_source_user_agent volume_replica_delete_deployment volume_replica_failover_deployment volume_replica_promote volume_replica_provision volume_replicas volume_replica_trigger_rejoin volume_size_bytes  Disable Telemetry It is possible to disable telemetry using the GUI, CLI, API, environment variables or the StorageOS Cluster Spec.\nStorageOS Cluster Spec Disable telemetry explicitly through the configurable spec parameters of the StorageOSCluster custom resource.\nEnvironment Variables You can use the following environmental variables to disable or enable telemetry.\nDISABLE_VERSION_CHECK # Disable the DNS version check DISABLE_TELEMETRY # Disable the once per day reporting DISABLE_ERROR_REPORTING # Disable sentry.io crash reports ","excerpt":"StorageOS collects telemetry and error reports from StorageOS clusters via two different methods for …","ref":"/docs/reference/telemetry/","title":"StorageOS Telemetry"},{"body":"The support bundle is a tool that gathers information both about StorageOS and the environment in which it is operating. It provides information about the state and configuration of the cluster, nodes and other Kubernetes objects, as well as system performance metrics.\nThe support bundle is an addition rather than a replacement to the diagnostic bundle.\nInstall Kubectl StorageOS plugin The support bundle is generated with the StorageOS kubectl plugin, which can be installed as follows:\n$ curl -sL https://docs.storageos.com/sh/storageos-support-bundle-install.sh | bash If you prefer to run the installation commands individually, we provide them here:\n# Create a temporary directory in which to store the plugin binaries TMP_DIR=$(mktemp -d /tmp/storageos-kubectl-plugin-XXXXX) # Download the plugin binaries and extract to the temporary directory TARGET=\u0026#34;kubectl-storageos_linux_amd64.tar.gz\u0026#34; curl -sSL -o kubectl-storageos.tar.gz https://github.com/storageos/storageos.github.io/raw/master/sh/$TARGET tar -xf kubectl-storageos.tar.gz -C $TMP_DIR/ # Clean up the tar file rm -f kubectl-storageos.tar.gz # Add executable permissions for the plugin binaries and move into system path # For details of the bundle-generation tool\u0026#39;s functionality, visit: #https://docs.storageos.com/docs/bundles/support-bundle/ chmod +x $TMP_DIR/bin/kubectl-storageos-bundle sudo mv $TMP_DIR/bin/kubectl-storageos-bundle /usr/local/bin/  For MacOS build, use TARGET=kubectl-storageos_darwin_amd64.tar.gz\n Generate Bundle To generate a bundle, use the following command, the specification of which can be viewed and edited by obtaining the bundle-configuration.yaml file, which is publicly available.\nStorageOS is usually installed in the kube-system namespace. If you have installed StorageOS in a different namespace please replace the namespace in the storage bundle config as below.\nSTORAGEOS_NS=my-namespace curl -s https://docs.storageos.com/yaml/bundle-configuration.yaml | sed \u0026#34;s/kube-system/$STORAGEOS_NS/g\u0026#34; \u0026gt; /tmp/storageos-kubectl-config.yaml kubectl storageos bundle /tmp/storageos-kubectl-config.yaml Please note that if you have a custom selector for your worker nodes you should update the bundle-configuration.yaml under spec.collectors.run.nodeselector to reflect this.\nNote also that the bundle tool expects there to be a StorageOS CLI running in kube-system as a Pod with the label run=cli. The tool will exec into this pod to get information from the StorageOS API. If the StorageOS CLI Pod does not match this criteria, you can either pull the YAML file and change the selector in the file, or add the label to the Pod. You can run the cli container following these instructions.\nData collected in the bundle The data collected covers both the state of the StorageOS cluster, and in particular information regarding the infrastructure on which StorageOS is operating. It includes:\n StorageOS CLI information StorageOS Operator logs StorageOS logs Cluster metadata Cluster resources (DaemonSets, Events, Services, Pods, etc.) Backend disk configuration and performance statistics Load average Network checks across StorageOS nodes and ports Running processes  Privacy StorageOS can only obtain the bundle if it is downloaded by the user and given to our engineering team, or uploaded for analysis. The data received by StorageOS is private and never leaves nor will leave StorageOS Inc.\nThe data contained in the support bundle has the sole purpose of helping customers troubleshoot their issues.\n","excerpt":"The support bundle is a tool that gathers information both about StorageOS and the environment in …","ref":"/docs/reference/bundles/support_bundle/","title":"Support bundle"},{"body":"Host Filesystems StorageOS will automatically use /var/lib/storageos on each host as a base directory for storing configuration and blob files. Supported host filesystem types are ext4 and xfs. If you require a specific filesystem please contact StorageOS.\nPersistent Volume Filesystems StorageOS provides a block device on which a file system can be created. The creation of the filesystem is either handled by StorageOS or by Kubernetes which affects what filesystems can be created.\nCSI Driver When using StorageOS with the CSI driver, StorageOS is responsible for running mkfs against the block device that pods mount. StorageOS is able to create ext2, ext3, ext4 and xfs file systems.\n","excerpt":"Host Filesystems StorageOS will automatically use /var/lib/storageos on each host as a base …","ref":"/docs/reference/filesystems/","title":"Supported File systems"},{"body":"Kubernetes Tolerations Tolerations are a Kubernetes pod property that allow pods to tolerate certain node taints. Taints can be thought of as the opposite of Node affinity in that taints repel pods. Node taints are automatically applied by Kubernetes in response to node resources coming under contention. As StorageOS provides storage to pods it should not be evicted during periods of resource contention, as any pods using StorageOS volumes on the same node would need to be restarted.\nAs StorageOS runs as a daemonset some Kubernetes tolerations are added by Kubernetes while others are automatically added by the StorageOS operator.\nFor more information about tolerations please see the Kubernetes documentation.\ntolerations:# The unreachable and not-ready tolerations are added by Kubernetes to daemonsets automatically- key:\u0026#34;node.kubernetes.io/unreachable\u0026#34;operator:\u0026#34;Exists\u0026#34;effect:\u0026#34;NoExecute\u0026#34;- key:\u0026#34;node.kubernetes.io/not-ready\u0026#34;operator:\u0026#34;Exists\u0026#34;effect:\u0026#34;NoExecute\u0026#34;# The following tolerations are added to the StorageOS daemonset by the StorageOS operator- key:node.kubernetes.io/disk-pressureoperator:Exists- key:node.kubernetes.io/memory-pressureoperator:Exists- key:node.kubernetes.io/network-unavailableoperator:Exists- key:node.kubernetes.io/out-of-diskoperator:Exists- key:node.kubernetes.io/pid-pressureoperator:Exists- key:node.kubernetes.io/unschedulableoperator:ExistsAdding Custom Tolerations To add custom tolerations to the StorageOS daemonset configure them in the StorageOSCluster resource.\n","excerpt":"Kubernetes Tolerations Tolerations are a Kubernetes pod property that allow pods to tolerate certain …","ref":"/docs/reference/tolerations/","title":"Tolerations"},{"body":"StorageOS volumes support TRIM/Unmap by default for all uncompressed volumes. StorageOS creates uncompressed volumes by default so all of these volumes will support TRIM/Unmap calls. TRIM\u0026rsquo;ing a StorageOS volume will release space taken up by deleted blocks in the StorageOS volume blob files.\nTRIM can either be called periodically via fstrim or continously using the -o discard mount option. StorageOS recommends using periodic discard.\nPeriodic Discard In order to TRIM a volume you can run fstrim against the volume filesystem. The volume filesystem will be presented at the mount point for the StorageOS device.\nThe example below shows the effect of fstrim on a StorageOS volume mounted on /mnt.\n# A StorageOS volume with some data written to it. $ ls -ls --block-size 1 /var/lib/storageos/data/dev1/vol.211585.*.blob | awk '!/^total/ {total = total + $1}END{print total}' 8.85838e+09 # A StorageOS volume is mounted on /mnt $ df -h /mnt Filesystem Size Used Avail Use% Mounted on /var/lib/storageos/volumes/v.bbfae475-3ce3-4238-bf33-cfe88256d813 4.9G 520M 4.1G 12% /mnt # Delete a file from the volume, sync all filesystems to ensure data has been written and then fstrim the volume filesystem $ rm /mnt/test; sync; fstrim /mnt # Observe that the space in the blob files, and thus space in the backend filesystem has been reclaimed. $ ls -ls --block-size 1 /var/lib/storageos/data/dev1/vol.211585.*.blob | awk '!/^total/ {total = total + $1}END{print total}' 469700608 Automating periodic discard Discard can be automated by running fstrim against mounted StorageOS volumes. When running in Kubernetes the Kubelet is responsible for mounting volumes into pods so the mount endpoints for pods are accessible under /var/lib/kubelet/pods. StorageOS volume mounts appear as mounts from /var/lib/storageos/volumes/v.${DEPLOYMENT_ID} on /var/lib/kubelet/pods/${POD_UID}/volumes/kubernetes.io~csi/${PV_ID}/mount.\nroot@unmap:/# mount | awk '/storageos\\/volumes\\/v.*/' /var/lib/storageos/volumes/v.4364e143-865e-45d7-a4a5-e0d964d9e200 on /var/lib/kubelet/pods/29ec4774-71d6-4ba3-b275-f80b47e9f6af/volumes/kubernetes.io~csi/pvc-357a9baa-7b74-49db-8d63-c540b5129ad8/mount type ext4 (rw,relatime,stripe=32) /var/lib/storageos/volumes/v.8dc449a4-1fb5-43d8-86d0-e18916a85c18 on /var/lib/kubelet/pods/6bb92a8f-afb5-48c8-837c-56f45e15e5c4/volumes/kubernetes.io~csi/pvc-2d30a2ba-2663-4036-bb1e-e795f226d6f3/mount type ext4 (rw,relatime,stripe=32) root@unmap:/# mount | awk '/storageos\\/volumes\\/v.*/ {print $3}' /var/lib/kubelet/pods/29ec4774-71d6-4ba3-b275-f80b47e9f6af/volumes/kubernetes.io~csi/pvc-357a9baa-7b74-49db-8d63-c540b5129ad8/mount /var/lib/kubelet/pods/6bb92a8f-afb5-48c8-837c-56f45e15e5c4/volumes/kubernetes.io~csi/pvc-2d30a2ba-2663-4036-bb1e-e795f226d6f3/mount With this information in mind it is therefore possible to use Kubernetes tools to automate periodic discard. The SYS_ADMIN capability is required to run fstrim and propagation of mounts from the host to the container will allow any new StorageOS mounts to be picked up by the unmap pod.\nThe naïve example below shows how a pod could be used to run fstrim against StorageOS volumes mounted on the same node that the pod is scheduled on.\napiVersion: v1 kind: Pod metadata: name: discard spec: containers: - name: fstrim ... command: [\u0026quot;/bin/bash\u0026quot;] args: [\u0026quot;-c\u0026quot;,\u0026quot;for mount in $(mount | awk '/storageos\\\\/volumes\\\\/v.*/ {print $3}'); do if [ -d ${mount} ]; then fstrim -v ${mount}; fi; sleep 5; done\u0026quot;] securityContext: capabilities: add: - SYS_ADMIN volumeMounts: - mountPath: /var/lib/kubelet mountPropagation: HostToContainer name: kubelet-dir volumes: - hostPath: path: /var/lib/kubelet type: Directory name: kubelet-dir Alternatively a cronjob on the node itself could be used to fstrim mounted StorageOS volumes using similar logic.\n N.B. TRIM can be an I/O intensive operation so care should be taken when running fstrim against multiple volumes at once.\n Continuous discard StorageOS volumes can be mounted using the discard option which will automatically send TRIM commands when blocks are removed. Caution should be used enabling this option as testing has shown that volumes with a lot of churn can experience performance degradation. The pathological case being a volume that is continuously filled with small files that are then all deleted, repeatedly. The RHEL documentation also recommends doing perodic discards.\nThe discard option can be enabled as a StorageClass or PersistentVolume option. Enabling discard as a StorageClass option will result in all volumes provisioned with that StorageClass being mounted with discard whereas setting it as a PersistentVolume option sets it on a per volume basis.\nThe StorageClass below would provision xfs volumes with the discard option enabled by default.\napiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:storageos-discardprovisioner:csi.storageos.comallowVolumeExpansion:trueparameters:...csi.storage.k8s.io/fstype:xfsmountOptions:- discardYou can also edit an existing PersistentVolume to add discard to .spec.mountOptions, this ensures that the next time the volume is mounted it will use -o discard.\napiVersion:v1kind:PersistentVolume...spec:...mountOptions:- discardvolumeMode:Filesystem","excerpt":"StorageOS volumes support TRIM/Unmap by default for all uncompressed volumes. StorageOS creates …","ref":"/docs/operations/trim/","title":"TRIM"},{"body":"This section is aimed to help you troubleshoot issues in your cluster, whether they are related to the StorageOS installation, integration with orchestrators or common misconfigurations.\nTools To be able to troubleshoot issues the StorageOS cli is required.\nPod in pending because of mount error Issue: The output of kubectl describe pod $POD_ID contains no such file or directory and references the StorageOS volume device file.\nroot@node1:~# kubectl -n kube-system describe $POD_ID (...) Events: (...) Normal Scheduled 11s default-scheduler Successfully assigned default/d1 to node3 Warning FailedMount 4s (x4 over 9s) kubelet, node3 MountVolume.SetUp failed for volume \u0026#34;pvc-f2a49198-c00c-11e8-ba01-0800278dc04d\u0026#34; : stat /var/lib/storageos/volumes/d9df3549-26c0-4cfc-62b4-724b443069a1: no such file or directory Reason: There are two main reasons this issue may arise:\n The StorageOS DEVICE_DIR location is wrongly configured when using Kubelet as a container Mount Propagation is not enabled  (Option 1) Misconfiguration of the DeviceDir/SharedDir\nSome Kubernetes distributions such as Rancher, DockerEE or some installations of OpenShift deploy the Kubelet as a container, because of this, the device files that StorageOS creates to mount into the containers need to be visible to the kubelet. StorageOS can be configured to share the device directory.\nModern installations use CSI, which handles the complexity internally.\nAssert: root@node1:~# kubectl -n default describe stos | grep \u0026#34;Shared Dir\u0026#34; Shared Dir: # \u0026amp;lt;-- Shouldn\u0026#39;t be blank Solution: The Cluster Operator Custom Definition should specify the SharedDir option as follows.\nspec: sharedDir: \u0026#39;/var/lib/kubelet/plugins/kubernetes.io~storageos\u0026#39; # Needed when Kubelet as a container ...  See example on how to configure the StorageOS Custom Resource.\n  (Option 2) Mount propagation is not enabled.\n Applies only if Option 1 is configured properly.\n Assert: If not using the Kubelet as a container, SSH into one of the nodes and check if /var/lib/storageos/volumes is empty. If so, exec into any StorageOS pod and check the same directory.\nroot@node1:~# ls /var/lib/storageos/volumes/ root@node1:~# # \u0026amp;lt;-- Shouldn\u0026#39;t be blank root@node1:~# kubectl exec $POD_ID -c storageos -- ls -l /var/lib/storageos/volumes bst-196004 d529b340-0189-15c7-f8f3-33bfc4cf03fa ff537c5b-e295-e518-a340-0b6308b69f74 If the directory inside the container and the device files are visible, disabled mount propagation is the cause.\nIf using the Kubelet as a container, SSH into one of the nodes and check if /var/lib/kubelet/plugins/kubernetes.io~storageos/devices is empty. If so, exec into any StorageOS pod and check the same directory.\nroot@node1:~# ls /var/lib/kubelet/plugins/kubernetes.io~storageos/devices root@node1:~# # \u0026amp;lt;-- Shouldn\u0026#39;t be blank root@node1:~# kubectl exec $POD_ID -c storageos -- ls -l /var/lib/kubelet/plugins/kubernetes.io~storageos/devices bst-196004 d529b340-0189-15c7-f8f3-33bfc4cf03fa ff537c5b-e295-e518-a340-0b6308b69f74 If the directory inside the container and the device files are visible, disabled mount propagation is the cause.\nSolution: Older versions of Kubernetes need to enable mount propagation as it is not enabled by default. Most Kubernetes distributions allow MountPropagation to be enabled using FeatureGates. Rancher specifically, needs to enable it in the \u0026ldquo;View in API\u0026rdquo; section of your cluster. You need to edit the section \u0026ldquo;rancherKubernetesEngineConfig\u0026rdquo; to enable the Kubelet feature gate.\nPVC pending state - Failed to dial StorageOS A created PVC remains in pending state making pods that need to mount that PVC unable to start.\nIssue: root@node1:~/# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE vol-1 Pending fast 7s kubectl describe pvc $PVC (...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 7s (x2 over 18s) persistentvolume-controller Failed to provision volume with StorageClass \u0026#34;fast\u0026#34;: Get http://storageos-cluster/version: failed to dial all known cluster members, (10.233.59.206:5705) Reason: For non CSI installations of StorageOS, Kubernetes uses the StorageOS API endpoint to communicate. If that communication fails, relevant actions such as create or mount volume can\u0026rsquo;t be transmitted to StorageOS, hence the PVC will remain in pending state. StorageOS never received the action to perform, so it never sent back an acknowledgement.\nIn this case, the Event message indicates that StorageOS API is not responding, implying that StorageOS is not running. For Kubernetes to define StorageOS pods ready, the health check must pass.\nAssert: Check the status of StorageOS pods.\nroot@node1:~/# kubectl -n kube-system get pod --selector app=storageos # for CSI add --selector kind=daemonset NAME READY STATUS RESTARTS AGE storageos-qrqkj 0/1 Running 0 1m storageos-s4bfv 0/1 Running 0 1m storageos-vcpfx 0/1 Running 0 1m storageos-w98f5 0/1 Running 0 1m If the pods are not READY, the service will not forward traffic to the API they serve hence PVC will remain in pending state until StorageOS pods are available.\n Kubernetes keeps trying to execute the action until it succeeds. If a PVC is created before StorageOS finish starting, the PVC will be created eventually.\n Solution:  StorageOS health check takes 60 seconds of grace before reporting as READY. If StorageOS is starting properly after that period, the volume will be created when StorageOS finishes its bootstrap. If StorageOS is not running or is not starting properly, the solution would be to troubleshoot the installation.  PVC pending state - Secret Missing A created PVC remains in pending state making pods that need to mount that PVC unable to start.\nIssue: kubectl describe pvc $PVC (...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 13s (x2 over 28s) persistentvolume-controller Failed to provision volume with StorageClass \u0026#34;fast\u0026#34;: failed to get secret from [\u0026#34;storageos\u0026#34;/\u0026#34;storageos-api\u0026#34;] Reason: For non CSI installations of StorageOS, Kubernetes uses the StorageOS API endpoint to communicate. If that communication fails, relevant actions such as create or mount a volume can\u0026rsquo;t be transmitted to StorageOS, and the PVC will remain in pending state. StorageOS never received the action to perform, so it never sent back an acknowledgement.\nThe StorageClass provisioned for StorageOS references a Secret from where it retrieves the API endpoint and the authentication parameters. If that secret is incorrect or missing, the connections won\u0026rsquo;t be established. It is common to see that the Secret has been deployed in a different namespace where the StorageClass expects it or that is has been deployed with a different name.\nAssert:   Check the StorageClass parameters to know where the Secret is expected to be found.\nkubectl get storageclass fast -o yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: creationTimestamp: 2018-09-25T08:44:57Z labels: app: storageos name: fast resourceVersion: \u0026#34;108853\u0026#34; selfLink: /apis/storage.k8s.io/v1/storageclasses/fast uid: 48490a9b-c09f-11e8-ba01-0800278dc04d parameters: adminSecretName: storageos-api adminSecretNamespace: storageos description: Kubernetes volume fsType: ext4 provisioner: kubernetes.io/storageos reclaimPolicy: Delete  Note that the parameters specify adminSecretName and adminSecretNamespace.\n   Check if the secret exists according to those parameters\nkubectl -n kube-system get secret storageos-api No resources found. Error from server (NotFound): secrets \u0026#34;storageos-api\u0026#34; not found If no resources are found, it is clear that the Secret doesn\u0026rsquo;t exist or it is not deployed in the right location.\n  Solution: Deploy StorageOS following the installation procedures. If you are using the manifests provided for Kubernetes to deploy StorageOS rather than using automated provisioners, make sure that the StorageClass parameters and the Secret reference match.\nNode name different from Hostname Issue: StorageOS nodes can\u0026rsquo;t join the cluster showing the following log entries.\ntime=\u0026#34;2018-09-24T13:47:02Z\u0026#34; level=error msg=\u0026#34;failed to start api\u0026#34; error=\u0026#34;error verifying UUID: UUID aed3275f-846b-1f75-43a1-adbfec8bf974 has already been registered and has hostname \u0026#39;debian-4\u0026#39;, not \u0026#39;node4\u0026#39;\u0026#34; module=command Reason: The StorageOS registration process to start the cluster uses the hostname of the node where the StorageOS container is running, provided by the Kubelet. However, StorageOS verifies the network hostname of the OS as a prestart check to make sure it can communicate with other nodes. If those names don\u0026rsquo;t match, StorageOS will be unable to start.\nSolution: Make sure the hostnames match with the Kubernetes advertised names. If you have changed the hostname of your nodes, make sure that you restart the nodes to apply the change.\nPeer discovery - Networking Issue: StorageOS nodes can\u0026rsquo;t join the cluster showing the following logs after one minute of container uptime.\ntime=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=info msg=\u0026#34;not first cluster node, joining first node\u0026#34; action=create address=172.28.128.5 category=etcd host=node3 module=cp target=172.28.128.6 time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=error msg=\u0026#34;could not retrieve cluster config from api\u0026#34; status_code=503 time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=error msg=\u0026#34;failed to join existing cluster\u0026#34; action=create category=etcd endpoint=\u0026#34;172.28.128.3,172.28.128.4,172.28.128.5,172.28.128.6\u0026#34; error=\u0026#34;503 Service Unavailable\u0026#34; module=cp time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=info msg=\u0026#34;retrying cluster join in 5 seconds...\u0026#34; action=create category=etcd module=cp Reason: StorageOS uses a gossip protocol to discover nodes in the cluster. When StorageOS starts, one or more nodes can be referenced so new nodes can query existing nodes for the list of members. This error indicates that the node can\u0026rsquo;t connect to any of the nodes in the known list. The known list is defined in the JOIN variable.\nAssert: It is likely that ports are block by a firewall.\nSSH into one of your nodes and check connectivity to the rest of the nodes.\n# Successfull execution: [root@node06 ~]# nc -zv node04 5705 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to 10.0.1.166:5705. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. StorageOS exposes network diagnostics in its API, viewable from the CLI. To use this feature, the CLI must query the API of a running node. The diagnostics show information from all known cluster members. If all the ports are blocked during the first bootstrap of the cluster, the diagnostics won\u0026rsquo;t show any data as nodes couldn\u0026rsquo;t register.\n StorageOS networks diagnostics are available for storageos-rc5 and storageos-cli-rc3 and above.\n # Example: root@node1:~# storageos cluster connectivity SOURCE NAME ADDRESS LATENCY STATUS MESSAGE node4 node2.nats 172.28.128.4:5708 1.949275ms OK node4 node3.api 172.28.128.5:5705 3.070574ms OK node4 node3.nats 172.28.128.5:5708 2.989238ms OK node4 node2.directfs 172.28.128.4:5703 2.925707ms OK node4 node3.etcd 172.28.128.5:5707 2.854726ms OK node4 node3.directfs 172.28.128.5:5703 2.833371ms OK node4 node1.api 172.28.128.3:5705 2.714467ms OK node4 node1.nats 172.28.128.3:5708 2.613752ms OK node4 node1.etcd 172.28.128.3:5707 2.594159ms OK node4 node1.directfs 172.28.128.3:5703 2.601834ms OK node4 node2.api 172.28.128.4:5705 2.598236ms OK node4 node2.etcd 172.28.128.4:5707 16.650625ms OK node3 node4.nats 172.28.128.6:5708 1.304126ms OK node3 node4.api 172.28.128.6:5705 1.515218ms OK node3 node2.directfs 172.28.128.4:5703 1.359827ms OK node3 node1.api 172.28.128.3:5705 1.185535ms OK node3 node4.directfs 172.28.128.6:5703 1.379765ms OK node3 node1.etcd 172.28.128.3:5707 1.221176ms OK node3 node1.nats 172.28.128.3:5708 1.330122ms OK node3 node2.api 172.28.128.4:5705 1.238541ms OK node3 node1.directfs 172.28.128.3:5703 1.413574ms OK node3 node2.etcd 172.28.128.4:5707 1.214273ms OK node3 node2.nats 172.28.128.4:5708 1.321145ms OK node1 node4.directfs 172.28.128.6:5703 1.140797ms OK node1 node3.api 172.28.128.5:5705 1.089252ms OK node1 node4.api 172.28.128.6:5705 1.178439ms OK node1 node4.nats 172.28.128.6:5708 1.176648ms OK node1 node2.directfs 172.28.128.4:5703 1.529612ms OK node1 node2.etcd 172.28.128.4:5707 1.165681ms OK node1 node2.api 172.28.128.4:5705 1.29602ms OK node1 node2.nats 172.28.128.4:5708 1.267454ms OK node1 node3.nats 172.28.128.5:5708 1.485657ms OK node1 node3.etcd 172.28.128.5:5707 1.469429ms OK node1 node3.directfs 172.28.128.5:5703 1.503015ms OK node2 node4.directfs 172.28.128.6:5703 1.484ms OK node2 node1.directfs 172.28.128.3:5703 1.275304ms OK node2 node4.nats 172.28.128.6:5708 1.261422ms OK node2 node4.api 172.28.128.6:5705 1.465532ms OK node2 node3.api 172.28.128.5:5705 1.252768ms OK node2 node3.nats 172.28.128.5:5708 1.212332ms OK node2 node3.directfs 172.28.128.5:5703 1.192792ms OK node2 node3.etcd 172.28.128.5:5707 1.270076ms OK node2 node1.etcd 172.28.128.3:5707 1.218522ms OK node2 node1.api 172.28.128.3:5705 1.363071ms OK node2 node1.nats 172.28.128.3:5708 1.349383ms OK Solution: Open ports following the prerequisites page.\nPeer discovery - Pod allocation Issue: StorageOS nodes can\u0026rsquo;t join the cluster and show the following log entries.\ntime=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=info msg=\u0026#34;not first cluster node, joining first node\u0026#34; action=create address=172.28.128.5 category=etcd host=node3 module=cp target=172.28.128.6 time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=error msg=\u0026#34;could not retrieve cluster config from api\u0026#34; status_code=503 time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=error msg=\u0026#34;failed to join existing cluster\u0026#34; action=create category=etcd endpoint=\u0026#34;172.28.128.3,172.28.128.4,172.28.128.5,172.28.128.6\u0026#34; error=\u0026#34;503 Service Unavailable\u0026#34; module=cp time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=info msg=\u0026#34;retrying cluster join in 5 seconds...\u0026#34; action=create category=etcd module=cp Reason: StorageOS uses a gossip protocol to discover the nodes in the cluster. When StorageOS starts, one or more active nodes must be referenced so new nodes can query existing nodes for the list of members. This error indicates that the node can\u0026rsquo;t connect to any of the nodes in the known list. The known list is defined in the JOIN variable.\nIf there are no active StorageOS nodes, the bootstrap process will elect the first node in the JOIN variable as master, and the rest will try to discover from it. In case of that node not starting, the whole cluster will remain unable to bootstrap.\nInstallations of StorageOS use a DaemonSet, and by default do not schedule StorageOS pods to master nodes, due to the presence of the node-role.kubernetes.io/master:NoSchedule taint that is typically present. In such cases the JOIN variable must not contain master nodes or the StorageOS cluster will remain unable to start.\nAssert: Check that the first node of the JOIN variable started properly.\nroot@node1:~/# kubectl -n kube-system describe ds/storageos | grep JOIN JOIN: 172.28.128.3,172.28.128.4,172.28.128.5 root@node1:~/# kubectl -n kube-system get pod -o wide | grep 172.28.128.3 storageos-8zqxl 1/1 Running 0 2m 172.28.128.3 node1 Solution: Make sure that the JOIN variable doesn\u0026rsquo;t specify the master nodes. In case you are using the discovery service, it is necessary to ensure that the DaemonSet won\u0026rsquo;t allocate Pods on the masters. This can be achieved with taints, node selectors or labels.\nFor installations with the StorageOS operator you can specify which nodes to deploy StorageOS on using nodeSelectors. See examples in the Cluster Operator Examples page.\nFor more advanced installations using compute-only and storage nodes, check the storageos.com/deployment=computeonly label that can be added to the nodes through Kubernetes node labels, or StorageOS in the Labels page.\nLIO Init:Error Issue: StorageOS pods not starting with Init:Error\nkubectl -n kube-system get pod NAME READY STATUS RESTARTS AGE storageos-2kwqx 0/3 Init:Err 0 6s storageos-cffcr 0/3 Init:Err 0 6s storageos-d4f69 0/3 Init:Err 0 6s storageos-nhq7m 0/3 Init:Err 0 6s Reason: This indicates that since the Linux open source SCSI drivers are not enabled, StorageOS cannot start. The StorageOS DaemonSet enables the required kernel modules on the host system. If you are seeing these errors it is because that container couldn\u0026rsquo;t load the modules.\nAssert Check the logs of the init container.\nkubectl -n kube-system logs $ANY_STORAGEOS_POD -c storageos-init In case of failure, it will show the following output, indicating which kernel modules couldn\u0026rsquo;t be loaded or that they are not properly configured:\nChecking configfs configfs mounted on sys/kernel/config Module target_core_mod is not running executing modprobe -b target_core_mod Module tcm_loop is not running executing modprobe -b tcm_loop modprobe: FATAL: Module tcm_loop not found. Solution: Install the required kernel modules (usually found in the linux-image-extra-$(uname -r) package of your distribution) on your nodes following this prerequisites page and delete StorageOS pods, allowing the DaemonSet to create the pods again.\nLIO not enabled Issue: StorageOS node can\u0026rsquo;t start and shows the following log entries.\ntime=\u0026#34;2018-09-24T14:34:40Z\u0026#34; level=error msg=\u0026#34;liocheck returned error\u0026#34; category=liocheck error=\u0026#34;exit status 1\u0026#34; module=dataplane stderr=\u0026#34;Sysfs root \u0026#39;/sys/kernel/config/target\u0026#39; is missing, is kernel configfs present and target_core_mod loaded? category=fslio level=warn\\nRuntime error checking stage \u0026#39;target_core_mod\u0026#39;: SysFs root missing category=fslio level=warn\\nliocheck: FAIL (lio_capable_system() returns failure) category=fslio level=fatal\\n\u0026#34; stdout= time=\u0026#34;2018-09-24T14:34:40Z\u0026#34; level=error msg=\u0026#34;failed to start dataplane services\u0026#34; error=\u0026#34;system dependency check failed: exit status 1\u0026#34; module=command Reason: This indicates that one or more kernel modules required for StorageOS are not loaded.\nAssert The following kernel modules must be enabled in the host.\nlsmod | egrep \u0026#34;^tcm_loop|^target_core_mod|^target_core_file|^configfs\u0026#34; Solution: Install the required kernel modules (usually found in the linux-image-extra-$(uname -r) package of your distribution) on your nodes following this prerequisites page and restart the container.\n(OpenShift) StorageOS pods missing \u0026ndash; DaemonSet error StorageOS DaemonSet doesn\u0026rsquo;t have any pod replicas. The DaemonSet couldn\u0026rsquo;t allocate any Pod due to security issues.\nIssue: [root@master02 standard]# oc get pod No resources found. [root@master02 standard]# oc describe daemonset storageos (...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 0s (x12 over 10s) daemonset-controller Error creating: pods \u0026#34;storageos-\u0026#34; is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used provider restricted: .spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.volumes[0]: Invalid value: \u0026#34;hostPath\u0026#34;: hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: \u0026#34;hostPath\u0026#34;: hostPath volumes are not allowed to be used spec.volumes[2]: Invalid value: \u0026#34;hostPath\u0026#34;: hostPath volumes are not allowed to be used spec.volumes[3]: Invalid value: \u0026#34;hostPath\u0026#34;: hostPath volumes are not allowed to be used spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed capabilities.add: Invalid value: \u0026#34;SYS_ADMIN\u0026#34;: capability may not be added spec.initContainers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.initContainers[0].securityContext.containers[0].hostPort: Invalid value: 5705: Host ports are not allowed to be used spec.initContainers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed capabilities.add: Invalid value: \u0026#34;SYS_ADMIN\u0026#34;: capability may not be added spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 5705: Host ports are not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used] Reason: The OpenShift cluster has security context constraint policies enabled that forbid any pod, without an explicitly set policy for the service account, to be allocated.\nAssert: Check if the StorageOS ServiceAccount can create pods with enough permissions\noc get scc privileged -o yaml # Or custom scc with enough privileges (...) users: - system:admin - system:serviceaccount:openshift-infra:build-controller - system:serviceaccount:management-infra:management-admin - system:serviceaccount:management-infra:inspector-admin - system:serviceaccount:storageos:storageos \u0026amp;lt;-- - system:serviceaccount:tiller:tiller If the StorageOS sa system:serviceaccount:storageos:storageos is in the privileged scc it will be able to create pods.\nSolution: Add the ServiceAccount system:serviceaccount:storageos:storageos to a scc with enough privileges.\noc adm policy add-scc-to-user privileged system:serviceaccount:storageos:storageos Getting Help If our troubleshooting guides do not help resolve your issue, please see our support section for details on how to get in touch with us.\n","excerpt":"This section is aimed to help you troubleshoot issues in your cluster, whether they are related to …","ref":"/docs/operations/troubleshooting/","title":"Troubleshooting"},{"body":"This document details a step-by-step procedure to remove StorageOS from a Kubernetes cluster.\nRemember that StorageOS enables the stateful applications within your cluster. It is very important to remove any applications that rely on StorageOS before you remove StorageOS itself, or those applications will suffer unrecoverable errors.\nRemove Stateful Workloads and Data   Delete any resources using StorageOS volumes\nDelete any statefulsets, deployments or pods that are using StorageOS Volumes.\n  Delete PVCs using StorageOS\nDelete any Persistent Volume Claims that are using StorageOS.\n$ kubectl -n $NS delete pvc $PVC  This will delete data held by StorageOS and won\u0026rsquo;t be recoverable.\n   Remove StorageOS Cluster   Delete StorageOS Cluster\n$ kubectl get storageoscluster --all-namespaces # Find the namespace where the Custom Resource runs $ kubectl -n $NS delete storageoscluster --all # Usually to be found in storageos-operator   Wait until the StorageOS resources are gone\n$ kubectl -n kube-system get pod # NS: Namespace where StorageOS Daemonset is running, usually \u0026#39;kube-system\u0026#39;   Uninstall the StorageOS Operator  Delete the Cluster Operator once the StorageOS Pods are terminated\n   Delete the StorageOS Operator deployment\n$ kubectl delete -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml   Procedure is finished. StorageOS is now uninstalled.\nRemove StorageOS contents and metadata (unrecoverable) The steps up until now have been recoverable - as long as the etcd backing StorageOS and the contents of /var/lib/storageos on your nodes are safe then StorageOS can be reinstalled. For complete removal and recovery of disk space, you can use the following procedure.\n The following steps will delete all data held by StorageOS and won\u0026rsquo;t be recoverable.\n   Remove the StorageOS data directory\nThere are two ways to remove the StorageOS data directory:\n  (Option 1) Login in to the hosts and execute the following commands\n$ sudo rm -rf /var/lib/storageos $ sudo umount /var/lib/kubelet/plugins_registry/storageos   (Option 2) Execute the following command to deploy a DaemonSet that removes the StorageOS data directory.\n N.B This step is irreversible and once the data is removed it cannot be recovered.\n  Run the following command where kubectl is installed and with the context set for your Kubernetes cluster.\n $ curl -s https://docs.storageos.com/sh/permanently-delete-storageos-data.sh | bash     Flush Etcd Data\nThis will remove any keys written by StorageOS.\n$ export ETCDCTL_API=3 $ etcdctl --endpoints=http://$ETCD_IP:2379 del --prefix \u0026#34;storageos\u0026#34; If running Etcd with mTLS, you can set the certificates location with the following command.\n$ export ETCDCTL_API=3 $ etcdctl --endpoints=https://$ETCD_IP:2379 \\  --cacert=/path/to/ca.pem \\  --cert=/path/to/client-cert.pem \\  --key=/path/to/client-key.pem \\  del --prefix \u0026#34;storageos\u0026#34;   ","excerpt":"This document details a step-by-step procedure to remove StorageOS from a Kubernetes cluster. …","ref":"/docs/operations/uninstall/","title":"Uninstall StorageOS"},{"body":"Update a volume Usage: storageos update volume [command] Available Commands: description Updates a volume\u0026#39;s description labels Updates a volume\u0026#39;s labels replicas Updates a volume\u0026#39;s target replica number size Updates a volume\u0026#39;s size Flags: -h, --help help for volume Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Update volume description $ storageos update volume description -h Updates a volume\u0026#39;s description Usage: storageos update volume description [volume name] [description] [flags] Examples: $ storageos update volume description my-volume-name \u0026#34;Volume for production IO\u0026#34; --namespace my-namespace-name Update volume labels $ storageos update volume labels -h Updates a volume\u0026#39;s labels Usage: storageos update volume labels [volume name] [labels] [flags] Examples: $ storageos update volume labels my-volume-name tier=production,app=my-app --namespace my-namespace-name $ storageos update volume labels my-volume-name --delete app=my-app --upsert tier=production --namespace my-namespace-name $ storageos update volume labels my-volume-name --upsert tier=production --namespace my-namespace-name $ storageos update volume labels my-volume-name --delete app=my-app --namespace my-namespace-name Update volume replicas $ storageos update volume replicas -h Updates a volume\u0026#39;s target replica number Usage: storageos update volume replicas [volume name] [target number] [flags] Examples: $ storageos update volume replicas my-volume-name 2 --namespace my-namespace-name Update volume size $ storageos update volume size -h Updates a volume\u0026#39;s size Usage: storageos update volume size [volume name] [size] [flags] Examples: $ storageos update volume size my-volume-name 42GiB --namespace my-namespace-name $ storageos update volume size my-volume-name 42gib --namespace my-namespace-name ","excerpt":"Update a volume Usage: storageos update volume [command] Available Commands: description Updates a …","ref":"/docs/reference/cli/update/","title":"Update volume"},{"body":"This document details a step-by-step procedure to upgrade a StorageOS v2 cluster.\nKeep in mind that upgrading a cluster will require minor downtime of applications using StorageOS volumes. However we will take steps to minimize the required downtime as much as possible.\nUpgrade StorageOS  Ensure that you have read the PIDs prerequisite introduced in StorageOS v2.3 and that you check the init container logs to ensure your environments PID limits are set correctly.\n  Warning: To reduce downtime, it is recommended to docker pull the new StorageOS container image storageos/node:v2.4.0 on the nodes beforehand so that the cluster spins up faster!\n   First make sure you keep a backup of all the StorageOS yaml files. You can reuse the StorageOSCluster configuration file to easily upgrade your cluster. You can also backup the Statefulset yaml files to keep track of the replicas.\nkubectl get pod -n storageos-operator -o yaml \u0026gt; storageos_operator.yaml kubectl get storageoscluster -n storageos-operator -o yaml \u0026gt; storageos_cr.yaml kubectl get statefulset --all-namespaces \u0026gt; statefulset-sizes.yaml   Scale all stateful applications that use StorageOS volumes to 0.\n  Delete the StorageOSCluster CR.\nkubectl delete storageoscluster cluster-storageos -n storageos-operator If you have renamed the StorageOSCluster resource you can find it by using the command below.\nkubectl get storageoscluster --all-namespaces   Deploy the new operator.\n# Do not apply the yaml resources. It requires a deletion and posterior creation kubectl delete -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml # Wait until the resources are deleted kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.0/storageos-operator.yaml  Do not run a kubectl apply. A delete and create are needed Warning: If you have made changes to the CRDs, Service Account or Cluster Role, make sure you migrate those changes in the StorageOS operator yaml.\n   Edit the StorageOSCluster Custom Resource (storageos_cr.yaml) with the new node image version.\nimages:nodeContainer:\u0026#34;storageos/node:v2.4.0\u0026#34;If you\u0026rsquo;re not using any of the hardcoded container image versions make sure to delete them as well and only leave the nodeContainer image as shown above.\nMake sure to also delete all metadata that you\u0026rsquo;re not using. The final metadata stanza should be something similar to that:\nmetadata:name:\u0026#34;cluster-storageos\u0026#34;namespace:\u0026#34;storageos-operator\u0026#34;  Re-create the StorageOSCluster\nkubectlcreate-fstorageos_cr.yaml  Wait for all the StorageOS pods to enter the RUNNING state\nkubectl get pods -l app=storageos -A -w   Scale your stateful applications back up.\n  Congratulations, you now have the latest version StorageOS!\n","excerpt":"This document details a step-by-step procedure to upgrade a StorageOS v2 cluster.\nKeep in mind that …","ref":"/docs/operations/upgrade/","title":"Upgrade StorageOS"},{"body":"A StorageOS cluster admin can create users and restrict their access rights to StorageOS namespaces using policies.\n Note: Users are created with access to the default namespace. This access is only revoked when a policy is created for the user or their group.\n Managing users Users can be created and deleted by navigating to the \u0026ldquo;Users\u0026rdquo; section of the GUI.\nAlternatively users can also be managed using the CLI.\nTo create a user with the CLI, run:\n$ storageos user create jim The above command will create a user named jim. The command will also prompt you to enter a password for the newly created user.\n","excerpt":"A StorageOS cluster admin can create users and restrict their access rights to StorageOS namespaces …","ref":"/docs/operations/users/","title":"User Management"},{"body":"Using Velero to backup your StorageOS volumes Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.\nVelero consists of the following components:\n A Velero Server in the cluster A CLI client A restic daemonset in the cluster  Here\u0026rsquo;s a diagram on how Velero backups work:\nAnd here\u0026rsquo;s how Velero restores work:\nIn our case, we will use MinIO in the Kubernetes cluster as the \u0026ldquo;cloud provider\u0026rdquo; pictured in the backup diagram. MinIO is an object store that uses an S3 compatible API and as such can be used to store our backed up resources.\nWe will set up MinIO through a StatefulSet with a 50GiB StorageOS volume and a ClusterIP service. The UI can be accessed through the browser by port-forwarding the MinIO service.\nVelero uses Restic to backup Kubernetes volumes. Restic is a fast and secure backup program for filesystems whose documentation can be found here. The way it works is that it scans the volume directory for its files and then splits those files into blobs which are then sent to MinIO. Here\u0026rsquo;s how it integrates with Velero.\nPrerequisites Here are the prerequisites for running Velero in your Kubernetes cluster:\n Kubernetes cluster version 1.17 to 1.21 with DNS Kubectl installed Velero cli installed https://Velero.io/docs/main/basic-install/   Velero can also be installed from a helm chart\n Install MinIO with a StorageOS volume   First, make sure to clone the StorageOS use cases repository and navigate to the Velero directory:\ngit clone https://github.com/storageos/use-cases.git cd use-cases/Velero   Installing MinIO is really simple, just deploy it using the minio-deploy.yaml manifest file:\nkubectl apply -f ./minio   Confirm that MinIO was deployed successfully:\n$ kubectl get pods -n velero NAME READY STATUS RESTARTS AGE minio-0 1/1 Running 0 3m48s minio-setup-zvcdg 0/1 Completed 1 3m47s You can access the web UI of MinIO by port-forwarding the MinIO service with this command:\nkubectl port-forward service/minio -n velero 9000   Install Velero Use the following command to install Velero via the Velero CLI or alternatively use the helm chart. To install it using the Velero cli, just run this command:\n The AWS plugin is being used because MinIO implements the S3 API. This is required even if you\u0026rsquo;re not using AWS.\n velero install \\ --provider aws \\ --plugins velero/velero-plugin-for-aws:v1.0.0 \\ --bucket velero \\ --secret-file ./credentials-Velero \\ --use-volume-snapshots=false \\ --backup-location-config region=minio,s3ForcePathStyle=\u0026#34;true\u0026#34;,s3Url=http://minio.velero.svc:9000 \\ --use-restic Make sure that Velero is installed correctly:\n$ kubectl logs deployment/velero -n velero ... time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Server started successfully\u0026#34; logSource=\u0026#34;pkg/cmd/server/server.go:881\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=restic-repository logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=restore logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=backup-sync logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=backup logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=backup-deletion logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Checking for expired DeleteBackupRequests\u0026#34; controller=backup-deletion logSource=\u0026#34;pkg/controller/backup_deletion_controller.go:551\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Done checking for expired DeleteBackupRequests\u0026#34; controller=backup-deletion logSource=\u0026#34;pkg/controller/backup_deletion_controller.go:579\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=schedule logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=downloadrequest logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=gc-controller logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=serverstatusrequest logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; Quiesce Before moving on to the MySQL use case, it is important to talk about quiescence. To quiesce is to pause or alter a device or application to achieve a consistent state, usually in preparation for a backup or other maintenance. Quiescence is very important when backing up real world applications because in flight or cached write operations of the filesystem during the backup process can cause corruption of the backed up data. It is a common mistake for engineers to back up their data without quiescing the filesystem and noticing corruption when they need to restore from that backup. A simple form of quiescence is the unix command fsfreeze, but in most cases, application specific quiescence should be used instead.\nMySQL use case For the our use case, we\u0026rsquo;ll use Velero with a real world application, MySQL.\n  First deploy MySQL, based on the MySQL use case:\nkubectl apply -f ./mysql Notice that the Statefulset also includes 5 annotations:\nannotations:backup.velero.io/backup-volumes:datapre.hook.backup.velero.io/command:\u0026#39;[\u0026#34;/sbin/fsfreeze\u0026#34;, \u0026#34;--freeze\u0026#34;, \u0026#34;/var/lib/mysql\u0026#34;]\u0026#39;pre.hook.backup.velero.io/container:fsfreezepost.hook.backup.velero.io/command:\u0026#39;[\u0026#34;/sbin/fsfreeze\u0026#34;, \u0026#34;--unfreeze\u0026#34;, \u0026#34;/var/lib/mysql\u0026#34;]\u0026#39;post.hook.backup.velero.io/container:fsfreezeThe first annotation specifies which volume to backup using restic. The other annotations are used to perform an fsfreeze on the volume mount point using pre and post backup hooks, for more details about Velero pre/post backup hooks please see their documentation here.\nWe have to specify to use the fsfreeze ubuntu container since the MySQL container doesn\u0026rsquo;t support fsfreeze\n- name:fsfreezeimage:ubuntu:bionicsecurityContext:privileged:truevolumeMounts:- name:datamountPath:/var/lib/mysqlcommand:- \u0026#34;/bin/bash\u0026#34;- \u0026#34;-c\u0026#34;- \u0026#34;sleep infinity\u0026#34;  Wait for the pod to spin up:\n$ kubectl get pods -n mysql NAME READY STATUS RESTARTS AGE client 1/1 Running 0 24m mysql-0 2/2 Running 0 24m   Exec into the MySQL pod and populate it with data using the commands below.\n$ kubectl exec mysql-0 -n mysql -ti -c mysql -- mysql mysql\u0026gt; create database shop; mysql\u0026gt; use shop; mysql\u0026gt; create table books (title VARCHAR(256), price decimal(4,2)); mysql\u0026gt; insert into books value (\u0026#39;Gates of Fire\u0026#39;, 13.99); mysql\u0026gt; select * from books; +---------------+-------+ | title | price | +---------------+-------+ | Gates of Fire | 13.99 | +---------------+-------+ 1 row in set (0.00 sec) mysql\u0026gt; exit   Create the Velero backup:\nvelero backup create mysql-backup --include-namespaces mysql --wait   Confirm that all the Kubernetes objects are there and the restic backups completed successfully:\nvelero backup describe mysql-backup --details   After the backup is finished, delete the StatefulSet and PVC.\n N.B. It\u0026rsquo;s important to make sure that the StatefulSet is deleted because the restore would be unable to complete if a StatefulSet pod is recreated during the restore process.\n kubectl delete statefulset mysql -n mysql kubectl delete pvc data-mysql-0 -n mysql   Make sure that the pod is fully terminated:\n$ kubectl get pods -n mysql NAME READY STATUS RESTARTS AGE client 1/1 Running 0 25m   Restore MySQL using Velero:\nvelero restore create --from-backup mysql-backup --wait   Wait for the MySQL pod to spin up and see if the data is backed up:\n$ kubectl exec mysql-0 -n mysql -ti -c mysql -- mysql mysql\u0026gt; use shop; mysql\u0026gt; select * from books; +---------------+-------+ | title | price | +---------------+-------+ | Gates of Fire | 13.99 | +---------------+-------+ 1 row in set (0.00 sec) mysql\u0026gt; exit   The data should be there and the backup was restored successfully!\n","excerpt":"Using Velero to backup your StorageOS volumes Velero is an open source tool to safely backup and …","ref":"/docs/usecases/velero-backups/","title":"Velero Backups"},{"body":"$ storageos version --help View version information for the StorageOS CLI Usage: storageos version [flags] Flags: -h, --help help for version Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos version --help View version information for the StorageOS CLI Usage: storageos version …","ref":"/docs/reference/cli/version/","title":"Version"},{"body":"StorageOS supports offline resize of volumes, either through editing a PVC storage request, or by updating the volume config via the CLI or UI. A volume cannot be resized while it is in use. To ensure that it is not, for a resize operation to take place the volume must not be attached to a node. This requires that any pods using a volume be scaled down for the resize to take place.\nPlease note that StorageOS only supports increasing volume size. For more information about how the resize works please see our Resize Concepts page.\nResizing a Volume In order to resize a PVC the storage request field must be updated.\napiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-1 spec: storageClassName: fast accessModes: - ReadWriteOnce resources: requests: storage: 5Gi The Volume must be offline for this to take effect. Therefore, before performing the operation, scale down any pods using the volume. This will ensure that it is not in use.\nIn order to edit a PVC you can use kubectl edit or kubectl apply to make changes.\n N.B. Resizing a volume without updating the PVC directly will NOT result in the PVC being updated. The methods below are included for completeness. In Kubernetes environments editing the PVC is the preferred method for resizing a volume.\n To resize a volume using the StorageOS CLI use the volume update command\n$ storageos update volume size pvc-a47cfa03-cc92-4ec9-84ab-00e5516c64fa 10GiB Name: pvc-a47cfa03-cc92-4ec9-84ab-00e5516c64fa ID: 925e667f-91d3-465a-9391-8fdb56d0c9ff Size: 11 GB Description: AttachedOn: Replicas: 1x ready Labels: - csi.storage.k8s.io/pv/name pvc-a47cfa03-cc92-4ec9-84ab-00e5516c64fa - csi.storage.k8s.io/pvc/name pvc-1 - csi.storage.k8s.io/pvc/namespace default - foo bar - pool default - storageos.com/replicas 1 Volume pvc-a47cfa03-cc92-4ec9-84ab-00e5516c64fa (925e667f-91d3-465a-9391-8fdb56d0c9ff) updated. Size changed. To resize a volume using the StorageOS UI, navigate to the volumes section and click the edit pencil in order to update the volume config.\n","excerpt":"StorageOS supports offline resize of volumes, either through editing a PVC storage request, or by …","ref":"/docs/operations/resize/","title":"Volume Resize"},{"body":"StorageOS volumes are a logical construct which represent a writeable volume and exhibit standard POSIX semantics. StorageOS presents volumes as mounts into containers via the Linux LIO subsystem.\nConceptually, StorageOS volumes have a frontend presentation, which is what the application sees, and a backend presentation, which is the actual on-disk format. Depending on the configuration, frontend and backend components may be on the same or different hosts.\nVolumes are formatted using the linux standard ext4 filesystem by default. Kubernetes users may change the default filesystem type to ext2, ext3, ext4, or xfs by setting the fsType parameter in their StorageClass (see Supported Filesystems for more information). Different filesystems may be supported in the future.\nStorageOS volumes are represented on disk in two parts. Actual volume data is written to blob files in /var/lib/storageos/data/dev[\\d+]. Inside these directories, each StorageOS block device gets two blob files of the form vol.xxxxxx.y.blob, where x is the inode number for the device, and y is an index between 0 and 1. We provide two blob files in order to ensure that certain operations which require locking do not impede in-flight writes to the volume.\nIn systems which have multiple /var/lib/storageos/data/dev[\\d+] directories, two blob files are created per block device. This allows us to load-balance writes across multiple devices. In cases where dev directories are added after a period of run time, later directories are favoured for writes until the data is distributed evenly across the blob files.\nMetadata is kept in directories named /var/lib/storageos/data/db[\\d+]. We maintain an index of all blocks written to the blob file inside the metadata store, including checksums. These checksums allow us to detect bitrot, and return errors on reads, rather than serve bad data. In future versions we may implement recovery from replicas for volumes with one or more replicas defined.\nStorageOS metadata requires approximately 2.7GB of storage per 1TB of allocated blocks in the associated volume. This size is consistent irrespective of data compression defined on the volume.\nTo ensure deterministic performance, individual StorageOS volumes must fit on a single node.\nMinimum Volume Size The minimum volume size StorageOS supports is 1 GiB.\nTRIM StorageOS volumes support TRIM/Unmap which allows the space allocated to deleted blocks to be reclaimed from the backend blob files that back each volume when a TRIM call is made. Support for TRIM is enabled by default for all uncompressed volumes, volumes are created without compression enabled by default. For more information on how to TRIM a filesystem see TRIM operations.\nVolume Resize StorageOS v2.1 supports offline resize of volumes. This means that a volume cannot be resized while it is in use. Furthermore, in order for a resize operation to take place the volume must not be attached to a node. This is to ensure that the volume is not in use.\nThis means that if a Kubernetes pod is currently consuming a volume that a resize request has been issued for, the resize will not be actioned until the pod is terminated and the volume is detached from the node. The StorageOS controlplane will then attach the volume to the node that holds the master deployment and resize the underlying block device and then run resize2fs to expand the filesystem.\nFor a walk through of how to resize a volume please see the Volume Resize operations page.\nVolume Encryption Volumes can be configured on creation to have encryption-at-rest. Data is encrypted with XTS-AES and decrypted upon use. Please see the Encryption reference page for more information.\n","excerpt":"StorageOS volumes are a logical construct which represent a writeable volume and exhibit standard …","ref":"/docs/concepts/volumes/","title":"Volumes"},{"body":"ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.\nUsing StorageOS persistent volumes with Apache Zookeeper means that if a pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up, the pod data is immediately available. Should Kubernetes schedule the Zookeeper pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not the original StorageOS master volume is located on the same node.\nAs Zookeeper has features to allow it to handle replication, and as such careful consideration of whether to allow StorageOS or Zookeeper to handle replication is required.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Zookeeper on Kubernetes Pre-requisites  StorageOS is assumed to have been installed; please check for the latest available version here.  Helm To simplify the deployment of Zookeeper, we\u0026rsquo;ve used this Zookeeper helm chart (incubator) (version 1.2.2, app version 3.4.10) and rendered it into the example deployment files you can find in our GitHub repo.\nDeployment Clone the use cases repo You can find the latest files in the StorageOS use cases repository in /zookeeper/\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion ---apiVersion:apps/v1beta1kind:StatefulSetmetadata:name:zookeeper...spec:replicas:3# \u0026lt;--- number of zookeeper pods...containers:- name:zookeeperimage:\u0026#34;gcr.io/google_samples/k8szk:v3\u0026#34;imagePullPolicy:IfNotPresent...volumeMounts:- name:datamountPath:/var/lib/zookeepervolumeClaimTemplates:- metadata:name:dataspec:accessModes:- \u0026#34;ReadWriteOnce\u0026#34;storageClassName:\u0026#34;fast\u0026#34;# \u0026lt;--- the StorageClass to useresources:requests:storage:\u0026#34;5Gi\u0026#34;# \u0026lt;--- storage requested per podThis excerpt is from the StatefulSet definition (10-statefulset.yaml). The file contains the PersistentVolumeClaim template that will dynamically provision the necessary storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.\nCreate the kubernetes objects cd storageos-usecases kubectl apply -f ./zookeeper/ Confirm Zookeeper is up and running $ kubectl get pods NAME READY STATUS RESTARTS AGE zookeeper-0 1/1 Running 0 2m30s zookeeper-1 1/1 Running 0 112s zookeeper-2 1/1 Running 0 56s zookeeper-test-client 1/1 Running 0 2m30s Connect to Zookeeper Connect to the zookeeper client pod and list existing topics using the service endpoint\nkubectl exec -it zookeeper-test-client /bin/bash and issue a command to the zookeeper service\nzkCli.sh -server zookeeper ls /zookeeper ","excerpt":"ZooKeeper is a centralized service for maintaining configuration information, naming, providing …","ref":"/docs/usecases/zookeeper/","title":"Zookeeper"}]