-
Notifications
You must be signed in to change notification settings - Fork 59
Custom configuration
The bundled Cassandra docker image includes a slightly customised Cassandra configuration that better suited for running inside a container, but leaves the bulk of the configuration up to Cassandra operator.
Cassandra operator automatically configures Cassandra and the JVM with (what we consider) sane defaults for the deployment, such as adjusting JVM heap sizes and GC strategies to values appropriate for the container resource limits.
That being said, different workloads require to tune the configuration to achieve best performance, and this tuning cannot be achieved automatically by the operator. Hence custom user configuration overrides are also supported.
The operator supports rack aware Cassandra deployments and will automatically configure and manage a separate StatefulSet per each of the defined racks. You can also assign node placement labels to each rack, so you can leverage Kubernetes fault domains or other placement strategies. To define racks, modify the racks object in the CRD. The operator will automatically balance replicas across your racks.
In case you do not specify any racks, all replicas will be placed into same "rack" called
rack1and there will be no balancing done for placing the pods.
Example yaml fragment:
racks:
- name: "west1-b"
labels:
failure-domain.beta.kubernetes.io/zone: europe-west1-b
- name: "west1-c"
labels:
failure-domain.beta.kubernetes.io/zone: europe-west1-c
- name: "west1-a"
labels:
failure-domain.beta.kubernetes.io/zone: europe-west1-aThe field called labels above in racks is a map which is eventually used as a nodeSelector for a pod. By this way, you can control where a pod will be scheduled.
If you tained a node or nodes for Cassandra deployment, in order to schedule pods to them, you need tolerations. Tolerations are exposed into racks above. For example, it would look like this:
racks:
- name: "west1-a"
tolerations:
- key: "key"
operator: "Equal"
value: "value"
effect: "NoSchedule"
- name: "west1-b"
tolerations:
- key: "key"
operator: "Equal"
value: "value"
effect: "NoSchedule"You can as well specify affinity for each rack.
If you do not use rack-aware deployment, you use it anyway, you just do not know about it. The default rack is called rack1 so you would have to specify tolerations under that rack name.
See the full example yaml here
cassandra-operator supports mounting a custom ConfigMap (via a ConfigMapVolumeSource) into the Cassandra container. The contents of this ConfigMap will be overlaid on top the images' out-of-box defaults and operator-generated configuration.
More specifically, all Cassandra and JVM configuration exists under /etc/cassandra inside the container (see Directory Layout).
The specified ConfigMap volume will have its contents extracted into /etc/cassandra when the container starts, allowing customisation to the Cassandra configuration by either adding file fragments to specific directories (preferred, see below) or by overwriting existing files entirely.
To customize the Cassandra configuration first create a ConfigMap object in the same K8s namespace as the Cassandra deployment, and fill it with the required configuration, where the data keys are arbitrary (they are referenced by the ConfigMapVolumeSource) and the values are the configuration files or file fragments.
Then set/update the CRD attribute userConfigMapVolumeSource to a ConfigMapVolumeSource object that
defines the ConfigMap key -> path mappings.
Let's say you want to modify the number of threads available for certain operations in Cassandra.
Create a YAML file 100-concurrent.yaml that just defines those properties:
concurrent_reads: 12
concurrent_writes: 12
concurrent_counter_writes: 12Create a config map from that YAML file:
$ kubectl create configmap concurrent-data --from-file=100-concurrent.yaml
Modify the CassandraDataCenter CRD to reference the newly created ConfigMap:
apiVersion: cassandraoperator.instaclustr.com/v1alpha1
kind: CassandraDataCenter
metadata:
name: example-cdc
...
spec:
...
userConfigMapVolumeSource:
# the name of the ConfigMap
name: concurrent-data
# ConfigMap keys -> file paths (relative to /etc/cassandra)
items:
- key: 100-concurrent.yaml
path: cassandra.yaml.d/100-concurrent.yamlCassandra will load the cassandra.yaml.d/100-concurrent.yaml file as well as the default settings managed by the operator!
It is possible to override / add options to JVM. All it takes is to
construct a file and mount it under path jvm.options.d.
In case you want to provide your own GC settings as this might be highly
specific to your load and application and so on, you can do so by
specifying an option file under path jvm.options.d/gc.options. If specified,
we will not step into GC settings and you can set it how you wish.
The Cassandra image that is included with this project has the cassandra-exporter built in, so it is ready to expose metrics to be gathered by a Prometheus instance. In order to use this capability automatically, one must have a running Prometheus operator in the same Kubernetes cluster as well as a Prometheus object.
Utilising the labels used in Prometheus selector, add the following configuration to the cluster CRD:
...
spec:
...
prometheusSupport: true
You need to as well specify labels on prometheusService as it is specified below so the integration
works correctly.
And then create the Cassandra cluster. After starting, the Prometheus will pick up the Cassandra targets and will begin collecting metrics.
You can attach custom labels to your objects by specifying a field in CDC spec like this:
spec:
operatorLabels:
nodesService:
mynodesservicelabel: labelvalue1
seedNodesService:
myseedlabel: somevalue
statefulSet:
mystatefullabel: labelvalue2
podTemplate:
mypodlabel: label1
myanotherpod: label2
prometheusService:
customPrometheusLabel: valueSimilarly as it is done for custom labels, you can annotate objects by specifying a field in CDC spec like this:
spec:
operatorAnnotations:
nodesService:
mynodesserviceannotation: annotationvalue1
seedNodesService:
myseedannotation: somevalue
statefulSet:
mystatefulsetannotation: annotationvalue2
podTemplate:
mypodannotation: annotation1
myanotherpod: annotation2
prometheusService:
customPrometheusAnnotation: valueCassandra operator is able to run with performance optimisation, this is driven by a parameter in spec called
optimizeKernelParams which has to be set to true. When enabled, it sets disk access mode to mmap and it starts
another init container which sets kernel parameters which are recommended by Cassandra team. Here we show
relevant code snippets which should be self-explanatory.
To use these features, one needs to use cassandra-performance service account which enables these parameters on K8S side:
// Set disk_access_mode to 'mmap' only when user specifies `optimizeKernelParams: true`.
if cdc.Spec.OptimizeKernelParams {
cc.DiskAccessMode = "mmap"
}
// Create sysctl init container only when user specifies `optimizeKernelParams: true`.
if rctx.cdc.Spec.OptimizeKernelParams {
initContainers = append(initContainers, *newSysctlLimitsContainer(rctx.cdc))
}
func newSysctlLimitsContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter) *corev1.Container {
return &corev1.Container{
Name: "sysctl-limits",
Image: "busybox:latest",
ImagePullPolicy: cdc.Spec.ImagePullPolicy,
SecurityContext: &corev1.SecurityContext{
Privileged: boolPointer(true),
},
Command: []string{"sh", "-xuec"},
Args: []string{
`sysctl -w vm.max_map_count=1048575`,
},
}
}
// Create C* container with capabilities required for performance tweaks only when user
// specifies `optimizeKernelParams: true`.
if cdc.Spec.OptimizeKernelParams {
container.SecurityContext = &corev1.SecurityContext{
Capabilities: &corev1.Capabilities{
Add: []corev1.Capability{
"IPC_LOCK", // C* wants to mlock/mlockall
"SYS_RESOURCE", // permit ulimit adjustments
},
},
}