Skip to content

Commit

Permalink
Feat Query-Scheduler (grafana#268)
Browse files Browse the repository at this point in the history
* docs: add query-scheduler

Signed-off-by: Joao Marques <[email protected]>

* improve apiVersion for pdb

Signed-off-by: nschad <[email protected]>

* fix changelog

Signed-off-by: nschad <[email protected]>

* add query-scheduler to test-deployment

Signed-off-by: nschad <[email protected]>

* fix query-scheduler to test-deployment

Signed-off-by: nschad <[email protected]>

Co-authored-by: nschad <[email protected]>
  • Loading branch information
marquezika and nschad authored Mar 14, 2022
1 parent 6a47771 commit f5b4a98
Show file tree
Hide file tree
Showing 12 changed files with 401 additions and 0 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

* [ENHANCEMENT] Allow StoreGateway podManagementPolicy to be changed #332
* [BUGFIX] Correct a typo in enabling distribtuor HPA #334
* [FEATURE] Add Query-Scheduler #268

## 1.4.0 / 2022-03-08

Expand Down
44 changes: 44 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -622,6 +622,50 @@ Kubernetes: `^1.19.0-0`
| query_frontend.&ZeroWidthSpace;strategy.&ZeroWidthSpace;type | string | `"RollingUpdate"` | |
| query_frontend.&ZeroWidthSpace;terminationGracePeriodSeconds | int | `180` | |
| query_frontend.&ZeroWidthSpace;tolerations | list | `[]` | |
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;podAffinityTerm.&ZeroWidthSpace;labelSelector.&ZeroWidthSpace;matchExpressions[0].&ZeroWidthSpace;key | string | `"app.kubernetes.io/component"` | |
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;podAffinityTerm.&ZeroWidthSpace;labelSelector.&ZeroWidthSpace;matchExpressions[0].&ZeroWidthSpace;operator | string | `"In"` | |
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;podAffinityTerm.&ZeroWidthSpace;labelSelector.&ZeroWidthSpace;matchExpressions[0].&ZeroWidthSpace;values[0] | string | `"query-scheduler"` | |
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;podAffinityTerm.&ZeroWidthSpace;topologyKey | string | `"kubernetes.io/hostname"` | |
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;weight | int | `100` | |
| query_scheduler.&ZeroWidthSpace;annotations | object | `{}` | |
| query_scheduler.&ZeroWidthSpace;containerSecurityContext.&ZeroWidthSpace;enabled | bool | `true` | |
| query_scheduler.&ZeroWidthSpace;containerSecurityContext.&ZeroWidthSpace;readOnlyRootFilesystem | bool | `true` | |
| query_scheduler.&ZeroWidthSpace;enabled | bool | `false` | If true, querier and query-frontend will connect to it (requires Cortex v1.6.0+) https://cortexmetrics.io/docs/operations/scaling-query-frontend/#query-scheduler |
| query_scheduler.&ZeroWidthSpace;env | list | `[]` | |
| query_scheduler.&ZeroWidthSpace;extraArgs | object | `{}` | Additional Cortex container arguments, e.g. log.level (debug, info, warn, error) |
| query_scheduler.&ZeroWidthSpace;extraContainers | list | `[]` | |
| query_scheduler.&ZeroWidthSpace;extraPorts | list | `[]` | |
| query_scheduler.&ZeroWidthSpace;extraVolumeMounts | list | `[]` | |
| query_scheduler.&ZeroWidthSpace;extraVolumes | list | `[]` | |
| query_scheduler.&ZeroWidthSpace;initContainers | list | `[]` | |
| query_scheduler.&ZeroWidthSpace;lifecycle | object | `{}` | |
| query_scheduler.&ZeroWidthSpace;livenessProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;path | string | `"/ready"` | |
| query_scheduler.&ZeroWidthSpace;livenessProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;port | string | `"http-metrics"` | |
| query_scheduler.&ZeroWidthSpace;nodeSelector | object | `{}` | |
| query_scheduler.&ZeroWidthSpace;persistentVolume.&ZeroWidthSpace;subPath | string | `nil` | |
| query_scheduler.&ZeroWidthSpace;podAnnotations | object | `{"prometheus.io/port":"http-metrics","prometheus.io/scrape":"true"}` | Pod Annotations |
| query_scheduler.&ZeroWidthSpace;podDisruptionBudget.&ZeroWidthSpace;maxUnavailable | int | `1` | |
| query_scheduler.&ZeroWidthSpace;podLabels | object | `{}` | Pod Labels |
| query_scheduler.&ZeroWidthSpace;readinessProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;path | string | `"/ready"` | |
| query_scheduler.&ZeroWidthSpace;readinessProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;port | string | `"http-metrics"` | |
| query_scheduler.&ZeroWidthSpace;replicas | int | `2` | |
| query_scheduler.&ZeroWidthSpace;resources | object | `{}` | |
| query_scheduler.&ZeroWidthSpace;securityContext | object | `{}` | |
| query_scheduler.&ZeroWidthSpace;service.&ZeroWidthSpace;annotations | object | `{}` | |
| query_scheduler.&ZeroWidthSpace;service.&ZeroWidthSpace;labels | object | `{}` | |
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;additionalLabels | object | `{}` | |
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;enabled | bool | `false` | |
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;extraEndpointSpec | object | `{}` | Additional endpoint configuration https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#endpoint |
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;metricRelabelings | list | `[]` | |
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;relabelings | list | `[]` | |
| query_scheduler.&ZeroWidthSpace;startupProbe.&ZeroWidthSpace;failureThreshold | int | `10` | |
| query_scheduler.&ZeroWidthSpace;startupProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;path | string | `"/ready"` | |
| query_scheduler.&ZeroWidthSpace;startupProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;port | string | `"http-metrics"` | |
| query_scheduler.&ZeroWidthSpace;strategy.&ZeroWidthSpace;rollingUpdate.&ZeroWidthSpace;maxSurge | int | `0` | |
| query_scheduler.&ZeroWidthSpace;strategy.&ZeroWidthSpace;rollingUpdate.&ZeroWidthSpace;maxUnavailable | int | `1` | |
| query_scheduler.&ZeroWidthSpace;strategy.&ZeroWidthSpace;type | string | `"RollingUpdate"` | |
| query_scheduler.&ZeroWidthSpace;terminationGracePeriodSeconds | int | `180` | |
| query_scheduler.&ZeroWidthSpace;tolerations | list | `[]` | |
| ruler.&ZeroWidthSpace;affinity | object | `{}` | |
| ruler.&ZeroWidthSpace;annotations | object | `{}` | |
| ruler.&ZeroWidthSpace;containerSecurityContext.&ZeroWidthSpace;enabled | bool | `true` | |
Expand Down
15 changes: 15 additions & 0 deletions ci/test-deployment-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@ config:
join_members:
- '{{ include "cortex.fullname" $ }}-memberlist'

frontend:
scheduler_address: 'dns+{{ include "cortex.querySchedulerFullname" $ }}-headless:9095'

frontend_worker:
scheduler_address: 'dns+{{ include "cortex.querySchedulerFullname" $ }}-headless:9095'

ingester:
replicas: 1
autoscaling:
Expand Down Expand Up @@ -98,6 +104,15 @@ query_frontend:
extraVolumeMounts:
- name: tmp-test
mountPath: /tmp-test
query_scheduler:
replicas: 1
enabled: true
extraVolumes:
- name: tmp-test
emptyDir: {}
extraVolumeMounts:
- name: tmp-test
mountPath: /tmp-test
nginx:
replicas: 1
autoscaling:
Expand Down
4 changes: 4 additions & 0 deletions templates/querier/querier-dep.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,11 @@ spec:
args:
- "-target=querier"
- "-config.file=/etc/cortex/cortex.yaml"
{{- if .Values.query_scheduler.enabled }}
- "-querier.scheduler-address={{ template "cortex.fullname" . }}-query-scheduler-headless.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}:{{ .Values.config.server.grpc_listen_port }}"
{{- else }}
- "-querier.frontend-address={{ template "cortex.fullname" . }}-query-frontend-headless.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}:{{ .Values.config.server.grpc_listen_port }}"
{{- end }}
{{- include "cortex.memcached" . | nindent 12}}
{{- range $key, $value := .Values.querier.extraArgs }}
- "-{{ $key }}={{ $value }}"
Expand Down
3 changes: 3 additions & 0 deletions templates/query-frontend/query-frontend-dep.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ spec:
- "-target=query-frontend"
- "-config.file=/etc/cortex/cortex.yaml"
{{- include "cortex.frontend-memcached" . | nindent 12 }}
{{- if .Values.query_scheduler.enabled }}
- "-frontend.scheduler-address={{ template "cortex.fullname" . }}-query-scheduler-headless.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}:{{ .Values.config.server.grpc_listen_port }}"
{{- end }}
{{- range $key, $value := .Values.query_frontend.extraArgs }}
- "-{{ $key }}={{ $value }}"
{{- end }}
Expand Down
23 changes: 23 additions & 0 deletions templates/query-scheduler/_helpers-query-scheduler.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@

{{/*
query-scheduler fullname
*/}}
{{- define "cortex.querySchedulerFullname" -}}
{{ include "cortex.fullname" . }}-query-scheduler
{{- end }}

{{/*
query-scheduler common labels
*/}}
{{- define "cortex.querySchedulerLabels" -}}
{{ include "cortex.labels" . }}
app.kubernetes.io/component: query-scheduler
{{- end }}

{{/*
query-scheduler selector labels
*/}}
{{- define "cortex.querySchedulerSelectorLabels" -}}
{{ include "cortex.selectorLabels" . }}
app.kubernetes.io/component: query-scheduler
{{- end }}
118 changes: 118 additions & 0 deletions templates/query-scheduler/query-scheduler-dep.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
{{- if .Values.query_scheduler.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "cortex.querySchedulerFullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
annotations:
{{- toYaml .Values.query_scheduler.annotations | nindent 4 }}
spec:
replicas: {{ .Values.query_scheduler.replicas }}
selector:
matchLabels:
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 6 }}
strategy:
{{- toYaml .Values.query_scheduler.strategy | nindent 4 }}
template:
metadata:
labels:
{{- include "cortex.querySchedulerLabels" . | nindent 8 }}
{{- with .Values.query_scheduler.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
annotations:
{{- if .Values.useExternalConfig }}
checksum/config: {{ .Values.externalConfigVersion }}
{{- else }}
checksum/config: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
{{- end }}
{{- with .Values.query_scheduler.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
serviceAccountName: {{ template "cortex.serviceAccountName" . }}
{{- if .Values.query_scheduler.priorityClassName }}
priorityClassName: {{ .Values.query_scheduler.priorityClassName }}
{{- end }}
{{- if .Values.query_scheduler.securityContext.enabled }}
securityContext: {{- omit .Values.query_scheduler.securityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
initContainers:
{{- toYaml .Values.query_scheduler.initContainers | nindent 8 }}
{{- if .Values.image.pullSecrets }}
imagePullSecrets:
{{- range .Values.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- end }}
containers:
- name: query-scheduler
image: "{{ .Values.image.repository }}:{{ default .Chart.AppVersion .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
- "-target=query-scheduler"
- "-config.file=/etc/cortex/cortex.yaml"
{{- range $key, $value := .Values.query_scheduler.extraArgs }}
- "-{{ $key }}={{ $value }}"
{{- end }}
volumeMounts:
{{- if .Values.query_scheduler.extraVolumeMounts }}
{{- toYaml .Values.query_scheduler.extraVolumeMounts | nindent 12 }}
{{- end }}
- name: config
mountPath: /etc/cortex
- name: runtime-config
mountPath: /etc/cortex-runtime-config
ports:
- name: http-metrics
containerPort: {{ .Values.config.server.http_listen_port }}
protocol: TCP
- name: grpc
containerPort: {{ .Values.config.server.grpc_listen_port }}
protocol: TCP
startupProbe:
{{- toYaml .Values.query_scheduler.startupProbe | nindent 12 }}
livenessProbe:
{{- toYaml .Values.query_scheduler.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.query_scheduler.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.query_scheduler.resources | nindent 12 }}
{{- if .Values.query_scheduler.containerSecurityContext.enabled }}
securityContext: {{- omit .Values.query_scheduler.containerSecurityContext "enabled" | toYaml | nindent 12 }}
{{- end }}
{{- if .Values.query_scheduler.env }}
env:
{{- toYaml .Values.query_scheduler.env | nindent 12 }}
{{- end }}
{{- with .Values.query_scheduler.lifecycle }}
lifecycle:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- if .Values.query_scheduler.extraContainers }}
{{- toYaml .Values.query_scheduler.extraContainers | nindent 8 }}
{{- end }}
nodeSelector:
{{- toYaml .Values.query_scheduler.nodeSelector | nindent 8 }}
affinity:
{{- toYaml .Values.query_scheduler.affinity | nindent 8 }}
tolerations:
{{- toYaml .Values.query_scheduler.tolerations | nindent 8 }}
terminationGracePeriodSeconds: {{ .Values.query_scheduler.terminationGracePeriodSeconds }}
volumes:
- name: config
secret:
{{- if .Values.useExternalConfig }}
secretName: {{ .Values.externalConfigSecretName }}
{{- else }}
secretName: {{ template "cortex.fullname" . }}
{{- end }}
- name: runtime-config
configMap:
name: {{ template "cortex.fullname" . }}-runtime-config
{{- if .Values.query_scheduler.extraVolumes }}
{{- toYaml .Values.query_scheduler.extraVolumes | nindent 8 }}
{{- end }}
{{- end }}
14 changes: 14 additions & 0 deletions templates/query-scheduler/query-scheduler-poddisruptionbudget.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{{- if and (.Values.query_scheduler.enabled) (gt (int .Values.query_scheduler.replicas) 1) (.Values.query_scheduler.podDisruptionBudget) }}
apiVersion: {{ include "cortex.pdbVersion" . }}
kind: PodDisruptionBudget
metadata:
name: {{ include "cortex.querySchedulerFullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
spec:
selector:
matchLabels:
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 6 }}
{{- toYaml .Values.query_scheduler.podDisruptionBudget | nindent 2 }}
{{- end }}
42 changes: 42 additions & 0 deletions templates/query-scheduler/query-scheduler-servicemonitor.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
{{- if and .Values.query_scheduler.enabled .Values.query_scheduler.serviceMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "cortex.querySchedulerFullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
{{- if .Values.query_scheduler.serviceMonitor.additionalLabels }}
{{ toYaml .Values.query_scheduler.serviceMonitor.additionalLabels | indent 4 }}
{{- end }}
{{- if .Values.query_scheduler.serviceMonitor.annotations }}
annotations:
{{ toYaml .Values.query_scheduler.serviceMonitor.annotations | indent 4 }}
{{- end }}
spec:
selector:
matchLabels:
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 6 }}
namespaceSelector:
matchNames:
- {{ .Release.Namespace | quote }}
endpoints:
- port: http-metrics
{{- if .Values.query_scheduler.serviceMonitor.interval }}
interval: {{ .Values.query_scheduler.serviceMonitor.interval }}
{{- end }}
{{- if .Values.query_scheduler.serviceMonitor.scrapeTimeout }}
scrapeTimeout: {{ .Values.query_scheduler.serviceMonitor.scrapeTimeout }}
{{- end }}
{{- if .Values.query_scheduler.serviceMonitor.relabelings }}
relabelings:
{{- toYaml .Values.query_scheduler.serviceMonitor.relabelings | nindent 4 }}
{{- end }}
{{- if .Values.query_scheduler.serviceMonitor.metricRelabelings }}
metricRelabelings:
{{- toYaml .Values.query_scheduler.serviceMonitor.metricRelabelings | nindent 4 }}
{{- end }}
{{- with .Values.query_scheduler.serviceMonitor.extraEndpointSpec }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
25 changes: 25 additions & 0 deletions templates/query-scheduler/query-scheduler-svc-headless.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{{- if .Values.query_scheduler.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ include "cortex.querySchedulerFullname" . }}-headless
namespace: {{ .Release.Namespace }}
labels:
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
{{- with .Values.query_scheduler.service.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
annotations:
{{- toYaml .Values.query_scheduler.service.annotations | nindent 4 }}
spec:
type: ClusterIP
clusterIP: None
publishNotReadyAddresses: true
ports:
- port: {{ .Values.config.server.grpc_listen_port }}
protocol: TCP
name: grpc
targetPort: grpc
selector:
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 4 }}
{{- end }}
23 changes: 23 additions & 0 deletions templates/query-scheduler/query-scheduler-svc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{{- if .Values.query_scheduler.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ include "cortex.querySchedulerFullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
{{- with .Values.query_scheduler.service.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
annotations:
{{- toYaml .Values.query_scheduler.service.annotations | nindent 4 }}
spec:
type: ClusterIP
ports:
- port: {{ .Values.config.server.http_listen_port }}
protocol: TCP
name: http-metrics
targetPort: http-metrics
selector:
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 4 }}
{{- end }}
Loading

0 comments on commit f5b4a98

Please sign in to comment.