Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Airflow chart #78

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions charts/airflow/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
apiVersion: v2
name: airflow
description: Apache Airflow is a tool to express and execute workflows as directed acyclic graphs (DAGs). It includes utilities to schedule tasks, monitor task progress and handle task dependencies.
icon: https://minio.lab.sspcloud.fr/projet-onyxia/assets/servicesImg/airflow.svg
keywords:
- apache
- airflow
- workflow
- dag
home: https://airflow.apache.org/
sources:
- https://github.com/bitnami/charts/tree/main/bitnami/airflow
type: application
version: 0.0.1
appVersion: "2.10.3"
dependencies:
- name: airflow
version: 22.4.1
repository: https://charts.bitnami.com/bitnami
- name: library-chart
version: 1.5.34
repository: https://inseefrlab.github.io/helm-charts-interactive-services
71 changes: 71 additions & 0 deletions charts/airflow/templates/NOTES.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
{{- $pvcList := list -}}
{{- if .Values.airflow.postgresql.enabled }}
{{- $pvcList = append $pvcList (printf "%s-%s-0" .Values.airflow.postgresql.primary.persistence.volumeName (include "postgresql.v1.primary.fullname" .Subcharts.airflow.Subcharts.postgresql)) -}}
{{- end -}}
{{- if .Values.airflow.redis.enabled }}
{{- $pvcList = append $pvcList (printf "redis-data-%s-master-0" (include "common.names.fullname" .Subcharts.airflow.Subcharts.redis)) -}}
{{- end -}}

{{- if eq .Values.userPreferences.language "fr" -}}

**Informations de connexion :**
- Votre nom d'utilisateur: **{{ .Values.airflow.auth.username }}**
- Votre password: **{{ .Values.airflow.auth.password }}**
{{- if .Values.ingress.enabled }}
- Vous pouvez vous connecter à Apache Airflow avec votre navigateur à ce [lien](http{{ if $.Values.ingress.tls }}s{{ end }}://{{ .Values.ingress.hostname }})
{{- end }}

**Synchronisation git des DAGs :**
{{- if not (and .Values.airflow.dags.enabled .Values.airflow.dags.repositories) }}
- **La synchronisation git est désactivée !**
Pour une utilisation optimale de ce service, il est recommandé d'activer la synchronisation git de dossiers de DAG.
Depuis la page de configuration de votre service, vous pouvez renseigner des identifiants git (si nécessaire) et lister des dépôts git à utiliser pour alimenter des dossiers de DAG.
{{- else }}
- La synchronisation git est configurée pour synchroniser les repertoires de DAGs suivants:
{{- range .Values.airflow.dags.repositories }}
- `{{ .repository }}{{ .path }}` (branche `{{ .branch }}`){{ with .name }} dans le dossier `{{ . }}`{{ end }}
{{- end -}}
{{- end -}}

{{- if not (empty $pvcList) }}

**REMARQUES concernant la suppression :**
- Vous pouvez supprimer ce service sans risque, ses volumes de données ne sont pas effacées automatiquement
- Un nouveau service Airflow pourra les réutiliser en indiquant leurs noms (`{{ join "`, `" $pvcList }}`) dans la configuration du service
- Si vous souhaitez supprimer définitivement ces volumes : `kubectl delete pvc {{ join " " $pvcList }}`
{{- end -}}

{{- else -}}
{{- if ne .Values.userPreferences.language "en" -}}
*NOTES are not supported for language `{{ .Values.userPreferences.language }}`... Defaulting to english.*

{{ end -}}

**Connection information:**
- Your username: **{{ .Values.airflow.auth.username }}**
- Your password: **{{ .Values.airflow.auth.password }}**
{{- if .Values.ingress.enabled }}
- You can connect to Apache Airflow with your browser on this [link](http{{ if $.Values.ingress.tls }}s{{ end }}://{{ .Values.ingress.hostname }})
{{- end }}

**DAG git sync:**
{{- if not (and .Values.airflow.dags.enabled .Values.airflow.dags.repositories) }}
- **git sync is disabled!**
For optimal use of this service, it is recommended to synchronize DAG folders using git.
In the configuration page of your service you can specify git credentials and list repositories that will be used to fill in the DAG folders.
{{- else }}
- git synchronizaton is configured to synchronize the following DAG directories:
{{- range .Values.airflow.dags.repositories }}
- `{{ .repository }}{{ .path }}` (branch `{{ .branch }}`){{ with .name }} in folder `{{ . }}`{{ end }}
{{- end -}}
{{- end -}}

{{- if not (empty $pvcList) }}

**NOTES about deletion:**
- You can safely delete this service, its data volumes are not automatically erased
- A new Airflow service can reuse them by specifying their names (`{{ join "`, `" $pvcList }}`) in the service configuration
- If you want to delete those volumes definitely: `kubectl delete pvc {{ join " " $pvcList }}`
{{- end -}}

{{- end -}}
33 changes: 33 additions & 0 deletions charts/airflow/templates/ingress.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{{- if .Values.ingress.enabled -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "library-chart.fullname" . }}
labels:
{{- include "library-chart.labels" . | nindent 4 }}
annotations:
{{- include "library-chart.ingress.annotations" . | nindent 4 }}
spec:
{{- if and .Values.ingress.ingressClassName (eq "true" (include "library-chart.ingress.supportsIngressClassname" .)) }}
ingressClassName: {{ .Values.ingress.ingressClassName | quote }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
- hosts:
- {{ .Values.ingress.hostname | quote }}
{{- if .Values.ingress.useCertManager }}
secretName: tls-cert-{{ include "library-chart.fullname" . }}
{{- end }}
{{- end }}
rules:
- host: {{ .Values.ingress.hostname | quote }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ include "common.names.fullname" .Subcharts.airflow }}
port:
number: {{ .Values.airflow.service.ports.http }}
{{- end }}
1 change: 1 addition & 0 deletions charts/airflow/templates/networkpolicy-ingress.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{ include "library-chart.networkPolicyIngress" . }}
1 change: 1 addition & 0 deletions charts/airflow/templates/networkpolicy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{ include "library-chart.networkPolicy" . }}
17 changes: 17 additions & 0 deletions charts/airflow/templates/test/test-connection.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
apiVersion: v1
kind: Pod
metadata:
name: "{{ include "library-chart.fullname" . }}-test-connection"
labels:
{{- include "library-chart.labels" . | nindent 4 }}
app.kubernetes.io/component: test
annotations:
"helm.sh/hook": test
"helm.sh/hook-delete-policy": hook-succeeded
spec:
containers:
- name: test-connection
image: busybox
command: ['wget']
args: ['http://{{ include "library-chart.fullname" . }}:8080']
restartPolicy: Never
Loading