Skip to content

Installation using Helm

Jump to bottom
Yuzhang Hu edited this page Aug 28, 2024 · 9 revisions

Auto-news loves Kubernetes (k8s), to deploy auto-news backend on Kubernetes, we can do it via either Helm or ArgoCD. This page, we are focusing on Helm path.

Before Start

We assume that we already:

  • Have the K8s environment and the client machine that can connect to and operate it (also kubectl installed)
  • Have Created a Notion page and enabled page connection for auto-news backend (We need to notion page id and permission for the rest of the setup)

Install Helm

Follow the Helm installation guide, and pick one of the installation methods for the environment.

After installation, from top directory of the auto-news project, run below command to update helm repos:

make helm-repo-update

Installation Steps

1. Env file creation

make prepare-env

Now, we have the default env file at build/.env.k8s, fill the necessary environment vars the same as Installation via docker-compose

Then, we create the env file only for k8s, here we won't save the confidential tokens/passwords into Docker image, instead, we will store them as Kubernetes Secret

make k8s-env-create

2. Namespace Creation

Run the below command to create a namespace auto-news

make k8s-namespace-create

Verification:

❯ kubectl get namespace | grep auto-news
auto-news              Active   28h

3. Secret Creation

make k8s-secret-create

Verification:

❯ kubectl get secret -n auto-news | grep airflow-secrets
airflow-secrets                    Opaque               13     19h

4. Build Docker image (Optional)

Skip this if we don't want to build a customized image, the default image can be found on Dockerhub.

If we still want to build a customized image and use it for the deployment, follow the below steps:

# Build docker image, replace <repo_and_imagename> and <x.y.z> accordingly
make k8s-docker-build repo=<repo_and_imagename> tag=<x.y.z>

# Push docker image to the registry (e.g. dockerhub)
docker login

# Replace <repo_and_imagename> and <x.y.z> accordingly
make k8s-docker-push repo=<repo_and_imagename> tag=<x.y.z>

After uploading the new image to the Registry, modify repository and tag in helm/values.yaml:

airflow:
  images:
    airflow:
      repository: finaldie/auto-news
      tag: 0.9.2

5. Helm Install Auto-News Chart

make k8s-helm-install

Verification:

❯ k get pods -n auto-news
NAME                                          READY   STATUS      RESTARTS   AGE
auto-news-statsd-6f64d89f4f-qpq5h             1/1     Running     0          19h
auto-news-milvus-attu-5b8c94484d-gjgwt        1/1     Running     0          19h
auto-news-minio-3                             1/1     Running     0          19h
auto-news-minio-1                             1/1     Running     0          19h
auto-news-minio-0                             1/1     Running     0          19h
auto-news-minio-2                             1/1     Running     0          19h
auto-news-redis-0                             1/1     Running     0          19h
auto-news-postgresql-0                        1/1     Running     0          19h
auto-news-milvus-standalone-8475ccfd5-5v4jd   1/1     Running     0          19h
auto-news-mysql-0                             2/2     Running     0          19h
auto-news-redis-master-0                      1/1     Running     0          4h43m
auto-news-redis-replicas-2                    1/1     Running     0          4h43m
auto-news-etcd-2                              1/1     Running     0          4h43m
auto-news-redis-replicas-1                    1/1     Running     0          4h42m
auto-news-redis-replicas-0                    1/1     Running     0          4h41m
auto-news-etcd-1                              1/1     Running     0          4h41m
auto-news-etcd-0                              1/1     Running     0          4h40m
auto-news-worker-0                            2/2     Running     0          4h39m
auto-news-scheduler-5f8d65589b-p642q          2/2     Running     0          4h43m
auto-news-webserver-556676647-4mnlk           1/1     Running     0          4h43m
auto-news-scheduler-5f8d65589b-25mqs          2/2     Running     0          4h35m
auto-news-triggerer-0                         2/2     Running     0          3h25m
auto-news-triggerer-1                         2/2     Running     0          3h25m
auto-news-adminer-5c754b9554-9cg89            1/1     Running     0          144m
airflow-init-user-b6k9f                       0/1     Completed   0          111m

6. Enable Airflow DAGs

make k8s-airflow-dags-enable

Verification all DAGs are enabled (except upgrade one):

  • Enable port-forwarding: kubectl port-forward service/auto-news-webserver 8080:8080 --namespace auto-news
  • Then open URL http://localhost:8080 from Browser

7. Verify Notion Inbox Page

If everything goes right, we will see the necessary sub-pages created, continue to set up the RSS list/Reddit list, and enjoy the peace of reading experience in the journey.

Any issues, feel free to drop one on Open Issue

Operations

Kubernetes Port-Forwarding

Service Command Browser URL
Airflow kubectl port-forward service/auto-news-webserver 8080:8080 --namespace auto-news --address=0.0.0.0 http://localhost:8080
Milvus kubectl port-forward service/auto-news-milvus-attu -n auto-news 9100:3001 --address=0.0.0.0 http://localhost:9100
Adminer kubectl port-forward service/auto-news-adminer -n auto-news 8070:8080 --address=0.0.0.0 http://localhost:8070

Upgrading

To upgrade to the latest version from the auto-news release, run:

# At the root directory of auto-news
> git pull

# Remove the latest completed airflow-init-user job first
> kubectl delete job -n auto-news airflow-init-user

# Install latest release
> make k8s-helm-install

Modify Env Vars

After modify values in build/.env, we need to re-create the k8s secret and restart the service

make k8s-secret-delete && make k8s-secret-create
make k8s-helm-uninstall && k8s-helm-install
Clone this wiki locally