argo/airflow部署

1. argo

1.1 准备离线helm部署文件

wget https://github.com/argoproj/argo-helm/releases/downloa/argo-workflows-0.45.8/argo-workflows-0.45.8.tgz
tar xvf argo-workflows-0.45.8.tgz
vim values.yaml # 替换server的ClusterIP为NodePort，修改持久化（包括日志和workflow）
server:
  serviceType: NodePort
  # -- Service port for server
  servicePort: 2746
  # -- Service node port
  serviceNodePort: 32746
  persistence:
    connectionPool:
      maxIdleConns: 100
      maxOpenConns: 0
    # save the entire workflow into etcd and DB
    nodeStatusOffLoad: false
    # enable archiving of old workflows
    archive: false
    postgresql:
      host: postgres.service.com
      port: 32635
      database: argo_workflows
      tableName: argo_workflows
  # postgresql:
  #   host: localhost
  #   port: 5432
  #   database: postgres
  #   tableName: argo_workflows
  #   # the database secrets must be in the same namespace of the controller
  #   userNameSecret:
  #     name: argo-postgres-config
  #     key: username
  #   passwordSecret:
  #     name: argo-postgres-config
  #     key: password
  #   ssl: true
  #   # sslMode must be one of: disable, require, verify-ca, verify-full
  #   # you can find more information about those ssl options here: https://godoc.org/github.com/lib/pq
  #   sslMode: requartifactRepository:
  # -- Archive the main container logs as an artifact
artifactRepository:
  archiveLogs: true
  # -- Store artifact in a S3-compliant object store
  # @default -- See [values.yaml]
  s3:                                                                                                                       # # Note the `key` attribute is not the actual secret, it's the PATH to
    # # the contents in the associated secret, as defined by the `name` attribute.
    accessKeySecret:
      name: argo-s3-config
      key: accesskey
    secretKeySecret:
      name: argo-s3-config
      key: secretkey
    # sessionTokenSecret:
    #   name: "{{ .Release.Name }}-minio"
    #   key: sessionToken
    # # insecure will disable TLS. Primarily used for minio installs not configured with TLS
    insecure: true
    bucket: argo_bucket
    endpoint: s3.service.com:80
    region: US

镜像列表：

quay.io/argoproj/workflow-controller:v3.6.4
quay.io/argoproj/argoexec:v3.6.4
quay.io/argoproj/argocli:v3.6.4

使用以下命令打包离线镜像文件：

#!/bin/bash
set -e

# first into the script directory
script_dir=$(c "$(dirname "$0")" && pwd)

cd $script_dir
rm -rf images && mkdir images && cd images

images_list=(
    # k8s
    "quay.io/argoproj/workflow-controller:v3.6.4"
    "quay.io/argoproj/argoexec:v3.6.4"
    "quay.io/argoproj/argocli:v3.6.4"
)

images_list="${images_list[*]}"

for img in $images_list; do
    echo -e "\e[94m    -> Preparing $img... \e[39m"
    ./bin/ctr -n k8s.io images pull --platform linux/amd64 $img --hosts-dir $script_dir
done

eval "ctr -n k8s.io images export --platform linux/amd64 ../containerd_images.tar ${images_list}"

1.2 部署

helm install argo-workflows /disk2/shared/build_offline_origin/argo/argo-workflows -n argo-workflows

1.3 创建role-binding

k create rolebinding argo-binding --role argo-argo-workflows-workflow --serviceaccount argo:default -n argo

1.4 获取登录token

kubectl exec -it argo-argo-workflows-server-667cddff87-5hg5m -n argo -- argo auth token

1.5 前提

已经创建postgres持久化数据库
已经创建s3存储
已经修改coredns
已经创建secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: argo-postgres-config
  namespace: argo
type: Opaque
stringData:
  username: postgres
  password: E6^a3)zVD48mMNYaA)bF@wPv
---
apiVersion: v1
kind: Secret
metadata:
  name: argo-s3-config
  namespace: argo
type: Opaque
stringData:
  accessKey: ENL7QVDGNNYNNEX3X3VS
  secretKey: vaUjPhUkR8yLAdqVD6FRnXGVNrxBNDs9bMWFb6Kb

1.6 升级

直接使用helm升级即可

在升级之前，可以考虑备份：

chmod 644 ~/.kube/config
docker run -v ~/.kube:/home/argocd/.kube --rm \
    argoproj/argocd:$VERSION argocd-util export -n argocd > backup.yaml
chmod 600 ~/.kube/config

恢复：

chmod 644 ~/.kube/config
docker run -i -v ~/.kube:/home/argocd/.kube --rm \
    argoproj/argocd:$VERSION argocd-util import -n argocd - < backup.yaml
chmod 600 ~/.kube/config

1.7 参考

Service Accounts - Argo Workflows - The workflow engine for Kubernetes

Argo Workflows 中文快速指南·-腾讯云开发者社区-腾讯云

Access Token - Argo Workflows - The workflow engine for Kubernetes

argo workflows — 配置持久化 persistencehelm方式安装的argo workflows需 - 掘金

Workflow Archive - Argo Workflows - The workflow engine for Kubernetes

Upgrading Argo CD — Roundtable Current documentation

2. airflow官方版本部署

2.1 准备离线helm部署文件

git clone git@github.com:apache/airflow.git
# wget https://codeload.github.com/apache/airflow/zip/refs/heads/v3-0-stable
# unzip v3-0-stable
cd chart
helm repo add binami https://charts.bitnami.com/bitnami
helm dependency build

2.2 修改values.yaml

workers.persistence.storageClassName: local-path
triggerer.persistence.storageClassName: local-path
redis.persistence.storageClassName: local-path
logs.persistence.storageClassName: local-path
dags.persistence.storageClassName: local-path
api-server.service.type: NodePort
webserver.service.type: NodePort
# flower.service.type: NodePort
flower.enabled: true
pgbouncer.enabled: true
# logs.persistence.enabled: true
dags.persistence.enabled: true
# 用于允许k8sPodOperator部署pod在其他命名空间内
multiNamespaceMode: true
# 多execotor并存
executor: "CeleryExecutor,KubernetesExecutor"
# 同时要修改files/pod-template-file.kubernetes-helm-yaml
        - name: AIRFLOW__CORE__EXECUTOR
          value: {{ .Values.executor | quote }}

其中，如果使用KubernetesExecutor，需要手动指明：

b = BashOperator(
    task_id="my_task_in_its_own_pod",
    executor="KubernetesExecutor",
    bash_command="echo hello & sleep 10",
)

2.3 其他建议

2.3.1 外置数据库

postgresql:
  enabled: false

data:
  metadataConnection:
    user: <username>
    pass: <password>
    protocol: postgresql
    host: <hostname>
    port: 5432
    db: <database name>

2.3.2 Webserver Secret Key

创建密钥

python3 -c 'import secrets; print(secrets.token_hex(16))'

修改values.yaml

webserverSecretKey: <secret_key>

2.3.3 修改broker

2.3.3.1 创建rabbitmq部署

git clone git@github.com:bitnami/charts.git
cd bitnami/charts-main/bitnami/rabbitmq
# helm dependency build # 由于无法下载而失败
mkdir charts && cd charts
wget https://charts.bitnami.com/bitnami/common-2.30.0.tgz
tar xvf common-2.30.0.tgz

修改values.yaml

global.defaultStorageClass: "local-path"
persistence.storageClass: "local-path"
auth.password: "password"
service.type: NodePort

获取连接地址

export NODE_IP=$(kubectl get nodes --namespace rabbitmq -o jsonpath="{.items[0].status.addresses[0].address}")
export NODE_PORT_AMQP=$(kubectl get --namespace rabbitmq -o jsonpath="{.spec.ports[?(@.name=='amqp')].nodePort}" services rabbitmq)
export NODE_PORT_STATS=$(kubectl get --namespace rabbitmq -o jsonpath="{.spec.ports[?(@.name=='http-stats')].nodePort}" services rabbitmq)
echo "URL : amqp://$NODE_IP:$NODE_PORT_AMQP/"

2.3.3.2 修改airflow的broker配置

redis:
  enabled: false
data:
  brokerUrl: amqp://user:password@10.144.66.217:31539
  # brokerUrl: redis://redis-user:password@redis-host:6379/0

2.3.4 修改环境变量

extraEnv: |
  - name: AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC
    value: '1'
  - name: AIRFLOW__SCHEDULER__PARSING_PROCESSES
    value: '4' # [core] 设置 parallelism 和 dag_concurrency
  - name: AIRFLOW__CORE__PARALLELISM
    value: '4'
  - name: AIRFLOW__CORE__DAG_CONCURRENCY
    value: '4'
  - name: AIRFLOW__CELERY__CELERYD_CONSUMER_POLL_INTERVAL
    value: '0.2'
  - name: AIRFLOW__WORKERS__MIN_HEARTBEAT_INTERVAL
    value: '1'

2.4 部署

helm install airflow ./ -n airflow # 时间可能会比较长
# 部署后可能flower会失败，删除等待重新部署即可。

部署完成提示：

Airflow Webserver:     kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow
Flower dashboard:      kubectl port-forward svc/airflow-flower 5555:5555 --namespace airflow
Default Webserver (Airflow UI) Login credentials:
    username: admin
    password: admin
Default Postgres connection credentials:
    username: postgres
    password: postgres
    port: 5432

You can get Fernet Key value by running the following:

    echo Fernet Key: $(kubectl get secret --namespace airflow airflow-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode)

2.5 问题

目前部署3.0会遇到以下问题：

JWT问题：
- Generate JWT secret during HELM install by bdsoha · Pull Request #49923 · apache/airflow
需要手动开启FabAuthManager，否则user-create会报错
- Configuring Flask Application for Airflow Webserver — apache-airflow-providers-fab Documentation

部署多Executor会遇到以下问题：

无法加载KubernetesExecutor运行时的日志信息（持久化日志应该就可以解决问题）

3. 社区版airflow

3.1 准备离线helm部署文件

wget https://github.com/airflow-helm/charts/releases/download/airflow-8.9.0/airflow-8.9.0.tgz
tar xvf airflow-8.9.0.tgz
vim values.yaml # 替换web和flower的ClusterIP为NodePort
server:
  service:
    annotations: {}
    sessionAffinity: "None"
    sessionAffinityConfig: {}
    type: NodePort

# 修改PVC of postgresql
persistence:
    ## if postgres will use Persistent Volume Claims to store data
    ## - [WARNING] if false, data will be LOST as postgres Pods restart
    ##
    enabled: true

    ## the name of the StorageClass used by the PVC
    ##
    storageClass: "local-path"

镜像列表：

apache/airflow:2.8.4-python3.9
registry.k8s.io/git-sync/git-sync:v3.6.9
ghcr.io/airflow-helm/pgbouncer:1.22.1-patch.0
ghcr.io/airflow-helm/postgresql-bitnami:11.22-patch.0
docker.io/bitnami/redis:6.2.14-debian-12-r17

使用以下命令打包离线镜像文件：

#!/bin/bash
set -e

# first into the script directory
script_dir=$(c "$(dirname "$0")" && pwd)

cd $script_dir
rm -rf images && mkdir images && cd images

images_list=(
    # k8s
    "apache/airflow:2.8.4-python3.9"
    "registry.k8s.io/git-sync/git-sync:v3.6.9"
    "ghcr.io/airflow-helm/pgbouncer:1.22.1-patch.0"
    "ghcr.io/airflow-helm/postgresql-bitnami:11.22-patch.0"
    "docker.io/bitnami/redis:6.2.14-debian-12-r17"
)

images_list="${images_list[*]}"

for img in $images_list; do
    echo -e "\e[94m    -> Preparing $img... \e[39m"
    ./bin/ctr -n k8s.io images pull --platform linux/amd64 $img --hosts-dir $script_dir
done

eval "ctr -n k8s.io images export --platform linux/amd64 ../containerd_images.tar ${images_list}"

3.2 部署

helm install airflow /disk2/shared/build_offline_origin/airflow/airflow -n airflow

Default Airflow Webserver login:
  * Username:  admin
  * Password:  admin

3.3 升级

直接安装新版本
运行中的容器需要重新指定AIRFLOW_VERSION环境变量，并且重新安装pip包
需要手动迁移数据库，要执行airflow db migrate命令

4. bitnami版本airflow部署

4.1 准备离线helm部署文件

# 下载整个仓库，或者也可以使用git clone
wget https://codeload.github.com/bitnami/charts/zip/refs/heads/main
# 进入postgres路径
cd bitnami/charts-main/bitnami/airflow
# 进入charts目录下载依赖
# helm dependency build # 由于无法下载而失败
mkdir charts && cd charts
wget https://charts.bitnami.com/bitnami/common-2.30.0.tgz
tar xvf common-2.30.0.tgz
# 也可以复制
cp -r ../../common ./
# 同理复制postgres与redis到charts目录内
# cp -r ../../postgresql ./
# cp -r ../../redis ./
cd ..

4.2 修改values.yaml

extraEnvVars: []
global.defaultStorageClass: "local-path"
triggerer.persistence.storageClass: "local-path"
service.type: NodePort

4.3 部署

helm install airflow ./ -n airflow

需要注意的是，bitnami版本的只能部署为使用redis的，不能为rabbitmq或者其他Broker的，而且也无法部署flower和pgbouncer。

输出：

This deployment will be incomplete until you configure Airflow with a resolvable
host. To configure Airflow with the URL of your service:

1. Get the Airflow URL by running:

    export AIRFLOW_HOST=$(kubectl get nodes --namespace airflow2 -o jsonpath="{.items[0].status.addresses[0].address}")
    export AIRFLOW_PORT=$(kubectl get --namespace airflow2 -o jsonpath="{.spec.ports[0].nodePort}" services airflow2-web)

2. Complete your Airflow deployment by running:

    export AIRFLOW_PASSWORD=$(kubectl get secret --namespace "airflow2" airflow2 -o jsonpath="{.data.airflow-password}" | base64 -d)
    export AIRFLOW_FERNET_KEY=$(kubectl get secret --namespace "airflow2" airflow2 -o jsonpath="{.data.airflow-fernet-key}" | base64 -d)
    export AIRFLOW_SECRET_KEY=$(kubectl get secret --namespace "airflow2" airflow2 -o jsonpath="{.data.airflow-secret-key}" | base64 -d)
    helm upgrade --namespace airflow2 airflow2 oci://registry-1.docker.io/bitnamicharts/airflow \
      --set service.type=NodePort \
      --set web.baseUrl=http://$AIRFLOW_HOST:$AIRFLOW_PORT \
      --set auth.password=$AIRFLOW_PASSWORD \
      --set auth.fernetKey=$AIRFLOW_FERNETKEY \
      --set auth.secretKey=$AIRFLOW_SECRETKEY