airflow-helm / charts Goto Github PK

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.

Home Page: https://github.com/airflow-helm/charts/tree/main/charts/airflow

License: Apache License 2.0

Python 0.14% Smarty 36.54% Dockerfile 4.50% Shell 58.83%

helm airflow kubernetes helm-charts helm-chart charts chart k8s

charts's Introduction

Airflow Helm Chart (User Community)

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.

↓ ↓ ↓

Chart Homepage

charts's People

Contributors

Stargazers

Watchers

Forkers

bgaechter toneill818 kangmaoyin vincentbellitto pkuong isaiah-turner capchriscap pedaling flavio-assis davidchenlondon chris-vest hedrickw kimkihyuk alexbegg michael-j-thomas mgvalverde vizbeth-oss ectogon arielshup ankurshaswat kot-behemoth minglunwu sisandi dstandish adanque dre1992 naturalett swalloow shoman2 makramjandar datgnomelife gfeldman sparbz hudsondba razius kulkinp bigsharkie rodolfo-picoreti patricol phobot povode billmcmath atulprusty srnbckr bhavishya101 jessicastewart-udemy abrahamgv tiqets chakir-alaoui darsi-an scheung38 abhaypartap edenbaus spagarwa bensta c0derust franchev sfc-gh-cfraleigh justincmoy johnsonheather odenio bledogit rafaelkonrath piboonsak harrydunham dominique-vassard dgadiraju meseta sk1tter rolanddb karakanb fernbach thetko85 maxcrom lord-y jgilme1 brain-buster roopainfoworks luozhaoyu davido912 griseau tonkonozhenko yehlo valentinovizner-sbt mevinfernando enniomorricone123 cocampbe sri-nidhi hussainsaify hanzala1234 yonatanm-kayhut moskitone rohitlaheri misterrnobe itsdipit jaegeunbang cftang0827 akshay2agarwal ramdharam momota10s

charts's Issues

Migrate to official `kubernetes/git-sync` container

Currently, we use alpine/git our own script to perform the git-sync, this is problematic, as it has a different syntax to the KubernetesExecutor git configs.

We should migrate to the kubernetes/git-sync container (which is hosted on k8s.gcr.io/git-sync), this will let us automatically set: AIRFLOW__KUBERNETES__GIT_REPO, AIRFLOW__KUBERNETES__GIT_PASSWORD, etc as this is what Airflow itself uses.

NOTE: we will have to remove the old git-sync configs from values.yaml (so this will need a major version bump)

Clone git repo as airflow user instead of root

What is your feature request?

By default, current airflow helm chart will clone source code git repo using root user, which cause permission problem if some tasks require to create some files or directory under the git repo.

What alternatives have you considered?

Not yet.

EKS Fargate Guincorn Error

What is the bug?

I am trying to deploy airflow using this helm chart on AWS EKS backed by Fargate. The problem I am facing is that airflow-scheduler gets deployed correctly, I am able to check the logs to verify as well but, airflow-web UI, Gunicorn, keeps failing/timing out and restarting the airflow-web UI. I also am not able to access my logs on airflow-web, I get airflow-web: net/http: TLS handshake timeout

03:05:17    [2020-04-03 03:05:17 +0000] [8] [INFO] Booting worker with pid: 8
03:05:17    [2020-04-03 03:05:17 +0000] [9] [INFO] Booting worker with pid: 9
03:05:17    [2020-04-03 03:05:17 +0000] [1] [DEBUG] 2 workers
03:07:17    [2020-04-03 03:07:17 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
03:07:17    [2020-04-03 03:07:17 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:9)
03:07:17    [2020-04-03 03:07:17 +0000] [12] [INFO] Booting worker with pid: 12
03:07:18    [2020-04-03 03:07:18 +0000] [13] [INFO] Booting worker with pid: 13

I've looked into the Gunicorn error time out error that others have fixed by either using gevent or updating gunicorn.conf for:

workers = 1
threads = 8
worker_class = 'sync'
worker_connections = 1000
timeout = 30
keepalive = 2

What are your Helm values?

Any relevant parts of your custom_values.yaml

  config: 
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"
    AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: "True"
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "True"

What is your Kubernetes Version?:

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:21:03Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

$ helm version

version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"clean", GoVersion:"go1.14.10"}

Is there any documented for previous set up for Airflow deployment?

What is your question?

Is there any documentation for airflow deploytment by helm?
Since I try to deploy airflow by helm, following error occured.

data-airflow-postgresql-0 ( for K8s PVC)
git-clone-secret (for K8s Secret)

Strangely following PVC works fine.

airflow-logs

Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "airflow-postgresql-0": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "airflow-postgresql-0": pod has unbound immediate PersistentVolumeClaims

Events:
  Type     Reason                  Age                 From                     Message
  ----     ------                  ----                ----                     -------
  Warning  FailedScheduling        <unknown>           default-scheduler        running "VolumeBinding" filter plugin for pod "airflow-scheduler-776c7dc794-j52f6": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling        <unknown>           default-scheduler        running "VolumeBinding" filter plugin for pod "airflow-scheduler-776c7dc794-j52f6": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled               <unknown>           default-scheduler        Successfully assigned airflow/airflow-scheduler-776c7dc794-j52f6 to zdlsserver
  Normal   SuccessfulAttachVolume  103s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-45323898-7106-4337-b2ad-cd6d16636328"
  Warning  FailedMount             39s (x8 over 102s)  kubelet, zdlsserver      MountVolume.SetUp failed for volume "git-clone-secret" : secret "airflow-git-key-files" not found

Events:
  Type     Reason                  Age                 From                     Message
  ----     ------                  ----                ----                     -------
  Warning  FailedScheduling        <unknown>           default-scheduler        running "VolumeBinding" filter plugin for pod "airflow-web-84d567b67f-ncwqb": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling        <unknown>           default-scheduler        running "VolumeBinding" filter plugin for pod "airflow-web-84d567b67f-ncwqb": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled               <unknown>           default-scheduler        Successfully assigned airflow/airflow-web-84d567b67f-ncwqb to zdlsserver
  Normal   SuccessfulAttachVolume  103s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-45323898-7106-4337-b2ad-cd6d16636328"
  Warning  FailedMount             39s (x8 over 103s)  kubelet, zdlsserver      MountVolume.SetUp failed for volume "git-clone-secret" : secret "airflow-git-key-files" not found

Pod status is follows.

NAME                                 READY   STATUS              RESTARTS   AGE
airflow-postgresql-0                 0/1     Pending             0          13m
airflow-redis-master-0               1/1     Running             0          13m
airflow-scheduler-776c7dc794-j52f6   0/2     ContainerCreating   0          13m
airflow-web-84d567b67f-ncwqb         0/2     ContainerCreating   0          13m

Upgrade to Airflow 2.0

Has anyone had successfully upgrade to Airflow 2.0 yet?
If yes, what you need to alter from the chart?

I have been trying to alter the files under the template folder and values.yaml but the best I got so far was to be able to see WebServer and Scheduler pods are running but not the DAG.

I trigger the DAG and quickly got the error telling me Failed to log action with (sqlite3.OperationalError) no such table....

The weird thing is I use an external Postgres database and I can confirm there are >20 tables are created by the scheduler but somehow I still get error related tosqlite3.OperationalError.

Easily support changing the image registry

It would be nice if airflow.image.repository supported defining the image registry separately from the image repository. This could be done either by invoking the tpl command when accessing the image repository from values, or by splitting them explicitly as is done by bitnami

Question about `Only works with the Celery or Kubernetes executors, sorry`

What is the bug?
I try to execute tutorial.py from web ui but failed with following error.

Only works with the Celery or Kubernetes executors, sorry

From digging this error, it seems airflow 1.10.12 problem.

KubernetesExecutor single task run error: Only works with the Celery or Kubernetes executors, sorry #12341

If I execute on command line airflow backfill, it works fine.
Is there any workaround?

What are your Helm values?

Any relevant parts of your custom_values.yaml

###################################
# Airflow - Common Configs
###################################
airflow:
  ## the airflow executor type to use
  ##
  executor: "KubernetesExecutor"
  ## environment variables for the web/scheduler/worker Pods (for airflow configs)
  ##
  config:
    AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "False"
    AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: "repo/"
    AIRFLOW__KUBERNETES__DAGS_VOLUME_CLAIM: "airflow"
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: "apache/airflow"
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: "1.10.12"
    AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"
    AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM: "airflow-logs"

workers:
  enabled: false # Celery workers

###################################
# Airflow - WebUI Configs
###################################
web:
  ## configs for the Service of the web Pods
  ##
  service:
    type: NodePort

###################################
# Airflow - Logs Configs
###################################
logs:
  persistence:
    enabled: true
    storageClass: basic-csi

###################################
# Airflow - DAGs Configs
###################################
dags:
  persistence:
    enabled: true
    storageClass: basic-csi
    accessMode: ReadWriteMany
    size: 1Gi

What is your Kubernetes Version?:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:39:24Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}

Patch permission on Airflow role to support new KubernetesPodOperator

What is your feature request?

The KubernetesPodOperator in the new provider package requires permission to patch a pod. See source. This applies to the CNCF backport provider as well and affects version that this chart supports.

A simple fix is to just add the patch permission, but for it to be more general maybe it is worth while to add a list of permissions that can be appended to the created role.

What alternatives have you considered?

I added two manifests in extraManifest one for a new role to allow patching of a pod and another to bind the role to the airflow service account. This works but introduces unnecessary complexity.

extraManifests:
    - apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: airflow-patcher
        namespace: "{{ .Release.Namespace }}"
        labels:
          app: airflow
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: airflow-patcher
      subjects:
        - kind: ServiceAccount
          name: airflow
          namespace: "{{ .Release.Namespace }}"
    - apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: airflow-patcher
        namespace: "{{ .Release.Namespace }}"
        labels:
          app: airflow
      rules:
        - apiGroups:
            - ""
          resources:
            - pods
          verbs:
            - "patch"

Task run faield on Kubernetes Executor after update to airflow 2.0

I was trying to upgrade airflow 1.10.12 to airflow 2.0.0 and I have encountered a lot of issues.
After taking a recommendation from @Dr1992 and @Swalloow. I was able to get airflow 2.0.0 up and running.

Now come to another issue when I trigger the example dag: example_kubeenetes_executor_config.
I check the log and here is what I get

[2021-01-12 04:03:53,149] {cli_action_loggers.py:105} WARNING - Failed to log action with (sqlite3.OperationalError) no such table: log
[SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('2021-01-12 04:03:53.145763', 'from_just_another_git_dag_kubernetes_executor', 'from_just_another_git_my_start_task', 'cli_task_run', '2021-01-12 04:03:39.626322', 'airflow', '{"host_name": "fromjustanothergitdagkubernetesexecutorfromjustanothergitmystar", "full_command": "[\'/home/airflow/.local/bin/airflow\', \'tasks\', \ ... (94 characters truncated) ... sk\', \'2021-01-12T04:03:39.626322+00:00\', \'--local\', \'--pool\', \'default_pool\', \'--subdir\', \'/opt/airflow/dags/just_another_git_dag.py\']"}')]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
[2021-01-12 04:03:53,150] {dagbag.py:440} INFO - Filling up the DagBag from /opt/airflow/dags/just_another_git_dag.py
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: task_instance

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 89, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 223, in task_run
    ti.refresh_from_db()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 556, in refresh_from_db
    ti = qry.first()
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3429, in first
    ret = list(self[0:1])
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3203, in __getitem__
    return list(res)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
    return self._execute_and_instances(context)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: task_instance
[SQL: SELECT task_instance.try_number AS task_instance_try_number, task_instance.task_id AS task_instance_task_id, task_instance.dag_id AS task_instance_dag_id, task_instance.execution_date AS task_instance_execution_date, task_instance.start_date AS task_instance_start_date, task_instance.end_date AS task_instance_end_date, task_instance.duration AS task_instance_duration, task_instance.state AS task_instance_state, task_instance.max_tries AS task_instance_max_tries, task_instance.hostname AS task_instance_hostname, task_instance.unixname AS task_instance_unixname, task_instance.job_id AS task_instance_job_id, task_instance.pool AS task_instance_pool, task_instance.pool_slots AS task_instance_pool_slots, task_instance.queue AS task_instance_queue, task_instance.priority_weight AS task_instance_priority_weight, task_instance.operator AS task_instance_operator, task_instance.queued_dttm AS task_instance_queued_dttm, task_instance.queued_by_job_id AS task_instance_queued_by_job_id, task_instance.pid AS task_instance_pid, task_instance.executor_config AS task_instance_executor_config, task_instance.external_executor_id AS task_instance_external_executor_id
FROM task_instance
WHERE task_instance.dag_id = ? AND task_instance.task_id = ? AND task_instance.execution_date = ?
 LIMIT ? OFFSET ?]
[parameters: ('from_just_another_git_dag_kubernetes_executor', 'from_just_another_git_my_start_task', '2021-01-12 04:03:39.626322', 1, 0)]
(Background on this error at: http://sqlalche.me/e/13/e3q8)

I check the DB and I confirm the log table and task_instance table are there.

So what causes this error?
I know we are trying to support both airflow 1 and airflow 2 but is there anything I need to concern or modify when coming to upgrade airflow 2.0?

Unable to initialize the pod

Hi guys,
I am losing my hair chasing the issue on KubernetesExecutor with git-sync.

The issue is I the pod is not able to initialize to run. It always gives me this Init:Error

Here is my values.yaml

airflow:
  ## configs for the docker image of the web/scheduler/worker
  ##
  image:
    repository: apache/airflow
    tag: 1.10.12-python3.6
    ## values: Always or IfNotPresent
    pullPolicy: IfNotPresent
    pullSecret: ""

  executor: KubernetesExecutor


  fernetKey: "PhyRqLI_7aoFYH4sTxtapY1xSTAAuZGU_dbntcUdUeI="

  config:
      AIRFLOW__CORE__REMOTE_LOGGING: "True"
      AIRFLOW__CORE__REMOTE_LOG_CONN_ID: my_aws
      AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://airflow-logs-kube"
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: "apache/airflow"
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: "1.10.12-python3.6"
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: "IfNotPresent"
      AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE: "10"
      AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: "False"
      AIRFLOW__KUBERNETES__IN_CLUSTER: "True"
      AIRFLOW__KUBERNETES__GIT_REPO: "[email protected]:ryanntk/k8s_airflow.git"
      AIRFLOW__KUBERNETES__GIT_BRANCH: "master"
      AIRFLOW__KUBERNETES__GIT_DAGS_FOLDER_MOUNT_POINT: "/opt/airflow/dags"
      AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: "repo/"

      AIRFLOW__KUBERNETES__GIT_SSH_KEY_SECRET_NAME: airflow-git-keys
      AIRFLOW__KUBERNETES__NAMESPACE: "airflow"
      AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "False"
      AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE: "False"
      AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"
      AIRFLOW__CORE__LOAD_EXAMPLES: "False"
      AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "60"


###################################
# Airflow - DAGs Configs
###################################
dags:
  path: /opt/airflow/dags
  doNotPickle: false
  installRequirements: true

  git:
    url: "[email protected]:ryanntk/k8s_airflow.git"
    ref: master
    secret: airflow-git-keys
    sshKeyscan: true
    privateKeyName: id_rsa
    repoHost: "bitbucket.org"
    repoPort: 22

    gitSync:
      enabled: true
      image:
        repository: alpine/git
        tag: latest
        pullPolicy: Always
      refreshTime: 60

  initContainer:
    enabled: true
    image:
      repository: alpine/git
      tag: latest
      pullPolicy: Always
    mountPath: "/opt/airflow/dags"
....

If I trigger the dag, here is what I get

Any help would be very much appreciated 💯

airflow logging to S3

We have been encountering some issues with the logging to S3. Seems like in some cases where the dag fails the log is missing as well and we get the following error. Has anyone come across this issue? any suggestions?

Sidecar pattern for gce-proxy when connecting to Cloud SQL instance

What is your feature request?
When deploying airflow on GKE, it would be nice to support connecting to a Cloud SQL instance through the sidecar pattern (as recommended in the docs), in a way similar to how git sync is supported currently.

What alternatives have you considered?

Deploying postgres on kubernetes with a persistent storage.
Running the proxy as a service in the airflow namespace.

Airflow 2.0 support for Cloud Providers

How far away is support for Airflow 2.0?

What is your feature request?

GKE
AWS
Azure

A clear description of what you want.
Airflow 2.0 support for cloud providers

What alternatives have you considered?
Managed Services...

Is there support for hybrid CeleryKubernetes Executor ?

What is your question?
Is there support for hybrid CeleryKubernetes Executor ?

According to comments in helm chart:

  ## the airflow executor type to use
  ##
  ## NOTE:
  ## - this should be `CeleryExecutor` or `KubernetesExecutor`
  ## - if set to `KubernetesExecutor`:
  ##   - ensure that `workers.enabled` is `false`
  ##   - ensure that `flower.enabled` is `false`
  ##   - ensure that `redis.enabled` is `false`
  ##   - ensure that K8S configs are set in `airflow.config`
  ##   - we set these configs automatically:
  ##     - `AIRFLOW__KUBERNETES__NAMESPACE`
  ##     - `AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME`
  ##     - `AIRFLOW__KUBERNETES__ENV_FROM_CONFIGMAP_REF`
  ##
  executor: CeleryExecutor

Upgrading 7.14.0 to 7.14.3

What is the bug?

When upgrading via Helm I get the following erros.

coalesce.go:196: warning: cannot overwrite table with non table for config (map[]) coalesce.go:196: warning: cannot overwrite table with non table for config (map[]) Error: UPGRADE FAILED: template: airflow/templates/worker/worker-statefulset.yaml:126:16: executing "airflow/templates/worker/worker-statefulset.yaml" at <include "airflow.mapenvsecrets" .>: error calling include: template: airflow/templates/_helpers.tpl:180:56: executing "airflow.mapenvsecrets" at <.Values.airflow.config.AIRFLOW__CELERY__FLOWER_BASIC_AUTH>: can't evaluate field AIRFLOW__CELERY__FLOWER_BASIC_AUTH in type interface {}

Some log from Harness:

INFO   2020-11-27T09:36:56.039+0000   �[1;97m�[40m
�[1;97m�[40mFetching values.yaml from helm chart for Service�[0m
INFO   2020-11-27T09:36:57.837+0000   Helm repository: Helm Airflow Helm
Chart name: airflow
Chart version: 7.14.3
Helm version: V3
Repo url: https://airflow-helm.github.io/charts

Successfully fetched values.yaml
INFO   2020-11-27T09:37:02.204+0000   Installing
List all existing deployed releases for release name: airflow-helm-tools-airflow-helm-10joyix
Executing command - KUBECONFIG=${KUBECONFIG_PATH} /opt/harness-delegate/client-tools/helm/v3.1.2/helm hist airflow-helm-tools-airflow-helm-10joyix   --max 5

REVISION	UPDATED                 	STATUS    	CHART         	APP VERSION	DESCRIPTION   
49      	Fri Nov 27 08:27:47 2020	superseded	airflow-7.14.0	1.10.12    	Rollback to 48
50      	Fri Nov 27 08:29:35 2020	superseded	airflow-7.14.0	1.10.12    	Rollback to 49
51      	Fri Nov 27 08:32:48 2020	superseded	airflow-7.14.0	1.10.12    	Rollback to 50
52      	Fri Nov 27 08:34:30 2020	superseded	airflow-7.14.0	1.10.12    	Rollback to 51
53      	Fri Nov 27 09:30:18 2020	deployed  	airflow-7.14.0	1.10.12    	Rollback to 52
INFO   2020-11-27T09:37:02.440+0000   Creating KubeConfig
Setting KubeConfig
KUBECONFIG_PATH=/opt/harness-delegate/./repository/helm/.kube/035d822e63b48531ced6bd6bc71b51c8
Finding helm version
Helm3 is installed at [/opt/harness-delegate/client-tools/helm/v3.1.2/helm]

Done.
INFO   2020-11-27T09:37:03.301+0000   Executing command - KUBECONFIG=${KUBECONFIG_PATH} /opt/harness-delegate/client-tools/helm/v3.1.2/helm list  --filter ^airflow-helm-tools-airflow-helm-10joyix$

Previous release exists for chart. Upgrading chart
Executing command - KUBECONFIG=${KUBECONFIG_PATH} /opt/harness-delegate/client-tools/helm/v3.1.2/helm upgrade  airflow-helm-tools-airflow-helm-10joyix ./repository/helm/source/vfzQOSoLTL2BwQWIPi5_4A/airflow  -f ./repository/helm/overrides/9fa4fa88585f83b369eb759875fc615f.yaml -f ./repository/helm/overrides/fefb6522ae5945dc0730454f3915980c.yaml

Error: failed to parse ./repository/helm/overrides/fefb6522ae5945dc0730454f3915980c.yaml: error converting YAML to JSON: yaml: line 8: did not find expected '-' indicator

Executing command - KUBECONFIG=${KUBECONFIG_PATH} /opt/harness-delegate/client-tools/helm/v3.1.2/helm list  --filter ^airflow-helm-tools-airflow-helm-10joyix$
INFO   2020-11-27T09:37:03.657+0000   Helm Chart Repo checked-out locally
Rendering Helm chart
Executing command - /opt/harness-delegate/client-tools/helm/v3.1.2/helm template airflow-helm-tools-airflow-helm-10joyix ./repository/helm/source/vfzQOSoLTL2BwQWIPi5_4A/airflow  --namespace airflow-helm  -f ./repository/helm/overrides/9fa4fa88585f83b369eb759875fc615f.yaml -f ./repository/helm/overrides/fefb6522ae5945dc0730454f3915980c.yaml

Failed to render chart location: ./repository/helm/source/vfzQOSoLTL2BwQWIPi5_4A/airflow. Reason Error: failed to parse ./repository/helm/overrides/fefb6522ae5945dc0730454f3915980c.yaml: error converting YAML to JSON: yaml: line 8: did not find expected '-' indicator
 
Failed to print Helm chart manifest, location: ./repository/helm/source/vfzQOSoLTL2BwQWIPi5_4A/airflow

Done
INFO   2020-11-27T09:37:04.516+0000   Command finished with status FAILURE

Support airflow 2.0

Has anyone used this helm chart with airflow 2.0 yet?
I have been trying with airflow by replacing 1.10.12-python3.6 to 2.0.0-python3.6
but get this error

airflow command error: argument GROUP_OR_COMMAND: airflow upgradedbcommand, has been removed, please useairflow db upgrade, see help above.

So friendly asking the plan to support airflow 2.0?

Dag_id could not found in KubernetesExecutor mode

What is the bug?
Hi everyone,
I am running stable/airflow on Azure Kubernetes with git-sync enable.
The example dags work just fine but not the dag in the Bitbucket repo.
My dag is pretty simple like this for testing purpose:

default_args = {
    'owner': 'Dev',
    'start_date': days_ago(2)
}

with DAG(
    dag_id='myown_example_kubernetes_executor_config',
    default_args=default_args,
    schedule_interval=None
) as dag:

    def print_stuff():
        print("Print stuff")

    # You can use annotations on your kubernetes pods!
    start_task = PythonOperator(
        task_id="start_task",
        python_callable=print_stuff,
        executor_config={
            "KubernetesExecutor": {
                "annotations": {"test": "annotation"}
            }
        }
    )

and the structure of Bitbucket repo like this:

I get the error from the pod when I trigger the dag:

I ssh into the airflow-web to the dags and it is there.

What are your Helm values?

airflow:
  image:
    repository: apache/airflow
    tag: 1.10.12-python3.6
    ## values: Always or IfNotPresent
    pullPolicy: IfNotPresent
    pullSecret: ""

  executor: KubernetesExecutor
  
  fernetKey: "PhyRqLI_7aoFYH4sTxtapY1xSTAAuZGU_dbntcUdUeI="

  config:
      AIRFLOW__CORE__DAGS_FOLDER: "/opt/airflow/dags/dags"
      AIRFLOW__CORE__REMOTE_LOGGING: "True"
      AIRFLOW__CORE__REMOTE_LOG_CONN_ID: my_aws
      AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://airflow-logs-kube"
      AIRFLOW__CORE__SQL_ALCHEMY_CONN: "postgresql+psycopg2://<usernamne>@ppsdb:<password>@<host>.database.azure.com:5432/postgres"
      AIRFLOW__CORE__DAG_RUN_CONF_OVERRIDES_PARAMS: "True"
      AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX: "True"
      AIRFLOW__KUBERNETES__NAMESPACE: "airflow"
      AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"
      AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "False"
      AIRFLOW__KUBERNETES__IN_CLUSTER: "True"
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: "apache/airflow"
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: "1.10.12"
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: "IfNotPresent"
      AIRFLOW__KUBERNETES__GIT_REPO: "[email protected]:ryanntk/k8s_airflow.git"
      AIRFLOW__KUBERNETES__GIT_BRANCH: "master"
      AIRFLOW__KUBERNETES__GIT_DAGS_FOLDER_MOUNT_POINT: "/opt/airflow/dags/dags"
      AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: "repo/"
      AIRFLOW__KUBERNETES__GIT_SSH_KEY_SECRET_NAME: "airflow-git-keys"
      AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE: "False"
      AIRFLOW__CORE__LOAD_EXAMPLES: "True"
      AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "60"
      AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: "True"

  extraEnv:
    - name: AIRFLOW__KUBERNETES__GIT_SYNC_RUN_AS_USER
      value: "65533"

scheduler:
  resources:
    requests:
      cpu: "1000m"
      memory: "1Gi"

  connections:
    - id: my_aws
      type: aws
      extra: |
        {
           "aws_access_key_id": "<aws_id>",
           "aws_secret_access_key": "<aws_key>",
           "region_name":"ap-southeast-2"
        }
  variables: |
    {}
  pools: |
    {}
  initdb: true
  initialStartupDelay: 0

web:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
  replicas: 1
  service:
    annotations: {}
    type: LoadBalancer
    externalPort: 8080
    loadBalancerSourceRanges: []
  baseUrl: "http://airflow.azure:8080"
  livenessProbe:
    enabled: true
    scheme: HTTP
    initialDelaySeconds: 300
    periodSeconds: 30
    timeoutSeconds: 3
    successThreshold: 1
    failureThreshold: 2


workers:
  enabled: false

flower:
  enabled: false

logs:

  path: /opt/airflow/logs
  persistence:
    enabled: false
    subPath: ""
    storageClass: ""
    accessMode: ReadWriteMany
    size: 1Gi

dags:
  doNotPickle: true
  persistence:
    enabled: false
    subPath: ""
    storageClass: ""
    accessMode: ReadOnlyMany
    size: 1Gi

  git:
    url: "[email protected]:ryanntk/k8s_airflow.git"
    ref: master
    secret: airflow-git-keys
    sshKeyscan: true
    privateKeyName: id_rsa
    repoHost: "bitbucket.org"
    repoPort: 22
    gitSync:
      enabled: true
      image:
        repository: alpine/git
        tag: latest
        pullPolicy: Always
      refreshTime: 60
  initContainer:
    enabled: true
	

ingress:
  enabled: false

rbac:
  create: true
  events: false
  
serviceAccount:
  create: true
  name: ""
  annotations: {}

extraManifests: []

postgresql:
  enabled: false


externalDatabase:

  type: postgres
  host: <host>.database.azure.com
  port: 5432
  database: postgres
  user: <usrename>
  passwordSecret: airflow-postgres-password
  passwordSecretKey: postgres-password
  properties: ""

redis:
  enabled: false

serviceMonitor:
  enabled: false


prometheusRule:
  enabled: false

What is your Kubernetes Version?:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.10", GitCommit:"62876fc6d93e891aa7fbe19771e6a6c03773b0f7", GitTreeState:"clean", BuildDate:"2020-10-16T20:43:34Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

version.BuildInfo{Version:"v3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.11"}

Any help would be very much appreciated

Error in database init and google OAuth login

Airflow database init command is throwing some WARNING logs and users table not created.

I am using AWS SQS instead of Redis as a broker and have modified the helm chart accordingly. As SQS needs pycurl python package I took the official Dockerfile and made these changes as pycurl have OS-level dependency

 ARG RUNTIME_APT_DEPS="\
+       gcc \
+       libcurl4-gnutls-dev \
+       librtmp-dev \
+       python3-pycurl \
        apt-transport-https \
        apt-utils \
        ca-certificates \

The connection with Postgres DB and SQS is working fine.

SOME ENVIRONMENT VARIABLES

AIRFLOW_VERSION=2.0.0.dev0
BROKER_CONNECTION_CMD=echo -n "sqs://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@"
AIRFLOW__CELERY__BROKER_URL_CMD=bash -c 'eval "$BROKER_CONNECTION_CMD"'
AIRFLOW__CELERY__RESULT_BACKEND_CMD=bash -c 'eval "$DATABASE_CELERY_CMD"'
AIRFLOW__WEBSERVER__AUTHENTICATE=True
AIRFLOW__WEBSERVER__WEB_SERVER_PORT=8080
AIRFLOW__WEBSERVER__AUTH_BACKEND=airflow.contrib.auth.backends.google_auth
AIRFLOW__GOOGLE__DOMAIN=XXX.com
AIRFLOW__GOOGLE__CLIENT_SECRET=XXXXXXXXXXXXXXX
AIRFLOW__GOOGLE__OAUTH_CALLBACK_ROUTE=/oauth2callback
AIRFLOW__GOOGLE__CLIENT_ID=XXXXXXXXXXXXXXX.apps.googleusercontent.com

Airflow with Google OAuth but when I'm login into Airflow. I am getting an error that users tables don't exist.

[2020-12-11 13:26:09,854] {app.py:1892} ERROR - Exception on /oauth2callback [GET]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "users" does not exist
LINE 2: FROM users 
             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/airflow/.local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/auth/backends/google_auth.py", line 172, in oauth_callback
    models.User.username == username).first()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3402, in first
    ret = list(self[0:1])
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3176, in __getitem__
    return list(res)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3508, in __iter__
    return self._execute_and_instances(context)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3533, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
    distilled_params,
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "users" does not exist
LINE 2: FROM users 
             ^

[SQL: SELECT users.id AS users_id, users.username AS users_username, users.email AS users_email, users.superuser AS users_superuser 
FROM users 
WHERE users.username = %(username_1)s 
 LIMIT %(param_1)s]
[parameters: {'username_1': 'Shubham Choudhary', 'param_1': 1}]
(Background on this error at: http://sqlalche.me/e/13/f405)

I also see this in the scheduler logs

INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table

After running airflow db init I have all these tables created.

                        List of relations
 Schema |             Name              | Type  |      Owner      
--------+-------------------------------+-------+-----------------
 public | ab_permission                 | table | airflow_qa_user
 public | ab_permission_view            | table | airflow_qa_user
 public | ab_permission_view_role       | table | airflow_qa_user
 public | ab_register_user              | table | airflow_qa_user
 public | ab_role                       | table | airflow_qa_user
 public | ab_user                       | table | airflow_qa_user
 public | ab_user_role                  | table | airflow_qa_user
 public | ab_view_menu                  | table | airflow_qa_user
 public | alembic_version               | table | airflow_qa_user
 public | celery_taskmeta               | table | airflow_qa_user
 public | celery_tasksetmeta            | table | airflow_qa_user
 public | connection                    | table | airflow_qa_user
 public | dag                           | table | airflow_qa_user
 public | dag_code                      | table | airflow_qa_user
 public | dag_pickle                    | table | airflow_qa_user
 public | dag_run                       | table | airflow_qa_user
 public | dag_tag                       | table | airflow_qa_user
 public | import_error                  | table | airflow_qa_user
 public | job                           | table | airflow_qa_user
 public | log                           | table | airflow_qa_user
 public | rendered_task_instance_fields | table | airflow_qa_user
 public | sensor_instance               | table | airflow_qa_user
 public | serialized_dag                | table | airflow_qa_user
 public | sla_miss                      | table | airflow_qa_user
 public | slot_pool                     | table | airflow_qa_user
 public | task_fail                     | table | airflow_qa_user
 public | task_instance                 | table | airflow_qa_user
 public | task_reschedule               | table | airflow_qa_user
 public | variable                      | table | airflow_qa_user
 public | xcom                          | table | airflow_qa_user
(30 rows)

My Scheduler and Worker the airflow points to apache-airflow @ file:///opt/airflow whereas in Web apache-airflow==1.10.12

I am assuming that I am modifying the wrong Dockerfile!!

What is your Kubernetes Version?:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

SQS Support for CELERY BROKER

SQS Support as CELERY BROKER

As SQS can also be used also as a broker for Celery the helm chart should provide a way to configure that.

https://docs.celeryproject.org/en/stable/getting-started/brokers/sqs.html

We can maintain a toggle for worker.broker as "redis" or "sqs"

templates/_helpers.tpl

+{{- define "SQS_CONNECTION_CMD" -}}
+echo -n "sqs://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@"
+{{- end -}}

templates/config/configmap-env.yaml

+  {{- if eq .Values.workers.broker "redis"}}
+  BROKER_CONNECTION_CMD: {{ include "REDIS_CONNECTION_CMD" . }}
+  {{ else if eq .Values.workers.broker "sqs"}}
+  BROKER_CONNECTION_CMD: {{ include "SQS_CONNECTION_CMD" . }}

   AIRFLOW__CELERY__BROKER_URL_CMD: |-
-    bash -c 'eval "$REDIS_CONNECTION_CMD"'
+    bash -c 'eval "$BROKER_CONNECTION_CMD"'

with redis.enabled: false

Task logs from kubernetes worker not showing up in UI. Instead stdout of pod is present.

I have deployed airflow in a minikube kubernetes cluster in my local system. The dags are running perfectly well.

I have set AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "False", and the log file at /opt/airflow/logs/Simple_DAG/task_2/2020-12-07T00:00:00+00:00/1.log in the worker pod has the logs but these are not visible in the airflow UI

The logs of the task runs (in airflow UI) & worker pods is always just four lines long and looks like

*** Trying to get logs (last 100 lines) from worker pod funlogslogtesttask-313988f7f8104750b4de1bea2debc7a5 ***

[2020-12-08 00:09:30,192] {__init__.py:50} INFO - Using executor LocalExecutor
[2020-12-08 00:09:30,193] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/dags/show_some_logs.py
Running %s on host %s <TaskInstance: fun_logs.log_test_task 2019-07-25T09:45:00+00:00 [queued]> funlogslogtesttask-313988f7f8104750b4de1bea2debc7a5

Full Values.yml file at https://github.com/saisujithkp/dags/blob/main/values.yml

How can I make the UI show the logs of the task instead of the STDOUT of the worker pod?

Errno 13 - Permission denied: '/opt/airflow/logs/scheduler

What is the bug?
Getting permission denied, when enabling persistent volume for logs

What is your Kubernetes Version?:
v1.17.11

What is your Helm version?:
Helm 3

Similar issue: helm/charts#23589

Add annotations block for airflow-env ConfigMap

What is your feature request?

Please add a way for us to add annotations directly to the airflow.cfg ConfigMap. We deploy with Spinnaker, which naturally versions Secrets and ConfigMaps when they change. As a result, worker pods search the ConfigMap to see what the name of the ConfigMap for the Env is; the problem is that it's a static value ( airflow-env ).

To get around this, Spinnaker allows the annotation on Secrets / ConfigMaps:

strategy.spinnaker.io/versioned: "false"

This would allow a working configuration for anyone else that deploys via Spinnaker.

What alternatives have you considered?

We had previously created a subchart or additional Manifests that mirror the env configmap, but this would be the easiest way to get an official implementation with an alternative block for the configuration.

Happy to contribute or make the change.

Thank you!

Release old chart versions

Thanks for getting this up and running guys. Would it be possible to release old versions of the charts? I'm sure there are some users out there (myself included) who aren't ready to perform chart or airflow version upgrades. Additionally, some of us are still running on the docker-puckel image and upgrading the chart would force a change in base image which is significant. Yes, I know we need to get off of the old version ;) Thanks again!

Special characters in `airflow.config` keys

Because we currently store the content of airflow.config inside ConfigMap/airflow-env, some airflow configs will not work, this is because ConfigMap keys must match the regex [-._a-zA-Z0-9]+.

There are cases where / is needed for airflow configs, for example:

airflow:
  config:
    AIRFLOW__KUBERNETES_LABELS__spotinst.io/restrict-scale-down : "true"

We should move to using env: definitions, rather than mounting ConfigMaps for user provided configs.

Redirect URI for Airflow to authenticate using Azure AD

Helm chart is deployed in Azure Kubernetes Service. And using Nginx Ingress Controller accessing the webserver UI in a URL: https://xyz.com/admin/

Configmap (airflow-webserver-config) is created as how mentioned here.

Mounting through airflow.extraConfigmapMounts authentication could be established.

Provided the URL "https://xyz.com" as Redirect URI in Azure App registration getting the following error:
AADSTS50011: The reply URL specified in the request does not match the reply URLs configured for the application:

Question:

What is the Redirect URI to be provided in the Azure AD App registration?
Do we need any API permissions to be added in Azure AD?

[GitSync and GitClone] Use shallow clone and single branch

What is your feature request?

For the Git Sync and Git Clone it would be helpful to use shallow clone (only clone the blobs and not the entire history) this goes with the single branch option.
https://git-scm.com/docs/shallow
https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt

What alternatives have you considered?
Moving DAGs out of the monorepo into a separate repository.

Update docs for Dag Storage option 1

What is the bug?

The following two points are related to Dag Storage Option 1. As someone new to airflow, helm and kubernetes, these are suggestions to minor issues that set me back a few days.

The example shown to create kubernetes secret did not work for me. I had to switch ~/.ssh/* to $HOME/.ssh/*. Suggestion to update the example shell statement to the following:

kubectl create secret generic \
  airflow-git-keys \
  --from-file=id_rsa=$HOME/.ssh/id_rsa \
  --from-file=id_rsa.pub=$HOME/.ssh/id_rsa.pub \
  --from-file=known_hosts=$HOME/.ssh/known_hosts \
  --namespace airflow

The preceding ssh:// for url was misleading that caused my git-sync container to throw a ssh: Could not resolve hostname... error message. A suggestion to amend the example's presentation:

dags:
  git:
    url: [email protected]/example.git # ssh url
    repoHost: repo.example.com
    secret: airflow-git-keys
    privateKeyName: id_rsa

    gitSync:
      enabled: true
      refreshTime: 60

What are your Helm values?

N.A.

What is your Kubernetes Version?:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:09:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

$ helm version
version.BuildInfo{Version:"v3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"dirty", GoVersion:"go1.15.4"}

Worker POD error WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)

Has anyone come across this issue where the worker crashes with this error ?

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/celery/worker/worker.py", line 208, in start
self.blueprint.start(self)
File "/home/airflow/.local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/airflow/.local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/home/airflow/.local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/home/airflow/.local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/airflow/.local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 599, in start
c.loop(*c.loop_args())
File "/home/airflow/.local/lib/python3.6/site-packages/celery/worker/loops.py", line 83, in asynloop
next(loop)
File "/home/airflow/.local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 308, in create_loop
events = poll(poll_timeout)
File "/home/airflow/.local/lib/python3.6/site-packages/kombu/utils/eventio.py", line 84, in poll
return self._epoll.poll(timeout if timeout is not None else -1)
File "/home/airflow/.local/lib/python3.6/site-packages/celery/apps/worker.py", line 285, in _handle_request
raise exc(exitcode)
celery.exceptions.WorkerShutdown: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/billiard/pool.py", line 1267, in mark_as_worker_lost
human_status(exitcode)),
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).

document Google CloudSQL Proxy

What is your feature request?

As discussed by @MishaVeldhoen in #15 (comment), you can use this chart with the GCP CloudSQL Proxy, we should include some docs about this.

Add extra volume mounts on worker pods

How do we add extra volume mounts onto worker pods

I have to add additional volumes on to the worker pods that are spun up by Kubernetes executor. Although the comments in the values file suggest we can do this with the following configs:

  extraVolumeMounts:
  extraVolumes:

These are only working for the scheduler and the webserver pod and the worker pod doesn't have the extra volume mount when the task runs. I have the plugins required for dag run as a different volume claim and the dag fails to run as the plugins are not found.

The config is not working as mentioned in the comments of the values.yaml file. Is there any other way to achieve this. Looking forward for any inputs.

Executor is always being set to Sequential although I change it to Kubernetes

What is your question?

I'm trying to install airflow with KubernetesExecutor on my cluster, but when I go into the airflow-web container(via interactive shell) and check the contents of airflow.cfg, it is Sequential Executor.

I'm unable to understand why the executor is not being updated even though I'm making the change in the values file.

The command I use to bring up the cluster: helm install airflow airflow-stable/airflow --values values.yml

My values.yml file

###################################
# Airflow - Common Configs
###################################
airflow:
  ## configs for the docker image of the web/scheduler/worker
  ##
  image:
    repository: apache/airflow
    tag: 1.10.12-python3.6
    ## values: Always or IfNotPresent
    pullPolicy: IfNotPresent
    pullSecret: ""


  ## the airflow executor type to use
  ##
  ## NOTE:
  ## - this should be `CeleryExecutor` or `KubernetesExecutor`
  ## - if set to `KubernetesExecutor`:
  ##   - ensure that `workers.enabled` is `false`
  ##   - ensure that `flower.enabled` is `false`
  ##   - ensure that `redis.enabled` is `false`
  ##   - ensure that K8S configs are set in `airflow.config`
  ##   - we set these configs automatically:
  ##     - `AIRFLOW__KUBERNETES__NAMESPACE`
  ##     - `AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME`
  ##     - `AIRFLOW__KUBERNETES__ENV_FROM_CONFIGMAP_REF`
  ##     - `AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY`
  ##     - `AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG`
  ##
  executor: "KubernetesExecutor"

  ## the fernet key used to encrypt the connections/variables in the database
  ##
  ## WARNING:
  ## - you MUST customise this value, otherwise the encryption will be somewhat pointless
  ##
  ## NOTE:
  ## - to prevent this value being stored in your values.yaml (and airflow-env ConfigMap),
  ##   consider using `airflow.extraEnv` to define it from a pre-created secret
  ##
  ## GENERATE:
  ##   python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)"
  ##
  fernetKey: "7T512UXSSmBOkpWimFHIVb8jK6lfmSAvx4mO6Arehnc="

      
  ## environment variables for the web/scheduler/worker Pods (for airflow configs)
  ##
  ## WARNING:
  ## - don't include sensitive variables in here, instead make use of `airflow.extraEnv` with Secrets
  ## - don't specify `AIRFLOW__CORE__SQL_ALCHEMY_CONN`, `AIRFLOW__CELERY__RESULT_BACKEND`,
  ##   or `AIRFLOW__CELERY__BROKER_URL`, they are dynamically created from chart values
  ##
  ## NOTE:
  ## - airflow allows environment configs to be set as environment variables
  ## - they take the form: AIRFLOW__<section>__<key>
  ## - see the Airflow documentation: https://airflow.apache.org/docs/stable/howto/set-config.html
  ##
  ## EXAMPLE:
  ##   config:
  ##     ## Security
  ##     AIRFLOW__CORE__SECURE_MODE: "True"
  ##     AIRFLOW__API__AUTH_BACKEND: "airflow.api.auth.backend.deny_all"
  ##     AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
  ##     AIRFLOW__WEBSERVER__RBAC: "False"
  ##
  ##     ## DAGS
  ##     AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "30"
  ##     AIRFLOW__CORE__LOAD_EXAMPLES: "False"
  ##
  ##     ## Email (SMTP)
  ##     AIRFLOW__EMAIL__EMAIL_BACKEND: "airflow.utils.email.send_email_smtp"
  ##     AIRFLOW__SMTP__SMTP_HOST: "smtpmail.example.com"
  ##     AIRFLOW__SMTP__SMTP_STARTTLS: "False"
  ##     AIRFLOW__SMTP__SMTP_SSL: "False"
  ##     AIRFLOW__SMTP__SMTP_PORT: "25"
  ##     AIRFLOW__SMTP__SMTP_MAIL_FROM: "[email protected]"
  ##
  ##     ## Disable noisy "Handling signal: ttou" Gunicorn log messages
  ##     GUNICORN_CMD_ARGS: "--log-level WARNING"
  ##
  ##     ## Proxy Config
  ##     HTTP_PROXY: "http://proxy.example.com:8080"
  ##
  config: {
    AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: "True"

  }

  ## extra annotations for the web/scheduler/worker Pods
  ##
  ## EXAMPLE:
  ##   podAnnotations:
  ##     iam.amazonaws.com/role: "airflow-Role"
  ##
  podAnnotations: {}

  ## extra environment variables for the web/scheduler/worker (AND flower) Pods
  ##
  ## EXAMPLE:
  ##   extraEnv:
  ##     - name: AIRFLOW__CORE__FERNET_KEY
  ##       valueFrom:
  ##         secretKeyRef:
  ##           name: airflow-fernet-key
  ##           key: value
  ##     - name: AIRFLOW__LDAP__BIND_PASSWORD
  ##       valueFrom:
  ##         secretKeyRef:
  ##           name: airflow-ldap-password
  ##           key: value
  ##
  extraEnv: []

  ## extra configMap volumeMounts for the web/scheduler/worker Pods
  ##
  ## EXAMPLE:
  ##   extraConfigmapMounts:
  ##     - name: airflow-webserver-config
  ##       mountPath: /opt/airflow/webserver_config.py
  ##       configMap: airflow-webserver-config
  ##       readOnly: true
  ##       subPath: webserver_config.py
  ##
  extraConfigmapMounts: []

The airflow.cfg in the airflow-web pod

add additional values files for chart-testing

Currently, we only test the default values.yaml in the CI workflow, we should also test more complex configs:

set airflow.configs
set airflow.extraPipPackages
set airflow.image.* (different airflow/python versions)
set scheduler.connections
set scheduler.variables
set scheduler.pools

The easiest way to do this is add a ci folder under charts/airflow, and place files named *-values.yaml in it.

NOTE:

we might need to extend --timeout from 10min in ct-config.yaml
we should think about if there is a way to run a test dag, and check its output
you can read more about helm/chart-testing here:
- https://github.com/helm/chart-testing/blob/master/doc/ct_lint.md
- https://github.com/helm/chart-testing/blob/master/doc/ct_install.md

Communication Channels

What are the current communication channels between maintainers and users?

How do I retain ownership of the mounted git repo? And is it possible to only sync a specific folder from git?

Hello maintainers. Thanks for putting together this project, I found documentation for almost every single step I was stuck at.

I am currently using the git-sync mount to pull my DAGs from a repository. I am using Data Build Tool (dbt) alongside with airflow, and wanted to use the sync'd git repo to mount both the dags and dbt models (files that need to be read/written-to by dbt) to keep everything in one repo. I found that after mounting, the dags folder ownership changes from the airflow user to become the root user. I have further documented this here and attempted to resolve it through StackOverflow before heading here.

My question(s) to you:

Can I (and should I?) mount other assets (not just Dags) via the git-sync container? If so, how can I retain the ownership of the mounted folders?
Let's say I only want to mount the dags. Is it possible to only mount a particular folder from the git repo? I am asking because maybe I want to separate dbt code from airflow code, yet keep in the same repo (so only sync the dags folder in the repo)

I am happy to provide any further details needed = )

Update:
Regarding the second question, I found a hack around that by using .airflowignore which has been discussed here

helm template generates manifests without namespace

What is the bug?

When used with helm template like this:

helm template airflow airflow-stable/airflow -f custom_values.yaml --namespace=airflow

The chart will generate manifests that, for the most part, don't contain namespace: key (RoleBinding is the only one I can see that will contain namespace).

This is problematic because helm template | kubectl apply -f - will result in a different state than helm install whcih can be surprising to users.

What are your Helm values?

This happens even without custom_values

What is your Helm version?:

$ helm version
version.BuildInfo{Version:"v3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.11"}

Other

Fix for this can be to include the following in all templates:

namespace: {{ .Release.Namespace }}

upgrading from stable/airflow

I have an existing Airflow installation on our Kubernetes cluster.
Can I upgrade it to the new chart without uninstallaing & reinstalling?

Make TLS secret name on Ingress optional

See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#ingresstls-v1-networking-k8s-io

"SecretName is the name of the secret used to terminate TLS traffic on port 443. Field is left optional to allow TLS routing based on SNI hostname alone. If the SNI host in a listener conflicts with the "Host" header field used by an IngressRule, the SNI host is used for termination and value of the Host header is used for routing."

I use an nginx-ingress controller and it supports a default certificate (https://kubernetes.github.io/ingress-nginx/user-guide/tls/#default-ssl-certificate - "The default certificate will also be used for ingress tls: sections that do not have a secretName option.") so there's no need to set this value. Currently I have to either manually modify the ingress object or duplicate the secret I want to use. This really should be optional and changing it wouldn't impact existing chart users.

Not adding the Connections via yaml/config

What is the bug?
I'm trying to set up the connections via yaml/config, but it seems that's not getting it. I can see the environment variable in the container, but it doesn't add the connection.

I tried with AIRFLOW_CONN_POSTGRES_DEFAULT, AIRFLOW__CONN_POSTGRES_DEFAULT and AIRFLOW__CONN__POSTGRES_DEFAULT.

I got the example from here:
https://airflow.apache.org/docs/stable/howto/connection/postgres.html
https://airflow.apache.org/docs/stable/howto/connection/index.html

Helm Chart Version?

helm upgrade --install --version 7.14.0

What are your Helm values?

airflow:
  config:
    AIRFLOW_CONN_POSTGRES_DEFAULT: 'postgresql://postgres_user:[email protected]:5432/postgresdb?sslmode=verify-ca&sslcert=%2Ftmp%2Fclient-cert.pem&sslkey=%2Ftmp%2Fclient-key.pem&sslrootcert=%2Ftmp%2Fserver-ca.pem'

What is your Kubernetes Version?:

$ kubectl version
v1.17.9-eks-4c6976

What is your Helm version?:

$ helm version
v3.4.0

stop using sub-charts for postgres and redis

Currently, we depend on stable/postgres and stable/redis charts when the user does not configure externalDatabase.* and externalRedis.*.

This is bad for a number of reasons:

we cant update postgres/redis versions
we cant fix bugs in those charts
those charts are deprecated

Given how simple a single node deployment of postgres/redis is, we should just include the required yaml in the airflow chart.

web-pod can't connect to postgres (password authentication failed)

Describe the bug
When using the stable chart, my web-pod can't connect to the postgres-pod. I've also tried to exec into the postgres-pod, but the same result there.

Version of Helm and Kubernetes:
kubernetes: v1.16.7
helm: v3.2.3

Which chart:
stable/airflow

What happened:
airflow-web logs:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "postgres"

airflow-postgres logs:

2020-07-03 11:43:19.358 GMT [196] FATAL:  password authentication failed for user "postgres"
2020-07-03 11:43:19.358 GMT [196] DETAIL:  Password does not match for user "postgres".
	Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"

What you expected to happen:
Expected it to get up and running

How to reproduce it (as minimally and precisely as possible):
helm install airflow stable/airflow

Anything else we need to know:
This is a copy of helm/charts#23061

v1.10.14 airflow requires secret_key for web instances

Airflow helm chart should support version 1.10.14 (https://github.com/apache/airflow/releases/tag/1.10.14) requirement for configuring SECRET_KEY for web instances (https://github.com/apache/airflow/blob/a878959b2c0e0a4c4d7aa96fd01bd997f4a7de1a/airflow/configuration.py#L775)

As the SECRET_KEY is specific for web instances and is only problematic when it has replica > 1, as each web replica will randomly generate its own secret key causing login to fail due to pod switch.

Is your feature request related to a problem? Please describe.
In airflow version 1.10.14 airflow require a secret key for web instances, if it is not specifically set it will be automatically created by os.urandom(16), each web replica will create its own SECRET_KEY causing login to fail.

This ticket is related to apache/airflow#13081

Describe the solution you'd like
airflow helm chart should support secret key just like fernet key.

web:
   replicas: 3
   secret_key: ${secret_key}

Describe alternatives you've considered
In the interim we set SECRET_KEY = os.environ.get("FLASK_SECRET_KEY") through webserver_config.py

The earlier solution we put in was to configure alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600

And verified when web replica=1 it is working fine.

Worker sidecar for DAG logs

Hi, i am very new to using airflow and im trying to figure out a way to get the DAG logs from the worker pods to stdout (currently using CeleryExecutor).

I know that it is common to use logging sidecars to extract logs from containers that do not emit logs to stdout. I am wondering if there is a way to use an extra container as a sidecar to pick up the DAG logs but only for the worker pods.

Currently i see that we have the option to use airflow.extraContainers but if i am not mistaken this way the container is created in all the pods, which is not ideal.

If there is currently another way (apart from configuring remote logging to S3 and elasticsearch) to get the logs in stdout, please let me know.

Thank you

Workaround found for: Unable to attach or mount volumes mount.nfs Connection timed out

What is the bug?

@thesuperzapper (Mathew Wicks) suggested I post the workaround that I found for this EFS problem I experienced setting up my PV/PVC for AirFlow. This is not an issue with AirFlow itself, it was a Persistent Volume issue that should work but did not. Please close this bug as the intention here is to be a source of documentation to anyone who might experience this same issue and would like to find a working solution.

When I specified a path: other than "/" in the PersistentVolume manifest, K8s failed to mount the volume to the AirFlow web and scheduler pods with this message:

 unmounted volumes=[k8s-cluster-dev], unattached volumes=[scripts k8s-cluster-dev airflow-devops-test-token-7bq9s]:
 timed out waiting for the condition
 
Warning  FailedMount  109s  kubelet, ip-10-4-4-44.ec2.internal  MountVolume.SetUp failed for volume "k8s-cluster-dev-efs-airflow-devops-test" :
 mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/xxxxxx-0d08-4b4f-b2e6-f97971d9dbd0/volumes/kubernetes.io~nfs/k8s-cluster-dev-efs-airflow-devops-test --scope -- mount -t nfs fs-xxxxxxxx.efs.us-x-x.amazonaws.com:/a.domain.com/airflow-devops-test /var/lib/kubelet/pods/xxxxxxx-0d08-4b4f-b2e6-xxxxxxxx/volumes/kubernetes.io~nfs/k8s-cluster-dev-efs-airflow-devops-test
Output: Running scope as unit run-28953.scope.
mount.nfs: Connection timed out

The PersistentVolume.yaml looks like this (redactions with xxxx here):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: k8s-cluster-dev-efs-airflow-devops-test
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: fs-xxxxxxxx.efs.us-x-x.amazonaws.com
    path: "/some/path"
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

There was no error from K8s when this manifest was used (with the real server name of course) and it showed Bound, and indeed the PVC could also be Bound to it. But when AirFlow tried to mount the volume, that's when the weird nfs timeout errors occurred. Changing the path: on the PV to "/" fixed the issue and allowed the volume to mount to AirFlow.

What are your Helm values?

Any relevant parts of your custom_values.yaml

  extraVolumeMounts:
    - name: k8s-cluster-dev
      mountPath: /opt/airflow/efs
  extraVolumes:
    - name: k8s-cluster-dev
      persistentVolumeClaim:
        claimName: k8s-cluster-dev-efs-airflow-devops-test

What is your Kubernetes Version?:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.10", GitCommit:"575467a0eaf3ca1f20eb86215b3bde40a5ae617a", GitTreeState:"clean", BuildDate:"2019-12-11T12:41:00Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:31:21Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}

Allow for $ characters in password / extra field of Airflow connections

What is the bug?
Whenever I try adding an Airflow connection that has a $-character in either the password or extra field, the value gets chopped off. I suspect this is because ultimately the value is passed to a bash command (airflow connections --add) that interprets $ as a bash-variable, which is obviously empty. For example, if I submit an airflow connection with password / extra 'abcd$ef', the password / extra stored in Airflow only contains 'abcd', since the variable $ef is not defined. If I manually escape the $-character with a \ in the bash-command, the value is properly added.

What are your Helm values?

scheduler:
  connections:
    - id: 'test'
      type: http
      login: 'test'
      password: 'A^Ci$c#h"GhjCofpn\R$'
      extra: 'A^Ci$c#h"GhjCofpn\R$'

What is your Kubernetes Version?:
1.17.13

What is your Helm version?:
3.4.1

Mount file as a secret to Airflow home or to any other directories

Is your feature request related to a problem? Please describe.
I'm using Airflow with LDAP+RBAC. Currently I'm mounting my webserver_config.py using extraConfigmapMounts functionality from chart. Here is how it looks like:

  extraConfigmapMounts:
    - name: airflow-webserver-config
      mountPath: /opt/airflow/webserver_config.py
      configMap: airflow-webserver-config
      readOnly: true
      subPath: webserver_config.py

Of course it is not secure because in webserver_config.py I have AUTH_LDAP_BIND_USER and AUTH_LDAP_BIND_PASSWORD.

Describe the solution you'd like
I want to have ability to mount this file from secret. For example I will create secret from file and want to mount it instead of ConfigMap as I mentioned above:
kubectl create secret generic airflow-webserver-config --from-file=webserver_config.py -n airflow

I know that chart have following functionality:

  secretsDir: /opt/airflow
  
  secrets:
    - airflow-webserver-config

But instead of overwriting webserver_config.py (existing/default) file in AIRFLOW_HOME (/opt/airflow) it will create directory /opt/airflow/airflow-webserver-config and put my webserver_config.py into it.

Additional context
I know that k8s secrets not really secrets and it is easy to decode it. But in any case it is better to keep password encoded at least with base64 than keep it as plain text...

Command error with Airflow v2

What is the bug?

Running airflow-helm with airflow 2 throws the following error.

airflow command error: argument GROUP_OR_COMMAND: `airflow worker` command, has been removed, please use `airflow celery worker`, see help above.

I think the problem lies in here:

charts/charts/airflow/templates/worker/worker-statefulset.yaml

Line 228 in 7c46d87

&& exec airflow worker

What are your Helm values?

image:
    repository: apache/airflow
    tag: 2.0.0-python3.6

What is your Kubernetes Version?:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.12-eks-7684af", GitCommit:"7684af4ac41370dd109ac13817023cb8063e3d45", GitTreeState:"clean", BuildDate:"2020-10-20T22:57:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

$ helm version
WARNING: "kubernetes-charts.storage.googleapis.com" is deprecated for "stable" and will be deleted Nov. 13, 2020.
WARNING: You should switch to "https://charts.helm.sh/stable" via:
WARNING: helm repo add "stable" "https://charts.helm.sh/stable" --force-update
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}

Move pip-install out of container start commands

We should move pip-install out of the container start commands, as this forces us to work around the liveness/readiness probes timing out, because pip-install can take long amounts of time.

My idea is to put all the pip install inside an initContainer (into a non-persistent Volume), and prefix the PYTHONPATH of the main container as it starts to include the install folder.

Multi-Attach error for volume for pods.

What is the bug?

I am trying to add the PVC for dags, so I added the relevant values for PVC in custom_value.yaml.
I Created PVC before installing the helm and the following is the PVC code:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: airflow-dag-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Error:
Multi-Attach error for volume "pvc-de5cae93-d519-4479-b24a-587083ffc642" Volume is already used by pod(s) airflow-chart-scheduler-75964ffbf4-drszp

What are your Helm values?

Any relevant parts of your custom_values.yaml

dags:
## configs for the dags PVC
##
  persistence:
    ## if a persistent volume is mounted at `dags.path`
    ##
    enabled: true

    ## the name of an existing PVC to use
    ##
    existingClaim: "airflow-dag-pvc"

What is your Kubernetes Version?:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:17:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.15-gke.4901", GitCommit:"7ed5ddc0e67cb68296994f0b754cec45450d6a64", GitTreeState:"clean", BuildDate:"2020-11-06T18:22:22Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

$ helm version
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}

Installing requirements on Kubernetes workers

What is your feature request?

Currently installRequirements only applies to the web and scheduler pods, however when using the Kubernetes executor you need to pre-bake the pip dependencies in order to run your DAGs.

What alternatives have you considered?

Baking in DAGs (developer experience kind of sucks)
Overriding the args on the Kubernetes executor worker pod to first run a script to install the requirements; this would also require mounting the requirements.txt file - this could be done in the same way as the web and scheduler pods

Notes

This issue is mostly to start a discussion around this, I'm sure others are facing the same limitation. I also don't think this is an issue with the Chart as such, so am keen to find out if Airflow already has the ability to do this.

External Postgres Database connects/disconnects randomly in scheduler

What is the bug?
Using an external database get random disconnects in the scheduler.

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 159, in _run_file_processor
    pickle_dags)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1609, in process_file
    dag.sync_to_db()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/dag.py", line 1515, in sync_to_db
    DagModel).filter(DagModel.dag_id == self.dag_id).first()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3429, in first
    ret = list(self[0:1])
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3203, in __getitem__
    return list(res)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
    return self._execute_and_instances(context)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3557, in _execute_and_instances
    querycontext, self._connection_from_session, close_with_result=True
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3572, in _get_bind_args
    mapper=self._bind_mapper(), clause=querycontext.statement, **kw
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3550, in _connection_from_session
    conn = self.session.connection(**kw)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1141, in connection
    execution_options=execution_options,
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1147, in _connection_for_bind
    engine, execution_options
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
    conn = bind._contextual_connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2340, in _wrap_pool_connect
    e, dialect, self
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1584, in _handle_dbapi_exception_noconnection
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
    return fn()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
    rec = pool._do_get()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
    self._dec_overflow()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    with_traceback=exc_tb,
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
    return self._create_connection()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
    return _ConnectionRecord(self)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
    self.__connect(first_connect_check=True)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    with_traceback=exc_tb,
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 493, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection timed out
	Is the server running on host "IPADRESSHERE" and accepting
	TCP/IP connections on port 5432?

(Background on this error at: http://sqlalche.me/e/13/e3q8)
[2020-12-10 01:16:23,212] {timeout.py:42} ERROR - Process timed out, PID: 3666

Process DagFileProcessor322-Process:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
    return fn()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
    rec = pool._do_get()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
    self._dec_overflow()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    with_traceback=exc_tb,
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
    return self._create_connection()
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
    return _ConnectionRecord(self)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
    self.__connect(first_connect_check=True)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    with_traceback=exc_tb,
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 493, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection timed out
	Is the server running on host "IPADDRESSHERE" and accepting
	TCP/IP connections on port 5432?

A clear description of the bug.

This bug is in the scheduler. After it starts up, without me triggering, it just randomly errors out to the database every few minutes. If I do start a job it give the second error above Process DagFileProcessor####-Process: and the triggered dag just hangs until it can reconnect again and keeps it connection for abit.

This is an external Database that is POSTGRESQL 10 in GCP
More details
vCPUs
2
Memory
6 GB
SSD storage
20 GB

What are your Helm values?

externalDatabase:
  ## the type of external database: {mysql,postgres}
  ##
  type: postgres

  ## the host of the external database
  ##
  host: IPADDRESS

  ## the port of the external database
  ##
  port: 5432

  ## the database/scheme to use within the the external database
  ##
  database: develop

  ## the user of the external database
  ##
  user: develop

  ## the name of a pre-created secret containing the external database password
  ##
  passwordSecret: postgresql

  ## the key within `externalDatabase.passwordSecret` containing the password string
  ##
  passwordSecretKey: postgresql-password

Any relevant parts of your custom_values.yaml

  image:
    repository: apache/airflow
    tag: 1.10.12-python3.6
    pullPolicy: IfNotPresent
    pullSecret: ""
  executor: CeleryExecutor
  config:
      ## Security
      AIRFLOW__CORE__SECURE_MODE: "True"
      AIRFLOW__API__AUTH_BACKEND: "airflow.api.auth.backend.deny_all"
      AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
      AIRFLOW__WEBSERVER__RBAC: "True"
      AIRFLOW__CORE__DEFAULT_TIMEZONE: "America/Chicago"
      AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS: "False"
   
      ## DAGS
      AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "20"

      ## GCP Remote Logging
      AIRFLOW__CORE__REMOTE_LOGGING: "True"
      AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "gs://somebucket/airflow/logs"
      AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "google_cloud_airflow"

      ## Disable noisy "Handling signal: ttou" Gunicorn log messages
      GUNICORN_CMD_ARGS: "--log-level WARNING"

What is your Kubernetes Version?:

1.16.15-gke.4300

What is your Helm version?:

 helm3 version
version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"clean", GoVersion:"go1.14.9"}

airflow-helm / charts Goto Github PK

charts's Introduction

Airflow Helm Chart (User Community)

↓ ↓ ↓

charts's People

Contributors

Stargazers

Watchers

Forkers

charts's Issues

The airflow.cfg in the airflow-web pod

Recommend Projects

Recommend Topics

Recommend Org