opendatahub-io-contrib / airflow-on-k8s-operator Goto Github PK

Describe the bug
I use airflow operator on okd 4, when cluster pod has been deployed, cluster pod could not connect to redis. I saw logs and I found this line:
timeout 3 nc -z pc-cluster-redis tcp://172.30.49.193:6379
it seems REDIS_PORT is set wrong,

After that I checked environment variables from container, then I saw this variables has been set

PC_CLUSTER_REDIS_PORT_6379_TCP_PROTO=tcp
REDIS_PORT=tcp://172.30.49.193:6379
REDIS_PORT_6379_TCP_ADDR=172.30.49.193
AIRFLOW_REDIS_PORT=tcp://172.30.26.111:6379
REDIS_PORT_6379_TCP_PORT=6379
REDIS_PORT_6379_TCP_PROTO=tcp
AIRFLOW_REDIS_PORT_6379_TCP_ADDR=172.30.26.111
PC_CLUSTER_REDIS_PORT_6379_TCP=tcp://172.30.32.239:6379
AIRFLOW_REDIS_PORT_6379_TCP_PORT=6379
AIRFLOW_REDIS_PORT_6379_TCP_PROTO=tcp
REDIS_PORT_6379_TCP=tcp://172.30.49.193:6379
AIRFLOW_REDIS_PORT_6379_TCP=tcp://172.30.26.111:6379
PC_CLUSTER_REDIS_PORT=tcp://172.30.32.239:6379
PC_CLUSTER_REDIS_PORT_6379_TCP_ADDR=172.30.32.239
PC_CLUSTER_REDIS_PORT_6379_TCP_PORT=6379

I used this manifest:

apiVersion: airflow.apache.org/v1alpha1
kind: AirflowCluster
metadata:
  name: pc-cluster
spec:
  executor: Celery
  redis:
    image: redis
    resources: {}
    version: '4.0'
  scheduler:
    image: "quay.io/opendatahub/docker-airflow"
    version: "openshift"
  ui:
    image: "quay.io/opendatahub/docker-airflow"
    replicas: 1
    version: "openshift"
    enableroutes: true
  worker:
    image: "quay.io/opendatahub/docker-airflow"
    replicas: 2
    version: "openshift"
    ForceRoot: "true"
  flower:
    image: "quay.io/opendatahub/docker-airflow"
    replicas: 1
    version: "openshift"
    enableroutes: true
  dags:
    subdir: "airflow/dags/"
    git:
      repo: "https://github.com/VedantMahabaleshwarkar/airflow-dags"
      once: true
  airflowbase:
    name: pc-base

would you guide me?

Features like OAuth/LDAP Group Sync likely won't get back-ported to Airflow 1.10.X [1], so if we care about providing the ability for customers to deploy Airflow properly in a multi-tenant environment we should probably take a look at upgrading (or at least providing a compatible branch) for the 2.0 Airflow community image.

I've done some testing and besides changing the build process, the only glaring incompatibilities between the Puckel image (ours) and the new community image is how some of the service environment variables are injected into the container at runtime. We should be able to easily shim around this using environment variable templating.

Another thing I've noticed in testing 2.0 is that

airflow worker 

Has changed to the more specific:

airflow celery worker

So we'll need to address that in the worker code as well.

• 1: https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html#support-for-airflow-1-10-x-releases

Is your feature request related to a problem? Please describe.
Upgrade the git-sync container to the latest release that adds support for additional git features and parameters.

Describe the solution you'd like
Update the airflow operator git-sync default repo and tag so a newer image and add a field to the dag.git spec to allow a user to specify a list of env vars that will be passed to git-sync

dags:
  git:
    git-sync-extra-env:
      GIT_SYNC_ROOT: $HOME/alt_dir/
      GIT_SYNC_SUBMODULES: shallow

Describe alternatives you've considered
N/A

Additional context
The primary goal is to use the updated image git-sync image.

Currently, the Airflow operator doesn't enable authentication/authorization for Airflow users. We would like to:

• Enable RBAC for the Airflow Webserver
• implement “Login with OpenShift” as a default auth

Here are some examples of existing "Login with OpenShift" implementations:

OAuth + JupyterHub Authenticator = OAuthenticator (login with OpenShift)
https://github.com/jupyterhub/oauthenticator
https://github.com/jupyterhub/oauthenticator/blob/master/oauthenticator/openshift.py
https://github.com/AICoE/jupyterhub-ocp-oauth/blob/master/.jupyter/jupyterhub_config.py

OpenShift Authentication Integration with ArgoCD
https://www.openshift.com/blog/openshift-authentication-integration-with-argocd
https://github.com/dexidp/dex/blob/master/Documentation/connectors/openshift.md
https://github.com/dexidp/dex/blob/master/connector/openshift/openshift.go

Users would like to leverage an external database (e.g. Amazon RDS) instead of the postgresql deployed by the operator. Allow users to configure the operator to use an external database.

Hey, AICoE-CI!

Please build and deliver the following git tag:

Tag: v0.0.1