KubernetesPodOperatorAsync not using AIRFLOW__KUBERNETES__CONFIG_FILE about astronomer-providers HOT 9 CLOSED

in the Triggerer

[2022-03-11 10:41:14,176] {triggerer_job.py:356} INFO - Trigger <astronomer.providers.cncf.kubernetes.triggers.wait_container.WaitContainerTrigger kubernetes_conn_id=None, hook_params={'cluster_context': None, 'config_file': None, 'in_cluster': None}, pod_name=airflow-test-pod.3279ee5034e046e2baa4b22d746cabf4, container_name=base, pod_namespace=default, pending_phase_timeout=120, poll_interval=10> (ID 9) starting
2022-03-11T10:41:14.186014105Z [2022-03-11 10:41:14,176] {kube_config.py:482} WARNING - Config not found: ~/.kube/config

I can fix the problem by doing this ->

    KubernetesPodOperatorAsync(
        config_file=os.getenv("AIRFLOW__KUBERNETES__CONFIG_FILE"),
        ...

from astronomer-providers.

actually i would prefer if we do not add support for using the config. as @raphaelauv has pointed out, users can set the config file in operator param. and once we add support for k8s hook in kpo, then users can just set it in k8s connection.

if we add support for taking values from conf, then we'll have to make a major release in astro operators to remove it. it seems that if we document this difference, then when switching to this operator they can also take care to set param.

what do yall think?

from astronomer-providers.

Yup agreed, thank you both

from astronomer-providers.

@raphaelauv Where did you see this log, on Triggerer or Scheduler ?

cc @dstandish

from astronomer-providers.

Looks like we have the same thing with KubernetesPodOperator, currently that setting is mostly for KubernetesExecutor and we can make it explicit in Airflow docs or start using the configs for KPO too :)

The other thing you could do is:

    KubernetesPodOperatorAsync(
        config_file=conf.get("kubernetes", "config_file"),
        namespace=conf.get("kubernetes", "namespace"),
        ...

That way, it will work for all different ways of settings Airflow configs: Secrets Backend, Env Vars, airflow.cfg :)

from astronomer-providers.

No AIRFLOW__KUBERNETES__CONFIG_FILE work with the KubernetesPodOperator because of this line

https://github.com/apache/airflow/blob/3452f7ce45607af04bade5e5edebaa18fdc13819/airflow/kubernetes/kube_client.py#L133

call by https://github.com/apache/airflow/blob/3452f7ce45607af04bade5e5edebaa18fdc13819/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L320

I've been using for long time the KPO without explicitly passing the config_file thanks to that env_var

from astronomer-providers.

Oh yes, you are absolutely right, we will fix it and release a new version soon :)

from astronomer-providers.

Yeah more work needs to be done to (1) make KPO use K8s hook and (2) no longer depend on internal kubernetes utils. There are a few PRs in flight concerning this. I am thinking we should add support for AIRFLOW__KUBERNETES__CONFIG_FILE with a deprecation warning and signal that in future everything will have to be configured through the hook. Let me look into this.

from astronomer-providers.

I agree with you, using a K8S connection will be nicer, so let's not add this feature to the KPOA.

from astronomer-providers.