Number of queries to Airflow database in "DAG File Processing Stats" about airflow HOT 6 OPEN

Good idea !

I would also love a global airflow option DAG_PARSING_METADATABASE_CALL_FORBIDEN

that could be set to True , so parsing of dags with variable / connection ... at top level fail. And promote usage of Dynamic Task mapping in the error log

from airflow.

@raphaelauv Wouldn't such a feature conflict with the theory of the Secrets Cache that was recently released too?

from airflow.

AIRFLOW__SECRETS__USE_CACHE is by default at false

and yes this feature will no more be usefull when DAG_PARSING_METADATABASE_CALL_FORBIDEN is set to True

from airflow.

@eladkal @potiuk @kaxil
Let's continue here our discussion about adding more logs in our output
@raphaelauv Thank you for your ideas, but I think this can be not directly related to this exact issue with adding exact logs to output. Maybe it will be more efficient to create a new issue for this? WDYT?

from airflow.

@raphaelauv Thank you for your ideas, but I think this can be not directly related to this exact issue with adding exact logs to output. Maybe it will be more efficient to create a new issue for this? WDYT?

Yeah, let's keep it separate

from airflow.

In my PR for this feature I have added a new column to the table with processing results in the log file. I have seen in the discussion thread some ideas about adding this information to the UI or DB, but I didn't notice a strong consensus about it and I decided not to make any UI or DB changes in this PR. I will be ready to create a new PR in future when we have a clear vision where on UI and DB are better to have this data.

from airflow.