andgineer / airflow Goto Github PK

Airflow do not support in-memory

DeprecationWarning: The namespace option in [kubernetes] has been moved to the namespace option in [kubernetes_executor] - the old setting has been used, but please update your config.

ERROR: relation "log" does not exist at character 13
airflow-airflow-db-1 | 2023-05-14 05:15:08.532 UTC [56] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2023-05-14T05:15:08.519949+00:00'::timestamptz, 'cli_webserver', 'root', '{"host_name": "7a3365f13543", "full_command": "[''/opt/conda/bin/airflow'', ''webserver'']"}') RETURNING log.id

DeprecationWarning: The logging_level option in [core] has been moved to the logging_level option in [logging] - the old setting has been used, but please update your config.
DeprecationWarning: The fab_logging_level option in [core] has been moved to the fab_logging_level option in [logging] - the old setting has been used, but please update your config.
DeprecationWarning: The log_format option in [core] has been moved to the log_format option in [logging] - the old setting has been used, but please update your config.
The colored_log_format option in [core] has been moved to the colored_log_format option in [logging] - the old setting has been used, but please update your config.
The colored_formatter_class option in [core] has been moved to the colored_formatter_class option in [logging] - the old setting has been used, but please update your config.
The base_log_folder option in [core] has been moved to the base_log_folder option in [logging] - the old setting has been used, but please update your config.
The dag_processor_manager_log_location option in [core] has been moved to the dag_processor_manager_log_location option in [logging] - the old setting has been used, but please update your config.
The log_processor_filename_template option in [core] has been moved to the log_processor_filename_template option in [logging] - the old setting has been used, but please update your config.
The remote_logging option in [core] has been moved to the remote_logging option in [logging] - the old setting has been used, but please update your config.

airflow-airflow-1 | /opt/conda/lib/python3.11/site-packages/airflow/utils/module_loading.py:40 DeprecationWarning: The airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG class is deprecated. Please use 'airflow.providers.celery.executors.default_celery.DEFAULT_CELERY_CONFIG'. The celery provider must be >= 3.3.0 for that..
airflow-airflow-1 | /opt/conda/lib/python3.11/site-packages/airflow/providers/celery/executors/celery_executor_utils.py:76 RemovedInAirflow3Warning: The celery.CELERY_APP_NAME configuration uses deprecated package name: 'airflow.executors.celery_executor'. Change it to airflow.providers.celery.executors.celery_executor, and update the -app flag in your Celery Health Checks to use airflow.providers.celery.executors.celery_executor.app.

RuntimeError: Detected recursive loop when walking DAG directory /etl: /etl/logs/scheduler/2023-05-14 has appeared more than once.

etl/pandas_hello.py:38: RemovedInAirflow3Warning: provide_context is deprecated as of 2.0 and is no longer required
/etl/pandas_hello.py:31: RemovedInAirflow3Warning: provide_context is deprecated as of 2.0 and is no longer required
etl/pandas_hello.py:24: RemovedInAirflow3Warning: provide_context is deprecated as of 2.0 and is no longer required
etl/pandas_hello.py:8: RemovedInAirflow3Warning: Param schedule_interval is deprecated and will be removed in a future release. Please use schedule instead.
dag = DAG(
etl/pandas_hello.py:2: DeprecationWarning: The airflow.operators.python_operator.PythonOperator class is deprecated. Please use 'airflow.operators.python.PythonOperator'.
from airflow.operators.python_operator import PythonOperator

now works only in CI

airflow-airflow-1 | Process ForkProcess-22:
airflow-airflow-1 | Traceback (most recent call last):
airflow-airflow-1 | File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
airflow-airflow-1 | self.run()
airflow-airflow-1 | File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
airflow-airflow-1 | self._target(*self._args, **self._kwargs)
airflow-airflow-1 | File "/opt/conda/lib/python3.10/site-packages/airflow/dag_processing/manager.py", line 259, in _run_processor_manager
airflow-airflow-1 | processor_manager.start()
airflow-airflow-1 | File "/opt/conda/lib/python3.10/site-packages/airflow/dag_processing/manager.py", line 493, in start
airflow-airflow-1 | return self._run_parsing_loop()
airflow-airflow-1 | File "/opt/conda/lib/python3.10/site-packages/airflow/dag_processing/manager.py", line 565, in _run_parsing_loop
airflow-airflow-1 | self._refresh_dag_dir()
airflow-airflow-1 | File "/opt/conda/lib/python3.10/site-packages/airflow/dag_processing/manager.py", line 759, in _refresh_dag_dir
airflow-airflow-1 | self._file_paths = list_py_file_paths(self._dag_directory)
airflow-airflow-1 | File "/opt/conda/lib/python3.10/site-packages/airflow/utils/file.py", line 306, in list_py_file_paths
airflow-airflow-1 | file_paths.extend(find_dag_file_paths(directory, safe_mode))
airflow-airflow-1 | File "/opt/conda/lib/python3.10/site-packages/airflow/utils/file.py", line 319, in find_dag_file_paths
airflow-airflow-1 | for file_path in find_path_from_directory(str(directory), ".airflowignore"):
airflow-airflow-1 | File "/opt/conda/lib/python3.10/site-packages/airflow/utils/file.py", line 246, in _find_path_from_directory
airflow-airflow-1 | raise RuntimeError(
airflow-airflow-1 | RuntimeError: Detected recursive loop when walking DAG directory /etl: /etl/logs/scheduler/2023-05-15 has appeared more than once.
airflow-airflow-1 | [2023-05-15T03:52:21.951+0000] {manager.py:290} WARNING - DagFileProcessorManager (PID=82) exited with exit code 1 - re-launching