Code Monkey home page Code Monkey logo

airflow-tutorial's Introduction

Airflow Tutorial

Learning resources for Airflow Tutorial article.

Contents

  • docker-compose.yml — An example of Airflow cluster with Celery executor. Contains:
    • Apache Airflow
    • PostgreSQL (Airflow metadata)
    • Redis (Task broker)
    • Celery workers
    • Flower (Celery monitoring)
  • docker-compose.db.yml — Additional database servers and sample data fill up:
    • SQL Server x3 (source database servers)
    • Vertica (target database)
    • mssql_init (initialize source data)
  • dags/ — Sample DAGs and common libraries.

Start and stop

To spin up Airflow cluster only (without databases), use:

$ docker-compose up --scale worker=3

To run all described images and create sample databases, execute:

$ docker-compose -f docker-compose.yml -f docker-compose.db.yml up --scale worker=3

To break down containers press Ctrl+C or Command+C and the following command:

$ docker-compose down

or

$ docker-compose -f docker-compose.yml -f docker-compose.db.yml down

Usage

Containers exposes a couple of WebUI's:

airflow-tutorial's People

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

valghar anatole72 dennterentyev jeeves1979 mnbulkin solarfrii-limbo drscat devrivne icyaltair rslngtn plakidinsv samsafonov2020 konons redextreme12 marynaprokhorenko

airflow-tutorial's Issues

ERROR when exec command $ docker-compose up --scale worker=3

error log

Stend12@DESKTOP12 MINGW64 /c/airflow/airflow-tutorial (master)
$ docker-compose up --scale worker=3
Found orphan containers (airflow-tutorial_mssql_2_1, airflow-tutorial_dwh_1, airflow-tutorial_mssql_init_1, airflow-tutorial_mssql_0_1, airflow-tutorial_mssql_1_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting airflow-tutorial_airflow-db_1 ... done
Starting airflow-tutorial_broker_1 ... done
Starting airflow-tutorial_airflow_1 ... done
Starting airflow-tutorial_worker_1 ... done
Starting airflow-tutorial_worker_2 ... done
Creating airflow-tutorial_worker_3 ... done
Attaching to airflow-tutorial_broker_1, airflow-tutorial_airflow-db_1, airflow-tutorial_airflow_1, airflow-tutorial_worker_1, airflow-tutorial_worker_2, airflow-tutorial_worker_3
airflow-db_1 |
airflow-db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
airflow-db_1 |
airflow-db_1 | 2020-08-12 10:27:36.640 UTC [1] FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
airflow-db_1 | 2020-08-12 10:27:36.640 UTC [1] HINT: The server must be started by the user that owns the data directory.
broker_1 | 1:C 12 Aug 2020 10:27:31.362 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
broker_1 | 1:C 12 Aug 2020 10:27:31.362 # Redis version=6.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
broker_1 | 1:C 12 Aug 2020 10:27:31.362 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
broker_1 | 1:M 12 Aug 2020 10:27:31.363 * Running mode=standalone, port=6379.
broker_1 | 1:M 12 Aug 2020 10:27:31.363 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
broker_1 | 1:M 12 Aug 2020 10:27:31.363 # Server initialized
broker_1 | 1:M 12 Aug 2020 10:27:31.363 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
broker_1 | 1:M 12 Aug 2020 10:27:31.363 * Loading RDB produced by version 6.0.5
broker_1 | 1:M 12 Aug 2020 10:27:31.363 * RDB age 116 seconds
broker_1 | 1:M 12 Aug 2020 10:27:31.363 * RDB memory usage when created 0.77 Mb
broker_1 | 1:M 12 Aug 2020 10:27:31.363 * DB loaded from disk: 0.000 seconds
broker_1 | 1:M 12 Aug 2020 10:27:31.363 * Ready to accept connections
airflow-tutorial_airflow-db_1 exited with code 1
airflow_1 | Requirement already satisfied: pandas in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 1)) (0.25.3)
airflow_1 | Requirement already satisfied: psycopg2-binary in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 2)) (2.8.4)
airflow_1 | Requirement already satisfied: pymssql in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 3)) (2.1.4)
airflow_1 | Requirement already satisfied: requests_toolbelt in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 4)) (0.9.1)
airflow_1 | Requirement already satisfied: vertica_python in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 5)) (0.11.0)
airflow_1 | Requirement already satisfied: python-dateutil>=2.6.1 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2.8.1)
airflow_1 | Requirement already satisfied: pytz>=2017.2 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2019.3)
airflow_1 | Requirement already satisfied: numpy>=1.13.3 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (1.18.2)
airflow_1 | Requirement already satisfied: requests<3.0.0,>=2.0.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests_toolbelt->-r /requirements.txt (line 4)) (2.23.0)
airflow_1 | Requirement already satisfied: six>=1.10.0 in /home/airflow/.local/lib/python3.7/site-packages (from vertica_python->-r /requirements.txt (line 5)) (1.14.0)
airflow_1 | Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (1.25.8)
airflow_1 | Requirement already satisfied: certifi>=2017.4.17 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2019.11.28)
airflow_1 | Requirement already satisfied: chardet<4,>=3.0.2 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (3.0.4)
airflow_1 | Requirement already satisfied: idna<3,>=2.5 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2.9)
worker_1 | Requirement already satisfied: pandas in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 1)) (0.25.3)
worker_1 | Requirement already satisfied: psycopg2-binary in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 2)) (2.8.4)
worker_1 | Requirement already satisfied: pymssql in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 3)) (2.1.4)
worker_1 | Requirement already satisfied: requests_toolbelt in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 4)) (0.9.1)
worker_1 | Requirement already satisfied: vertica_python in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 5)) (0.11.0)
worker_1 | Requirement already satisfied: pytz>=2017.2 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2019.3)
worker_1 | Requirement already satisfied: numpy>=1.13.3 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (1.18.2)
worker_1 | Requirement already satisfied: python-dateutil>=2.6.1 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2.8.1)
worker_1 | Requirement already satisfied: requests<3.0.0,>=2.0.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests_toolbelt->-r /requirements.txt (line 4)) (2.23.0)
worker_1 | Requirement already satisfied: six>=1.10.0 in /home/airflow/.local/lib/python3.7/site-packages (from vertica_python->-r /requirements.txt (line 5)) (1.14.0)
worker_1 | Requirement already satisfied: idna<3,>=2.5 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2.9)
worker_1 | Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (1.25.8)
worker_1 | Requirement already satisfied: certifi>=2017.4.17 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2019.11.28)
worker_1 | Requirement already satisfied: chardet<4,>=3.0.2 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (3.0.4)
worker_2 | Requirement already satisfied: pandas in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 1)) (0.25.3)
worker_2 | Requirement already satisfied: psycopg2-binary in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 2)) (2.8.4)
worker_2 | Requirement already satisfied: pymssql in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 3)) (2.1.4)
worker_2 | Requirement already satisfied: requests_toolbelt in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 4)) (0.9.1)
worker_2 | Requirement already satisfied: vertica_python in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 5)) (0.11.0)
worker_2 | Requirement already satisfied: numpy>=1.13.3 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (1.18.2)
airflow_1 | You are using pip version 19.0.2, however version 20.2.2 is available.
airflow_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
worker_2 | Requirement already satisfied: python-dateutil>=2.6.1 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2.8.1)
worker_2 | Requirement already satisfied: pytz>=2017.2 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2019.3)
worker_2 | Requirement already satisfied: requests<3.0.0,>=2.0.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests_toolbelt->-r /requirements.txt (line 4)) (2.23.0)
worker_2 | Requirement already satisfied: six>=1.10.0 in /home/airflow/.local/lib/python3.7/site-packages (from vertica_python->-r /requirements.txt (line 5)) (1.14.0)
worker_2 | Requirement already satisfied: certifi>=2017.4.17 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2019.11.28)
worker_2 | Requirement already satisfied: idna<3,>=2.5 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2.9)
airflow_1 | DB_BACKEND=postgres+psycopg2
airflow_1 | DB_HOST=airflow-db
airflow_1 | DB_PORT=5432
worker_2 | Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (1.25.8)
worker_2 | Requirement already satisfied: chardet<4,>=3.0.2 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (3.0.4)
worker_1 | You are using pip version 19.0.2, however version 20.2.2 is available.
worker_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
worker_1 | DB_BACKEND=postgres+psycopg2
worker_1 | DB_HOST=airflow-db
worker_1 | DB_PORT=5432
worker_2 | You are using pip version 19.0.2, however version 20.2.2 is available.
worker_2 | You should consider upgrading via the 'pip install --upgrade pip' command.
worker_2 | DB_BACKEND=postgres+psycopg2
worker_2 | DB_HOST=airflow-db
worker_2 | DB_PORT=5432
worker_3 | Requirement already satisfied: pandas in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 1)) (0.25.3)
worker_3 | Requirement already satisfied: psycopg2-binary in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 2)) (2.8.4)
worker_3 | Collecting pymssql (from -r /requirements.txt (line 3))
worker_3 | Downloading https://files.pythonhosted.org/packages/be/09/455835d22f84636dc646f4dc6afcc79c226d8bc253ef767ca8a922eec74a/pymssql-2.1.4-cp37-cp37m-manylinux1_x86_64.whl (1.4MB)
worker_3 | Collecting requests_toolbelt (from -r /requirements.txt (line 4))
worker_3 | Downloading https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl (54kB)
worker_3 | Collecting vertica_python (from -r /requirements.txt (line 5))
worker_3 | Downloading https://files.pythonhosted.org/packages/5e/19/a215a0dbe0c63f9c40fa0dc0bd8b3517b0e1221b050586d6a78a90929b5d/vertica_python-0.11.0-py2.py3-none-any.whl (167kB)
worker_3 | Requirement already satisfied: pytz>=2017.2 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2019.3)
worker_3 | Requirement already satisfied: numpy>=1.13.3 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (1.18.2)
worker_3 | Requirement already satisfied: python-dateutil>=2.6.1 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2.8.1)
worker_3 | Requirement already satisfied: requests<3.0.0,>=2.0.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests_toolbelt->-r /requirements.txt (line 4)) (2.23.0)
worker_3 | Requirement already satisfied: six>=1.10.0 in /home/airflow/.local/lib/python3.7/site-packages (from vertica_python->-r /requirements.txt (line 5)) (1.14.0)
worker_3 | Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (1.25.8)
worker_3 | Requirement already satisfied: certifi>=2017.4.17 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2019.11.28)
worker_3 | Requirement already satisfied: chardet<4,>=3.0.2 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (3.0.4)
worker_3 | Requirement already satisfied: idna<3,>=2.5 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2.9)
worker_3 | Installing collected packages: pymssql, requests-toolbelt, vertica-python
worker_3 | Successfully installed pymssql-2.1.4 requests-toolbelt-0.9.1 vertica-python-0.11.0
worker_3 | You are using pip version 19.0.2, however version 20.2.2 is available.
worker_3 | You should consider upgrading via the 'pip install --upgrade pip' command.
worker_3 | DB_BACKEND=postgres+psycopg2
worker_3 | DB_HOST=airflow-db
worker_3 | DB_PORT=5432
airflow_1 | ..........
airflow_1 | ERROR! Maximum number of retries (0) reached while checking postgres+psycopg2 db. Exiting
airflow_1 |
airflow_1 | /entrypoint: line 84: BACKEND: unbound variable
worker_1 | ........./entrypoint: line 84: BACKEND: unbound variable
worker_1 | .
worker_1 | ERROR! Maximum number of retries (0) reached while checking postgres+psycopg2 db. Exiting
worker_1 |
worker_2 | ..........
worker_2 | ERROR! Maximum number of retries (0) reached while checking postgres+psycopg2 db. Exiting
worker_2 |
worker_2 | /entrypoint: line 84: BACKEND: unbound variable
worker_3 | ........./entrypoint: line 84: BACKEND: unbound variable
worker_3 | .
worker_3 | ERROR! Maximum number of retries (0) reached while checking postgres+psycopg2 db. Exiting
worker_3 |
airflow_1 | Requirement already satisfied: pandas in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 1)) (0.25.3)
airflow_1 | Requirement already satisfied: psycopg2-binary in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 2)) (2.8.4)
airflow_1 | Requirement already satisfied: pymssql in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 3)) (2.1.4)
airflow_1 | Requirement already satisfied: requests_toolbelt in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 4)) (0.9.1)
airflow_1 | Requirement already satisfied: vertica_python in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 5)) (0.11.0)
airflow_1 | Requirement already satisfied: python-dateutil>=2.6.1 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2.8.1)
airflow_1 | Requirement already satisfied: pytz>=2017.2 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2019.3)
airflow_1 | Requirement already satisfied: numpy>=1.13.3 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (1.18.2)
airflow_1 | Requirement already satisfied: requests<3.0.0,>=2.0.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests_toolbelt->-r /requirements.txt (line 4)) (2.23.0)
airflow_1 | Requirement already satisfied: six>=1.10.0 in /home/airflow/.local/lib/python3.7/site-packages (from vertica_python->-r /requirements.txt (line 5)) (1.14.0)
airflow_1 | Requirement already satisfied: chardet<4,>=3.0.2 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (3.0.4)
airflow_1 | Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (1.25.8)
airflow_1 | Requirement already satisfied: certifi>=2017.4.17 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2019.11.28)
airflow_1 | Requirement already satisfied: idna<3,>=2.5 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2.9)
airflow_1 | You are using pip version 19.0.2, however version 20.2.2 is available.
airflow_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
airflow_1 | DB_BACKEND=postgres+psycopg2
airflow_1 | DB_HOST=airflow-db
airflow_1 | DB_PORT=5432
worker_1 | Requirement already satisfied: pandas in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 1)) (0.25.3)
worker_1 | Requirement already satisfied: psycopg2-binary in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 2)) (2.8.4)
worker_1 | Requirement already satisfied: pymssql in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 3)) (2.1.4)
worker_1 | Requirement already satisfied: requests_toolbelt in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 4)) (0.9.1)
worker_1 | Requirement already satisfied: vertica_python in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 5)) (0.11.0)
worker_1 | Requirement already satisfied: pytz>=2017.2 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2019.3)
worker_1 | Requirement already satisfied: python-dateutil>=2.6.1 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2.8.1)
worker_1 | Requirement already satisfied: numpy>=1.13.3 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (1.18.2)
worker_1 | Requirement already satisfied: requests<3.0.0,>=2.0.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests_toolbelt->-r /requirements.txt (line 4)) (2.23.0)
worker_1 | Requirement already satisfied: six>=1.10.0 in /home/airflow/.local/lib/python3.7/site-packages (from vertica_python->-r /requirements.txt (line 5)) (1.14.0)
worker_1 | Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (1.25.8)
worker_1 | Requirement already satisfied: idna<3,>=2.5 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2.9)
worker_1 | Requirement already satisfied: chardet<4,>=3.0.2 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (3.0.4)
worker_1 | Requirement already satisfied: certifi>=2017.4.17 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2019.11.28)
worker_2 | Requirement already satisfied: pandas in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 1)) (0.25.3)
worker_2 | Requirement already satisfied: psycopg2-binary in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 2)) (2.8.4)
worker_2 | Requirement already satisfied: pymssql in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 3)) (2.1.4)
worker_2 | Requirement already satisfied: requests_toolbelt in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 4)) (0.9.1)
worker_2 | Requirement already satisfied: vertica_python in /home/airflow/.local/lib/python3.7/site-packages (from -r /requirements.txt (line 5)) (0.11.0)
worker_2 | Requirement already satisfied: numpy>=1.13.3 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (1.18.2)
worker_2 | Requirement already satisfied: pytz>=2017.2 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2019.3)
worker_2 | Requirement already satisfied: python-dateutil>=2.6.1 in /home/airflow/.local/lib/python3.7/site-packages (from pandas->-r /requirements.txt (line 1)) (2.8.1)
worker_2 | Requirement already satisfied: requests<3.0.0,>=2.0.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests_toolbelt->-r /requirements.txt (line 4)) (2.23.0)
worker_2 | Requirement already satisfied: six>=1.10.0 in /home/airflow/.local/lib/python3.7/site-packages (from vertica_python->-r /requirements.txt (line 5)) (1.14.0)
worker_2 | Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (1.25.8)
worker_2 | Requirement already satisfied: certifi>=2017.4.17 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2019.11.28)
worker_2 | Requirement already satisfied: chardet<4,>=3.0.2 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (3.0.4)
worker_2 | Requirement already satisfied: idna<3,>=2.5 in /home/airflow/.local/lib/python3.7/site-packages (from requests<3.0.0,>=2.0.1->requests_toolbelt->-r /requirements.txt (line 4)) (2.9)
worker_1 | You are using pip version 19.0.2, however version 20.2.2 is available.
worker_1 | You should consider upgrading via the 'pip install --upgrade pip' command.
worker_1 | DB_BACKEND=postgres+psycopg2
worker_1 | DB_HOST=airflow-db
worker_1 | DB_PORT=5432
Stopping airflow-tutorial_worker_3 ... done
Stopping airflow-tutorial_worker_1 ... done
Stopping airflow-tutorial_worker_2 ... done
Stopping airflow-tutorial_airflow_1 ... done
Stopping airflow-tutorial_broker_1 ... done
Gracefully stopping... (press Ctrl+C again to force)

Stend12@DESKTOP12 MINGW64 /c/airflow/airflow-tutorial (master)
$

DAG's error

I created empty tg_main connection, but I get ×Broken DAG: [/dags/dags/update_reports.py] 'bot_token' anyway.
Just clonned your project, didnt change anything. Pls help

[2021-02-17 15:51:47,629] {dagbag.py:239} ERROR - Failed to import: /dags/dags/update_reports.py airflow_1 | Traceback (most recent call last): airflow_1 | File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 236, in process_file airflow_1 | m = imp.load_source(mod_name, filepath) airflow_1 | File "/usr/local/lib/python3.7/imp.py", line 171, in load_source airflow_1 | module = _load(spec) airflow_1 | File "<frozen importlib._bootstrap>", line 696, in _load airflow_1 | File "<frozen importlib._bootstrap>", line 677, in _load_unlocked airflow_1 | File "<frozen importlib._bootstrap_external>", line 728, in exec_module airflow_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed airflow_1 | File "/dags/dags/update_reports.py", line 43, in <module> airflow_1 | trigger_rule=TriggerRule.ONE_FAILED) airflow_1 | File "/dags/commons/operators.py", line 28, in __init__ airflow_1 | self._hook = TelegramBotHook(tg_bot_conn_id) airflow_1 | File "/dags/commons/hooks.py", line 22, in __init__ airflow_1 | self.get_conn() airflow_1 | File "/dags/commons/hooks.py", line 26, in get_conn airflow_1 | self.tg_bot_token = extra['bot_token'] airflow_1 | KeyError: 'bot_token'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.