abhioncbr / docker-airflow Goto Github PK

enhance the repository for docker-compose functionality

Describe the bug
Readme.md file is the old documentation. Current, image usage is quite different from the old one.

Describe the bug
Currently, for different airflow version docker container, same config file template exists. Because of one config template, multiple warnings messages in the latest version of airflow.
Better management of airflow config is required.

The inclusion of the Prometheus plugin in the airflow docker container.

RBAC based UI & authorization is introduced in airflow 1.10 in beta mode. Include support for that in the Docker container.

Currently, airflow container only uses MySQL as database & Celery as a scheduler. For exploration, purpose container should have support for Sequential Scheduler & Sqllite Database.

In-case of RBAC based UI default 'airflow' needs to be present for airflow web-portal usage. Although, the script is present but seems still it is not working.

Recently most of the changes are done in the airflow1.10, for eg. instead of multiple 'entrypoint.sh' one file. Same changes need to be done in airflow1.9.0 branch too.

**Is your feature request related to a problem? Please describe.
Apart from running the Docker container through 'docker-compose' & 'docker run' container, many users start the container through some UI tool like Kitematic. Add the default configurations in the DockerFile so that container could start from Kitematic.

In airflow docker container for version 1.1.0, user creation script i.e user_add.py is failing and hence no user is created by default.

Reason - Because of RBAC, multiple new tables are added to airflow databases and hence user_add.py is failing.

In 'standalone' mode dag task execution logs are not visible/accessible through airflow UI. Although tasks log files are present in the log folder.

Celery Flower UI is good for monitoring tasks and getting stats about how many tasks are triggered, processed, failed & succeeded. In airflow-docker UI failed & succeeded are not getting updated and always showing 0.

Is your feature request related to a problem? Please describe.
For enabling the managed deployment of docker-airflow, include Terraform scripts for infrastructure creation and Ansible scripts for environment setup.

Describe the solution you'd like
Terraform scripts for infrastructure creation and Ansible scripts for environment setup.

Setup the detail Github-Pages based project website.

Is your feature request related to a problem? Please describe.
Currently, airflow processes logs are getting accumulated in one file which results into restarting of the container for cleaning up of space occupied by large files.

Describe the solution you'd like
Process logs should be rotated into multiple files on a daily so that that old logs deletion would be easy and quick.

Additional context
Log rotation along with archival of old logs would be great.

Currently, there are 4 different 'entrypoints.sh' files for bootstrapping the airflow-container. Is should be merged into one file with all the options available.