Comments (2)
hasher1705
commented on May 23, 2024
You can manually set up a cron job to delete log files in the master node. We have got a similar setup as yours.
find /home/ubuntu/airflow/logs/*/* -type f -mtime +7 -exec rm -rf {} \;
We run this once every day.
from airflow-maintenance-dags.
prakshalj0512
commented on May 23, 2024
@chetkhatri the DAG task deletes logs on the node on which it is running so that's the reason why only logs on worker nodes are being removed.
You can try the new airflow-log-cleanup-pwdless-ssh.py DAG and specify the hosts to run the process on Line 41.
Follow the readme for this DAG and make sure you're able to passwordless SSH onto the nodes from the worker nodes.
from airflow-maintenance-dags.
Related Issues (20)
- Naive datetime is disallowed in db cleanup dag
- Another task is already deleting logs on this worker node. HOT 1
- Airflow 2.0.2 renames the last_scheduler_run column (used in db cleanup job) HOT 1
- DagModel cleanup fails due to dependency record in the DagTag table HOT 3
- Airflow 2.2.0 compatibility HOT 13
- [Airflow 2.0] Using log-cleanup in Docker HOT 7
- Folder are not deleted
- Db cleanup failes after upgrade to python3.9 and airflow2.2.2 HOT 9
- cleanup_XCom Task | Task exited with return code -9 HOT 1
- Airflow 2.2.5 cleanup_DagModel issue HOT 2
- Task 'cleanup_DagModel' fails bcs it cannot delete on table 'dag' - violates foreign key constraint HOT 1
- sla_readme_images HOT 1
- Dependency rule tried to blank-out primary key column 'serialized_dag.dag_id' HOT 1
- issue on airflow 2.6.0 - airflow-db-cleanup.py - No module named 'airflow.jobs.base_job' HOT 5
- [docs] add airflow docs by markdown
- Add compress feature to the log-cleanup HOT 1
- clear-missing-dags failuring HOT 7
- cleanup-db problem when querying some airflow db models HOT 1
- Add support for Mysql Backed airflow. HOT 1
- airflow-log-cleanup Can not find log file correctly HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from airflow-maintenance-dags.