Comments (2)
@aspain
I had the same issue only happening for xcom task. For me it helped to set PRINT_DELETES = False
in the DAG. The conditional branch for this output here is running entries_to_delete = query.all()
, which is printing multiple rows you've mentioned while consuming a lot of RAM trying to get all records. If you set it to False
, the records will be deleted just fine, but you won't get an output for every row you're about to delete.
from airflow-maintenance-dags.
@gulyash unfortunately I did already have PRINT_DELETES = False set in the DAG. Since we had so many (millions) of rows being deleted, I didn't want to clog up the logs with that many lines. So I still received that error unfortunately. Because of this I've switched to a more straight-forward, mysql-only setup (I have yet to understand sqlalchemy to this extent)
from airflow-maintenance-dags.
Related Issues (20)
- scripts break airflow?
- Naive datetime is disallowed in db cleanup dag
- Another task is already deleting logs on this worker node. HOT 1
- Airflow 2.0.2 renames the last_scheduler_run column (used in db cleanup job) HOT 1
- DagModel cleanup fails due to dependency record in the DagTag table HOT 3
- Airflow 2.2.0 compatibility HOT 13
- [Airflow 2.0] Using log-cleanup in Docker HOT 7
- Folder are not deleted
- Db cleanup failes after upgrade to python3.9 and airflow2.2.2 HOT 9
- cleanup_XCom Task | Task exited with return code -9 HOT 1
- Airflow 2.2.5 cleanup_DagModel issue HOT 2
- Task 'cleanup_DagModel' fails bcs it cannot delete on table 'dag' - violates foreign key constraint HOT 1
- sla_readme_images HOT 1
- Dependency rule tried to blank-out primary key column 'serialized_dag.dag_id' HOT 1
- issue on airflow 2.6.0 - airflow-db-cleanup.py - No module named 'airflow.jobs.base_job' HOT 5
- [docs] add airflow docs by markdown
- Add compress feature to the log-cleanup HOT 1
- clear-missing-dags failuring HOT 7
- cleanup-db problem when querying some airflow db models HOT 1
- Add support for Mysql Backed airflow. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from airflow-maintenance-dags.