Comments (12)
Marked them to be included in 2.9.2rc2
from airflow.
Hey @vincbeck @o-nikolas -> looks like we got it now when 2.9.2rc1 is using the old celery provider.
Way to reproduce:
- Run this:
breeze start-airflow --use-airflow-version 2.9.2rc1 --load-example-dags --load-default-connections --executor CeleryExecutor --airflow-extras celery
- Check installed packages:
root@c45830039533:/opt/airflow# pip freeze | grep celery
apache-airflow-providers-celery==3.7.1
celery==5.4.0
root@c45830039533:/opt/airflow# pip freeze | grep apache-airflow
apache-airflow==2.9.2rc1
apache-airflow-providers-celery==3.7.1
apache-airflow-providers-common-io==1.3.2
apache-airflow-providers-common-sql==1.14.0
apache-airflow-providers-fab==1.1.1
apache-airflow-providers-ftp==3.9.1
apache-airflow-providers-http==4.11.1
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-sqlite==3.8.1
root@c45830039533:/opt/airflow#
-
Connect to http://localhost:28080/
-
Run "example_complex" dag
-
You will see
BaseExecutor.change_state() got an unexpected keyword argument 'remove_running' airflow_scheduler
- errors. I am afraid we will need to to handle back-compatibility there.
from airflow.
#40035 (this one is just a unit test, but it links to the other issues and PR that fixes it as wel. That was merged a couple days ago)
from airflow.
@o-nikolas -> but I think the problem is that the new executor interface does not work with old celery provider.
from airflow.
(or the other way round - maybe the celery executor 3.7.1 does not work with 2.9.2 - and likely 2.8, 2.9 in general - not sure as I am on a meeting today).
from airflow.
I'm AFK right now, I can look later, but the traceback seems to be base exec being called with a param it doesn't expect (and the type error is now caught correctly as of a couple days ago)
TypeError: BaseExecutor.change_state() got an unexpected keyword argument 'remove_running'
from airflow.
Ok. I see those were not marked for 2.9.2 so they should be in this case.
from airflow.
I marked it to be in 2.9.2 but it got removed :( But anyway, yes #40012 is the fix to this issue
from airflow.
Actually - it's not a full fix. It will only work with "fufure" celery provider when it is released. The older versions of celery providers will not work with it @vincbeck @o-nikolas. I guess that means that we
- have to wait with 2.9.2 until new celery provider is released
- have to yank celery provider 3.7.1 because #39212 essentially broke the executor for all past versions of Airflow, where #40012 fixes it in the future celery provider
from airflow.
This might actually justify slightly acceleratte celery /k8s provider voting, because we won't be able to effectively test 2.9.1rc1 until celery/k8s providers are released cc: @eladkal .
from airflow.
- Celery provider 3.7.2rc1 will be released before Airflow 2.9.2
- Celery provider 3.7.1 is yanked
I guess we can close this issue?
from airflow.
Yep.
from airflow.
Related Issues (20)
- wait_for_downstream doesn't seem to work as expected HOT 4
- Deprecation of operators should be prominently highlighted instead of the one line remark in the documentation HOT 2
- Abilitiy to set dag priority
- Access control error after upgrading from version 2.9.3 to 2.10.0 HOT 3
- Add Backfill Button to DAGs Overview and Individual DAG Pages in the Webserver UI HOT 1
- `airflow db migrate` throws dataset alias reference columns and Cryptographic Paramiko warnings
- Values in SparkKubernetesOperator YAML needs to be defined HOT 1
- callback without __name__ are not allowed when taskinstance executing HOT 2
- Gets a `DatasetEvent` that was created later than `logical_date`. HOT 1
- Status of testing Providers that were prepared on August 19, 2024 HOT 18
- Cannot execute Dynamic Task Mapping when `default_arg` is present. HOT 2
- Logout link in no roles and permissions error page is broken HOT 1
- Support for workload identity authentication from the Databricks provider HOT 3
- Nested task group names are not generated properly HOT 1
- BashOperator is killed before it can execute HOT 7
- XCom with / in key results in UI error
- Running Breeze static checks (all files) on local machine results in failures HOT 6
- OpenTelemetry metrics are not getting sent HOT 1
- Multiple Airflow in one k8s namespace HOT 2
- end_from_trigger leaves deferred tasks without an end_date and increasing duration
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from airflow.