Comments (6)
My 2 cents, this code use internals methods, as result it is expected that this approach could be broken on changes in airflow codebase
def deco(cls):
orig_init = cls.__init__
def new_init(self, *args, default_args=None, **kwargs):
orig_init(self, *args, **kwargs)
self.default_args = default_args
cls.__init__ = cls._apply_defaults(new_init)
return cls
from airflow.
Thanks for the investigation and feedback, @Taragolis and @dabla!
Yes, everything still works despite the warning message. I created this issue just as a "preventive" measure because @potiuk wrote in the closing comment of #37937:
Having a warning is "good enough" for now. If we decide to do something more (for example allow such use to behave more "as expected") we can always add it later.
What would you recommend if we need to inject some common logic into the constructor and the execute
method of around 15 different (standard) operators? Or shall I patch the TaskInsntance
and BaseOperator
classes to insert the __sentinel
property?
from airflow.
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
from airflow.
cc @dabla WDYT?
from airflow.
WDYT
Seems like the deco method is considered as calling he execute method of the PythonOperator outside of the TaskInstance, this is the protection mechanism which would warn you when mixing task decorated tasks with native operators. But normally it should still work atm, as this only triggers a warning.
Ok think I understand, it's because the class AlloyPythonOperator is instantiated outside the class TaskInstance, due to the deco method, so the sentinel which acts as a guard isn't set and thus Airflow considers that the execute method is called outside the TaskInstance
from airflow.
Thanks for the investigation and feedback, @Taragolis and @dabla!
Yes, everything still works despite the warning message. I created this issue just as a "preventive" measure because @potiuk wrote in the closing comment of #37937:
Having a warning is "good enough" for now. If we decide to do something more (for example allow such use to behave more "as expected") we can always add it later.
What would you recommend if we need to inject some common logic into the constructor and the
execute
method of around 15 different (standard) operators? Or shall I patch theTaskInsntance
andBaseOperator
classes to insert the__sentinel
property?
In your example above, if you only have this "issue" of common logic with the PythonOpertor, then I would suggest you use the @task decorated method, there you could delegate to another non decorated method having all the common logic. Personally, I would not try to "mess" with native operators and start extending them.
from airflow.
Related Issues (20)
- Databricks Provider (Version 6.1.0) seems to bump the airflow version to latest HOT 2
- duplicate key value violates unique constraint serialized_dag_pkey for dynamic dag generation HOT 17
- Unable to Scroll Down to See All DAG Runs in Calendar View HOT 2
- Env_vars from KubernetesPodOperator does not assume secret (deploy as env var) because secrets are loaded after env vars into the pods. HOT 2
- Status of testing of Apache Airflow 2.9.2rc1 HOT 8
- Chart: Allow to extend cleanup cronjob env with custom envs from secrets or config maps HOT 1
- celery_executor got an unexpected keyword argument 'remove_running' HOT 12
- celery_executor.py remove_running got an unexpected keyword HOT 6
- Airflow 2.9.1 : AirflowContextDeprecationWarning: Accessing 'yesterday_ds_nodash' from the template is deprecated and will be removed in a future version. HOT 2
- EmrServerlessStartJobOperator causes dag load failure when using XComArg HOT 5
- Connection edit page says "Changed Row" even if field changes were not saved HOT 2
- Tasks Stuck at Scheduled State HOT 9
- local installation problem HOT 2
- Status of testing Providers that were prepared on June 07, 2024 HOT 14
- Operator `SQLExecuteQueryOperator` not logging `RAISE NOTICE` statements in PostgreSQL functions HOT 10
- EKS Overrides for AWS Batch submit_job HOT 1
- Rendered Template Fields missing for some tasks HOT 2
- dag callbacks not called on SSHOperator (base operator?) HOT 17
- Password autocompletion set to off on login page HOT 3
- Amazon Provider v8.23.0 `importlib_metadata` module import error HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from airflow.