Comments (20)
Hi, since there haven’t been any recent updates for the Teradata Provider beyond version 2.5.0, could you explain why a new release version is needed?
According to our rules - periodically we bump all providers min-airflow version. https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow -> then we relaease all providers with min-airflow version bumped - we also remove all pre-min-airflow backports, this allows to keep airflow providers free from back-compatibility issues.
from airflow.
Hi,
#41527, #41262 work as expected
from airflow.
- #41356 works fine.
- #41358 I didn't use
spark
so I can't check, but given that_sql
is simply an alias forsql
, it should be fine. - #41461 I couldn't find a good way to verify
common.sql
, but since we've only removed the part that generates a more detailed error message, it should be fine.
from airflow.
@eladkal
#40008 works fine! (unittest & example run)
from airflow.
@eladkal the ElasticSearchSQLHook
is now working as expected.
from airflow.
Works fine
from airflow.
#41494 tried to fix the OOM error in the scheduler that the OpenLineage can cause when generating a dag_tree from a huge DAG (related issue: #41505). It works but we've just got information about another production case where the scheduler went OOM with another complex DAG. There is a fix prepared in #41587 that will remove the dag_tree entirely so there will no more errors like this. I'd like to request an rc2 for OpenLineage provider (@eladkal) that will include that fix, as it is a bug that can cause some problems in bigger deployments.
from airflow.
Tested #40703 with both access_control
formats and it's working fine:
![image](https://private-user-images.githubusercontent.com/7281460/359229515-7ade62db-13cf-448c-a489-6f8d327af0b7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NTY1NDMsIm5iZiI6MTcyNDQ1NjI0MywicGF0aCI6Ii83MjgxNDYwLzM1OTIyOTUxNS03YWRlNjJkYi0xM2NmLTQ0OGMtYTQ4OS02ZjhkMzI3YWYwYjcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjNUMjMzNzIzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZjliNzA1ZGM4OGU4MmVhYzFjYzczYzUzZDMyMGYyMjllYzZkY2YxZWYxMzU2ZWQ3ZjQ3Mjk3MzQ1NTJmZGUwNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.uMCSgTgYi_ri2PcVtx776_IYdJvHYxS23SLVgl5JFlc)
from airflow.
@kacpermuda I will exclude openlineage from this release
from airflow.
ydb provider works fine: #41303
from airflow.
Hi, #40356 work as expected
from airflow.
Hi,
#41372 address the documentation changes. So, no functionality change.
from airflow.
#41150 - Works as expected - unit test + functional check, details below:
- Connection definition (different role and warehouse than is used on Operator definition)
- DAG definition, mind the different warehouse, role and schema
- Queries executed on a warehouse defined when declaring the SnowflakeSqlApiOperator and not the connection
- DDL executing role is also properly forwarded through the SnowflakeSqlApiOperator
from airflow.
I did not find any regression from 10.3.0 to 10.4.0rc1.
Then it's not blocking the release.
Feel free to raise PR to address the bugs you mentioned
from airflow.
Hi, since there haven’t been any recent updates for the Teradata Provider beyond version 2.5.0, could you explain why a new release version is needed?
from airflow.
Checked that all my changes are in.
from airflow.
41142 passes test, but:
I noticed is that starting with 2.9.3 with microsoft-azure providers 10.3.0 the return value quits being put into XCOM (it is blank in the UI). Then with airflow 2.10.0 with microsoft-azure providers 10.3.0 the key shows up in XCOM but it says "No value found for XCom key". So something broke between 2.9.2 / 10.1.2 and 2.10.0/10.3.0.
With all that said, my actual change to put the last line (or all lines) of the logs into XCOM worked. So, I think we need a new defect logged to see why return value isn't showing up correctly any more.
Here is my operator. I'd been testing my changes with do_xcom_push=False since I didn't care about the normal return_value...obscured the fact something else broke :(
aciOperator = AzureContainerInstancesOperator(
ci_conn_id="azure-container-instance-conn-id",
registry_conn_id="acr-conn-id",
resource_group="redacted",
name="http2blob{{ ds }}",
image='redacted',
region="WestUS2",
environment_variables={
redacted
},
volumes=[],
memory_in_gb=1.0,
cpu=1.0,
task_id="start-download-aci",
retries=0,
do_xcom_push=True,
# xcom_all=True,
post_execute=_post_execute,
)
Also, I did some more testing and multiple_outputs=True also fails back in 2.9.2 with microsoft-azure providers 10.1.2. This actually makes sense because the operator returns a single value, not a dictionary so I think this wasn't actually an issue. So, I'm saying this was tester error unless someone tells me otherwise.
from airflow.
@perry2of5 there is only 1 question relevant here. Is there regression in apache-airflow-providers-microsoft-azure
from 10.4.0rc1
to 10.3.0
? All the rest is possible bugs that do not affect our decision about releasing.
Please clarify explicitly what worked on 10.3.0
and does not work anymore on 10.4.0rc1
from airflow.
I did not find any regression from 10.3.0 to 10.4.0rc1.
from airflow.
Thank you everyone. Providers are released.
Provider openlineage
is excluded and will followup with rc2
I invite everyone to help improve providers for the next release, a list of open issues can be found here.
from airflow.
Related Issues (20)
- Ineffiction dag_run endpoint HOT 1
- Airflow 2 to 3 auto migration rules HOT 4
- Evaluate DB migration restriction when upgrading to AF 3 HOT 2
- Confirmation text "Wait a minute" is confusing HOT 1
- Incorrect link to task detail from task instance list HOT 1
- S3ToGCSOperator broken in deferable mode HOT 7
- Unable to specify env variables using fieldPath HOT 3
- can't scroll task tries
- Success callback not being executed in Custom kubernetesPodOperator HOT 1
- Logs issue HOT 1
- Bad display of xcom values HOT 2
- "cannot pickle '_cffi_backend.FFI' object" error in python/branch operator processing HOT 10
- OpenLineage - dag_run.update_state() - AssertionError: daemonic processes are not allowed to have children
- Simple auth manager HOT 3
- Airflow DAG access control permissions are not working HOT 3
- worker: Warm shutdown (MainProcess) HOT 1
- ONNX Model Inference Operator HOT 1
- GKEStartJobOperator's job_poll_interval parameter is not used by its GKEJobTrigger HOT 1
- kubernetes connection defined via env variable does not work with deferrable=True HOT 2
- post_execute() method does not work correctly when used along with a trigger HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from airflow.