Status of testing Providers that were prepared on August 19, 2024 about airflow HOT 20 CLOSED

Hi, since there haven’t been any recent updates for the Teradata Provider beyond version 2.5.0, could you explain why a new release version is needed?

According to our rules - periodically we bump all providers min-airflow version. https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow -> then we relaease all providers with min-airflow version bumped - we also remove all pre-min-airflow backports, this allows to keep airflow providers free from back-compatibility issues.

from airflow.

Hi,
#41527, #41262 work as expected

from airflow.

1. #41356 works fine.
2. #41358 I didn't use spark so I can't check, but given that _sql is simply an alias for sql, it should be fine.
3. #41461 I couldn't find a good way to verify common.sql, but since we've only removed the part that generates a more detailed error message, it should be fine.

from airflow.

@eladkal
#40008 works fine! (unittest & example run)

from airflow.

@eladkal the ElasticSearchSQLHook is now working as expected.

from airflow.

#41256: @vikramaditya91

Works fine

from airflow.

#41494 tried to fix the OOM error in the scheduler that the OpenLineage can cause when generating a dag_tree from a huge DAG (related issue: #41505). It works but we've just got information about another production case where the scheduler went OOM with another complex DAG. There is a fix prepared in #41587 that will remove the dag_tree entirely so there will no more errors like this. I'd like to request an rc2 for OpenLineage provider (@eladkal) that will include that fix, as it is a bug that can cause some problems in bigger deployments.

from airflow.

Tested #40703 with both access_control formats and it's working fine:

from airflow.

@kacpermuda I will exclude openlineage from this release

from airflow.

ydb provider works fine: #41303

from airflow.

Hi, #40356 work as expected

from airflow.

Hi,
#41372 address the documentation changes. So, no functionality change.

from airflow.

#41150 - Works as expected - unit test + functional check, details below:

1. Connection definition (different role and warehouse than is used on Operator definition)
   
2. DAG definition, mind the different warehouse, role and schema
   
3. Queries executed on a warehouse defined when declaring the SnowflakeSqlApiOperator and not the connection
   
4. DDL executing role is also properly forwarded through the SnowflakeSqlApiOperator
   

from airflow.

I did not find any regression from 10.3.0 to 10.4.0rc1.

Then it's not blocking the release.
Feel free to raise PR to address the bugs you mentioned

from airflow.

Hi, since there haven’t been any recent updates for the Teradata Provider beyond version 2.5.0, could you explain why a new release version is needed?

https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpypi.org%2Fproject%2Fapache-airflow-providers-teradata%2F2.6.0rc1%2F&data=05%7C02%7CSATISH.CHINTHANIPPU%40teradata.com%7Cc798de64676b411f9b7a08dcc0279df7%7C9151cbaafc6b4f4889bb8c4a71982138%7C0%7C0%7C638596523614516170%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=uL1wmpzD4EDpAJZPPvDPVbqC%2BbuDSIhwniZPqKyMQ3M%3D&reserved=0

from airflow.

Checked that all my changes are in.

from airflow.

41142 passes test, but:

I noticed is that starting with 2.9.3 with microsoft-azure providers 10.3.0 the return value quits being put into XCOM (it is blank in the UI). Then with airflow 2.10.0 with microsoft-azure providers 10.3.0 the key shows up in XCOM but it says "No value found for XCom key". So something broke between 2.9.2 / 10.1.2 and 2.10.0/10.3.0.

With all that said, my actual change to put the last line (or all lines) of the logs into XCOM worked. So, I think we need a new defect logged to see why return value isn't showing up correctly any more.

Here is my operator. I'd been testing my changes with do_xcom_push=False since I didn't care about the normal return_value...obscured the fact something else broke :(

  aciOperator = AzureContainerInstancesOperator(
      ci_conn_id="azure-container-instance-conn-id",
      registry_conn_id="acr-conn-id",
      resource_group="redacted",
      name="http2blob{{ ds }}",
      image='redacted',
      region="WestUS2",
      environment_variables={
      redacted
      },
      volumes=[],
      memory_in_gb=1.0,
      cpu=1.0,
      task_id="start-download-aci",
      retries=0,
      do_xcom_push=True,
      # xcom_all=True,
      post_execute=_post_execute,
  )

Also, I did some more testing and multiple_outputs=True also fails back in 2.9.2 with microsoft-azure providers 10.1.2. This actually makes sense because the operator returns a single value, not a dictionary so I think this wasn't actually an issue. So, I'm saying this was tester error unless someone tells me otherwise.

from airflow.

@perry2of5 there is only 1 question relevant here. Is there regression in apache-airflow-providers-microsoft-azure from 10.4.0rc1 to 10.3.0 ? All the rest is possible bugs that do not affect our decision about releasing.

Please clarify explicitly what worked on 10.3.0 and does not work anymore on 10.4.0rc1

from airflow.

I did not find any regression from 10.3.0 to 10.4.0rc1.

from airflow.

Thank you everyone. Providers are released.
Provider openlineage is excluded and will followup with rc2

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

from airflow.