`__extra__` being passed to SQLalchemy engine url erroneously about airflow HOT 8 CLOSED

Hmm, thanks. I'll try reproducing it later today or tomorrow. Will assign it to myself for now.

from airflow.

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

from airflow.

@MialLewis could you share a complete example, ie a complete dag that could be used to replicate the behaviour and any other configs/etc?

from airflow.

Here is a minimal dag that reproduces the issue:

from airflow import DAG
from airflow.utils.dates import days_ago

default_args = {
    'owner': 'airflow',
}

with DAG(
    dag_id='issue_dag',
    default_args=default_args,
    start_date=days_ago(2),
    tags=['issue'],
) as dag:

    @dag.task
    def error_fn():
        import pandas as pd
        output_df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
        engine = _get_engine_from_conn(conn_id='MariaDB')
        output_df.to_sql(name='test_table', con=engine, schema='temp', if_exists='replace')

    def _get_engine_from_conn(conn_id):
        from airflow.providers.mysql.hooks.mysql import MySqlHook as hook
        db_hook = hook(mysql_conn_id=conn_id)
        engine = db_hook.get_sqlalchemy_engine()
        return engine

error_task = error_fn()

The connection I am using to a simple MariaDB of the latest version:

from airflow.

I tried reproducing it, I see that the issue is because of the values in extra, ie the {}. Removing them removes the extra field from the engine and doesn't cause the issue. Though the existence of the extra field is odd to be because to my knowledge mysql doesn't accept it... 🤔 @potiuk would you know if that should probably not exist in the mysql connection type? Or maybe it should be encoded some other way?

Also @MialLewis to fix your issue you can just remove the {} from extras and you should be good.

from airflow.

I can’t remove the ‘{}’, it gets put there by default when the extra field is left empty.

I mentioned this in the issue post:

“Looking at previous versions of airflow, the default value in the extras field in a connection appears to have gone from None to {}.”

Thanks for taking the time to look into this.

from airflow.

from airflow.

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

from airflow.