OpenLineage Provider issue: scheduler shuts down after pickle of OpenLineageListener fails about airflow HOT 5 CLOSED

In general airflow standalone doesn't really work with OpenLineage, even with earlier versions (at least that's what I recall).
More importantly, this is not reproducible in any environment other than airflow standalone - checked with breeze & set of executors, Google Composer and Astro Cloud.

from airflow.

@JDarDagran I understand there may be limitations with running OL in Airflow standalone, but we should do better than shutting down the scheduler (if not in 2.9, in future versions). Would it be a possibility to capture these errors and raise warnings?

from airflow.

Totally agree with @tatiana maybe we should raise warning instead of scheduler getting crashed

from airflow.

Agree. "Can't pickle" error in standalone does not say much :)

from airflow.

Instead of a warning, maybe we can try avoiding the pickling error completely. For now the initializer is simply a single call to another function, so we can pass it directly and avoid using local functions #40353. Does it make sense?

from airflow.