Airflow timeout error in chicago_taxi_pipeline example about tfx HOT 4 CLOSED

Seems like the default airflow timeout threshold was set too low: might be a good idea to add to the README that dagbag_import_timeout should be bumped to something higher e.g. 100 in airflow.cfg.

from tfx.

Just curious, does it happen everytime without dagbag_import_timeout change?

from tfx.

It did for me: the default configuration spawns 4 webserver workers and it seems that all of them each individually attempt to load the dagbag. At least one of them would experience a timeout error upon startup.

I also don't have the beefiest computer (Macbook pro 2015, dual core w/ hyperthreading).

from tfx.

Hi, Richard, I noticed that unit of dagbag_import_timeout is second, so 30s should be enough, I also find they have a discuss related to decrease it further more, they mentioned some case that it might take a long time importing, e.g., scheduler machine is overloaded, it might be the reason

from tfx.