Difference between logs which stores in files and logs in UI about airflow HOT 5 OPEN

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

from airflow.

This happen due to log deduplication, which might happen when logs streaming from remote logging

from airflow.

As I understood, the main problem is in log.splitlines(), which split log-string by simple lines and not by log-messages. Then function analyzes line by line and deduplicates lines, but we need to analyze and deduplicate log-messages.
As Airflow can be configured with a custom log-format, then we need to store the pattern in config (custom patterns) to split the whole log into log-messages.

from airflow.

If you have a suggestion how improve logging feel free to raise a PR which will work with any type of existed loggers without breaking changes.

from airflow.

I had the same multiple times, for example using DockerOperator which logs all stdout of the container upon failure. Also the logs are messed up not only because of the split lines but also because the file log handler per default tries to sort messages. This not only causes a lot of overhead on the server, it also changes the order any makes a confusion.

Looking forward that somebody raises a PR allowing log sorting and merging to be turned off :-)

from airflow.