KEDA v2 Supports StatefulSet about airflow-chart HOT 5 CLOSED

Hey @jacobhjkim, just curious, what would be your use-case for stateful KEDA workers?

We also just merged a slightly more open-source friendly version of this chart into the official Apache Airflow repository, so a PR over on the main Airflow repo would be great! It lives under the chart subdirectory.

from airflow-chart.

I currently have an issue with webserver not being able to fetch logs from each worker. I thought that this issue would be solved by using StatefulSet and assigning a fixed DNS name to each individual worker.

It's great to hear that this chart repo has been merged into the official Apache Airflow repo!

from airflow-chart.

Oh, that's a good point. Even with a StatefulSet, if you attempt to view the logs from a worker that has been scaled away, you'd get he same error.

I think the only way to get around this would be to use a centralized logging solution, like one of the object store backends (s3, gcs, wasb) or elastic.

from airflow-chart.

@jacobhjkim That's an interesting point re: logs. How would this work if a worker is scaled down? Would a worker need to spin up again to use that volume to pull the logs?=

from airflow-chart.

Right, using StatefulSet won't solve the issue mentioned above. Will close this issue. (Sorry for the late reply 😅)

from airflow-chart.