Compactor: Deadlock on S3 error during meta sync about thanos HOT 3 OPEN

When using select we should add a default condition or timeout in the part of the code below
select { case <-ctx.Done(): return ctx.Err() case metaChan <- id: }

from thanos.

@palamvmw In general that would help getting unstuck at the very least.

I am trying to wrap my head around what happened in the first place, and I believe this is caused by a large number of errors "clogging up" the work queue. Essentially any Exists call that throws an Error takes out one of the worker threads with it, and there is no check for all of them going away, so when we reach the 64th Exists Error all things just stop. No more workers picking up items, and no more items pushed to the channel. Since we are not on a timed Context, we also never get ctx.Done().

I would think that if we error out a worker, we should start a new one in its place. (Or just not stop it in the first place, and send errors to a different channel to be collected.)

I am also not quite sure that this line is correct:
https://github.com/thanos-io/thanos/blame/086a698b2195adb6f3463ebbd032e780f39d2050/pkg/block/fetcher.go#L252
An Exists call can be failing for a bunch of reasons some of which could be retried successfully.

We have had S3 503 Slow Down responses previously, so it is not at all unlikely that the reason we clogged the 64 workers is because all of a sudden all of them started getting 503 Slow Downs from AWS.

from thanos.

I think I am facing the same issue but on Azure Object store so I tend to agree with the comment on your draft PR.
we are not using any minio lib for azure :/

from thanos.