Thanos Store Bucket Operation Latency since v0.33.0 about thanos HOT 3 OPEN

We went from bitnami chart 12.13.0 (thanos 0.32.1) to chart 12.20.1 (thanos 0.33) and see exactly the same behavior and we also use S3 objstore as backend:

version:

  build user:       root@293b7b0c32a8
  build date:       20231219-10:37:39
  go version:       go1.21.5
  platform:         linux/amd64
  tags:             netgo

args:

      --log.level=info
      --log.format=logfmt
      --grpc-address=0.0.0.0:10901
      --http-address=0.0.0.0:10902
      --data-dir=/data
      --objstore.config-file=/conf/objstore.yml

*** 2nd jan is the date of version change, so it really looks like it is something in this version

from thanos.

We also observed this exact issue when we upgraded from 0.32.5 to 0.33.0, and rolled back. I had a look at the changes between versions that could have caused this, and I'm now suspecting this might not be an issue per-se, but actually something that wasn't being correctly measured: thanos-io/objstore#79 instrumented the iter operation, which was previously always at 0. In my environment, this operation took a lot longer than all the other ones, which made the p99 latency of bucket operations increase dramatically (purple line is the p99 of iter, green one is overall p99):

Maybe it's the 2s threshold of the ThanosStoreObjstoreOperationLatencyHigh alert that needs to be tweaked to account for the newly instrumented operation?

from thanos.

We haven't looked enough in it yet, though also questioned ourself if simply the reporting is changed. This feeling is fuelled by the facts that in all the other metrics we have, we dont see any behavioural change in S3.

from thanos.