Comments (15)
MichaHoffmann
commented on July 21, 2024
Hey,
can you please share your configuration of the related components? Something is odd ~ thanos sidecars usually dont issue range queries.
from thanos.
AlexDCraig
commented on July 21, 2024
@MichaHoffmann Just want to highlight on this that 0.34 with the same exact config doesn't have this problem. Here's what I'm supplying to the various components.
Query:
- args:
- query
- --log.level=info
- --log.format=logfmt
- --grpc-address=0.0.0.0:10901
- --http-address=0.0.0.0:10902
- --query.replica-label=prometheus_replica
- --endpoint=dnssrv+_grpc._tcp.thanos-bitnami-storegateway-headless.thanos-bitnami.svc.cluster.local
- --endpoint=t-1-0.thanos.mydomain.com:443
- --endpoint=t-1-1.thanos.mydomain.com:443
- --endpoint=s2-0.thanos.mydomain.com:443
- --endpoint=s2-1.thanos.mydomain.com:443
- --endpoint=p2-0.thanos.mydomain.com:443
- --endpoint=p2-1.thanos.mydomain.com:443
- --endpoint=ci-0.thanos.mydomain.com:443
- --endpoint=d3-0.thanos.mydomain.com:443
- --endpoint=d3-1.thanos.mydomain.com:443
- --endpoint=pt-0.thanos.mydomain.com:443
- --endpoint=pt-1.thanos.mydomain.com:443
- --endpoint=ps-0.thanos.mydomain.com:443
- --endpoint=ps-1.thanos.mydomain.com:443
- --endpoint=ss-0.thanos.mydomain.com:443
- --endpoint=ss-1.thanos.mydomain.com:443
- --endpoint=i0.thanos.mydomain.com:443
- --endpoint=i1.thanos.mydomain.com:443
- --endpoint=cu-0.thanos.mydomain.com:443
- --endpoint=cu-1.thanos.mydomain.com:443
- --endpoint=ee-0.thanos.mydomain.com:443
- --endpoint=ee-1.thanos.mydomain.com:443
- --endpoint=l2-0.thanos.mydomain.com:443
- --endpoint=l2-1.thanos.mydomain.com:443
- --endpoint=dnssrv+_grpc._tcp.thanos-receiver-headless.thanos-receiver.svc.cluster.local
- --alert.query-url=https://thanos-query-frontend-bitnami.mydomain.com
- --query.auto-downsampling
- --grpc-client-tls-secure
- --grpc-client-tls-skip-verify
- --grpc-client-tls-cert=/etc/certs/client.crt
- --grpc-client-tls-key=/etc/certs/client.key
- --grpc-client-tls-ca=/etc/certs/ca.crt
Query Frontend:
- args:
- query-frontend
- --log.level=info
- --log.format=logfmt
- --http-address=0.0.0.0:9090
- --query-frontend.downstream-url=http://thanos-bitnami-query:9090
- --query-range.split-interval=12h
- --query-frontend.compress-responses
- |
--query-range.response-cache-config=
type: IN-MEMORY
config:
max_size: 2GB
Store:
- args:
- store
- --log.level=info
- --log.format=logfmt
- --grpc-address=0.0.0.0:10901
- --http-address=0.0.0.0:10902
- --data-dir=/data
- --objstore.config-file=/conf/objstore.yml
- --sync-block-duration=3m
- --grpc-server-tls-cert=/etc/certs/server.crt
- --grpc-server-tls-key=/etc/certs/server.key
- --grpc-server-tls-client-ca=/etc/certs/ca.crt
Let me know if this sufficient, or there's more config you'd like to see. Thanks!
from thanos.
MichaHoffmann
commented on July 21, 2024
Can you also please share the configuration of the sidecar that is logging the error?
from thanos.
AlexDCraig
commented on July 21, 2024
Sidecar:
- args:
- sidecar
- --prometheus.url=http://127.0.0.1:9090/
- '--prometheus.http-client={"tls_config": {"insecure_skip_verify":true}}'
- --grpc-address=:10901
- --http-address=:10902
- --objstore.config=$(OBJSTORE_CONFIG)
- --tsdb.path=/prometheus
- --log.level=debug
- --log.format=logfmt
from thanos.
MichaHoffmann
commented on July 21, 2024
The thing that is really weird to me is that the only thing that really runs that code (QueryRange from promclient.go) is the thanos ruler but the log statement you passed indicates that its from a container named "thanos-sidecar". Do you by chance run a ruler too?
from thanos.
MichaHoffmann
commented on July 21, 2024
Could it be that some sidecars are on a version pre 0.34.0 ? And use the queryPushdown feature? We removed all raw promql queries from sidecars in f29b338
from thanos.
AlexDCraig
commented on July 21, 2024
@MichaHoffmann No, all Thanos sidecars are version 0.34.1:
- --thanos-default-base-image=quay.io/thanos/thanos:v0.34.1
Also, we don't use Thanos Ruler. or at least, we don't have a Thanos Ruler deployment running, or intend to. The Thanos sidecars on the remote clusters are configured in the Prometheus Operator.
from thanos.
MichaHoffmann
commented on July 21, 2024
Sorry, it changes nothing but just to correct myself: that change was released in 0.34.1. The only way I could understand this is if you would run a sidecar with version before 0.34.1. Something is pretty weird here; can you spot check the Thanos version of that sidecar that logs that line maybe just to be extra sure?
from thanos.
AlexDCraig
commented on July 21, 2024
k get pod prometheus-k-prom-prometheus-operator-prometheus-0 -n monitoring -o yaml
apiVersion: v1
kind: Pod
metadata:
name: prometheus-k-prom-prometheus-operator-prometheus-0
namespace: monitoring
spec:
containers:
...
- args:
- sidecar
- --prometheus.url=http://127.0.0.1:9090/
- '--prometheus.http-client={"tls_config": {"insecure_skip_verify":true}}'
- --grpc-address=:10901
- --http-address=:10902
- --objstore.config=$(OBJSTORE_CONFIG)
- --tsdb.path=/prometheus
- --log.level=debug
- --log.format=logfmt
image: quay.io/thanos/thanos:v0.34.1
imagePullPolicy: IfNotPresent
name: thanos-sidecar
from thanos.
MichaHoffmann
commented on July 21, 2024
I mean can you run something like "Thanos --version" inside the container?
from thanos.
AlexDCraig
commented on July 21, 2024
Certainly:
k exec -it prometheus-k-prom-prometheus-operator-prometheus-0 -c thanos-sidecar -n monitoring -- /bin/sh
~ $ thanos --version
thanos, version 0.34.1 (branch: HEAD, revision: 4cf1559998bf6d8db3f9ca0fde2a00d217d4e23e)
build user: root@61db75277a55
build date: 20240219-17:13:48
go version: go1.21.7
platform: linux/amd64
tags: netgo
from thanos.
pvlltvk
commented on July 21, 2024
Hi guys!
I can confirm that we have the same issue in our environment. We use 0.34.0 for a sidecar and after upgrading Thanos Query to 0.35.0 min/max operators don't work in the same way as @AlexDCraig described
from thanos.
MichaHoffmann
commented on July 21, 2024
@pvlltvk does it work again if you upgrade sidecars?
from thanos.
pvlltvk
commented on July 21, 2024
@MichaHoffmann Yes, I can confirm that with after sidecar upgrade to 0.35.0 it works again
from thanos.
MichaHoffmann
commented on July 21, 2024
@MichaHoffmann Yes, I can confirm that with after sidecar upgrade to 0.35.0 it works again
Awesome, thanks for confirming
from thanos.
Related Issues (20)
- Read value of remote_user in Slow Query Logs of Query Frontend from a HTTP header HOT 3
- Thanos Receive doesn't announce external_labels which are set in hashrings.json when it works in routing and ingesting mode. HOT 1
- Issue with deduplication alogrithm in Thanos HOT 4
- Query Stats Returned with query including query bytes fetched HOT 5
- `ThanosSidecarBucketOperationsFailed` alert is flaky
- PR Title Validation
- Thanos Receive Pod is crashing with Readiness and livness Probe Failed
- Thanos ruler vs. eventual consistency of metrics
- Can Huawei's OBS storage be supported? HOT 2
- Thanos React-app : Proxy server for thanos-query
- Query: update of endpoint failed...context deadline exceeded
- Thanos Chart 0.34.0 app version 12.23.1
- Thanos receive fails "no space left on device"
- sidecar: Greatly increased Thanos sidecar memory usage from 0.32.2 to 0.32.3, still exists in 0.35.0 HOT 3
- api/v1/label returns wrong values HOT 3
- Regression in thanos v0.35.1 HOT 3
- Thanos Receiver: Router/Ingestor setup no longer returns `thanos_receive_write_timeseries_*` and `thanos_receive_write_samples_*` metrics with thanos v0.35.1 HOT 3
- Extend Thanos bucket rewrite to support filtered archiving of existing blocks
- Support additional aggregates for downsampling
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.