Multiple prometheus instances about prometheus-adapter HOT 14 CLOSED

TL;DR: there's no support right now.

So, this is a bit tougher than it looks. From what I've seen, there are two cases here: the case where you know that each Prometheus instance has unique metric families, and the case where instances can have overlapping metrics series. Neither case is currently covered, but the former case is fairly trivial to implement, due to our discovery mechanisms (just remember where each series lives during discovery)

The latter case is tougher for us to implement, but should still be possible -- we just have to remember which Prometheus is associated with each namespace. However, the design there has to be considered carefully -- what if you have some system-collected metrics that you also want to expose as custom metrics, but those flow through a central Prometheus?

I'll probably try to tackle this problem next after I get the advanced configuration work finished, because it's a commonly requested feature.

from prometheus-adapter.

I have just started doing research on this as well. Each team has their own Prometheus for their application they want to autoscale off of. Do I need to deploy multiple adapters or is this something we just cannot support right now?

from prometheus-adapter.

right now it's unsupported -- you can only have one Prometheus instance that's used for scaling.

from prometheus-adapter.

Is this likely to get implemented any time soon?
If not, would you be willing to write up a design to guide someone else in contributing the change?

Thanks

from prometheus-adapter.

@DirectXMan12 why can't we just add an attribute called prometheusUrl to the metric definition to start with. This is blocker for lot of people and this issue has been open since 2018

from prometheus-adapter.

@smitthakkar96 would you be interested in contributing to this feature?

On a side note, running multiple Prometheus instances is already possible by using Thanos querier as explained here: #267

from prometheus-adapter.

from prometheus-adapter.

Awesome! Would you mind opening a feature request to give more details about how you want to solve this and answer the few concerns that were raised earlier in this issue?

from prometheus-adapter.

I think we should stick to what we proposed in #267. Thanos Querier is a stateless component which is easy to deploy and implements many edge-cases and concerns that can become quite complex and intricate to be implemented here.

For the sake of simplicity I would not implement this here.

@smitthakkar96 Is there any use case that is not covered by Thanos Querier?

from prometheus-adapter.

to achieve this you need to use partial response strategy for which you need to specify partial_response in the query args which I guess is not possible to do with prometheus-adapter. https://github.com/thanos-io/thanos/blob/master/docs/components/query.md#partial-response

There is no clear way to tell querier that for this metric look at this prometheus. People usually use thanos querier incase where they have 2 prometheus running in HA scraping same metrics. Another usecase is where two prometheus are scraping same metrics but different sources for eg two different namespaces, two different prometheus. Querier doesn't support metric based routing

from prometheus-adapter.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

from prometheus-adapter.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

from prometheus-adapter.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

from prometheus-adapter.

@fejta-bot: Closing this issue.

from prometheus-adapter.