Distributed mode HowTo? about jmeter-prometheus-plugin HOT 11 CLOSED

Yes it would see likely, on the prometheus side I'm fairly sure it's trivial to just use, as you say, <kubernetes_sd_config> with parameter role=pod.

On the Jmeter minions pods side, you'll have to open multiple ports. A few for communication with the master and 9270 for this plugin. You'll probably need to specify this port (9270) in the prometheus related annotations.

I honestly don't have a recipe for that deployment yml, else I'd give it to you. If anyone else does, I'm happy to add it somewhere in the documentation of this repo.

from jmeter-prometheus-plugin.

I quickly found this rabbit mq example for the annotations you need to specify on the Jmeter deployment side.

from jmeter-prometheus-plugin.

Thanks, i've configured prometheus to automatically discover jmeter-slave endpoints:

  - job_name: jmeter-slaves
    scheme: http
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        separator: ;
        regex: jmeter-slaves-svc
        replacement: $1
        action: keep
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        separator: ;
        regex: prom
        replacement: $1
        action: keep
      - source_labels: [__meta_kubernetes_namespace]
        separator: ;
        regex: (.*)
        target_label: namespace

from jmeter-prometheus-plugin.

I believe because it's a role: endpoint you're only going to scrape 1/N instances. That is, if you have 3 instances deployed, Prometheus is going to get loadbalanced across those 3. So you won't be picking up correct totals, etc. I think you where right the first time suggesting role: pods.

from jmeter-prometheus-plugin.

In my case metrics are collected from all jmeter slaves. I've created jmeter-slave service with exposed 9270 port named "prom" and when i scale up jmeter-slave replicas i see every endpoints ( every slave address) at this service.
For example, for 5 jmeter-slave replicas i have:

kubectl describe endpoints jmeter-slaves-svc
Name:         jmeter-slaves-svc
Namespace:    restapimock
Labels:       jmeter_mode=slave
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2019-09-05T15:00:32Z
Subsets:
  Addresses:          10.244.0.13,10.244.0.14,10.244.1.219,10.244.1.221,10.244.1.222
  NotReadyAddresses:  <none>
  Ports:
    Name    Port   Protocol
    ----    ----   --------
    prom    9270   TCP

from jmeter-prometheus-plugin.

Oh! OK I guess it's been a minute since I've been out of the k8s game. I misunderstood, I thought endpoints was a load balancing type resource.

OK, thanks for the info!

from jmeter-prometheus-plugin.

Hello All,

I have gotten all of this to work properly. Thanks for the comments.
One issue i experience is that things are running only once.

I setup the jmeter distributed, and prometheus and my pods (which I'm going to test some load against).
Prometheus shows the endpoints as DOWN, before i run any tests.
Prometheus shows the endpoints as UP, while the jmeter tests are running
When the jmeter tests finish, Prometheus shows the endpoints as DOWN.
Up to this point everything is fine.

If i want to do another run of the distributed jmeter, the endpoints never become UP anymore.
even doing a curl localhost:9270 in the pod, seems unreachable.
but running this command kubectl describe endpoints jmeter-slaves-svc
shows that the port 9270 is properly listed.

Any ideas?
The workaround is to redeploy over and over my jmeter slaves pods. but that's not ideal.

from jmeter-prometheus-plugin.

That sounds like a bug where the jmeter server instance can boot this plugin once, the first time you run a test, then it won't start this plugin a second time?

Is that a correct understanding of your situation?

from jmeter-prometheus-plugin.

yes. that's correct.

I was wondering if it is a docker image thing, or just as you are saying a jmeter server thing. I ll turn on more logging, to see what i can get.

from jmeter-prometheus-plugin.

Thanks! I've opened #67, let's talk about it there as this may be a bug on my side.

from jmeter-prometheus-plugin.

i am facing same issues as i have one master and 3 slaves and running prometheus on master to collect metrics from slaves and send to new-relic but we are not seeing consolidate data.(as we can see 1/3 data only at at time)

from jmeter-prometheus-plugin.