Provide a mean to reproduce benchmarks results about cassandra-exporter HOT 5 CLOSED

I'll look into adding some more official benchmarking tooling/guidelines.

But for now I'll outline the general approach below.

I started with a CCM 3-node 3.11.3 cluster.

I then created a keyspace with 1000 tables, using Java code similar to the following:

import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Session;

public class LotsOTables {
    public static void main(String[] args) {
        final Cluster cluster = Cluster.builder().addContactPoint("127.0.0.1").build();

        final Session session = cluster.connect();

        session.execute("CREATE KEYSPACE lotsotables WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 3}");

        for (int i = 0; i < 1000; i++) {
            session.execute(String.format("CREATE TABLE lotsotables.table%d (name text PRIMARY KEY);", i));
        }
    }
}

To benchmark your exporter I downloaded the latest release and used the example config.yml file from the repository. I started it like so:

$ java -jar cassandra_exporter-2.0.2-all.jar config.yml

I then let it run for a period of time and observed its log output, which gives timing information. Since your exporter collects in a background thread and caches the results, I had to use the timing information from the logs rather than timing curl as I did for the other benchmarks.

[main] INFO com.criteo.nosql.cassandra.exporter.Config - Loading yaml config from config.yml
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 27821ms for the whole run
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 25786ms for the whole run
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 25243ms for the whole run
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 27656ms for the whole run
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 25332ms for the whole run
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 26222ms for the whole run
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 26284ms for the whole run
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 26075ms for the whole run
[main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 25511ms for the whole run

Note that all the benchmarks were run locally on my laptop which has a Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz CPU. And while I created a 3-node CCM cluster, I only collected metrics from one node.

from cassandra-exporter.

Hi,
Thanks for the quick reply :)
I will look into that later in the week and will keep you posted.

As a side note, while performance is nice, you should warn/advertise the downside of running the exporter as an agent, most notably:

• You have to restart your cluster if you want to upgrade the exporter plugin (may not be an issue if you are on kubernetes as you have to do it anyway to upgrade your pod)
• You are sharing the same jvm/heap than cassandra itself, so requesting too frequently the endpoint or during a bad time for cassandra may result in triggering GC that will cause to pause

Also if you want to improve further your performance, you can look to add gzip support for you http server, it make a big difference.
Example from the official prometheus java client
https://github.com/prometheus/client_java/blob/master/simpleclient_httpserver/src/main/java/io/prometheus/client/exporter/HTTPServer.java#L71

from cassandra-exporter.

Hi back,

So indeed my exporter was way slower with a lot of tables.
I made some change in the 2.0.3 in order to reduce the gap (you should expect a 2x improvement) but a gap will still remain. The dominant factor compared to other exporters, are my calls to mbean.getAttributes in order for the exporter to be able to not export them.
This is a design choice as letting the exporter filter on attributes let us scale more easily without increasing the size of our Prometheus.

I have updated the readme of the project in order to reflect the design decisions:
https://github.com/criteo/cassandra_exporter/blob/master/README.md#design-explanation

Thanks for having taking time to benchmarks

from cassandra-exporter.

Thanks for the suggestions of notes/warnings about running an exporter as an agent. I'll consider adding them to the README.

Also if you want to improve further your performance, you can look to add gzip support for you http server, it make a big difference.

GZip and Deflate is already supported -- Netty's HttpContentCompressor is part of the pipeline.

For example, the following request returns GZip encoded content which is successfully decompressed with gunzip:

curl -H "Accept-Encoding: gzip" localhost:9500/metrics | gunzip

... are my calls to mbean.getAttributes in order for the exporter to be able to not export them.

Yes, getAttributes is one of the slowest JMX calls in my experience.

from cassandra-exporter.

Ok for gzip, I should have missed it :)
Good luck

from cassandra-exporter.