logzio / elasticsearch-benchmark-tool Goto Github PK

View Code? Open in Web Editor NEW

58.0 72.0 16.0 85 KB

Stress test tool that benchmark indexing and searching in Elasticsearch

License: Apache License 2.0

Shell 1.57% Java 98.13% Dockerfile 0.30%

opensource

elasticsearch-benchmark-tool's Introduction

THIS PROJECT IS NO LONGER MAINTAINED

elasticsearch-benchmark-tool

Stress test tool that benchmark indexing and searching in Elasticsearch

How it is working

The tool is getting a configuration file, which represent the test plan.
Each step can have 1 or more of the following controllers:

Indexing - Index 1k bulks of log like documents (5 different options, with some fields getting different data based on common cardinality factor and always changing timestamp)
Search - Iterate over 5 different Kibana like searches
Optimize - Force merge the index
Noop - Do nothing

The tool produce metrics as JMX counters, and the Jmx2Graphite tool is sending those to your graphite server.

Configuration Example

Can be found under ConfigurationExample.conf

Run

 docker run --rm -it -e GRAPHITE_SERVER="your-graphite-server.com" \
                     -e GRAPHITE_PREFIX="Prefix.under.graphite.root" \
                     -e SERVICE_HOST="BENCHMARK_TEST_NAME" \
                     -v /your/configuration.conf:/config.conf \
                     -v /your/templates:/templates \
                     logzio/elasticsearch-benchmark-tool

Build

mvn clean package docker:build

elasticsearch-benchmark-tool's People

Contributors

Stargazers

Watchers

Forkers

zhuan77241 lawulu oribaldi nobaksan dongbin86 venkateshsampath rajaprabus27 diggleweb stek0v bestjex rachio astridstar ekmixon rajesh16702 791837060 abratanich

elasticsearch-benchmark-tool's Issues

CVE-2019-20330 (High) detected in jackson-databind-2.9.10.1.jar

CVE-2019-20330 - High Severity Vulnerability

Vulnerable Library - jackson-databind-2.9.10.1.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /tmp/ws-scm/elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: epository/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar

Dependency Hierarchy:

❌ jackson-databind-2.9.10.1.jar (Vulnerable Library)

Vulnerability Details

FasterXML jackson-databind 2.x before 2.9.10.2 lacks certain net.sf.ehcache blocking.

Publish Date: 2020-01-05

URL: CVE-2019-20330

CVSS 3 Score Details (9.8)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: High
- Integrity Impact: High
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://github.com/FasterXML/jackson-databind/tree/jackson-databind-2.9.10.2

Release Date: 2020-01-03

Fix Resolution: com.fasterxml.jackson.core:jackson-databind:2.9.10.2

Check this box to open an automated fix PR

CVE-2020-9546 (Medium) detected in jackson-databind-2.9.10.1.jar

CVE-2020-9546 - Medium Severity Vulnerability

Vulnerable Library - jackson-databind-2.9.10.1.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /tmp/ws-scm/elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: epository/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar

Dependency Hierarchy:

❌ jackson-databind-2.9.10.1.jar (Vulnerable Library)

Vulnerability Details

FasterXML jackson-databind 2.x before 2.9.10.4 mishandles the interaction between serialization gadgets and typing, related to org.apache.hadoop.shaded.com.zaxxer.hikari.HikariConfig (aka shaded hikari-config).

Publish Date: 2020-03-02

URL: CVE-2020-9546

CVSS 2 Score Details (5.0)

Base Score Metrics not available

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-9546

Release Date: 2020-03-02

Fix Resolution: com.fasterxml.jackson.core:jackson-databind:2.10.3

Check this box to open an automated fix PR

CVE-2020-8840 (Medium) detected in jackson-databind-2.9.10.1.jar

CVE-2020-8840 - Medium Severity Vulnerability

Vulnerable Library - jackson-databind-2.9.10.1.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /tmp/ws-scm/elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: epository/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar

Dependency Hierarchy:

❌ jackson-databind-2.9.10.1.jar (Vulnerable Library)

Vulnerability Details

FasterXML jackson-databind 2.0.0 through 2.9.10.2 lacks certain xbean-reflect/JNDI blocking, as demonstrated by org.apache.xbean.propertyeditor.JndiConverter.

Publish Date: 2020-02-10

URL: CVE-2020-8840

CVSS 2 Score Details (5.0)

Base Score Metrics not available

Suggested Fix

Type: Upgrade version

Origin: FasterXML/jackson-databind#2620

Release Date: 2020-02-10

Fix Resolution: 2.8.115,2.9.10.3

CVE-2019-7611 (High) detected in elasticsearch-2.4.0.jar

CVE-2019-7611 - High Severity Vulnerability

Vulnerable Library - elasticsearch-2.4.0.jar

Elasticsearch - Open Source, Distributed, RESTful Search Engine

Library home page: http://nexus.sonatype.org/oss-repository-hosting.html/parent/elasticsearch

Path to dependency file: /tmp/ws-scm/elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: epository/org/elasticsearch/elasticsearch/2.4.0/elasticsearch-2.4.0.jar

Dependency Hierarchy:

❌ elasticsearch-2.4.0.jar (Vulnerable Library)

Vulnerability Details

A permission issue was found in Elasticsearch versions before 5.6.15 and 6.6.1 when Field Level Security and Document Level Security are disabled and the _aliases, _shrink, or _split endpoints are used . If the elasticsearch.yml file has xpack.security.dls_fls.enabled set to false, certain permission checks are skipped when users perform one of the actions mentioned above, to make existing data available under a new index/alias name. This could result in an attacker gaining additional permissions against a restricted index.

Publish Date: 2019-03-25

URL: CVE-2019-7611

CVSS 3 Score Details (8.1)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: High
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: High
- Integrity Impact: High
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-7611

Release Date: 2019-03-25

Fix Resolution: 5.6.15,6.6.1

WS-2009-0001 (Low) detected in commons-codec-1.9.jar

WS-2009-0001 - Low Severity Vulnerability

Vulnerable Library - commons-codec-1.9.jar

The Apache Commons Codec package contains simple encoder and decoders for various formats such as Base64 and Hexadecimal. In addition to these widely used encoders and decoders, the codec package also maintains a collection of phonetic encoding utilities.

Library home page: http://commons.apache.org/proper/commons-codec/

Path to dependency file: /elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: /root/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar

Dependency Hierarchy:

jest-2.0.4.jar (Root Library)
- httpclient-4.5.2.jar
  - ❌ commons-codec-1.9.jar (Vulnerable Library)

Found in HEAD commit: d1db845afad5b66f5924c3f0a3979eefe3cf9ff5

Vulnerability Details

Not all "business" method implementations of public API in Apache Commons Codec 1.x are thread safe, which might disclose the wrong data or allow an attacker to change non-private fields.

Updated 2018-10-07 - an additional review by WhiteSource research team could not indicate on a clear security vulnerability

Publish Date: 2007-10-07

URL: WS-2009-0001

CVSS 2 Score Details (0.0)

Base Score Metrics not available

Step up your Open Source Security Game with WhiteSource here

CVE-2020-9548 (Medium) detected in jackson-databind-2.9.10.1.jar

CVE-2020-9548 - Medium Severity Vulnerability

Vulnerable Library - jackson-databind-2.9.10.1.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /tmp/ws-scm/elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: epository/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar

Dependency Hierarchy:

❌ jackson-databind-2.9.10.1.jar (Vulnerable Library)

Vulnerability Details

FasterXML jackson-databind 2.x before 2.9.10.4 mishandles the interaction between serialization gadgets and typing, related to br.com.anteros.dbcp.AnterosDBCPConfig (aka anteros-core).

Publish Date: 2020-03-02

URL: CVE-2020-9548

CVSS 2 Score Details (5.0)

Base Score Metrics not available

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-9548

Release Date: 2020-03-02

Fix Resolution: com.fasterxml.jackson.core:jackson-databind:2.10.3

Check this box to open an automated fix PR

CVE-2017-18640 (High) detected in snakeyaml-1.15.jar

CVE-2017-18640 - High Severity Vulnerability

Vulnerable Library - snakeyaml-1.15.jar

YAML 1.1 parser and emitter for Java

Library home page: http://www.snakeyaml.org

Path to dependency file: /tmp/ws-scm/elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: /root/.m2/repository/org/yaml/snakeyaml/1.15/snakeyaml-1.15.jar

Dependency Hierarchy:

elasticsearch-2.4.0.jar (Root Library)
- ❌ snakeyaml-1.15.jar (Vulnerable Library)

Vulnerability Details

The Alias feature in SnakeYAML 1.18 allows entity expansion during a load operation, a related issue to CVE-2003-1564.

Publish Date: 2019-12-12

URL: CVE-2017-18640

CVSS 3 Score Details (7.5)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: None
- Integrity Impact: None
- Availability Impact: High

For more information on CVSS3 Scores, click here.

CVE-2018-10237 Medium Severity Vulnerability detected by WhiteSource

CVE-2018-10237 - Medium Severity Vulnerability

Vulnerable Library - guava-19.0.jar

Guava is a suite of core and expanded libraries that include utility classes, google's collections, io classes, and much much more.

Guava has only one code dependency - javax.annotation,
per the JSR-305 spec.</p>

path: 2/repository/com/google/guava/guava/19.0/guava-19.0.jar

Library home page: https://github.com/google/guava/guava

Dependency Hierarchy:

❌ guava-19.0.jar (Vulnerable Library)

Vulnerability Details

Unbounded memory allocation in Google Guava 11.0 through 24.x before 24.1.1 allows remote attackers to conduct denial of service attacks against servers that depend on this library and deserialize attacker-provided data, because the AtomicDoubleArray class (when serialized with Java serialization) and the CompoundOrdering class (when serialized with GWT serialization) perform eager allocation without appropriate checks on what a client has sent and whether the data size is reasonable.

Publish Date: 2018-04-26

URL: CVE-2018-10237

CVSS 3 Score Details (5.9)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: High
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: None
- Integrity Impact: None
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2018-10237

Release Date: 2018-04-26

Fix Resolution: 24.1.1

Step up your Open Source Security Game with WhiteSource here

WS-2017-3734 (Medium) detected in httpclient-4.5.2.jar

WS-2017-3734 - Medium Severity Vulnerability

Vulnerable Library - httpclient-4.5.2.jar

Apache HttpComponents Client

Library home page: http://hc.apache.org/httpcomponents-client

Path to dependency file: /elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: /root/.m2/repository/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar

Dependency Hierarchy:

jest-2.0.4.jar (Root Library)
- ❌ httpclient-4.5.2.jar (Vulnerable Library)

Found in HEAD commit: 38d5d675af8dde261d87251af53f34785aaf4f74

Vulnerability Details

Apache httpclient before 4.5.3 are vulnerable to Directory Traversal. The user-provided path was able to override the specified host, resulting in giving network access to a sensitive environment.

Publish Date: 2019-05-30

URL: WS-2017-3734

CVSS 2 Score Details (5.5)

Base Score Metrics not available

Suggested Fix

Type: Upgrade version

Origin: https://issues.apache.org/jira/browse/HTTPCLIENT-1803

Release Date: 2019-05-30

Fix Resolution: httpcore-5.0-alpha3-RC1

CVE-2020-10673 (Medium) detected in jackson-databind-2.9.10.3.jar

CVE-2020-10673 - Medium Severity Vulnerability

Vulnerable Library - jackson-databind-2.9.10.3.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /tmp/ws-scm/elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: downloadResource_eea4cdfb-6c06-4f4f-af5b-5acf3f94ba31/20200309160515/jackson-databind-2.9.10.3.jar

Dependency Hierarchy:

❌ jackson-databind-2.9.10.3.jar (Vulnerable Library)

Vulnerability Details

FasterXML jackson-databind 2.x before 2.9.10.4 mishandles the interaction between serialization gadgets and typing, related to com.caucho.config.types.ResourceRef (aka caucho-quercus).

Publish Date: 2020-03-18

URL: CVE-2020-10673

CVSS 3 Score Details (5.0)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: N/A
- Attack Complexity: N/A
- Privileges Required: N/A
- User Interaction: N/A
- Scope: N/A
Impact Metrics:
- Confidentiality Impact: N/A
- Integrity Impact: N/A
- Availability Impact: N/A

For more information on CVSS3 Scores, click here.

CVE-2017-5929 High Severity Vulnerability detected by WhiteSource

CVE-2017-5929 - High Severity Vulnerability

Vulnerable Library - logback-classic-1.1.7.jar

logback-classic module

path: 2/repository/ch/qos/logback/logback-classic/1.1.7/logback-classic-1.1.7.jar

Library home page: http://logback.qos.ch/logback-classic

Dependency Hierarchy:

❌ logback-classic-1.1.7.jar (Vulnerable Library)

Vulnerability Details

QOS.ch Logback before 1.2.0 has a serialization vulnerability affecting the SocketServer and ServerSocketReceiver components.

Publish Date: 2017-03-13

URL: CVE-2017-5929

CVSS 3 Score Details (9.8)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: High
- Integrity Impact: High
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Change files

Origin: qos-ch/logback@f46044b

Release Date: 2017-02-07

Fix Resolution: Replace or update the following files: LogbackClassicSerializationHelper.java, Innocent.java, SimpleSocketServer.java, HardenedObjectInputStream.java, LoggerSerializationTest.java, HardenedObjectInputStreamTest.java

Step up your Open Source Security Game with WhiteSource here

CVE-2017-12629 (High) detected in lucene-queryparser-5.5.2.jar

CVE-2017-12629 - High Severity Vulnerability

Vulnerable Library - lucene-queryparser-5.5.2.jar

Lucene QueryParsers module

Library home page: http://lucene.apache.org/lucene-parent/lucene-queryparser

Path to dependency file: /elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: /root/.m2/repository/org/apache/lucene/lucene-queryparser/5.5.2/lucene-queryparser-5.5.2.jar

Dependency Hierarchy:

elasticsearch-2.4.0.jar (Root Library)
- ❌ lucene-queryparser-5.5.2.jar (Vulnerable Library)

Found in HEAD commit: d1db845afad5b66f5924c3f0a3979eefe3cf9ff5

Vulnerability Details

Remote code execution occurs in Apache Solr before 7.1 with Apache Lucene before 7.1 by exploiting XXE in conjunction with use of a Config API add-listener command to reach the RunExecutableListener class. Elasticsearch, although it uses Lucene, is NOT vulnerable to this. Note that the XML external entity expansion vulnerability occurs in the XML Query Parser which is available, by default, for any query request with parameters deftype=xmlparser and can be exploited to upload malicious data to the /upload request handler or as Blind XXE using ftp wrapper in order to read arbitrary local files from the Solr server. Note also that the second vulnerability relates to remote code execution using the RunExecutableListener available on all affected versions of Solr.

Publish Date: 2017-10-14

URL: CVE-2017-12629

CVSS 3 Score Details (9.8)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: High
- Integrity Impact: High
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2017-12629

Release Date: 2017-10-14

Fix Resolution: Apache Solr - 7.1;Apache Lucene - 7.1

Step up your Open Source Security Game with WhiteSource here

Some Advices

At first I use es-rally tool to run benchmark on es, actually, it really not working so good .

After I found this project, I like the theory you put here: https://logz.io/blog/benchmarking-elasticsearch-magnetic-ebs/?utm_content=buffer804f1&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

Such as it use docker, and I can launch instance as many as I want.

But I have few questions:

The metrics in Graphite is not easy to understand, can you provide a table with description.
Actually I use Grafana to visualize the status of es, can you provide a Grafana template?
The parameter like 'SERVICE_HOST' is not very clearly, I really don't know what it meas.

CVE-2020-9547 (Medium) detected in jackson-databind-2.9.10.1.jar

CVE-2020-9547 - Medium Severity Vulnerability

Vulnerable Library - jackson-databind-2.9.10.1.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /tmp/ws-scm/elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: epository/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar

Dependency Hierarchy:

❌ jackson-databind-2.9.10.1.jar (Vulnerable Library)

Vulnerability Details

FasterXML jackson-databind 2.x before 2.9.10.4 mishandles the interaction between serialization gadgets and typing, related to com.ibatis.sqlmap.engine.transaction.jta.JtaTransactionConfig (aka ibatis-sqlmap).

Publish Date: 2020-03-02

URL: CVE-2020-9547

CVSS 2 Score Details (5.0)

Base Score Metrics not available

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-9547

Release Date: 2020-03-02

Fix Resolution: com.fasterxml.jackson.core:jackson-databind:2.10.3

Check this box to open an automated fix PR

CVE-2019-7614 (Medium) detected in elasticsearch-2.4.0.jar

CVE-2019-7614 - Medium Severity Vulnerability

Vulnerable Library - elasticsearch-2.4.0.jar

Elasticsearch - Open Source, Distributed, RESTful Search Engine

Library home page: http://nexus.sonatype.org/oss-repository-hosting.html/parent/elasticsearch

Path to dependency file: /tmp/ws-scm/elasticsearch-benchmark-tool/pom.xml

Path to vulnerable library: epository/org/elasticsearch/elasticsearch/2.4.0/elasticsearch-2.4.0.jar

Dependency Hierarchy:

❌ elasticsearch-2.4.0.jar (Vulnerable Library)

Vulnerability Details

A race condition flaw was found in the response headers Elasticsearch versions before 7.2.1 and 6.8.2 returns to a request. On a system with multiple users submitting requests, it could be possible for an attacker to gain access to response header containing sensitive data from another user.

Publish Date: 2019-07-30

URL: CVE-2019-7614

CVSS 3 Score Details (5.9)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: High
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: High
- Integrity Impact: None
- Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-7614

Release Date: 2019-07-30

Fix Resolution: org.elasticsearch:elasticsearch:6.8.2,7.2.1

logzio / elasticsearch-benchmark-tool Goto Github PK

elasticsearch-benchmark-tool's Introduction

THIS PROJECT IS NO LONGER MAINTAINED

elasticsearch-benchmark-tool

How it is working

Configuration Example

Run

Build

elasticsearch-benchmark-tool's People

Contributors

Stargazers

Watchers

Forkers

elasticsearch-benchmark-tool's Issues

CVE-2019-20330 - High Severity Vulnerability

CVE-2020-9546 - Medium Severity Vulnerability

CVE-2020-8840 - Medium Severity Vulnerability

CVE-2019-7611 - High Severity Vulnerability

WS-2009-0001 - Low Severity Vulnerability

CVE-2020-9548 - Medium Severity Vulnerability

CVE-2017-18640 - High Severity Vulnerability

CVE-2018-10237 - Medium Severity Vulnerability

WS-2017-3734 - Medium Severity Vulnerability

CVE-2020-10673 - Medium Severity Vulnerability

CVE-2017-5929 - High Severity Vulnerability

CVE-2017-12629 - High Severity Vulnerability

CVE-2020-9547 - Medium Severity Vulnerability

CVE-2019-7614 - Medium Severity Vulnerability

Recommend Projects

Recommend Topics

Recommend Org