zloster / frameworkbenchmarks Goto Github PK

Problem raised here: https://groups.google.com/d/topic/framework-benchmarks/VQZICSeiEcM

Currently the MySQL and PostgreSQL configurations are using max 256 connections to the DB.
The jdbi is configured for max 25 connections.
It is possible to have some other missing properties.
As result the configurations should look almost identical.

The usage of transactions have to be checked because currently the performance in the update and multiple queries test is slower than the hibernate and servlet-postgres-raw versions.

Some material for thought about the async processing: https://youtu.be/EDB24onZxsA?t=33m18s
Currently the undertow have implementation using normal blocking driver and the async driver: undertow and undertow-mongodb-async. And the results are not favorable for the async version. Maybe it needs polishing?
https://www.techempower.com/benchmarks/previews/round15/#section=data-r15&hw=ph&test=fortune&p=zdk8an-zik0zj-zik0zj-v&d=m&a=2 - this is the data from the Round 15 preview 1.

See: https://github.com/zloster/FrameworkBenchmarks/blob/master/frameworks/Java/wildfly-ee7/setup.sh#L5

-verbosegc -Xloggc:/tmp/wildfly_gc.log

This should be removed.

Commit the local branch with the Jackson Afterburner module enabled for the servlet test.
Edit: also adding implementation with the Jackson streaming API for comparison will be good. I'll do new issue.

Currently the random numbers are accumulated in a list and requested from the cache with one API call. But if there are duplicated keys the cache will omit the second fetch.

Provide setup scripts so it can be used as dependency to be installed from the shell scripts. Write shell scripts and benchmark variation for the servlet tests to run on Tomcat.
Which version? Check here: https://tomcat.apache.org/whichversion.html
Currently 9.0.x is in alpha stage. Now in beta

[From the mailing list}(https://groups.google.com/d/msg/framework-benchmarks/7KzlbOfnm30/KODCEcxIBAAJ):

I am experiencing database deadlock issue with Updates test of ActFramework, and when I tried to debug some other framework to see if I can get some clue, I found at least Dropwizard and ninjar-standalone tests did not commit the changes into the database, i.e. the World table not changed after sending request to updates test endpoints.
I found this issue by setting a breakpoint into the relevant code of updates test, and checking the world record been updated from MySql workbench, after the response sent back, I check the same world record again in the MySql workbench and found the data is not updated. I specifically checked Act and minijax framework using the same technique and found the record did change after response sent back.

I've done a quick check and indeed the value in the returned JSON for given id is different than in the DB.

the recommended DB driver class or maybe it will help with the transaction deadlocks?
Ordering of the updates should help with the transaction deadlocks. (https://gist.github.com/isopov/9af3864888b85c6cffc4 and some comments in the TFB issues/PRs).
Also the Resin documentation strongly recommends usage of the JDBC more advanced connection options if available. The Driver class is a last resort.

Also check the setting of the PostgreSQL about the server-side prepared statement threshold.

Update the benchmark_config.json to have correct versus data.
For example:

jdbi-postgres is configured to be compared with "versus": "servlet-raw-postgres"

The actual name must be verified by checking the JSON configuration of the project you want to compare with.
MySQL test should compare against servlet-raw. servlet is the default name but there are no DB tests and we need to point the config with the MySQL code which is servlet-raw.

Update:
We need to break up things as the servlet's benchmark_config.json. There are 3 (three) parts. One for the plaintext and JSON and 2 (two) for the MySQL and PostgreSQL tests. I'm not sure what should be the base comparison for MongoDB tests. Maybe MySQL.

Areas that need improvement:

• Servlet code duplication for the db tests. The only difference is the annotation which is injecting the datasource - using DataSource with the same name and different parameters
• The exception handling currently is swallowing the exceptions - the lost Async servlets were having good production grade error-handling
• Move the application to the root context to have shorter URLs. Currently it is http://TFB-server/servlet/plaintext - Moved to #42
• Check if there is actual difference in the performance if the DB connection pool is using the recommended DB driver class or maybe it will help with the transaction deadlocks? - Moved to #43

It is a big endeavor to attempt fine tuning the JVM arguments.
But some basic stuff will not harm. Use the other Java projects for reference. But note that some of them are using outdated or currently default settings.

Update:
Track the progress in this branch: https://github.com/zloster/FrameworkBenchmarks/tree/jvm-params

Review again the rules about the transactions and the update test: https://github.com/zloster/FrameworkBenchmarks/blob/dw8-jdbi/frameworks/Java/dropwizard/src/main/java/com/example/helloworld/db/jdbi/WorldRepository.java#L46-L61 - I think this is forbidden by the rules - to have one big transaction around the reading and updating the values.

After this cleanup: https://github.com/zloster/FrameworkBenchmarks/blob/dw8-jdbi/frameworks/Java/dropwizard/src/main/java/com/example/helloworld/db/jdbi/WorldJDBIImpl.java

It will be interesting.

• Add Grizzly 2.4 test implementation - currently in beta, see the GitHub releases: https://github.com/javaee/grizzly/releases https://github.com/eclipse-ee4j/grizzly
• Add NIO-test implementations
• Fix missing log library - #63

Cancelled:

• Add Grizzly 3.0 test implementation - it is not compatible with the 2.x versions. Currently I don't see the branch with the 3.x

Round 14 preview 4 results are not good:

Round 14 preview 4 results are not good:
Undertow Plaintext and JSON results are suffering. Jetty-servlet also. Dropwizard Plaintext is giving very strange behaviour at higher concurrences. The fluctuations of the Servlet (Resin) tests (Plaintext and JSON) are not gone as I hoped. Only the Jetty JSON have positive reaction.

Basically - total failure.

This will need more granular approach. Maybe project by project will be better.

Also I've checked the logs from R14 preview 4 and noticed that I haven't use shell expansion - note the ' that is visible at export 'JAVA_OPTS_TFB=. Also I'm not sure why the " have become ' at the next line: JAVA_OPTS_TFB='-

Setup dropwizard: ++++ export PATH=/usr/lib/jvm/java-8-oracle/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
Setup dropwizard: ++++ PATH=/usr/lib/jvm/java-8-oracle/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
Setup dropwizard: ++++ export 'JAVA_OPTS_TFB=-Xms1G -Xmx1G -Xss320k -XX:+UseNUMA -XX:+UseParallelGC -XX:+AggressiveOpts'
Setup dropwizard: ++++ JAVA_OPTS_TFB='-Xms1G -Xmx1G -Xss320k -XX:+UseNUMA -XX:+UseParallelGC -XX:+AggressiveOpts'
Setup dropwizard: +++ return 0

Work toward their restoration. See issue 2598 in the upstream repository.

gcc 6 was reintroduced in the toolset so play if there is a difference.
Also maybe add processor family flags?

Currently the dropwizard performance for the simpler testcases (plaintext and json) is almost flat. That suggests that there is some serious restriction somewhere. The other jetty tests suggest that there should be better performance. Good place to start seems to be the default dropwizard config about the max request to queue before blocking the acceptor threads.

maxQueuedRequests 1024 The maximum number of requests to queue before blocking the acceptors.

http://www.dropwizard.io/1.1.0/docs/manual/configuration.html#all

This is the warning in the console:

Setup dropwizard: [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!

Investigate what is the table bloat because there are many writes to the World/world table. Autovacuum will run around 200 000 000 writes to the table with the default settings Autovacuum is not enabled. Maybe this will impact some of the projects?

Additional information materials:

1. https://www.joyent.com/blog/manta-postmortem-7-27-2015 - Joyent had a problem with the vacuum operation on the tables
2. https://blog.sentry.io/2015/07/23/transaction-id-wraparound-in-postgres.html via the above Joyent post

Suggested here: https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Suggested-Frameworks,-Languages-and-Features#java #

This framework has two modes: default and Jackson. https://github.com/zloster/FrameworkBenchmarks/blob/master/frameworks/Java/activeweb/benchmark_config.json
But the JSON controller source suggests that only JSON test has different implementation. Note that Jackson is instantiated in the controller and used explicitly in the method.
This needs investigation/corrections.
Each controller have to entry points (see benchmark_config.json): one is returning the data via template that produces JSON, the other is calling Jackson for the serialisation.

Both the PostgreSQL and MySQL tests are suffering a lot - 30% performance decrease, except the updates test.
Changes that I recall:

1. the code clean up - exception handling;
2. Resin versions
3. Database drivers

They are using very old. Update to the version other benchmarks are using: 9.4.1208

Watch out for the:

• benchmark_config.json metadata
• pom.xml updates
• the setup.sh script contents

This is interesting: https://lwn.net/Articles/713963/

So, if you rely on 3.18.y, you should upgrade. But really, go use 4.4.y
or 4.9.y, those are both much better, and far more secure.

14.04 is using 3.13 as default (Ubuntu provides extended support for it by backporting patches and so) but it worth considering moving to Linux kernel 4.4 which is officially supported by Ubuntu and is apt-get install away.

The Vagrant development box and the ServerCentral servers are on Ubuntu 14.04 and kernel 3.13 (the default).

4.4-family has quite a lot of improvements including a major patches about the NUMA scheduling which seems to affect the current preview of Round 14.

The idea is mentioned here about Vagrant image: 2496.

Some frameworks see a lot more commits than others. It will be interesting to track the distribution of the development effort.
Such information can be partially tracked by git and the commits hitting the given framework directory. Remember to not count the merge requests!

TechEmpower currently provides git tags for the releases in the form Rxx where xx is a digit. There is also also a single R9-ec2 tag.

Two tickets describing a similar behaviour with Grizzly/Jersey:

• https://java.net/jira/browse/GLASSFISH-21005
• https://java.net/jira/browse/JERSEY-3084

Currently tbf is using nonofficial PPA for the maven: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/toolset/setup/linux/systools/maven.sh#L5
Newer maven packages should be available in the more recent Ubuntu versions from the official repositories. Check what is available.

Add implementation with the Jackson serialization API for comparison.

Update Resin 4.0.48 to the latest 4.0.53: http://caucho.com/products/resin/download/gpl#download

Investigate how to enable and if the large memory pages help?

-XX:+UseHugeTLBFS

This option for Linux is the equivalent of specifying -XX:+UseLargePages. This option is disabled by default. This option pre-allocates all large pages up-front, when memory is reserved; consequently the JVM cannot dynamically grow or shrink large pages memory areas; see -XX:UseTransparentHugePages if you want this behavior.

For more information, see "Large Pages".

-XX:+UseLargePages

Enables the use of large page memory. By default, this option is disabled and large page memory is not used.

For more information, see "Large Pages".

Source: http://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html

1. Commit the branch with the experiment.
2. Verify again the performance and the configuration settings because the first time there was a big difference: the newer version had significantly lower results.

Update: seem the wal-level setting needs to be tweaked. The defaults have changed.

wal_level deserves a mention; the default in Pg9.6 was minimal and in Pg10 and later the default is replica. This change requires a lot of extra WAL activity that seriously hinders performance on the Pi. All tests here used wal_level = minimal. (Again, initial tests with replica were noticably slower!)

Via: https://blog.rustprooflabs.com/2019/04/postgrseql-pgbench-raspberry-pi

See 593 for details

For quite a while there are:

• hstore (string key, value pairs)
• JSON (since 9.2 but seems like a little half-baked, see the link below, depends on the use-cases)
• JSONB (since 9.4, best full feature integration)
• UNLOGGED SQL tables - they skip the WAL and are not safe for data in case of crash. They are automatically truncated when crash is detected - so they are quite good for temporary data.

It will be interesting to see these against MongoDB.

Introduction here: https://www.citusdata.com/blog/2016/07/14/choosing-nosql-hstore-json-jsonb/

The current command for dstat is:

dstat -afilmprsT ......

T is for timestamp. Currently it is somewhere after 80*6 columns of CPU utilization data. Seem better to be at the first column:

dstat -Tafilmprs

Currently all tests don't mandate the usage of DB transactions but in PostgreSQL it is not very easy to turn them off.
Setting autocommit=true should do the trick - for example Revenj.jvm is using this.
Affected projects (minimum): servlet and dropwizard.

Currently there are: realistic and striped.
I have the idea about: experimental.
The results should not be visible by default in the normal results page of a round. Maybe even to have separate runs for this test implementations - to improve the feedback speed. Maybe limit the total number per framework to stay around 24 hours wall clock duration.
In general it should help the project participants to play and experiment with tuning the "knobs": how many threads to spawn, how many connections to the database, how many requests to queue.
Currently this is very hard if you don't have similar hardware to play with and the time-span is huge.

Currently it is http://TFB-server/servlet/plaintext - this will be moved to another issue

Probably it is some configuration in Jetty. Investigate the logs for all tests that respond with gzip header. Correct the problem in Dropwizard and report the others.
Update: http://www.dropwizard.io/1.1.0/docs/manual/configuration.html#gzip Straightforward configuration

The current metadata says there is NO FRONTEND server.
But it is using Jetty.

Some projects are not yet configured to use Java 8 bytecode and don't have stripped debugging symbols. These two should give small improvement in the test results.
Review the pom.xmls and change the settings.

Make the shell scripts setup_mysql.sh and setup_postgresql.sh the same. Currently the later is calling mvn clean install and it should not.
Make the servlet URLs the same. There is another issue to use the same code and inject the different DataSource which is now possible.
Finish README.md update.

For example see Dropwizard or undertow projects.
This will resolve the verification warnings.

This affects the plaintext results. I can't be sure to what extent. But 330 000+ (Round 13, cloud environment, plaintext) replies per second with each reply has + 14 bytes it will add up.