When SQLancer cannot connect to the DBMS under test, it currently repeatedly fails while printing an output such as the following:

--com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
--
--The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
--	at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
--	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
--	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
--	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
--	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
--	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
--	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
--	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
--	at sqlancer.mysql.MySQLProvider.createDatabase(MySQLProvider.java:243)
--	at sqlancer.mysql.MySQLProvider.createDatabase(MySQLProvider.java:1)
--	at sqlancer.Main$DBMSExecutor.run(Main.java:312)
--	at sqlancer.Main$1.runThread(Main.java:430)
--	at sqlancer.Main$1.run(Main.java:423)
--	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
--	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
--	at java.base/java.lang.Thread.run(Thread.java:834)
--Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

This might be confusing to users. It would be better to print an error message that SQLancer cannot connect, and refer to the relevant settings to set, for example, the user credentials.

I see an option --username, it is also possible to specify the host, port and usage of password? Thank you!

The SQLite implementation provides an option to print statements on stdout:

It would be nice to also add this feature to the other DBMS implementations.

We have been running SQLancer on DuckDB with 20 threads for a while now and noticed the following: In the beginning, loads of queries are run at a high speed, e.g.

Executed 219835 queries (6854 queries/s;... on our rather fast box, but 12 hours later we see
Executed 49502209 queries (159 queries/s;...

Is there a conceptual reason for this or do we need to dig into DuckDB performance degradations?

In #101, we disabled the ClickHouse CI due to some new failures that blocked us from merging the PR. We should fix the issue and enable the CI again as soon as possible, to prevent that we break the ClickHouse implementation during refactoring.

https://travis-ci.com/github/sqlancer/sqlancer/jobs/358720280

[2020/07/08 08:17:24] Executed 61181 queries (30 queries/s; 2.20/s dbs, successful statements: 56%). Threads shut down: 4.
[2020/07/08 08:17:24] Executed 61184 queries (12236 queries/s; 197.00/s dbs, successful statements: 56%). Threads shut down: 4.
[2020/07/08 08:17:29] Executed 61395 queries (43 queries/s; 1.20/s dbs, successful statements: 56%). Threads shut down: 4.

The Travis CI tests just failed with an error direct joins between distributed and local tables are not supported for a query SELECT t4.c1, t4.c0 FROM ONLY t4 RIGHT OUTER JOIN (SELECT 0.79263866 FROM ONLY t2, t0*, t1, ONLY t4 LIMIT 5032437055157358207) AS sub0 ON upper((1422540800)::VARCHAR(710))!~*upper((('pt')||((0.0034468356)::MONEY))); @nukoyluoglu, do you want to fix this or should I add the error to the list of expected errors?

Currently, the JDBC string containing the host name and port is hardcoded. For example, consider the connection string for MySQL:

It would be desirable to make this configurable. We could provide an option to set the JDBC string for each DBMS. For example, for MySQL, the string could look like this:

"jdbc:mysql://$(HOST_NAME):$(PORT)/?serverTimezone=UTC&useSSL=false&allowPublicKeyRetrieval=true"

We could then also provide two options to set the host name and port, which are then used to replace the placeholder in the JDBC string.

This would make everything configurable, but also be convenient for users. Most users could go with the default options. Some users might need to configure the port and host. Only very few users probably would need to change the JDBC query string.

It would be useful to keep track of coverage information when running the smoke tests in the Travis CI to spot dead code and potentially test additional options.

It would be useful to provide an option to capture how long individual statements take to execute, which could be appended to the log after each statement as a comment. For example:

CREATE TABLE .... -- 1.2 ms

It would be very useful to provide an external SQLancer API, which would allow exposing concerns like

• Database generation
• Query generation
• Checking a result set with a test oracle (probably in generation with generating a random query).

Used as the following:

me:sqlancer$ java -cp ./target/SQLancer-0.0.1-SNAPSHOT.jar --num-threads 4 sqlite3 --oracle NoREC
Unrecognized option: --num-threads
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

me:sqlancer$ java -cp ./target/SQLancer-0.0.1-SNAPSHOT.jar -num-threads 4 sqlite3 --oracle NoREC
Unrecognized option: -num-threads
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

After merging #39, SQLancer now prints SLF4J warnings on startup:

$ java -jar SQLancer-0.0.1-SNAPSHOT.jar sqlite3
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2020/07/01 09:22:57] Executed 150665 queries (30120 queries/s; 3.00/s dbs, successful statements: 92%). Threads shut down: 0.

@qoega do you have any ideas on how to fix this?

ClickHouse produced a syntactically-invalid test case (see https://travis-ci.com/github/sqlancer/sqlancer/jobs/390341131):

DROP DATABASE IF EXISTS database4TLPHaving;
CREATE DATABASE IF NOT EXISTS database4TLPHaving;
CREATE TABLE t0 (c0 String MATERIALIZED ('-1255315139') CODEC (NONE)) ENGINE = Memory() ;
INSERT INTO t0() VALUES (), ();
INSERT INTO t0() VALUES (), ();
INSERT INTO t0() VALUES (), ();
INSERT INTO t0() VALUES (), (), (), ();
INSERT INTO t0() VALUES (), ();
INSERT INTO t0() VALUES (), (), ();
INSERT INTO t0() VALUES (), ();
[...]

When I execute the CREATE TABLE locally (on an older ClickHouse version), I get the following error:

Code: 90, e.displayText() = DB::Exception: Empty list of columns passed (version 20.4.1.2742 (official build))

I'm a bit confused that the CREATE TABLE and many of the INSERTs are executed without an error. @qoega, do you have any ideas on what the issue could be?

As part of the OOPSLA artifact evaluation, an anonymous reviewer reported the following issue (thank you!):

java.lang.ArrayIndexOutOfBoundsException: 10
    at java.util.ArrayList.add(ArrayList.java:463)
    at sqlancer.Main$StateLogger.ensureExistsAndIsEmpty(Main.java:120)
    at sqlancer.Main$StateLogger.<init>(Main.java:94)
    at sqlancer.Main$DBMSExecutor.run(Main.java:327)
    at sqlancer.Main$2.run(Main.java:500)
    at sqlancer.Main$2.runThread(Main.java:477)
    at sqlancer.Main$2.run(Main.java:468)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Exception in thread "database14" java.lang.NullPointerException
    at sqlancer.Main$2.run(Main.java:513)
    at sqlancer.Main$2.runThread(Main.java:477)
    at sqlancer.Main$2.run(Main.java:468)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

This seems to be a race condition, since adding an element to an ArrayList should never cause an exception.

Travis CI switched to a paid model (see https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing), and no longer runs pull requests for SQLancer. While the blog post mentions OSS credits, the credits seem to no longer be awarded as clarified in an email:

[...] At the moment, credit allocation for OSS projects is on hold as per directives from management. [...]

It might be worthwhile to investigate GitHub Actions, which seems to (still) be free for open-source projects.

"we have been working on a SQL-specific reducer"

See https://github.com/mariadb-corporation/mariadb-qa/blob/master/reducer.sh

Info links at the top of the file.

Let me know if you have any questions.

I received a bug report that, when using MacOS and OpenJDK 13, compilation fails with the following error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile (default-testCompile) on project SQLancer: Compilation failure: Compilation failure: 
[ERROR] GitHub/sqlancer/test/sqlancer/dbms/TestConfig.java:[3,14] Implicit super constructor java.lang.Object() is undefined for default constructor. Must define an explicit constructor
[ERROR] GitHub/sqlancer/test/sqlancer/dbms/TestConfig.java:[4,25] String cannot be resolved to a type
[ERROR] GitHub/sqlancer/test/sqlancer/dbms/TestConfig.java:[5,25] String cannot be resolved to a type
[ERROR] -> [Help 1]

JDBC drivers for embedded DBMS like SQLite and DuckDB run the respective DBMS directly within the JVM's application process. When testing DBMS, often crashes can be triggered. For embedded DBMS, this means that a crash results in the JVM terminating, which is annoying and inhibits the bug-finding process. To address this, we should add an option to run the DBMS in a separate process.

I was wondering if sqlancer now supports generating random queries based on a pre-defined database schema.

Hi Manuel,
A couple of options, num-threads, num-tries, max-expression-depthand and num-queries don't work right now. I've tried both --num_threads and --num-threads, but get the same exception in both cases:

Exception in thread "main" com.beust.jcommander.ParameterException: Was passed main parameter '--num-threads' but no main parameter was defined in your arg class at com.beust.jcommander.JCommander.initMainParameterValue(JCommander.java:954) at com.beust.jcommander.JCommander.parseValues(JCommander.java:755) at com.beust.jcommander.JCommander.parse(JCommander.java:356) at com.beust.jcommander.JCommander.parseValues(JCommander.java:796) at com.beust.jcommander.JCommander.parse(JCommander.java:356) at com.beust.jcommander.JCommander.parse(JCommander.java:335) at sqlancer.Main.executeMain(Main.java:296) at sqlancer.Main.main(Main.java:273)

It would be desirable that the test oracles give an informative message when the detect an error.

UCDetector (see http://www.ucdetector.org/) just detected that the port variable in the PostgresProvider class is never read. @nukoyluoglu, should I delete the variable or do you want to fix this?

The commands

CREATE UNLOGGED TABLE t0(c0 DECIMAL  UNIQUE CHECK (((- (290659171))::INT) IN (+ (+ (-552712090)), ((CAST(B'1111111111111111111111111111111111011001011111011110111101110011' AS INT))/(+ (850767994))))) DEFAULT (0.6609091545033620729299173035542480647563934326171875), c1 DECIMAL ) WITH (parallel_workers=568, fillfactor=69, autovacuum_vacuum_cost_limit=5135, autovacuum_freeze_table_age=1697368661, autovacuum_freeze_max_age=1365356848, autovacuum_vacuum_threshold=949719417, autovacuum_analyze_threshold=993547691, autovacuum_analyze_scale_factor=0.01);
ALTER TABLE t0 ALTER COLUMN c0 SET NOT NULL;

were produced by SQLancer, which raises the error:
org.postgresql.util.PSQLException: ERROR: integer out of range.
This is likely because the CHECK statement always evaluates to FALSE, which is why a NOT NULL c0 value is not possible.

hi
I try using intellij build , but build print error ! below
Error:java: java.lang.IllegalArgumentException: properties file .settings/org.eclipse.jdt.core.prefs does not exist
Error:java: properties file .settings/org.eclipse.jdt.core.prefs does not exist

Is there any way to solve it

Query generateInto()

for (int row = 0; row < nrRows; row++) {
if (row != 0) {
sb.append(", ");
}
sb.append("(");
for (int c = 0; c < columns.size(); c++) {
if (c++ != 0) { <---- typo, should be c
sb.append(", ");
}
sb.append(MySQLVisitor.asString(gen.generateConstant()));

        }
        sb.append(")");
    }

I went through the option list but didn't see such an option to specify the table number in a database.
Is it possible to do this?

Currently, SQLite uses both general, and DBMS-specific commands. For example, consider the following command arguments:

$java -jar SQLancer-0.0.1-SNAPSHOT.jar --timeout-seconds 60 sqlite3 --oracle NoREC

The --timeout-seconds 60 option is a general one, and has to precede the name of the DBMS to be tested, that is, sqlite3. The --oracle option is DBMS-specific, and has to come after the DBMS name. This is error prone, and it's easy to mix up the order between general and DBMS-specific arguments. This has been highlighted in #31 and https://github.com/researchart/fse20/issues/93#issuecomment-646859320. For example, the following causes an error:

$ java -jar SQLancer-0.0.1-SNAPSHOT.jar --timeout-seconds 60 --oracle NoREC sqlite3
Exception in thread "main" com.beust.jcommander.MissingCommandException: Expected a command, got --oracle
	at com.beust.jcommander.JCommander.parseValues(JCommander.java:788)
	at com.beust.jcommander.JCommander.parse(JCommander.java:356)
	at com.beust.jcommander.JCommander.parse(JCommander.java:335)
	at sqlancer.Main.executeMain(Main.java:296)
	at sqlancer.Main.main(Main.java:273)

We should address this by allowing either kind of options to precede or succeed the DBMS name.

Do not know how to use the tool if no help information is provided, suggest to give a --help or -h option first.

Currently, the Travis CI gate fails every two to three runs, since SQLancer seems to trigger a crash bug in DuckDB. I assume this is already fixed in DuckDB, but has not yet been propagated to the latest Maven JDBC driver. @hannesmuehleisen is a release planned for the near future?

I followed hints from #267 to try to generate random queries based on an existing sqlite3 database.
However, sqlancer will stop generating queries after generating the first few queries.

An example snapshot for the stdout is as follows:
$ java -jar SQLancer-0.0.1-SNAPSHOT.jar --timeout-seconds=60 --print-statements True --num-threads 1 sqlite3 --execute-queries=False --oracle QUERY_PARTITIONING --generate-new-database false
[2020/12/14 22:34:33] Executed 2 queries (0 queries/s; 0.20/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:34:38] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:34:43] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:34:48] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:34:53] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:34:58] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:35:03] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:35:08] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:35:13] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:35:18] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:35:23] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.
[2020/12/14 22:35:28] Executed 2 queries (0 queries/s; 0.00/s dbs, successful statements: 75%). Threads shut down: 0.

An example content of the logfile is as follows:

SELECT ALL REGION.R_NAME, PART.P_CONTAINER, SUPPLIER.S_NAME, PART.P_BRAND, SUPPLIER.S_ADDRESS FROM SUPPLIER, PART LEFT OUTER JOIN REGION ON ((LIKELIHOOD(SUPPLIER.S_NAME, REGION.R_NAME))||(x''));
SELECT DISTINCT SUPPLIER.S_COMMENT, SUPPLIER.S_NATIONKEY, SUPPLIER.S_ADDRESS FROM SUPPLIER;
SELECT DISTINCT SUPPLIER.S_COMMENT, SUPPLIER.S_NATIONKEY, SUPPLIER.S_ADDRESS FROM SUPPLIER WHERE ((((SUPPLIER.S_COMMENT IN (SUPPLIER.S_NAME)))+(CASE WHEN SUPPLIER.S_NAME THEN SUPPLIER.S_COMMENT WHEN SUPPLIER.S_NAME THEN SUPPLIER.S_COMMENT WHEN SUPPLIER.S_PHONE THEN SUPPLIER.S_COMMENT WHEN SUPPLIER.S_ACCTBAL THEN SUPPLIER.S_COMMENT END))) UNION SELECT DISTINCT SUPPLIER.S_COMMENT, SUPPLIER.S_NATIONKEY, SUPPLIER.S_ADDRESS FROM SUPPLIER WHERE ((NOT ((((SUPPLIER.S_COMMENT IN (SUPPLIER.S_NAME)))+(CASE WHEN SUPPLIER.S_NAME THEN SUPPLIER.S_COMMENT WHEN SUPPLIER.S_NAME THEN SUPPLIER.S_COMMENT WHEN SUPPLIER.S_PHONE THEN SUPPLIER.S_COMMENT WHEN SUPPLIER.S_ACCTBAL THEN SUPPLIER.S_COMMENT END))))) UNION SELECT DISTINCT SUPPLIER.S_COMMENT, SUPPLIER.S_NATIONKEY, SUPPLIER.S_ADDRESS FROM SUPPLIER WHERE ((((((SUPPLIER.S_COMMENT IN (SUPPLIER.S_NAME)))+(CASE WHEN SUPPLIER.S_NAME THEN SUPPLIER.S_COMMENT WHEN SUPPLIER.S_NAME THEN SUPPLIER.S_COMMENT WHEN SUPPLIER.S_PHONE THEN SUPPLIER.S_COMMENT WHEN SUPPLIER.S_ACCTBAL THEN SUPPLIER.S_COMMENT END))) ISNULL));
SELECT CUSTOMER.C_CUSTKEY FROM CUSTOMER, PART, REGION, SUPPLIER, NATION GROUP BY CUSTOMER.C_CUSTKEY;

Expected Behavior

I would have expected sqlancer to generate many random queries based on the existing schema.

Reproduce Bug

1. set up a TPCH.db (download using the following link: https://drive.google.com/file/d/17QXAJoSbQiqoOajjiBDALavZg7U0swTH/view?usp=sharing)

2. change code in SQLite3Provider.java following hints from #267

• Specifically, change the url in function createDatabase in order to connect to an existing database

  sqlancer/src/sqlancer/sqlite3/SQLite3Provider.java

  Line 285 in 0b7c724

  public SQLConnection createDatabase(SQLite3GlobalState globalState) throws SQLException {

• You can download the resulting SQLite3Provider.java using the following link (https://drive.google.com/file/d/1KCs2fJkNDA9FdsqsIuu4_UUMmF0k_-A9/view?usp=sharing) (you still need to add your own absolute path of the TPCH.db in function createDatabase)

3. Build the new sqlancer

mvn package -DskipTests

3. run the sqlancer using the following command (setting --generate-new-database to be false to avoid modifying the existing database)

$ java -jar SQLancer-0.0.1-SNAPSHOT.jar --timeout-seconds=60 --print-statements True --num-threads 1 sqlite3 --execute-queries=False --oracle QUERY_PARTITIONING --generate-new-database false

It would be extremely helpful if you could provide any suggestion on how to fix this bug or point out if I was using the wrong command to fulfill my goal. Thank you!!!

The H2 implementation was merged in #217. It would be desirable to add Travis tests. However, the currently released Maven version of H2 is 1.4.200, which was published in 2019, and thus is still affected by the bugs that SQLancer found, and which were fixed by the H2 maintainers. Thus, we need to wait for a new release (see h2database/h2database#2491).

I was wondering which --oracle parameter I need to use if my goal is to test DBMSs only using Ternary Logic Query Partitioning (TLP):
java -jar SQLancer-0.0.1-SNAPSHOT.jar --num-threads 4 sqlite3 --oracle NoREC (according to the Quick Start Guide)
or
java -jar SQLancer-0.0.1-SNAPSHOT.jar --num-threads 4 sqlite3 --oracle QUERY_PARTITIONING (according to my understanding of the SQLite3Options.java).

CentOS Linux release 8.1.1911

openjdk version "1.8.0_252"                                                                                                                       
OpenJDK Runtime Environment (build 1.8.0_252-b09)                                                                                                 
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

$ sudo java -jar SQLancer-0.0.1-SNAPSHOT.jar --num_threads 16 --num_tries 5 --max_expression_depth 3 --num_queries 100000 --max_num_inserts 30 sqlite3 --oracle query_partitioning                                                                     
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.UnsupportedClassVersionError: sqlancer/Main has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)
$ 

It would be useful to log the random seed to be able to reproduce crashes.

Since NoREC and TLP have been implemented, I haven't been paying attention to the PQS implementation, which is now probably outdated. It would be desirable to make it work again, and modernize the code.

I believe the SELECT queries generated for testing by the Postgres implementation do not involve WINDOW clauses right now. I was wondering if they will be incorporated into the implementation, as well as other clauses that are currently not supported.

We should a Travis CI gate to check PRs.

One or two days ago, the TiDB gate has started failing nondeterministically (e.g., see https://travis-ci.com/github/sqlancer/sqlancer/jobs/364361274):

[ERROR] Please refer to /home/travis/build/sqlancer/sqlancer/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /home/travis/build/sqlancer/sqlancer && /home/travis/oraclejdk11/bin/java -javaagent:/home/travis/.m2/repository/org/jacoco/org.jacoco.agent/0.8.5/org.jacoco.agent-0.8.5-runtime.jar=destfile=/home/travis/build/sqlancer/sqlancer/target/jacoco.exec -jar /home/travis/build/sqlancer/sqlancer/target/surefire/surefirebooter8125434434466104958.jar /home/travis/build/sqlancer/sqlancer/target/surefire 2020-07-23T19-25-26_427-jvmRun1 surefire9478951949353639248tmp surefire_02698655768766759830tmp
[ERROR] Process Exit Code: 0
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /home/travis/build/sqlancer/sqlancer && /home/travis/oraclejdk11/bin/java -javaagent:/home/travis/.m2/repository/org/jacoco/org.jacoco.agent/0.8.5/org.jacoco.agent-0.8.5-runtime.jar=destfile=/home/travis/build/sqlancer/sqlancer/target/jacoco.exec -jar /home/travis/build/sqlancer/sqlancer/target/surefire/surefirebooter8125434434466104958.jar /home/travis/build/sqlancer/sqlancer/target/surefire 2020-07-23T19-25-26_427-jvmRun1 surefire9478951949353639248tmp surefire_02698655768766759830tmp
[ERROR] Process Exit Code: 0
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:282)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
[ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
[ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

This should be temporarily addressed by making it optional that the job succeeds.

Compiler warnings should let the Travis CI gate fail. It seems that currently, not (all) warnings are detected, see #110.

It would be useful to create a Docker image that includes all the DBMS that SQLancer supports.

The command SET SESSION jit_inline_above_cost=-134588496167444481; was produced by SQLancer, which raises the error: org.postgresql.util.PSQLException: ERROR: -1.34588e+17 is outside the valid range for parameter "jit_inline_above_cost" (-1 .. 1.79769e+308).

To specify the expected error messages for a SQL statement, we currently use Collection<String>, see

It would be desirable to introduce an abstraction to also allow, for example, specifying regular expressions to cover multiple expected error messages at once, and thus make expected error messages more descriptive (since without using regular expressions, we need to check for the shortest common string).

Mike Freedman, CTO and Co-founder of Timescale, expressed interest in having TimescaleDB tested:

Hey Manuel: would welcome any testing to @timescaledb , which is implemented as extension to Postgres and thus probably already fits your model.

The implementation could probably be based on top of SQLancer's PostgreSQL implementation.

It would be interesting to test H2, especially since one of H2's maintainers expressed interest on Hacker News in the SQLancer thread:

Neat! Perhaps you could run this against H2 (I’m a maintainer). Although best to run against our current master, release is a little old.

Sqlancer has unreasonable virtual memory usage when running:

We should make the SQLancer core independent from the DBMS implementations using a plugin system (e.g., by using the ServiceLoader infrastructure).

When launching SQLancer, a message is printed by the ClickHouse JDBC driver:

$ java -jar SQLancer-0.0.1-SNAPSHOT.jar sqlite3
[database9] INFO ru.yandex.clickhouse.ClickHouseDriver - Driver registered
[2020/07/01 22:20:07] Executed 133033 queries (26585 queries/s; 3.20/s dbs, successful statements: 92%). Threads shut down: 0.

@qoega do you know how we can disable this output?

It would be desirable to add smoke testing to the CI, to verify that the testing implementation for one of the DBMS is not obviously broken.

See the following:

root@39d4ee9ee7de:~/sqlancer/target# java -jar SQLancer-0.0.1-SNAPSHOT.jar --use-constant-caching false mysql   
Exception in thread "main" com.beust.jcommander.MissingCommandException: Expected a command, got false
        at com.beust.jcommander.JCommander.parseValues(JCommander.java:788)
        at com.beust.jcommander.JCommander.parse(JCommander.java:356)
        at com.beust.jcommander.JCommander.parse(JCommander.java:335)
        at sqlancer.Main.executeMain(Main.java:424)
        at sqlancer.Main.main(Main.java:271)

Try another option should work:

root@39d4ee9ee7de:~/sqlancer/target# java -jar SQLancer-0.0.1-SNAPSHOT.jar --test-only-nonempty-tables false mysql 
[2020/10/12 10:05:18] Executed 7718 queries (1542 queries/s; 1.00/s dbs, successful statements: 99%). Threads shut down: 0.
[2020/10/12 10:05:23] Executed 24691 queries (3397 queries/s; 2.20/s dbs, successful statements: 99%). Threads shut down: 0.