Describe This Problem

MySQL support is already fixed in #26, it would be great if PostgreSQL is also supported.

Proposal

Implement database trait for PostgreSQL

• https://docs.rs/postgres/latest/postgres/struct.Client.html

Additional Context

No response

Describe This Problem

Sometimes we don't care about the order in the result. It might be caused by undetermined execution ordering etc.

Proposal

Implement IGNORE_ORDER interceptor that makes the compare process ignore the different order between result lines. Like the rowsort mode in sqllogictest https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki

Additional Context

No response

Describe This Problem

Target of Sqlness is to ensure SQL's correction of one database, so if a database want to use sqlness, it need to implement Database trait, although this works as expected, this seems a tedious work.

In SQL world, there are only a few wire protocols, such as MySQL or PostgreSQL. If we can supply a default database with MySQL protocols, then any MySQL-compatible databases can use sqlness with less work.

Proposal

Add one default database implementation: MySQL

Additional Context

No response

Describe This Problem

In current implementation, there is only one case directory allowed for one environment, although this works, but this makes it awkward to share cases between multiple environment.

Proposal

Support multiple case directories

Additional Context

Another solution to this is symlinks, although this works, but I think it's a little ugly.

Describe this problem

Detected by GitHub Dependabot

The earliest fixed version is 0.10.0.

Steps to reproduce

None

Expected behavior

No response

Additional Information

In function Table::as_ref, a reference of vector is force cast to slice. There are multiple problems here:

To guarantee the size is correct, we have to first do Vec::shrink_to_fit. The function requires a mutable reference, so we have to force cast from immutable to mutable, which is undefined behavior (UB).
Even if (1) is sound, &Vec and &[T] still might not have the same layout. Treating them equally may lead to undefinted behavior (UB).

The earliest fixed version is 0.10.0.

Describe this problem

For libraries, cargo.lock shouldn't be checked in git.

Steps to reproduce

Not required

Expected behavior

No response

Additional Information

https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html

Describe This Problem

We have a scenario like this:

CREATE EXTERNAL TABLE WITH (LOCATION='/home/data/tests/data/csv/taxi_zone_lookup.csv', FORMAT='csv');

It's trying to create an external file table from an absolute path, specifying the absolute path in the local development environment is acceptable. However, Using the same absolute path in other environments (i.g., CI) will be problematic.

Proposal

Therefore, we can treat the test cases as templates and inject the environment variables at runtime.

I propose a template syntax:

CREATE EXTERNAL TABLE WITH (location='{{$PWD}}/tests/data/csv/taxi_zone_lookup.csv', FORMAT='csv');

it will replace all {{\$.*}} with Env(.*) in a test case file.

Additional Context

No response

Describe This Problem

When make a new release, there are severals things need to be done:

1. Bump version in Cargo.toml
2. Push new tag
3. Make new release

It would be great to automate this process.

Proposal

Use GitHub Actions to make new release

• https://pratikpc.medium.com/publishing-crates-using-github-actions-165ee67780e1

We can trigger this action when new tags pushed.

Additional Context

No response

Describe This Problem

Currently sqlness outputs everything with println!. This is hard to maintain.

Proposal

Consider a structured way to organize our outputs. Either using existing util like TestReporter from nextest or wrap our own.

Additional Context

No response

Some TODO I think we need to address before publish to crates.io

• Design review
• More testcase
• Demo, multiple envs setup
• CI
• Issue/PR template

Context

• apache/horaedb#397
• https://github.com/MariaDB/server/blob/10.11/mysql-test/mariadb-test-run.pl

Similar projects

• sqllogictest-rs
• ....

Describe This Problem

In current usage pattern, sqlness will write query response to .out file, and it's used to be compared with corresponding .result, while this works for basic case, it becomes verbose when we change some behaviors of server, and lots of .result file need to be updated ONE AFTER ONE.

Proposal

I suggest another pattern to compare result:

Write query result directly to result file

It works like this

• When a SQL test file is first added, write result file directly can save us one mv command
• When execute testcases next time, save original result file in Vec or tempfile before write query response, then compare it with result file,
  • If updated result file is what we expected, just check into git with git add ...
  • if wrong, discard result file with git checkout ..., update testcase and run again

I think this usage pattern is more efficient than before.

Additional Context

No response

          Why this method return Optione before? it seems Result is more reasonable, otherwise we cannot know why an interceptor isn't working as expected.

Maybe we can file an issue first.

Originally posted by @jiacai2050 in #58 (comment)

Describe This Problem

In current implementation, sqlness use tokio to do file IO, which is not a requirement, and this cause sqlness to depend on a heavy dependency.

Proposal

Use std to do file IO, and remove hard requirement of tokio.

Additional Context

No response

Describe This Problem

It's a bit frustrating when we want to repeat SQL several times. Like insert 100 rows

INSERT INTO t (c) VALUES
(1),
(2),
...
(100);

or query with all the aggregators

SELECT count(c) FROM t;
SELECT avg(c) FROM t;
...

Proposal

Implement a query interceptor that can do this for us. It can automatically replace variables from the given value set. It will looks like

-- SQLNESS TEMPLATE i=0..100
INSERT INTO t (c) VALUES {{i}};

-- SQLNESS TEMPLATE aggr={avg, count, stddev, stdvar, sum}
SELECT {{aggr}}(c) from t;

Additional Context

As @jiacai2050 mentioned in #50 (comment), crate minijinja may help to implement this

Describe This Problem

"Interceptor" is a hook point in sqlness, which allows people to write tests with extra functionality. E.g., provide per-case config, parameterized SQL or resource limitation etc.

Proposal

Some interceptable points I can think of are:

• before query execution
• after query execution
• before one .sql file start

And I plan to add one simple interceptor implementation for each point as an example, and to see if this mechanism actually works.

Additional Context

No response

          https://crates.io/crates/mysql is a popular crates, so we can go with it first.

We can add a feature gate to enable this feature.

By doing this, we can make it easier to get started with sqlness, user only needs to handle the start/stop of their services.

I think we can go further, we can provide a cli for this default server, and let user start/stop service all by themselves, users can execute this cli against their servers directly, something like

sqlness-cli -t mysql -h localhost -p 3306 --case-dir some-directory

We can implement this in two steps, first to add default database implementation, then to add this cli.

Originally posted by @jiacai2050 in #15 (comment)

Describe This Problem

In CeresDB, we use docker-compose to setup integration tests for cluster mode, and it requires some additional step to run it correctly.

In local dev, setup those steps is non-trivial, so I prefer to add an option to support run test for only one env.

Proposal

See above

Additional Context

No response

Describe this problem

When cases fail to run, sqlness return 0as the result which means success.
run_env needs to return errors up instead of logging.
Refer to

Steps to reproduce

1. Define a wrong case, run sqlness.
2. echo $? will return 0 while the correct result should be 1.

Expected behavior

No response

Additional Information

No response