artorias1024 / readyset Goto Github PK

ReadySet is a SQL caching engine designed to help developers enhance the performance and scalability of their existing database-backed applications.

• What is ReadySet?
• Quickstart
• Documentation
• Join the Community
• Development
• License

ReadySet is a lightweight SQL caching engine that sits between your application and database and turns even the most complex SQL reads into lightning-fast lookups. Unlike other caching solutions, ReadySet keeps the cache up-to-date automatically and requires no changes to your application code.

Based on years of dataflow research at MIT¹, ReadySet stores the results of queries in-memory and automatically keeps these results up-to-date as the underlying database changes. ReadySet can do this automatically because it listens to your database's replication stream.

This means:

• No extra code to keep your cache and database in sync
• No extra code to evict stale records
• No TTLs to set - your cache is as up-to-date as your replication lag

ReadySet is wire-compatible with MySQL and Postgres.

These instruction show you how to run ReadySet against a new Postgres or MySQL database in Docker. To use an existing database, see the Quickstart page in our docs. For a full walk-through of ReadySet's capabilities and features, see the ReadySet Tutorial.

• Install and start Docker for your OS.
• Install the psql shell or the mysql shell.

Create a Docker container and start the database inside it.

For Postgres:

docker run -d \
--name=postgres \
--publish=5432:5432 \
-e POSTGRES_PASSWORD=readyset \
-e POSTGRES_DB=testdb \
postgres:14 \
-c wal_level=logical

For MySQL:

docker run -d \
--name=mysql \
--publish=3306:3306 \
-e MYSQL_ROOT_PASSWORD=readyset \
-e MYSQL_DATABASE=testdb \
mysql

Download a sample data file and load it into the database via the SQL shell.

For Postgres:

curl -O https://raw.githubusercontent.com/readysettech/docs/main/docs/assets/quickstart-data-postgres.sql

PGPASSWORD=readyset psql \
--host=127.0.0.1 \
--port=5432 \
--username=postgres \
--dbname=testdb

\i quickstart-data-postgres.sql

\q

For MySQL:

curl -O https://raw.githubusercontent.com/readysettech/docs/main/docs/assets/quickstart-data-mysql.sql

mysql \
--host=127.0.0.1 \
--port=3306 \
--user=root \
--password=readyset \
--database=testdb

source quickstart-data-mysql.sql;

\q

Create a Docker container and start ReadySet inside it, connecting ReadySet to the database via the connection string in --upstream-db-url:

For Postgres:

docker run -d \
--name=readyset \
--publish=5433:5433 \
--platform=linux/amd64 \
--volume='readyset:/state' \
--pull=always \
-e DEPLOYMENT_ENV=quickstart_github \
public.ecr.aws/readyset/readyset:beta-2023-01-18 \
--standalone \
--deployment='github-postgres' \
--database-type=postgresql \
--upstream-db-url=postgresql://postgres:[email protected]:5432/testdb \
--address=0.0.0.0:5433 \
--username='postgres' \
--password='readyset' \
--db-dir='/state'

For MySQL:

docker run -d \
--name=readyset \
--publish=3307:3307 \
--platform=linux/amd64 \
--volume='readyset:/state' \
--pull=always \
-e DEPLOYMENT_ENV=quickstart_github \
public.ecr.aws/readyset/readyset:beta-2023-01-18 \
--standalone \
--deployment='github-mysql' \
--database-type=mysql \
--upstream-db-url=mysql://root:[email protected]:3306/testdb \
--address=0.0.0.0:3307 \
--username='root' \
--password='readyset' \
--db-dir='/state'

• Connect an app

  Once you have a ReadySet instance up and running, the next step is to connect your application by swapping out your database connection string to point to ReadySet instead. The specifics of how to do this vary by database client library, ORM, and programming language. See Connect an App for examples.

  Note: By default, ReadySet will proxy all queries to the database, so changing your app to connect to ReadySet should not impact performance. You will explicitly tell ReadySet which queries to cache.

• Cache queries

  Once you are running queries against ReadySet, connect a database SQL shell to ReadySet and use the custom SHOW PROXIED QUERIES SQL command to view the queries that ReadySet has proxied to your upstream database and identify which queries are supported by ReadySet. Then use the custom CREATE CACHE SQL command to cache supported queries.

  Note: To successfully cache the results of a query, ReadySet must support the SQL features and syntax in the query. For more details, see SQL Support.

• Tear down

  When you are done testing, stop and remove the Docker resources:

  docker rm -f readyset postgres mysql \
  && docker volume rm readyset

• Deploy to production

  To run ReadySet in a cloud environment, see the Deploy with ReadySet Cloud and Deploy with Kubernetes pages in our docs. You can also run ReadySet yourself using binaries.

Q: How does ReadySet work under the hood?

A: The heart of ReadySet is a query engine based on partially-stateful, streaming data flow. To learn more about how it works, see our docs.

Q: How does ReadySet keep cached state up to date?

A: ReadySet receives updates about data changes from your backing database via binlog replication and uses those updates to automatically update its internal state.

Q: Do I have to send all of my database traffic to ReadySet?

A: You can if you want to, but it’s not required. You can manually route a subset of your traffic to ReadySet (as you would with a traditional database read replica), or you can send all of it to ReadySet. It's important to note that not all of the queries sent to ReadySet need to be cached– you have fine-grained control over what is cached vs. what is proxied.

For more information, check out our full docs site here.

For questions or support, join us on the ReadySet Community Discord, post questions on our Github forum, or schedule an office hours chat with our team.

Everyone is welcome!

This section is for developers who want to build ReadySet from source as they work on the ReadySet codebase.

1. Install ReadySet dependencies.

   macOS with homebrew:

   brew install lz4 [email protected] rocksdb

   Add the following to your cargo config to make it discover lz4:

   [env]
   LIBRARY_PATH = "/opt/homebrew/lib"
   ROCKSDB_LIB_DIR = "/opt/homebrew/lib"

   Ubuntu:

   sudo apt update && sudo apt install -y build-essential libssl-dev pkg-config llvm clang liblz4-dev librocksdb-dev
   sudo apt-get -y install cmake

   Arch:

   sudo pacman -S base-devel clang lz4 rocksdb-static

   CentOS/Amazon Linux:

   sudo yum -y update
   sudo yum -y groupinstall "Development Tools"
   sudo yum -y install clang lz4-devel openssl-devel

   Nix:

   nix-shell

2. Install Rust via rustup.rs.

   ReadySet is written entirely in Rust.

3. Install Docker via Get Docker and Docker Compose via Install Docker Compose.

   ReadySet runs alongside a backing Postgres or MySQL database and, when run in distributed fashion, uses Consul for leader election, failure detection, and internal cluster state management. You'll use Docker Compose to create and manage these resources for local development.

1. Clone the repo using git and navigate into it:

   git clone https://github.com/readysettech/readyset.git
   cd readyset

2. Start a backing database:

   docker-compose up -d

   This starts both Postgres and MySQL in containers, although you will run ReadySet against only one at a time. If you don't want to run both databases, edit docker-compose.yml and comment out the mysql or postgres fields.

3. Compile and run ReadySet, replacing <deployment name> with a unique identifier for the deployment.

   Run against Postgres:

   cargo run --bin readyset --release -- --standalone --database-type=postgresql --upstream-db-url=postgresql://postgres:[email protected]:5432/testdb --username=postgres --password=readyset --address=0.0.0.0:5433 --deployment=<deployment name> --prometheus-metrics --query-log --query-log-ad-hoc

   Run against MySQL:

   cargo run --bin readyset --release -- --standalone --database-type=mysql --upstream-db-url=mysql://root:[email protected]:3306/testdb --username=root --password=readyset --address=0.0.0.0:3307 --deployment=<deployment name> --prometheus-metrics --query-log --query-log-ad-hoc

   This runs the ReadySet Server and Adapter as a single process, a simple, standard way to run ReadySet that is also the easiest approach when developing. For production deployments, however, it is possible to run the Server and Adapter as separate processes. See the scale out deployment pattern in the docs.

   For details about ReadySet options, see the CLI docs (coming soon).

4. With ReadySet up and running, you can now connect the Postgres or MySQL shell:

   Postgres:

   PGPASSWORD=readyset psql
   --host=127.0.0.1 \
   --port=5433 \
   --username=postgres \
   --dbname=testdb

   MySQL:

   mysql \
   --host=127.0.0.1 \
   --port=3307 \
   --user=root \
   --password=readyset \
   --database=testdb

To run tests for the project, run the following command:

cargo test --skip integration_serial

Note:

• Certain tests cannot be run in parallel with others, and these tests are typically in files postfixed with _serial. Running the entire set of tests for a package (e.g., cargo test -p readyset-server) may fail if serial tests are included.

• Running tests may require increasing file descriptor limits. You can do so by running ulimit -Sn 65535.

When running ReadySet in a performance-critical setting, make sure you compile with the --release flag.

Note: This repository contains a snapshot of the current state of the ReadySet internal repository. Our team is currently working on getting our live codebase, including all development history, ready to be hosted on GitHub, at which point we'll force push to this repository.

ReadySet is licensed under the BSL 1.1 license, converting to the open-source Apache 2.0 license after 4 years. It is free to use on any number of nodes. The ReadySet team is hard at work getting the codebase ready to be hosted on Github.

ReadySet is also available as a paid cloud service that can be deployed to your accounts or hosted entirely by ReadySet. You can get early access to our cloud product by signing up here.