clockworklabs / spacetimedb Goto Github PK

I noticed this line at the top of the readme:
"You can think of SpacetimeDB as both a database and server combined into one."

This is confusing to me, I think it should read:
"You can think of SpacetimeDB as both a database and application server combined into one."

A server could be anything! Initially I thought you wanted to replace the OS as well.

When mutating subscribed tables spacetimedb does not trigger on_insert, on_update or on_delete events when using spacetime sql or HTTP SQL API interface, it seems these events fire currently only when using rust-land functions inside server-side code.

Reproduction

Create server-side module that creates User and adds it to the table in identity_connected, when we connect, the user gets inserted and on_insert on the client-side triggers correctly, but when we call e.g spacetime sql spacetime "DELETE FROM User" the User::on_delete listener does not trigger at all but it would trigger if we deleted the record using server-side User::delete or User::delete_by_ function.

mod module_bindings;
use module_bindings::*;

const URI: &str = "http://localhost:3000";
const DB_NAME: &str = "spacetime";

use spacetimedb_sdk::{
    identity::on_connect,
    on_subscription_applied, subscribe,
    table::{TableType, TableWithPrimaryKey},
};

fn main() {
    User::on_insert(|user, _| println!("User inserted: {:?}", user));
    User::on_delete(|user, _| println!("User deleted: {:?}", user));

    on_connect(|_| subscribe(&["SELECT * FROM User;"]).expect("Subscribe failed"));

    connect(URI, DB_NAME, None).expect("Connect failed");

    loop {}
}

We need benchmarks that exercise the query engine (currently only SQL) including subscriptions.

We would like to experiment with running this in the socket runtime. I realize there is a wasm version but im interested in the high performance version of this, is there a C++ port somewhere already started?

Hi there -

A few things I noticed for the TypeScript QuickStart:

1. useRef is used in the code but not imported, I added it to the import on line 1, so it is now import React, {useEffect, useState, useRef} from "react";
2. The logo import on line 2 is not used, so I removed that line.
3. For the line : var user = User.filterByIdentity(local_identity?.current?.toUint8Array()!); It should not be converted to a Uint8Array, so the line I got working was: var user = User.filterByIdentity(local_identity?.current!);
4. I had to update the tsconfig.json file that was generated by create-react-app to say target=es6 in order to get the iterators to work. I'd suggest changing the instruction to tell folks they'll have to do that, as it doesn't appear to be possible to make it es6 as part of create-react-app.
5. For the line: var identityStr = new Identity(user.identity).toHexString(); it does need to be converted to match the constructor, so the line I got working was: var identityStr = new Identity(user.identity.toUint8Array()).toHexString();
6. For the line: var spacetimeDBClient = new SpacetimeDBClient("ws://localhost:3000", "chat", token); the text says 'replace <module_name> with the name you chose', but <module_name> isn't in the code, it's hard-set to chat. So I believe that line should be var spacetimeDBClient = new SpacetimeDBClient("ws://localhost:3000", "<module_name>", token);
7. local_identity isn't a very Typescript-y variable name, so it should probably be tweaked to localIdentity or something like that.

It's possible this Issue should be over in this repo, since thats where this example code lives, but I'm not sure.

I think that's all the tweaks I had to make to get it running. I hope that helps! 👍🏻

One general piece of feedback would be to put working versions of the whole thing into a repo somewhere so it can be copied in one go rather than pieced together. I like the explanations a lot, but some of the "go back to this function and add this line" stuff was a bit tricky. I see there is a PR on the typescript-sdk repo that tries to do this, but has extra stuff.

Cheers!

sometimes when publishing (with clear fwiw) you get this error:

WARN crates/client-api/src/routes/subscribe.rs:177: error closing: WebSocket protocol error: Sending after closing is not allowed

and after that no new connection (including logs) can be made until you clear and re-publish

update: this seems to happen when you publish with clear flag while there's a connected client (like editor window that we always have running)

device: Apple M1 Max
im following the quickstart guide (https://spacetimedb.com/docs/server-languages/rust/rust-module-quickstart-guide)

when i try to publish the module, i get this: (reduced by a lot)

error[E0425]: cannot find function, tuple struct or tuple variant `Ok` in this scope
   --> /Users/cc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gimli-0.27.3/src/read/mod.rs:676:13
    |
676 |             Ok(Register(y))
    |             ^^ not found in this scope

error[E0425]: cannot find function, tuple struct or tuple variant `Err` in this scope
   --> /Users/cc/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gimli-0.27.3/src/read/mod.rs:678:13
    |
678 |             Err(Error::UnsupportedRegister(x))
    |             ^^^ not found in this scope

Some errors have detailed explanations: E0405, E0412, E0425, E0433, E0463, E0531.
error: could not compile `gimli` (lib) due to 1892 previous errors
Error: command ["cargo", "--config=net.git-fetch-with-cli=true", "build", "--target=wasm32-unknown-unknown", "--release", "--message-format=json-render-diagnostics"] exited with code 101

im on rust 1.71, not sure how to fix

This is a mix of server/protobuf/C# client SDK issues, but I'm sticking them all here for now.

There are three cases I'm aware of:

• 1. Malformed Subscribe calls are not reported to the client at all

SpacetimeDBClient.instance.Subscribe(new List<string> { "SELECT * FROM User", "SELECT * FROM Message" });

// this will cause an unreported server error
SpacetimeDBClient.instance.Subscribe(new List<string> { "SELECT * FROM HI MY NAME IS JAMES User", "SELECT * FROM Message" });
// invoking Subscribe again will succeed, and the client will proceed as if nothing has happened
SpacetimeDBClient.instance.Subscribe(new List<string> { "SELECT * FROM User", "SELECT * FROM Message" });

2023-08-25T18:19:08.124171Z TRACE crates\client-api\src\routes\subscribe.rs:232: Received heartbeat from client ClientActorId(90cf6f2168def766e073b6bedf063aea07785fd751a625fde7dec9d5d26dde3e/0)
2023-08-25T18:19:08.265946Z ERROR crates\core\src\subscription\module_subscription_actor.rs:66: error occurred in ModuleSubscriptionActor: SqlParserError: sql parser error: Expected end of statement, found: NAME, executing: `SELECT * FROM HI MY NAME IS JAMES User`
2023-08-25T18:19:38.118225Z TRACE crates\client-api\src\routes\subscribe.rs:232: Received heartbeat from client ClientActorId(90cf6f2168def766e073b6bedf063aea07785fd751a625fde7dec9d5d26dde3e/0)

• 2. Errors in reducers are reported to the client, but not logged by default in the C# client SDK

• 3. Certain unusual errors can cause the server to close the client websocket, or prevent it from opening in the first place. In the C# SDK, this causes the client to hang with no issue of an error. Both of my repros for this are fairly contrived though.

webSocket.Send(Encoding.ASCII.GetBytes("{ \"subscribe\": { \"query_strings\": " + json + " }}"));

webSocket.Send(Encoding.ASCII.GetBytes("{ \"subscrib\": { \"query_strings\": " + json + " }}"));

2023-08-25T18:42:52.908700Z TRACE crates\client-api\src\routes\subscribe.rs:232: Received heartbeat from client ClientActorId(90cf6f2168def766e073b6bedf063aea07785fd751a625fde7dec9d5d26dde3e/0)
2023-08-25T18:42:53.012411Z DEBUG crates\client-api\src\routes\subscribe.rs:216: Client caused error on text message: 
unknown variant `subscrib`, expected one of `call`, `subscribe`, `one_off_query` at line 1 column 12
2023-08-25T18:42:53.018879Z TRACE crates\client-api\src\routes\subscribe.rs:247: Close frame Some(CloseFrame { code: Normal, reason: "" })
2023-08-25T18:42:53.020439Z DEBUG crates\client-api\src\routes\subscribe.rs:251: Client connection ended

2023-08-25T19:18:41.090945Z TRACE crates\client-api\src\auth.rs:149: Authorization rejection: Jwt(InvalidSignature)   

On further reflection, I'm not sure 3 is worth fixing. 1 and 2 are relatively easy. 3 has more to do with malformed handshakes and low-level protocol messages, which aren't things that are very user-facing.

I guess it might be worth adding a generic ProtocolError protobuf, which would work for 3A. I have no idea about 3B.

For game analytics we need a mechanism to stream the db transaction log

Apple M1 / Ventura

XCode installed
clang version 14.0.3 (clang-1403.0.22.14.1)
rustc 1.71.1 (eb26296b5 2023-08-03)

mkdir quickstart-chat
cd quickstart-chat
spacetime init --lang rust server

cd server
cargo build

error: linking with cc failed: exit status: 1
= note: LC_ALL="C" PATH="/
......
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation

Full log.txt

Apparently linking some C code together doesn't work.

What fixed the issue for me was creating a cargo config file

vi ~/.cargo/config

and adding the following:

rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]

[target.aarch64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]

Then cargo build finishes:

cargo build
...
Finished dev [unoptimized + debuginfo] target(s) in 13.67

Would be great to see this kind of stuff in a troubleshooting document or FAQ.

Remove table scoped Header field from RelValue which represents a single row in the query engine.

Derek Brinkmann:
C# modules that use Camel case are not generated in snake case. NOTE: This is working in Typescript SDK

Hi! I am looking for database witch able to work without internet connection only with local data. And with ability to sync data between local storage and server when connection return online.
Is SpacetimeDB fits?

We have this definition in crates/bindings/src/lib.rs:

    /// Erase the `R` type parameter from the token.
    ///
    /// In other words, forget what reducer this is for.
    #[inline]
    pub fn erase(self) -> ScheduleToken {
        ScheduleToken::new(self.id)
    }

It seems unused so it's unclear if it should be provided.

From the Discord:

PS G:\GitHub\spacetime_sandbox\quickstart-chat\server> spacetime call chat-server send_message '["Hello"]'        
Error: Invalid arguments provided for reducer `send_message` for database `chat-server` resolving to address `5ebd1221afb10aaf61d6c8a527413d82`.

The reducer has the following signature:
        send_message(text: String)

Caused by:
    0: Response text: invalid arguments for reducer send_message: [0]: expected value at line 1 column 2
    1: HTTP status client error (400 Bad Request) for url (http://localhost:3000/database/call/5ebd1221afb10aaf61d6c8a527413d82/send_message)

Okay, this issue is about libc / bindgen.

spacetime init --lang=rs bindgen
cd bindgen
cargo add rand
spacetime build

Fixed wasm build error with adding getrandom to cargo.toml, as stated in the manual.

getrandom = { version = "0.2", features = ["js"] }

Added the following to lib. rs.

use spacetimedb::{spacetimedb, ReducerContext, println};
use rand::Rng; // 0.8.5

#[spacetimedb(init)]
pub fn init() {
    let num = rand::thread_rng().gen_range(0..100);
    println!("{}", num);
}

spacetime build

Error: wasm-bindgen detected.

It seems like either you or a crate in your dependency tree is depending on
wasm-bindgen. wasm-bindgen is only for webassembly modules that target the web
platform, and will not work in the context of SpacetimeDB.

To find the offending dependency, run `cargo tree -i wasm-bindgen`. Try checking
its cargo features for 'js' or 'web' or 'wasm-bindgen' to see if there's a way
to disable it.

In a nutshell, rand requires libc, which is supposed to be handled by wasm-bindgen, which is not allowed in SpaceTimeDB.
That also categorically rules out most if not all crypto crates since they rely on libc for random number generation.

Conventionally when I work with a normal DB, I have to encrypt some user data for legal compliance reasons. As I understand it now, I cannot do that with the wasm modules meaning I would have to generate client bindings, encrypt elsewhere, and do basically the exact same as I would do with a normal relational DB.

Questions:

• Are there any plans to support libc / wasm-bindgen at any point?
• A lot of this has been fixed in wasm32-wasi, so is that an avenue you have considered?
• AFAIK, no_std crypto is still out of reach, so how do I encrypt my data in SpaceTimeDB?
• Is there any particular reason the libc / bindgen issue isn't mentioned in the docs?

The guys at ICP had almost an identical issue with their WASM runtime for canisters, but eventually provided encryption through the API with some weirdo keyless algorithm. Meaning, it can be solved, but none of the options is particular easy.

I understand it's early days and much of these issues result from the rather mixed WASM implementation in Rust (WASM-unknown vs WASM-wasi), but some guidance on the libc issue would be very much appreciated since the absence of cryptography has a very meaningful impact on application architecture.

I believe this is a drop in replacement to JWT but this could allow for attestations to a user/pid allowing for finer grained control using the same token strategy.

For example in the Payload of the JWT we just add a signed attestation by some trusted authority like so

{
  "iss": "did:key:theserversDID/PublicKey",
  "aud": "did:key:ourPublicKey",
  // Time bounds of token
  "nbf": 1529496683,
  "exp": 9256939505,
  "att": [
    {
      "with": "servername/DB",
      "can": "READ"
    }
  ],
  "prf": []
}

Do I plan on implementing this??

Possibly if I have time :)

I was trying to run spacetimedb on Docker in a default GitHub Codespace, and apparently the image is only built for arm64. To work on intel CPUs, the image needs to be multi-arch on Dockerhub.

$ docker run --name spacetimedb clockworklabs/spacetimedb:latest start
Unable to find image 'clockworklabs/spacetimedb:latest' locally
latest: Pulling from clockworklabs/spacetimedb
fc9ce7290e7e: Pull complete 
be081aa8c78b: Pull complete 
3ee2becf8478: Pull complete 
Digest: sha256:3415fd15cdcbc25e36ee0ca4c9a7dd0af949629a3fb18818885e275e844427f6
Status: Downloaded newer image for clockworklabs/spacetimedb:latest
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested

$ docker logs spacetimedb
exec /usr/local/bin/spacetimedb: exec format error

On the landing page it says:

The reason for the name SpacetimeDB is that we store a full-transaction history for your application. This lets you reset the state of your database to any previous point in time and replay transactions from there. This means you get replays for free. It's basically time-travel, and it's very cool.

I'm quite curious about these features, but I haven't seen them mentioned in the docs. How can I found out more about them? I do hope you'll document these soon! 🙏

We current use Keccak256 for computing a product value's DataKey. This can account for over 90% of the cost of generating the DataKey in some cases, the other portion being the bsatn encoding. Note this hash is invoked once for each row, both on the write path as well as the read path.

Is a cryptographic hash necessary for identifying rows? Can we use something cheaper?

Clients can lose connection to the server for a variety of reasons. We need to deal with this behavior in client SDKs.
Things we may want:

• Log in console when connection is lost.
• Add hooks to let users add custom behavior when connection is lost.
• Possibly perform some sort of reconciliation with old state when the server sends over all the new state.
  • Alternatively, let the client tell the server "i got everything up to here" and server fills in the rest. This seems expensive, but it means clients don't need to worry about reconciliation, everything just goes through the on-update logic they already coded.
• Auto-reconnect logic with exponential backoff (on by default?)

Client SDKs:

• Rust
• C#
• Typescript
• Python

To ensure consistent expectations BSL 1.1 enforced limitations on eg Change Date and Change License.

First, we set a cap of four years for the duration of the time window prior to code becoming FOSS. That said, we encourage a shorter window (and for MariaDB MaxScale, the window is between two and three years).

Current change date seems to be 5 years into the future: https://github.com/clockworklabs/SpacetimeDB/blame/8eb1743b75435c7b3b9ea7179c923262105b2b2b/LICENSE.txt#L24

One year more than the max allowed under BSL 1.1

Second, we now require the Change License to be GPL compatible with either GPLv2, GPLv3, or any other license that can be combined properly with GPL software

AGPL 3.0 is set as the change license: https://github.com/clockworklabs/SpacetimeDB/blame/8eb1743b75435c7b3b9ea7179c923262105b2b2b/LICENSE.txt#L26

Though AGPL 3.0 is not a GPL 2.0 compatible license: https://www.gnu.org/licenses/license-list.en.html#AGPLv3.0

Hence violators this part of the BSL 1.1:

software provided under the Change License can be included in a program with software provided under GPL Version 2.0 or a later version

On top of this, you are using a modified AGPL 3.0 license even, which I would highly discourage, as modified licenses by their very nature are not OSI approved and sets unclear expectations and has unclear compatibility with other licenses.

To be able to build the analytics we need for the game we need additional information to be available in the transaction log. Currently only db operations are stored.
The bare minimum extra information we would need is some way of identifying which reducer ran, and the timestamp of the transaction.
Reducer parameters would also be useful but not absolutely necessary.

Edit. Reducer parameters would actually be important currently because they contain the actor which is needed to know which player called the reducer.

In release 0.6.1, the server starts on port 80, not 3000 as specified in the instructions

• Networking
  • Connect / Disconnect
  • Errors
  • ingress/egress bytes + messages
• Client Cache
  • Memory consumption per table

On M1 Mac with Ventura (13.4)

brew install clockworklabs/tap/spacetime

Results in:

Error: spacetime: SHA256 mismatch
Expected: 64534d142f6618c7bb6be9a0b19f7b7185069cb842afa0df086d1e2fcf7e7f29
Actual: 40c553be7246d709a08ccf07a20e73db2503127853c256b3de9fff4a95062db2

Looks like somehow the binary was updated without updating the hash.

Any idea?

The current PrimaryKey type is just a wrapper for DataKey which uniquely identifies each row in the database. It is not a column or row attribute and therefore does correspond to a traditional primary key in a relational database. We should remove to avoid confusion.

• Allow the connection to be managed explicitly, such that a node can attempt to reconnect in case of errors. (#235)
• It should also be possible to configure the underlying websocket connection (e.g. frame size, message size).

Currently we have an iterator for equality index scans supporting equality predicates, i.e point queries. We need a similar iterator to support range predicates.

The protocol should be embedded in the URI argument

Using the SpacetimeDB CLI v0.6.1 on Windows, following https://spacetimedb.com/docs/spacetimedb-cloud.

It instructs the user to execute

spacetime server set 'https://testnet.spacetimedb.com'

but this returns Error: Invalid protocol: 'https

The user needs to execute it without the quotes for it to work:

spacetime server set https://testnet.spacetimedb.com

Depends on #296.

Incorporate storage level index range iterators into query execution.

All queries for a particular subscription are evaluated within a single thread of execution. An orders of magnitude speedup can be achieved using a multithreaded execution model. Not this will require some refactoring of the datastore as currently a query acquires a global exclusive lock in order to run.

See

Hi!

There are a lot of examples of different software, where PGO helps a lot with performance - you can check it here. E.g. in this list are a lot of databases like Redis, MongoDB, and PostgreSQL.

We need to check the PGO effects on SpacetimeDB, and if it works well - write a note in the SpacetimeDB documentation about PGO. I'd appreciate your providing an easy way to build SpacetimeDB with PGO. And experienced users will be able to do it on their own for their own usage scenarios. Another option is to optimize SpacetimeDB build with a generic-enough profile. Providing PGO-optimized binaries could be a trickier task (since it requires preparing a good-enough profile) but as an option would be great to see too. Also, you even could optimize distributed from the official sites binaries with PGO (if would be able to collect a generic-enough profile).

As an additional optimization, I suggest looking at LLVM BOLT. But from my experience, starting with PGO and then using BOLT would be better.

For the Rust projects, I recommend starting with cargo-pgo.

I understand that the project is still in the early development stages. So I think the issue is just an improvement idea for future developments.

Tyler: We should use the Apache 2.0 license

Queries and subscriptions currently default to full table scans. We should incorporate index scans into query planning and utilize them when appropriate.

@jdetter

Hello, a quick question: how does this scale on multiple nodes? is there a cluster mode? I couldn't find any info on that.

Trying to install spacetime on Linux, I ran into missing dependencies.

My steps were:

1. Create an empty codespace
2. Run curl --proto '=https' --tlsv1.2 -sSf https://install.spacetimedb.com | sudo sh in the terminal
3. Run spacetime

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3435  100  3435    0     0   9703      0 --:--:-- --:--:-- --:--:--  9703
This script will install the spacetimedb-cli command line tool. platform=linux-amd64 os=linux
Press [enter] to install the spacetimedb-cli binary. Use Ctrl-C now to exit.


Downloading from https://install.spacetimedb.com...
Extracting...
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
Default: /usr/bin/spacetime
If you do not have sudo access for your computer, we recommend using /home/root/bin/spacetime
Press enter or provide the full install path you would like to use: 
copied '/tmp/tmp.ymLB5CTR6I/spacetime' -> '/usr/bin/spacetime'
removed '/tmp/tmp.ymLB5CTR6I/spacetime'

spacetime is installed into /usr/bin/spacetime Note: we recommend making sure that this executable is in your PATH.
The install process is complete, head over to our quickstart guide to get started!

        https://spacetimedb.com/docs/quick-start

codespace@codespaces-7030e8:/workspaces/spacetime:(main *)$ spacetime 
spacetime: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by spacetime)
spacetime: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by spacetime)
spacetime: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by spacetime)

How to deploy distributed SpacetimeDB on the server side.

I've just tried your demo's to get a feel for your system, and I ran into a few issues (running WSL2, Ubuntu 22.04, node.js 18.17). Perhaps you can update the documentation.

Rust server

• I had a fresh install of WSL2, and needed to install pkg-config in order to get the example to compile (Open SSL issue): sudo apt install pkg-config.
• When publishing the server module, I received an authorization error, indicating that my current identity was not authorized to publish. I needed to remove all identities first using: spacetime identity remove --all --force
• I could not find any documentation to delete existing modules (which is sometimes needed, e.g. when you publish the default Rust server project, and want to later update it with the example server - the Person schema's are not compatible). spacetime delete <module-name>

Rust client

• Worked smoothly: I would probably start with the connection to the DB, but that's just my preference. And it would be nice if the code would allow me to set my name, for example when starting the app.

Typescript client:

• There is another OpenSSL issue: in this case, I needed to update the react-scripts: npm i react-scripts@latest.
• There were 3 code errors related to the user identity type, which needed to be updated to Uint8Array: user.identity.toUint8Array()
• The tsconfig.json needed to be updated: target: ESnext.
• The default npm start command starts the client on port 3000, which is a bit annoying, since spacetime is using the same port.
• At the end of the react example, there is a link to the full example source code Congratulations! You've built a simple chat app with SpacetimeDB. You can find the full source code for this app [here](https://github.com/clockworklabs/spacetimedb-typescript-sdk/tree/main/examples/quickstart). However, that page no longer exists.
• When I set my name, the system displays 'name has connected' 15 times instead of once.

Currently subscriptions compute the RowID for each row returned in the result set. This can account for upwards of 50% of the execution time.

Table access methods should return the RowIDs along with the rows themselves which would minimize the need for queries and subscriptions to recompute them.