Code Monkey home page Code Monkey logo

Comments (12)

haaawk avatar haaawk commented on September 27, 2024 1

It seems that the lock in ProxyService::execute is being held during the execution of the query even though it is just protecting the creation of the Libsql connection before the execution of the query. This could lead to a situation when a transaction keeps lock but is block by another transaction and then the other transaction can't proceed because the first transaction keeps the lock.

Is this lock only held during execution of a query on an embedded replica connections? Since, txns on remote connections are working fine. Or, is the ProxyService only used for embedded replicas and not remote connections?

I believe ProxyService is only used for embedded replicas.
Here's a fix: #1339

from libsql.

haaawk avatar haaawk commented on September 27, 2024

A bit simplified reproducer:

use libsql::{Builder, Connection, Result};

#[tokio::main]
async fn main() {
    let db_url = "http://localhost:8080";
    let replica = Builder::new_remote_replica(
        "/tmp/embedded_transaction.db",
        db_url.to_string(),
        String::new(),
    )
    .build()
    .await
    .unwrap();
    let replica_conn_2 = replica.connect().unwrap();
    let replica_conn_3 = replica.connect().unwrap();

    setup_db(replica_conn_2.clone()).await.unwrap();

    // If we execute concurrently, two tasks
    let replica_task_2 = db_work(replica_conn_2);
    let replica_task_3 = db_work(replica_conn_3);

    let (replica_task_2_res, replica_task_3_res) = tokio::join!(replica_task_2, replica_task_3);

    if replica_task_2_res.is_err() {
        eprintln!("Task 2 failed: {:?}", replica_task_2_res);
    }
    if replica_task_3_res.is_err() {
        eprintln!("Task 3 failed: {:?}", replica_task_3_res);
    }

    // One of these concurrent tasks fail currently. Both tasks should succeed.
    assert!(replica_task_2_res.is_ok());
    assert!(replica_task_3_res.is_ok());
}

async fn db_work(conn: Connection) -> Result<()> {
    let tx = conn.transaction().await?;
    // Some business logic here...
    tx.execute("INSERT INTO test (name) VALUES (?1)", ["somename"])
        .await?;
    tx.commit().await?;
    Ok(())
}

async fn setup_db(conn: Connection) -> Result<()> {
    conn.execute(
        "CREATE TABLE IF NOT EXISTS test (id INTEGER PRIMARY KEY, name TEXT)",
        (),
    )
    .await?;
    Ok(())
}

from libsql.

haaawk avatar haaawk commented on September 27, 2024

Works just fine without embedded replica:

let replica = Builder::new_remote(
        db_url.to_string(),
        String::new(),
    )

from libsql.

haaawk avatar haaawk commented on September 27, 2024

It seems that one transaction hangs on commit while the other hangs on execute.

from libsql.

haaawk avatar haaawk commented on September 27, 2024

One issue it might be is read_your_writes.
The first transaction is waiting for embedded replica to sync up to the write the transaction did and this is why commit takes that long.
The second transaction is blocked by the first.

One plot I got was:

Starting db work
Starting db work
Executing db work
Executing db work
Committing db work
Task 2 failed: Err(RemoteSqliteFailure(2, 0, "Transaction timed-out"))
Task 3 failed: Err(Replication(Client(Status { code: Unknown, message: "connection closed before message completed", source: Some(hyper::Error(IncompleteMessage)) })))
thread 'main' panicked at src/main.rs:33:5:
assertion failed: replica_task_2_res.is_ok()

from libsql.

haaawk avatar haaawk commented on September 27, 2024

Building the db with .read_your_writes(false) should help for read your writes issue but there is apparently another issue as this does not make the code to finish successfully.

let replica = Builder::new_remote_replica(
        "/tmp/embedded_transaction.db",
        db_url.to_string(),
        String::new(),
    )
    .read_your_writes(false)
    .build()
    .await
    .unwrap();

from libsql.

haaawk avatar haaawk commented on September 27, 2024

Here's a deterministic reproducer:

    eprintln!("Start tx1");
    let tx2 = replica_conn_2.transaction().await.unwrap();
    eprintln!("Start tx2");
    let tx3 = replica_conn_3.transaction().await.unwrap();
    eprintln!("Execute tx1");
    tx2.execute("INSERT INTO test (name) VALUES (?1)", ["somename"])
        .await.unwrap();
    eprintln!("Execute tx2");
    tx3.execute("INSERT INTO test (name) VALUES (?1)", ["somename"])
        .await.unwrap();
    eprintln!("Commit tx1");
    tx2.commit().await.unwrap();
    eprintln!("Commit tx2");
    tx3.commit().await.unwrap();
    eprintln!("Done");

Start tx1
Start tx2
Execute tx1
Execute tx2
Commit tx1
thread 'main' panicked at src/main.rs:44:24:
called `Result::unwrap()` on an `Err` value: RemoteSqliteFailure(2, 0, "Transaction timed-out")

from libsql.

haaawk avatar haaawk commented on September 27, 2024

Here's a deterministic reproducer:

    eprintln!("Start tx1");
    let tx2 = replica_conn_2.transaction().await.unwrap();
    eprintln!("Start tx2");
    let tx3 = replica_conn_3.transaction().await.unwrap();
    eprintln!("Execute tx1");
    tx2.execute("INSERT INTO test (name) VALUES (?1)", ["somename"])
        .await.unwrap();
    eprintln!("Execute tx2");
    tx3.execute("INSERT INTO test (name) VALUES (?1)", ["somename"])
        .await.unwrap();
    eprintln!("Commit tx1");
    tx2.commit().await.unwrap();
    eprintln!("Commit tx2");
    tx3.commit().await.unwrap();
    eprintln!("Done");

Start tx1
Start tx2
Execute tx1
Execute tx2
Commit tx1
thread 'main' panicked at src/main.rs:44:24:
called `Result::unwrap()` on an `Err` value: RemoteSqliteFailure(2, 0, "Transaction timed-out")

Ok this is a bit too simplistic because Execute tx2 has to wait for Commit tx1 so it's a deadlock

from libsql.

haaawk avatar haaawk commented on September 27, 2024

Deterministic reproducer this time without deadlock:

    eprintln!("Start tx1");
    let before = Instant::now();
    let tx1 = replica_conn_2.transaction().await.unwrap();
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    let before = Instant::now();
    eprintln!("Start tx2");
    let tx2 = replica_conn_3.transaction().await.unwrap();
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    let before = Instant::now();
    eprintln!("Execute tx1");
    tx1.execute("INSERT INTO test (name) VALUES (?1)", ["somename"])
        .await.unwrap();
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    let before = Instant::now();
    eprintln!("Execute tx2 and Commit tx1 concurrently");
    let exec_fut = tx2.execute("INSERT INTO test (name) VALUES (?1)", ["somename"]);
    let commit_fut = tx1.commit();
    let (exec, commit) = join!(
        exec_fut,
        commit_fut);
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    exec.unwrap();
    commit.unwrap();
    let before = Instant::now();
    eprintln!("Commit tx2");
    tx2.commit().await.unwrap();
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    eprintln!("Done");

Start tx1
Finished in 17.477393ms
Start tx2
Finished in 22.61203ms
Execute tx1
Finished in 25.201501ms
Execute tx2 and Commit tx1 concurrently
Finished in 5.368149362s
thread 'main' panicked at src/main.rs:63:12:
called `Result::unwrap()` on an `Err` value: RemoteSqliteFailure(2, 0, "Transaction timed-out")

from libsql.

haaawk avatar haaawk commented on September 27, 2024

After adding more logging it seems that exec for tx2 finishes before commit for tx1 so it seems that exec for tx2 blocks commit for tx1.

    eprintln!("Start tx1");
    let before = Instant::now();
    let tx1 = replica_conn_2.transaction().await.unwrap();
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    let before = Instant::now();
    eprintln!("Start tx2");
    let tx2 = replica_conn_3.transaction().await.unwrap();
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    let before = Instant::now();
    eprintln!("Execute tx1");
    tx1.execute("INSERT INTO test (name) VALUES (?1)", ["somename"])
        .await.unwrap();
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    let before = Instant::now();
    eprintln!("Execute tx2 and Commit tx1 concurrently");
    let exec_fut = async {
        let res = tx2.execute("INSERT INTO test (name) VALUES (?1)", ["somename"]).await;
        let after = Instant::now();
        eprintln!("Exec finished in {:?}", after.duration_since(before));
        res
    };
    let commit_fut = async {
        let res = tx1.commit().await;
        let after = Instant::now();
        eprintln!("Commit finished in {:?}", after.duration_since(before));
        res
    };
    let (exec, commit) = join!(
        exec_fut,
        commit_fut);
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    exec.unwrap();
    commit.unwrap();
    let before = Instant::now();
    eprintln!("Commit tx2");
    tx2.commit().await.unwrap();
    let after = Instant::now();
    eprintln!("Finished in {:?}", after.duration_since(before));
    eprintln!("Done");

Start tx1
Finished in 20.555148ms
Start tx2
Finished in 17.230765ms
Execute tx1
Finished in 20.49774ms
Execute tx2 and Commit tx1 concurrently
Exec finished in 5.005591167s
Commit finished in 5.0059767s
Finished in 5.005993392s
thread 'main' panicked at src/main.rs:73:12:
called `Result::unwrap()` on an `Err` value: RemoteSqliteFailure(2, 0, "Transaction timed-out")

from libsql.

haaawk avatar haaawk commented on September 27, 2024

It seems that the lock in ProxyService::execute is being held during the execution of the query even though it is just protecting the creation of the Libsql connection before the execution of the query. This could lead to a situation when a transaction keeps lock but is block by another transaction and then the other transaction can't proceed because the first transaction keeps the lock.

from libsql.

sveltespot avatar sveltespot commented on September 27, 2024

It seems that the lock in ProxyService::execute is being held during the execution of the query even though it is just protecting the creation of the Libsql connection before the execution of the query. This could lead to a situation when a transaction keeps lock but is block by another transaction and then the other transaction can't proceed because the first transaction keeps the lock.

Is this lock only held during execution of a query on an embedded replica connections? Since, txns on remote connections are working fine. Or, is the ProxyService only used for embedded replicas and not remote connections?

from libsql.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.