backtrace-labs / verneuil Goto Github PK

View Code? Open in Web Editor NEW

385.0 385.0 16.0 3.61 MB

Verneuil is a VFS extension for SQLite that asynchronously replicates databases to S3-compatible blob stores.

License: MIT License

Rust 4.09% C 95.87% Shell 0.04%

s3 sqlite

verneuil's People

Contributors

Stargazers

Watchers

Forkers

doytsujin fujohnwang cxz syllogy warmchang laplacekorea adfernandes hiboyang paulfitz frew

verneuil's Issues

Audit the public interface for usage of the internal runtime

In #15, we observed an issue with the public interface's transitive use of Verneuil's internal async runtime. I don't believe there's any left (everything else just pushes work to the internal threads or uses fresh worker threads), but let's make extra sure.

Consider a different/dedicated type for chunk ids

Sqlite tops out at 2^31 pages, and 64K is its (current) maximum page size. Simply using u32 for the chunk index type would prevent some type confusion errors between offsets in a file and chunk index.

Operations on the chunk cache are too fine grained

We don't expose the cache's lock (fine), so we should expose atomic higher-level operations like fetch-or-create.

Bundle high-churn chunks with the manifest

The first chunk (which contains sqlite's first page, and thus starts with a header that includes a file change counter) is unlikely to see meaningful deduplication from content addressing. It also tends to be highly compressible (it holds the sqlite schema btree, which is usually small and mostly consists of SQL text).

We should bundle it with the manifest.

Check for NAME_MAX

Verneuil mangles long paths into long filenames, without checking for NAME_MAX. We should make sure filenames and directory names don't exceed 255 bytes.

It's not clear if we can simply lower the VFS's path size limit in sqlite: the default unix VFS"s limit is 512 bytes, and:

going lower would break compatibility
mangling can increases a database's size
in one use case, we concatenate the hostname to the database file's path

It probably makes sense to detect long mangled paths, and replace their middle characters with a hash of the filename. This would be seamless for mangled paths <= 254 bytes. Mangled paths >= 255 bytes would be truncated down to 255, and still unique, thanks to the hash.

We'll solve the hostname problem differently... potentially by making copiers interact more deeply with manifest protos.

Lack of sqlite flush on shutdown

When running a program that has a oneshot mode rather then an ongoing daemon process, due to the model verneuil operates under (wrt lag in replication of data) we may see data loss at the end of a program, this will generally only hit short running programs but may effect long running ones.

As a work around we can force on program end the usage of verneuil::copy_all_spool_paths to force an upload of any locally replicated data that has not made it to the remote.

Issues running verneuil operations within a async runtime

As noted in the title, when invoking verneuil operations other then the main verneuil async replication loop inside a async runtime your program will error at runtime with:

thread 'main' panicked at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/runtime/enter.rs:39:9
stack backtrace:
   0:     0x5608fe2ebff0 - std::backtrace_rs::backtrace::libunwind::trace::h706b838f5bbd876b
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:     0x5608fe2ebff0 - std::backtrace_rs::backtrace::trace_unsynchronized::hc15f29ae7822b7b8
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x5608fe2ebff0 - std::sys_common::backtrace::_print_fmt::h7c580c971f91926c
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x5608fe2ebff0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h2a772198c4032452
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x5608fe310b7c - core::fmt::write::h9a6d9c74526a6c1b
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/core/src/fmt/mod.rs:1115:17
   5:     0x5608fe2e4c65 - std::io::Write::write_fmt::h00f38d9eb2c32e02
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/io/mod.rs:1663:15
   6:     0x5608fe2ee53b - std::sys_common::backtrace::_print::h5b3c171e6f864ae4
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x5608fe2ee53b - std::sys_common::backtrace::print::h93b9e9ed2a98e611
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x5608fe2ee53b - std::panicking::default_hook::{{closure}}::hd8da92bb68d520c5
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panicking.rs:208:50
   9:     0x5608fe2ee011 - std::panicking::default_hook::hdbc8b2951c5afbab
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panicking.rs:225:9
  10:     0x5608fe2eec04 - std::panicking::rust_panic_with_hook::h8a4c841655926f4e
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panicking.rs:622:17
  11:     0x5608fe2d02e7 - std::panicking::begin_panic::{{closure}}::hb0d33eb2055508d2
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panicking.rs:542:9
  12:     0x5608fe2d02c0 - std::sys_common::backtrace::__rust_end_short_backtrace::hd3b4c97b5a7cad4c
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/sys_common/backtrace.rs:141:18
  13:     0x5608fde1800e - std::panicking::begin_panic::hba99d266457f13fb
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panicking.rs:541:12
  14:     0x5608fe2b7b69 - tokio::runtime::enter::enter::h9f1b3f7a8528ef6d
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/runtime/enter.rs:39:9
  15:     0x5608fde81ab9 - tokio::runtime::handle::Handle::block_on::h83d62ebbad7b513c
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/runtime/handle.rs:352:34
  16:     0x5608fdee64b1 - verneuil::tracker::Tracker::flush_spooled_data::hb8ee1dce4a49165a
                               at /home/noah/code/reddit_exporter/rust/verneuil/src/tracker/mod.rs:384:13
  17:     0x5608fdecbea7 - verneuil__file_flush_replication_data
                               at /home/noah/code/reddit_exporter/rust/verneuil/src/vfs_ops.rs:88:11
  18:     0x5608fdf085e8 - linux_file_control
                               at /home/noah/code/reddit_exporter/rust/verneuil/c/vfs.c:2150:36
  19:     0x5608fdf085e8 - linux_file_control
                               at /home/noah/code/reddit_exporter/rust/verneuil/c/vfs.c:2107:1
  20:     0x7f458b082da0 - sqlite3_file_control
  21:     0x7f458b1161a1 - <unknown>
  22:     0x7f458b11a9b3 - <unknown>
  23:     0x7f458b11f809 - <unknown>
  24:     0x7f458b11fbef - <unknown>
  25:     0x7f458b11fcea - sqlite3_prepare_v2
  26:     0x5608fe126be0 - rusqlite::inner_connection::InnerConnection::prepare::h8dbdc4b6e92dad97
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/rusqlite-0.26.3/src/inner_connection.rs:247:17
  27:     0x5608fde28950 - rusqlite::Connection::prepare::hfcb7f155e657f120
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/rusqlite-0.26.3/src/lib.rs:711:9
  28:     0x5608fde28950 - rusqlite::pragma::<impl rusqlite::Connection>::pragma::h7c49312f1339a295
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/rusqlite-0.26.3/src/pragma.rs:234:24
  29:     0x5608fde2ac77 - reddit_exporter::run::{{closure}}::h0b08dcc4bebf950f
                               at /home/noah/code/reddit_exporter/rust/reddit_exporter/src/lib.rs:84:9
  30:     0x5608fde2ac77 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h23826ff10c1568e1
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/core/src/future/mod.rs:80:19
  31:     0x5608fde2ac77 - reddit_exporter::main::{{closure}}::h4a70bb1ad8b599de
                               at /home/noah/code/reddit_exporter/rust/reddit_exporter/src/main.rs:5:5
  32:     0x5608fde2ac77 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h0c01af9e689a1fba
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/core/src/future/mod.rs:80:19
  33:     0x5608fde271df - tokio::park::thread::CachedParkThread::block_on::{{closure}}::hbeb1622ff3aac55c
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/park/thread.rs:263:54
  34:     0x5608fde271df - tokio::coop::with_budget::{{closure}}::h5bfc9a4967bb24f0
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/coop.rs:102:9
  35:     0x5608fde271df - std::thread::local::LocalKey<T>::try_with::h0bc854f57be45633
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/thread/local.rs:399:16
  36:     0x5608fde271df - std::thread::local::LocalKey<T>::with::hbcf97c3248396ffc
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/thread/local.rs:375:9
  37:     0x5608fde29c5e - tokio::coop::with_budget::hae6ca180114e01a9
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/coop.rs:95:5
  38:     0x5608fde29c5e - tokio::coop::budget::h2509df568e464e85
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/coop.rs:72:5
  39:     0x5608fde29c5e - tokio::park::thread::CachedParkThread::block_on::h12239001b61cbb4f
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/park/thread.rs:263:31
  40:     0x5608fde30fdf - tokio::runtime::enter::Enter::block_on::h6fa1fb9e9d8e58e2
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/runtime/enter.rs:151:13
  41:     0x5608fde30fdf - tokio::runtime::thread_pool::ThreadPool::block_on::ha300f1a646b9eff9
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/runtime/thread_pool/mod.rs:73:9
  42:     0x5608fde27725 - tokio::runtime::Runtime::block_on::h68b28e80d59b6be7
                               at /home/noah/.cargo/registry/src/localhost-20d513fd1fb2345e/tokio-1.17.0/src/runtime/mod.rs:477:43
  43:     0x5608fde27725 - reddit_exporter::main::h1bfb50ff936ba5b0
                               at /home/noah/code/reddit_exporter/rust/reddit_exporter/src/main.rs:6:5
  44:     0x5608fde28400 - core::ops::function::FnOnce::call_once::hbfe679e2efad3469
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/core/src/ops/function.rs:227:5
  45:     0x5608fde28400 - std::sys_common::backtrace::__rust_begin_short_backtrace::hd96ca07378af8c4f
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/sys_common/backtrace.rs:125:18
  46:     0x5608fde2f7fd - std::rt::lang_start::{{closure}}::h8b17894e59eb2d0d
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/rt.rs:63:18
  47:     0x5608fe2ef20a - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hcbe5785d54a49ce2
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/core/src/ops/function.rs:259:13
  48:     0x5608fe2ef20a - std::panicking::try::do_call::h1c15e7ee8563c29d
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panicking.rs:401:40
  49:     0x5608fe2ef20a - std::panicking::try::hb8b07cef009cb4fc
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panicking.rs:365:19
  50:     0x5608fe2ef20a - std::panic::catch_unwind::h65a8efbe596bcbb1
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panic.rs:434:14
  51:     0x5608fe2ef20a - std::rt::lang_start_internal::{{closure}}::h66ded7ab0f18e9fb
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/rt.rs:45:48
  52:     0x5608fe2ef20a - std::panicking::try::do_call::h29cafa65efb3f3f3
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panicking.rs:401:40
  53:     0x5608fe2ef20a - std::panicking::try::h412fc5378b4dbfc4
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panicking.rs:365:19
  54:     0x5608fe2ef20a - std::panic::catch_unwind::h555def6aac5bf192
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/panic.rs:434:14
  55:     0x5608fe2ef20a - std::rt::lang_start_internal::hfff48202d7fee6e3
                               at /rustc/240ff4c4a0d0936c9eeb783fa9ff5c0507a6ffb4/library/std/src/rt.rs:45:20
  56:     0x5608fde2790a - main
  57:     0x7f458a99b0b3 - __libc_start_main
  58:     0x5608fde1b50e - _start
  59:                0x0 - <unknown>
Aborted (core dumped)

We tried an attempted patch earlier to fix this to no effect:

diff --git a/src/tracker/mod.rs b/src/tracker/mod.rs
index 63b6c78..ae63cf4 100644
--- a/src/tracker/mod.rs
+++ b/src/tracker/mod.rs
@@ -372,6 +372,20 @@ impl Tracker {
     /// tracked database.
     #[instrument(skip(self))]
     pub fn flush_spooled_data(&self) -> Result<()> {
-        crate::copier::copy_spool_path(self.buffer.spooling_directory())
+        let to_flush = self.buffer.spooling_directory();
+
+        if let Ok(handle) = tokio::runtime::Handle::try_current() {
+            // We use our own Tokio runtime in `copy_spool_path`.
+            // Avoid panics due to nested runtimes by flushing in a
+            // temporary thread.  Flushes have to upload data over the
+            // network, so they're not exactly fast anyway.
+            let to_flush = to_flush.to_owned();
+
+            handle
+                .block_on(handle.spawn_blocking(move || crate::copier::copy_spool_path(&to_flush)))
+                .map_err(|e| chain_error!(e, "copy_spool_path thread failed"))?
+        } else {
+            crate::copier::copy_spool_path(to_flush)
+        }
     }
 }

Somewhat large manifest protos for > 1 GB databases

We currently refer to each 64KB page in a database with a 16 byte fingerprint. That's a good 4000x reduction, but still means a 256 KB incompressible manifest for a 1 GB db file.

Most DB writes don't actually change that many pages, so we should be able to expose that redundancy to the compressor. Do so by stashing a base list of fingerprints (as raw bytes in a content-addressed chunk), and xor-ing that with the list of fingerprints before serialisation and after deserialisation. This should result in a lot of zeros for zstd to shrink.

In order to achieve this:

Decode manifest protos with an optional base chunk
Encode manifest protos with a base chunk
Try to decompress manifest blobs when they look zstd-compressed
Compress manifest blobs
~~Avoid re-uploading chunks back-to-back in copier.rs~~ already handled! (RecentWorkSet)
Figure out a policy to reset the base chunk
Make sure not to use base chunks for dbs < a certain size
Stash the latest Arc<Chunk> for each db somewhere: this guarantees we keep them in the global cache.

tracker.rs's `snapshot_file_contents` is a bit long

It's a linear pipeline, but a lot of operations. Splitting it up a little should help readers.

Let snapshots start with partial data

For large databases, fully loading all the chunks in a snapshot can be useless, and is always slow. Sqlite page allocation tends to put root btree pages at the beginning of the file, so we could want to load only the first few chunks in a snapshot, and the rest on demand.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.