Comments (14)
In my opinion, &mut self
has semantic meaning, that is, if you're going to change it you need to borrow it as mutable. Of course we can make it all &self
but by doing that we lost the borrow semantics. For concurrent, you can always wrap a repo in Arc
or other synchronous types, and that might suit your need.
from zbox.
@theduke , the File
struct should be able support read-write lock concurrently, could you share some example code?
from zbox.
Ah, I realized that I can open multiple file handles and then use them concurrently.
Since zbox does support internal concurrency/locking, it would be quite nice to hoist all of this into the Repo
struct and have all methods on Repo
take &self
rather than &mut self
.
from zbox.
In my opinion, &mut self has semantic meaning, that is, if you're going to change it you need to borrow it as mutable.
mut
should not be interpreted as "mutable", but as "exclusive".
from zbox.
It seems to fail to support working on multiple files even without threads.
I get zbox::Error::InTrans
if I try to work on multiple files simultaneously.
zbox::File
behaves as if it wants exclusive access to Repo.
from zbox.
It seems to fail to support working on multiple files even without threads.
I get
zbox::Error::InTrans
if I try to work on multiple files simultaneously.
zbox::File
behaves as if it wants exclusive access to Repo.
Can you share your code about this issue? One File
cannot be mutated simultaneously, but multiple files should able to do that.
from zbox.
Can you share your code about this issue?
One File cannot be mutated simultaneously, but multiple files should able to do that.
fn main() -> Result<(), Box<dyn std::error::Error>> {
env_logger::init();
zbox::init_env();
use std::io::Write;
let mut r = zbox::RepoOpener::new().create(true).force(true).open("file:///tmp/limit/q", "qwe123")?;
eprintln!("Opening file1");
let mut f1 = r.create_file("/test1")?;
eprintln!("Opening file2");
let mut f2 = r.create_file("/test2")?;
eprintln!("Writing file1");
f1.write_all(b"12345")?;
eprintln!("Writing file2");
f2.write_all(b"qwerty")?;
eprintln!("Committing file1");
f1.finish()?;
eprintln!("Committing file2");
f2.finish()?;
Ok(())
}
Running `target/debug/zboxtest`
Opening file1
Opening file2
Writing file1
Writing file2
Error: Custom { kind: Other, error: "Already in transaction" }
from zbox.
Thanks @vi for your code. The reason for this because content manager needs to be exclusively accessed. In writing file1, its transaction will exclusively include the content manager, thus when writing file2 the transaction cannot get access to it any more.
from zbox.
One File cannot be mutated simultaneously, but multiple files should able to do that.
content manager needs to be exclusively accessed
Is it a bug or by design?
from zbox.
One File cannot be mutated simultaneously, but multiple files should able to do that.
content manager needs to be exclusively accessedIs it a bug or by design?
Well, it is by design currently but I think a tx should not hold a shared object too long until finish
is called. Some optimization could be done to deal with this situation.
from zbox.
Can two zbox::File
s be sent to two threads that will write and finish multiple transactions on their files independently, without coordination?
If no then it should be documented.
from zbox.
Ok, after digging through this issue deeper I found it is actually not the exclusive access to content manager caused this issue, sorry about my misleading explanation before.
The real reason is that each thread can only run one transaction at a time. In the code below,
let mut r = zbox::RepoOpener::new().create(true).force(true).open("file:///tmp/limit/q", "qwe123")?;
let mut f1 = r.create_file("/test1")?;
let mut f2 = r.create_file("/test2")?;
f1.write_all(b"12345")?; // this will create one ongoing tx
f2.write_all(b"qwerty")?; // this will create another ongoing tx
f1.finish()?;
f2.finish()?;
There are 2 files are using multi-part writing, 2 write_all()
created 2 parallel txs in one thread, thus it reports an error. There are 2 solutions:
- run
finish()
sequentially
f1.write_all(b"12345")?;
f1.finish()?;
f2.write_all(b"qwerty")?;
f2.finish()?;
- using 2 threads to write 2 files separately
let child1 = std::thread::spawn(move || {
f1.write_all(b"12345").unwrap();
f1.finish().unwrap();
});
let child2 = std::thread::spawn(move || {
f2.write_all(b"qwerty").unwrap();
f2.finish().unwrap();
});
child1.join().unwrap();
child2.join().unwrap();
from zbox.
Is ZboxFS supposed to be used in async/Tokio context? (where thread may be unpredictable).
Does it rely on thread-local storage?
Can transactions be untied from threads? There can be explicit object zbox::Transaction
that can open files, which would be bound to that transaction instead of to thread-default transaction.
Or is there some definite way to check if it is OK to write a file? (when running from thread pool where tasks can migrate between threads on its own)
Is only way to reliably use ZboxFS without fully controlling threading in application is to consult a map from std::thread::ThreadId
before every write?
from zbox.
Is ZboxFS supposed to be used in async/Tokio context? (where thread may be unpredictable).
It hasn't used async yet because that is a big change and may change the internal structure as well.
Does it rely on thread-local storage?
Yes, it stores tx id in thread local storage.
Can transactions be untied from threads? There can be explicit object
zbox::Transaction
that can open files, which would be bound to that transaction instead of to thread-default transaction.Or is there some definite way to check if it is OK to write a file? (when running from thread pool where tasks can migrate between threads on its own)
Is only way to reliably use ZboxFS without fully controlling threading in application is to consult a map fromstd::thread::ThreadId
before every write?
Currently, threading is tightly coupled with tx control internally, so it is hard to expose it as API and get it right. Also, this will add complexity for developing using ZboxFS.
As for the thread pool use case, I think app has to stick each file to different thread at this moment. If app cannot control that, it might be a limitation of ZboxFS.
from zbox.
Related Issues (20)
- Absolute file URLs on Windows HOT 5
- Zbox storage needs destroy method
- RepoInfo.uri() should not mask secretive data
- Is zbox secure agains storage file tampering? HOT 3
- Confusing abort if omit zbox::init_env HOT 7
- zbox::Error is not accessible from within std::io::Error HOT 1
- Stress-testing multiple reads, writes and seeks produces an unreadable file (IO error: File not found)
- Problems when `file://` storage runs out of space HOT 42
- Panic when writing to `Mutex<zbox::File>` from multiple threads. HOT 1
- Panic when trying to write a file after interrupted previous attempt. HOT 1
- FR: `zbox::Repo::ongoing_transaction` HOT 5
- Is zbox::Repo::rename atomic? HOT 1
- Add dedup file option when creating repo
- RepoInfo lack of dedup_file entry
- Snapshots? HOT 2
- Failed open repo in file mode
- Question: reason not support symbolic link?
- storage-sqlite RepoOpener not creating path
- Future plans and continuation HOT 1
- Python support
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zbox.