tokio-rs / io-uring Goto Github PK
View Code? Open in Web Editor NEWThe `io_uring` library for Rust
License: Apache License 2.0
The `io_uring` library for Rust
License: Apache License 2.0
I notice the 0.5.2 version remove the ownsplit mod, it breaks program which use 0.5 version io_uring
with concurrent IoUring, I create a thread and it will call submit_and_wait(1)
to waiting for the entry, when it is waiting, I create another thread and push a new entry into sqe. but the thread one doesn't know a new entry is pushed.
How to let submit_and_wait
know a new entry is pushed and return it when it ready? I've tried to call submit
after push, it works with Readv
and Writev
, the submit_and_wait
return after read or write complete, but Openat2
doesn't work, the submit_and_wait
still blocking even the file is created.
Is there a better way to do this?
We can implement concurrent SubmissionQueue
and CompletionQueue
so that multiple threads share a uring.
It would be neat to have that as well as the bring docs :)
I've started a prototype, using this crate and eventfd, to try an implement efficent async file-backed Stream
and Sink
types. In review, I noticed this comment on IoUring
:
Line 72 in 5653104
Given the self-referential nature, should SubmissionQueue
, CompletionQueue
, and IoUring
use PhantomPinnned
? As far as I can tell at present they all implement Unpin
, automatically. Or is this not relevant because these types don't own memory buffers, just contain pointers to memory mapped regions that won't be moved? If the latter, then it might be possible to say something more assuring in place of the current comment? Thanks!
When I try to run the example on Ubuntu 20.04 (with the linux kernel version 5.4), it produces this error:
$ cargo run --example tcp_echo ✹
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `target/debug/examples/tcp_echo`
listen 127.0.0.1:3456
token Some(Accept) error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }
token Some(Accept) error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }
token Some(Accept) error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }
FWIW, running the readme example with the fix described in #87 worked but it seems that calling accept
is failing. Any ideas what the issue might be?
see #56 (comment)
A pretty weird problem (to me).
Symptom
By learning this io_uring library, I started from trying out the echo server code in example. The code is here:
https://github.com/JeepYiheihou/rust-io-uring-echo-server
However the behavior is different on two of my machines. One unexpected and another one expected.
The unexpected one happens on my fedora machine. It only returns contents after 3 lines of your input, and returns them altogether:
[jiachenbai@localhost 23:10:18 io_uring-echo-server]$ telnet 127.0.0.1 3456
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
a
b
c
a
b
c
And the expected one happens on my ubuntu machine. This time it runs well:
jiachen@jiachen-dev:~/workspace/rust/echo-server$ telnet localhost 3456
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
a
a
b
b
c
c
Investigation
I suspected it might be the problem with kernel's io_uring functionality, but I tried some other echo servers in C implemetation, for example https://github.com/frevib/io_uring-echo-server, but they work consistently on both systems.
To follow up with a previous discussion before in the email, the echo server is submitting sqe with opcode IORING_OP_READV
, which should trigger the kernel to perform a preadv
call. However, if I understand correctly, a preadv doesn't wait until the buffer is filled up. It just returns whatever it reads and it only blocks & waits when the socket is really empty. (The system call that really waits untill buffer is full should be recv
if I'm right. Please correct me.) So in this regard, the echo server even betrays the behavior of the system call under the hood. To make things more weird, the server just returns every 3 times, regardless how many chars you type in each time. Look at this:
[jiachenbai@localhost 23:26:36 io_uring-echo-server]$ telnet 127.0.0.1 3456
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
a looong string
an even loooooonger string
a verrrrrrrrry looooooooooooooooooooooong string
a looong string
an even loooooonger string
a verrrrrrrrry looooooooooooooooooooooong string
It looks like it's not even about "whether the buffer is full". It's just some weird fixed 3-times-rule and I totally have no clue where it comes from. I'm now guessing it might be issue with the implementation of this rust io_uring library, or maybe kernel. But I just don't know how to dive deeper. Anyway, this shows a risk that the library will perform some unexpected behaviors in Linux world.
Context
The system of the fedora machine is:
OS: Fedora 33,
kernel: 5.9.16-200.fc33.x86_64
CPU: Intel Core i5-7200U
The system of the ubuntu machine is:
OS: Ubuntu 20.10
kernel: 5.8.0-36-generic
CPU: Intel Core i7-4720HQ
I'm not sure if this issue is strange/severe enough to draw your attention to take a look into. If so, I would be grad to help.
io_uring
requires at least one unsafe function; currently this is push
on the submission queue. However moving this unsafety to the creation of the squeue::Entry
s (in the opcode module) has a few benefits:
Nop
can be made fully safe.Timeout
writing "timespec
must be a valid pointer for the lifetime of this entry" in Timeout::new
is clearer than writing "the entry must be valid for the entire operation" in push
.In practice, concurrent sq is useful, while concurrent cq is almost useless. we can split the two and provide concurrent sq and non-concurrent cq to avoid some atomic overhead.
And, I hope to implement lock-free squeue.
Linux VM_0_13_centos 5.13.7-1.el7.elrepo.x86_64
io-uring v0.5.1
using io-uring
build my application, after run several days, got error:
traps: my-exec[25434] general protection fault ip:7f18294bf3b2 sp:7fffffed4ab0 error:0 in libc-2.17.so[7f182943d000+1c3000]
I'm sure my code don't call any libc function.
rebuild with:
[profile.release]
debug = true
addr2line show nothing.
addr2line 7f18294bf3b2 -e my-exec -f -C -s
??
??:0
There're couple io_uring bindings available in the community. Is there any reason to start another binding effort? Compare to ringbahn, is there anything new in this new binding? Since both repos claim to provide safe API, what's the difference between these two repos in design? I'm not familiar with ringbahn nor tokio's io-uring but I'm interested to know the differences.
before use io-uring i want to check whether features are supported or not. currently opcodes like IORING_OP_PROVIDE_BUFFERS
do not export. there is other way to check or we can make the sys module
public?
// check if IORING_FEAT_FAST_POLL is supported
let ring = IoUring::new(256).expect("init io uring fail");
if !ring.params().is_feature_fast_poll() {
panic!("IORING_FEAT_FAST_POLL not available in the kernel");
}
// check if buffer selection is supported
let mut probe = io_uring::Probe::new();
ring.submitter().register_probe(&mut probe);
// compile error: error[E0603]: module `sys` is private
if probe.is_supported(io_uring::sys::IORING_OP_PROVIDE_BUFFERS) {
panic!("IORING_OP_PROVIDE_BUFFERS not supported");
}
I just cloned it and executed the cargo run.
Linux pktserver 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Hi,
I created a connection with a TCP socket using the following code:
let poll_e = opcode::Connect::new(
Fd(new_fd),
&mut addr as *mut sockaddr_in as *mut sockaddr,
mem::size_of::<sockaddr_in>() as u32,
)
.build()
.user_data(poll_token as _);
Here the event loop immediately acknowledges the connect with a ret of 0, but the socket is not connected yet. Is there any way to defer the Connect opcode until the socket is readable and fully connected?
The current test quality is unsatisfactory, I hope to achieve the following points.
uname -r
)IoUring
for each test.Hi,
man 2 fallocate
says len
should be an off_t
, and this library implements that as an u32
.
Related:
https://github.com/axboe/liburing/blob/master/src/include/liburing.h#L331-L337
Reproduction:
Step one: place a file named update_file_registrations.rs
into the example directory containing this text:
use io_uring::{reg, unreg, IoUring};
use std::{io::Result, iter, os::unix::io::IntoRawFd};
use tempfile::tempfile;
fn open() -> Vec<i32> {
iter::repeat_with(|| tempfile().unwrap().into_raw_fd()).take(10).collect()
}
fn main() -> Result<()> {
let uring = IoUring::new(256)?;
let fds = open();
let new_fds = open();
uring.register(reg::Target::Files(&fds))?;
uring.register(reg::Target::FilesUpdate { offset: 0, fds: &new_fds })?;
uring.unregister(unreg::Target::Files)?;
Ok(())
}
Step two: execute $ cargo run --example update_file_registrations
.
Expected result: example completes with no output.
Actual result: example returns Error: Os { code: 0, kind: Other, message: "Success" }
.
My code is based on this test that successfully passes on my system:
https://git.kernel.dk/cgit/liburing/tree/test/file-update.c#n62
When enable setup_sqpoll, if run the writev request, it will return -9 (EBADF).
There is no such issue to disable the sqpoll.
see reddit comment.
The current API gives out mutable references to the inner SubmissionQueue
and CompletionQueue
of the io_uring instance. However this gives the ability for users to replace the queues entirely with a different one, leading to code like this causing unsoundness:
let mut ring_a = IoUring::new(16)?;
let mut ring_b = IoUring::new(16)?;
mem::swap(ring_a.completion(), ring_b.completion());
drop(ring_a); // ring_a's cq is freed...
let freed_queue = ring_b.completion(); // ...but we can still access it here.
I think this is best solved by instead handing out a CompletionQueueMut
/cqueue::Mut
type or something like that. I'm working on a fix for this now.
formats all files using rustfmt #37
Feel free to close this issue if you think this is not needed.
Currently we have squeue::AvailableQueue
and cqueue::AvailableQueue
that represent a snapshot of the submission queue and completion queue respectively. But I don't think either are necessary: performance-wise, atomic loads and stores are incredibly cheap, and utility-wise there isn't much benefit in freezing the queue at a specific point in time, and if users wish to do that they can just cap it off at the length of the queue manually.
I think it would be a lot simpler if we got rid of them entirely, and made CompletionQueue
implement Iterator
and have the push
method on SubmissionQueue
.
I tried running the readme.rs example on Ubuntu 20.04 & Linux kernel 5.4. However, it fails with the following error:
thread 'main' panicked at 'read error: -22', examples/readme.rs:28:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Please let me know if additional information would be helpful
When running cargo run --package io-uring-test
my system eventually crashes while it's hanging on test queue_nodrop
. Looking at htop while running the test I can see my memory usage jump from 4GB to 16GB, 20GB+, then 32GB and the system then crashes due to OOM. Memory usage for the test process shows as 0% in htop.
Version: 0.5.1
Rust version: rustc 1.56.0-nightly (9c25eb7aa 2021-07-25)
Kernel version: 5.13.4-arch1-1
Edit: changed the title to better reflect the issue
I did some benchmark about your echo server example, and got very good results. Well done man! The performance of rio is a disaster.
I'm curious about the implementation, but I'm not familiar with rust. Do you use poll
before accept
? What method are you using? recv
/send
or read
/write
? Do you use fixed buffers and files? IOSQE_IO_LINK?
Line 33 in 29ae5cb
I was just wondering why you chose to use a lock instead of casing tail?
I download this repository and run
cargo run --example readme
The program panics and outputs
thread 'main' panicked at 'read error: -22', examples/readme.rs:28:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
My OS envirenment is
Linux version 5.4.0-67-generic (buildd@lcy01-amd64-025) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #75-Ubuntu SMP Fri Feb 19 18:03:38 UTC 2021
My rust version is 1.51
and io-uring
version is 0.5.1
.
liburing has a way to block until completion events come in. I'd be nice to have that functionality in this library. Some like https://unixism.net/loti/ref-liburing/completion.html#c.io_uring_wait_cqe
hello,
Ran the tests on my machine and encountered some failures. Be forewarned that I updated my kernel to run this in the first place, so possibly it's some kind of compatibility or versioning issue of the underlying system I'm on.
Am running ubuntu 18.04 on a workstation with older xeons, and the kernel had been at 4.15 (I think). Now running 5.1.21:
$ uname -a
Linux coolidge 5.1.21-050121-generic #201907280731 SMP Sun Jul 28 07:34:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
When I run the tests, the ones in tests/cancel.rs hang. I've let them run more than 5 minutes without completion. When I run strace on the test runner process, that somehow triggers the test actually failing.
During the hang, there are actually two processes running. strace
on one of them results in this:
$ sudo strace -p 18074
strace: Process 18074 attached
restart_syscall(<... resuming interrupted futex ...>^Cstrace: Process 18074 detached
<detached ...>
Then, strace
on the other pushes the test runner past its deadlock:
strace: Process 18076 attached
syscall_0x1aa(0x4, 0x1, 0x2, 0x1, 0, 0x80) = 0x1
close(7) = 0
close(5) = 0
munmap(0x7fd7b8071000, 160) = 0
munmap(0x7fd7b8072000, 256) = 0
munmap(0x7fd7b8070000, 272) = 0
close(4) = 0
futex(0x557afd529a9c, FUTEX_WAKE_PRIVATE, 1) = 1
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x7fd7b8076000, 8192) = ? <unavailable>
+++ exited with 101 +++
The test failure output is:
running 2 tests
test test_cancel_nop ... FAILED
test test_cancel_pipe ... test test_cancel_pipe has been running for over 60 seconds
test test_cancel_pipe ... FAILED
failures:
---- test_cancel_nop stdout ----
thread 'test_cancel_nop' panicked at 'assertion failed: `(left == right)`
left: `-9`,
right: `-2`', tests/cancel.rs:42:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- test_cancel_pipe stdout ----
thread 'test_cancel_pipe' panicked at 'assertion failed: `(left == right)`
left: `-4`,
right: `-125`', tests/cancel.rs:96:5
The values (-9/-2, -4/-125) are the same on successive runs. Switching which features are enabled does not seem to change this behavior.
Hopefully this is useful to you. This is my first experience with either the rust io-uring library or the io_uring syscalls, so apologies if I am doing anything wrong. Thanks!
This can be used to share a ring between separate users or processes.
I'm using v0.3.5 and alpine linux
here is the errors message
Compiling io-uring v0.3.5
error[E0412]: cannot find type `statx` in crate `libc`
--> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/io-uring-0.3.5/src/opcode.rs:750:30
|
750 | statxbuf: *mut libc::statx,
| ^^^^^ help: a struct with a similar name exists: `stat`
|
::: /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.69/src/macros.rs:79:13
|
79 | pub struct $i { $($field)* }
| ------------- similarly named struct `stat` defined here
error: aborting due to previous error
here is musl version
$ apk info musl
musl-1.1.24-r2 description:
the musl c library (libc) implementation
musl-1.1.24-r2 webpage:
https://musl.libc.org/
musl-1.1.24-r2 installed size:
614400
The git log says " release 0.5.0-alpha" but it's not on crates.io. In the mean time, the repository has moved to a new home. Would you consider releasing an -alpha or -alpha.1 to crates.io so that we can have the latest (admittedy small) fixes, and so that crates.io points to the correct repo ?
Thanks for this crate btw. I'm just playing around with io_uring but I'm glad I did not have to mess around with the C liburing or raw system calls.
Clippy says it might be UB.
$ cargo clippy --features unstable
error: casting from `*mut u8` to a more-strictly-aligned pointer (`*mut sys::io_uring_probe`) (1 < 4 bytes)
--> src/register.rs:144:15
|
144 | Probe(ptr as *mut _)
| ^^^^^^^^^^^^^
|
= note: `#[deny(clippy::cast_ptr_alignment)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#cast_ptr_alignment
I hope to stabilize more APIs in 0.5. To stabilize a new API, the following conditions must be meet.
for new API of 5.10, we must upgrade the cross
kernel version to pass ci.
https://docs.rs/io-uring/0.5.1/io_uring/struct.Builder.html#method.setup_sqpoll says
After idle seconds,
but exactly it is milliseconds
uname -a
Linux centos 5.12.12-1.el7.elrepo.x86_64 #1 SMP Thu Jun 17 10:00:40 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
when process run a period of time, got the following error.
segfault at 55de015568bb ip 000055de015568bb sp 00007ffda51fcde0 error 14 in libc-2.17.so[7efc2f3c2000+1c3000]
Code: Unable to access opcode bytes at RIP 0x55de01556891
main process exited, code=killed, status=11/SEGV
it's seems like a kernel bug fixed in 5.13, according to https://lore.kernel.org/io-uring/[email protected]/t/ .
OS: Ubuntu 20.04.2 LTS
/proc/version
: Linux version 5.4.0-70-generic (buildd@lcy01-amd64-004) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021
Cargo.toml:
[package]
name = "io-uring-rs"
version = "0.1.0"
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
io-uring = "0.5"
Program running results :
Finished dev [unoptimized + debuginfo] target(s) in 0.29s
Running `target/debug/io-uring-rs`
thread 'main' panicked at 'read error: -22', src/main.rs:28:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Is this io-uring
bug? Or is it a problem with the program itself? Please help to answer, thanks.
The sys
module defines several constants (syscalls, operations, etc...), and it would be useful from the caller's perspective, to be able to access them.
Currently the IoUring type isn't Send or Sync because it contains raw pointers. Is there any reason IoUring isn't thread safe? If not can it be marked as such?
I try to run this codes
pub fn ttt() {
let mut io_uring = IoUring::builder().setup_sqpoll(1000).build(4096).unwrap();
let (submitter, sq, cq) = io_uring.owned_split();
/*drop(sq);
drop(cq);*/
submitter.submitter().submit().unwrap();
}
it will failed with
thread 'main' panicked at 'called
Result::unwrap()
on anErr
value: Os { code: 130, kind: Uncategorized, message: "Owner died" }'
I search io_uring EOWNERDEAD
on Google, and got somethings
link1
link2
I tried to use io_uring statx but fail with errno 22. The same parameters can run successfully with libc::statx directly.
I didn't find statx sample code or test code. Is there any known issue?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.