Comments (9)
Ablu
commented on June 13, 2024
The backtrace seems to indicate that the frontend waits for an ACK from the backend. Is the SET_LOG_MESSAGE acked by the backend here?
The reproduction steps are fairly involved... Do you have a link to the code of the backend that you pair this with?
from vhost.
rveerama1
commented on June 13, 2024
@Ablu Thanks for the reply.
Do you have a link to the code of the backend that you pair this with?
https://github.com/rust-vmm/vhost/blob/main/vhost/src/vhost_user/connection.rs#L538 do you mean this part or something else?
The reproduction steps are fairly involved
There is another simple method to reproduce this scenario is
Clone cloud hypervisor
$ git clone [email protected]:cloud-hypervisor/cloud-hypervisor.git
$ cd cloud-hypervisor
apply my PR patch
https://patch-diff.githubusercontent.com/raw/cloud-hypervisor/cloud-hypervisor/pull/5877.patch
and run this particular test
$ scripts/dev_cli.sh tests --integration-live-migration -- --test-filter test_live_migration_ovs_dpdk
This test will run inside the container. I don't know adding gdb there. If you would like to run it on host all those steps are involved.
from vhost.
rveerama1
commented on June 13, 2024
I am trying to debug it further.
It fails/blocks at self.sock.recv_with_fds(iovs, &mut fd_array) :
vhost/vhost/src/vhost_user/connection.rs
Lines 328 to 329 in 0669474
I went further and it fails/blocks at socket recvmsg:
Do you have a link to the code of the backend that you pair this with?
I couldn't figure out from ovs-dpdk or dpdk source code exact location. ovs-dpdk is a package installed from Ubuntu 22.04.
They don't have an issues tab https://github.com/openvswitch/ovs, but I sent a mail to [email protected] . Let's see.
from vhost.
Ablu
commented on June 13, 2024
@rveerama1: Yeah, as I said before, it looks like we expect an ACK from the backend does not arrive. It would be interesting to know if the SET_LOG_MESSAGE is acked by the backend here.
from vhost.
rveerama1
commented on June 13, 2024
It would be interesting to know if the SET_LOG_MESSAGE is acked by the backend here.
do you mean reply to this call?
vhost/vhost/src/vhost_user/frontend.rs
Line 215 in 0669474
from vhost.
rveerama1
commented on June 13, 2024
logs from /var/log/openvswitch/ovs-vswitchd.log
on host Ubuntu 22.04
2024-01-16T16:01:25.143Z|00033|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_LOG_BASE
2024-01-16T16:01:25.143Z|00034|dpdk|INFO|VHOST_CONFIG: log mmap size: 32768, offset: 0
This is where vhost doesn't get ACK
on host Ubuntu 20.04
2024-01-16T14:17:03.687Z|00033|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_LOG_BASE
2024-01-16T14:17:03.688Z|00034|dpdk|INFO|VHOST_CONFIG: log mmap size: 32768, offset: 0
2024-01-16T14:17:03.688Z|00035|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
2024-01-16T14:17:03.688Z|00036|dpdk|INFO|VHOST_CONFIG: negotiated Virtio features: 0x16442a782
2024-01-16T14:17:03.688Z|00037|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
2024-01-16T14:17:03.688Z|00038|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
2024-01-16T14:17:04.767Z|00039|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_LOG_BASE
2024-01-16T14:17:04.767Z|00040|dpdk|INFO|VHOST_CONFIG: log mmap size: 32768, offset: 0
2024-01-16T14:17:04.776Z|00041|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_LOG_BASE
2024-01-16T14:17:04.776Z|00042|dpdk|INFO|VHOST_CONFIG: log mmap size: 32768, offset: 0
2024-01-16T14:17:04.783Z|00043|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2024-01-16T14:17:04.783Z|00044|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 0
2024-01-16T14:17:04.783Z|00045|netdev_dpdk|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/tmp/dpdkvhostclient2' changed to 'disabled'
2024-01-16T14:17:04.783Z|00046|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2024-01-16T14:17:04.783Z|00047|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 1
2024-01-16T14:17:04.784Z|00048|netdev_dpdk|INFO|State of queue 1 ( rx_qid 0 ) of vhost device '/tmp/dpdkvhostclient2' changed to 'disabled'
2024-01-16T14:17:04.789Z|00049|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_LOG_BASE
2024-01-16T14:17:04.789Z|00050|dpdk|INFO|VHOST_CONFIG: log mmap size: 32768, offset: 0
2024-01-16T14:17:04.797Z|00051|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
2024-01-16T14:17:04.797Z|00052|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
2024-01-16T14:17:04.797Z|00053|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
2024-01-16T14:17:04.797Z|00054|dpdk|INFO|VHOST_CONFIG: negotiated Virtio features: 0x16042a782
from vhost.
Ablu
commented on June 13, 2024
do you mean reply to this call?
Yes, that sends a SET_LOG_BASE message and then waits for the ACK (at:
vhost/vhost/src/vhost_user/frontend.rs
Line 232 in 0669474
You will probably want to double check that our behavior is correct according to the vhost-user spec, then try to find out if your backend does not actually send the ACK or whether it gets lost somewhere.
from vhost.
stefano-garzarella
commented on June 13, 2024
You will probably want to double check that our behavior is correct according to the vhost-user spec
https://qemu-project.gitlab.io/qemu/interop/vhost-user.html#communication
VHOST_USER_SET_LOG_BASE is in the list of those that require it, but only when VHOST_USER_PROTOCOL_F_LOG_SHMFD is negotiated.
Looking at our frontend (and also the one in QEMU) it seems to require the ack in that condition. Can you check that the backend does the same? (i.e. it sends the ack when F_LOG_SHMFD is negotiated)
from vhost.
rveerama1
commented on June 13, 2024
@Ablu @stefano-garzarella thanks for the replies. I will check and get back to you.
from vhost.
Related Issues (20)
- vdpa: Incorrect implementation of get_iova_range() HOT 3
- Vring addresses are not inside guest memory range if placed behind a vIOMMU HOT 10
- Mac build support HOT 5
- Typo `vhots v0.4.0` in GitHub Release HOT 1
- Clarification of return type of `handle_event` HOT 3
- vhost: check better if `GET_VRING_BASE` message should reset the vring HOT 1
- Potential regression for vDPA support HOT 6
- Vhost crate point release for fixing vDPA regression HOT 5
- vhost-user-backend: Release new version (fix GET_VRING_BASE behavior) HOT 2
- Don't always enable vm-memory/backend-mmap
- Regenerate vhost_binding.rs and move custom changes in another file
- Missing Input Parameter Validation
- Remove non-standard BackedReq types
- ByteValued implementation for VhostUserInflight is UB HOT 11
- Inconsistent licensing HOT 11
- Allow `Master` to set flags in the header HOT 4
- Add support for Inflight I/O tracking HOT 1
- Just a misoperation
- Change VhostUserSlaveReqHandler::set_slave_req_fd() to take an UnixStream instead of SlaveFsCacheReq HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vhost.