High CPU usage for cdrs_tokio::transport::AsyncTransport::start_processing about cdrs-tokio HOT 19 OPEN

It might help out of the box, but also remember you can fine tune many settings. If you still get those spikes, try lowering heartbeat interval and/or pool size.

from cdrs-tokio.

start_processing is a bit misleading - it starts read/write loops per connection, so it will naturally show up on the flamegraph as the main culprit, especially when connections live as long as the whole application. Can you share debug logs from the moment it happens?

from cdrs-tokio.

I will redeploy my app to run with debug symbols and at debug log level and I will update the logs here once I have them.

from cdrs-tokio.

You don't have to run a debug build - just debug logs will be fine. The problem might be in any part of the application, maybe even not in cdrs, so it's important to get the logs from the exact moment it happens.

from cdrs-tokio.

I will run a release build itself but with debug symbols.

from cdrs-tokio.

I ran again and captured the flamegraph with same results as above. I finally decided to use another library named scylla which does not have this problem. Is there any possibility that start_processing::{closure}::{..} can somehow be stuck in a infinite loop?

from cdrs-tokio.

No it can't, since it's simply a read/write loop which waits for data. That's why you're seeing it on the flamegraph - it runs for the entire duration of the session, but doesn't actually do anything until data comes in or out. That's why you can see AsynRead and AsyncWrite there which don't actually take any CPU cycles - they wait an do work only when data is available. It would be great if you provided the debug logs, so we could see what's actually going on.

from cdrs-tokio.

I ran with RUST_LOG=debug but it cdrs_tokio did not logged anything. Did i missed something? I use rust nightly and tokio-tracing.

from cdrs-tokio.

Are you using the latest version? Do you have any tracing subscriber, which outputs the logs, e.g. https://docs.rs/tracing-subscriber/latest/tracing_subscriber/fmt/index.html?

from cdrs-tokio.

Are you using the latest version?

Yes, (I usually perform cargo update once a week)

Do you have any tracing subscriber, which outputs the logs, e.g. https://docs.rs/tracing-subscriber/latest/tracing_subscriber/fmt/index.html?

Yes, I use the above tracing_subscriber itself. I am able to get logs from hyper, h2 and other crates.

I have been running the service for a long time and this has been happening all the time. I am able to get some logs from cdrs_tokio target at ERROR.

For now, I have moved to scylla which does not have this problem. This library used to somehow cause so much IO, that our istio-sidecar would also consume more CPU, resulting in a high overall pod CPU usage. There must be some tight loop somewhere in transport.

from cdrs-tokio.

"Connection reset by peer" means the cluster shut the connection down. Can you verify two things:

1. You are using version 8.1, not 8.0 which was yanked.
2. RUST_LOG is set to DEBUG, since it seems you have some other log level set, as if the env variable was not set for your application. cdrs uses tracing for logging, so it follows all the usual rules.

from cdrs-tokio.

Just had a thought - if you are using 8.0 and the nodes drop connections, the io spikes might be related to connection pools being re-established. Lowering pool size might be the solution.

from cdrs-tokio.

You are using version 8.1, not 8.0 which was yanked.

Now that I check, I was using cdrs-tokio = { version = "7.0.4", features = ["derive"] }, Let me check back these things with latest version.

from cdrs-tokio.

I'm facing similar issue wherein CPU usage is ~50% and I tried both lowering connection pooling (local: 1 and remote:0) and upgrading to 8.1.0. Both didn't help as of now.

from cdrs-tokio.

Do you have a flamegraph and/or debug logs?

from cdrs-tokio.