Comments (6)
@RomanBelozerov @krizhanovsky @const-t
The test case does not set the socket's rcvbuf size, so the default size is big enough to include a bunch of control frame acks. Although asyncore
stops reading, the underlying socket will continue reading until the sockbuf is full. Therefore, there is a high probability that the control frame acks are successfully transmitted to the client. It is better not to set such a small number of control frames. That's why the default size of 10000 (http2_general.test_max_queued_control_frames.TestH2ControlFramesFlood.test_reset_stream_default
) succeeds always, because it fills up the sockbuf.
The test script tries to fill rcvbuf by requesting a large file, but note that control frame acknowledgments and response chunks are interleaved in the downstream flow, so the control frame acks may be mostly transferred to the client before the response chunk transmission, that is, before sockbuf is full, so the blocking requirement of the test case may become false.
Example:
[Fri May 31 00:50:25 2024] [tempesta fw] new client socket is accepted: sk=0000000038642ad1, conn=00000000862a4ec7, cli=000000006c6660d1
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=0, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=1, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] [tempesta fw] Matching request: 00000000eb9cca60, list: 0000000093a7703d
[Fri May 31 00:50:25 2024] [tempesta fw] Matching request: 00000000eb9cca60, list: 00000000aa32c3b1
[Fri May 31 00:50:25 2024] [tempesta fw] rule: 000000008994914a, field: 0x1, op: 0x1, arg:1:0''
[Fri May 31 00:50:25 2024] [tempesta fw] frang: check request for client 127.0.0.2, acc=000000009e905f95
[Fri May 31 00:50:25 2024] [tempesta fw] frang: check incomplete request headers for client 127.0.0.2, acc=000000009e905f95
[Fri May 31 00:50:25 2024] [tempesta fw] enter frang FSM at state 0x0(Frang_Req_0)
[Fri May 31 00:50:25 2024] [tempesta fw] enter frang FSM at state 0x1(Frang_Req_Hdr_Method)
[Fri May 31 00:50:25 2024] [tempesta fw] enter frang FSM at state 0x2(Frang_Req_Hdr_UriLen)
[Fri May 31 00:50:25 2024] [tempesta fw] enter frang FSM at state 0x3(Frang_Req_Hdr_Check)
[Fri May 31 00:50:25 2024] [tempesta fw] enter frang FSM at state 0x5(Frang_Req_Body_Start)
[Fri May 31 00:50:25 2024] [tempesta fw] frang: check incomplete request body for client 127.0.0.2, acc=000000009e905f95
[Fri May 31 00:50:25 2024] [tempesta fw] enter frang FSM at state 0x6(Frang_Req_Body)
[Fri May 31 00:50:25 2024] [tempesta fw] enter frang FSM at state 0x8(Frang_Req_Trailer)
[Fri May 31 00:50:25 2024] [tempesta fw] enter frang FSM at state 0x9(Frang_Req_Done)
[Fri May 31 00:50:25 2024] [tempesta fw] frang: checks done for client 127.0.0.2
[Fri May 31 00:50:25 2024] [tempesta fw] Finish frang FSM at state 0x9, ret=0
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=0, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=0, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] [tempesta fw] tfw_match_ctext_vchar: str[0]=0x20 len=32540 r=3
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=0, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=0, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=0, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=0, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=1, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] [tempesta fw] connection lost: close client socket: sk=0000000038642ad1, conn=00000000862a4ec7, client=000000006c6660d1
[Fri May 31 00:50:25 2024] [tempesta fw] put client 000000006c6660d1, users=2
[Fri May 31 00:50:25 2024] [tempesta fw] put client 000000006c6660d1, users=1
[Fri May 31 00:50:25 2024] [tempesta fw] put client: cli=000000006c6660d1
[Fri May 31 00:50:25 2024] [tempesta fw] client was found in tdb
[Fri May 31 00:50:25 2024] [tempesta fw] client 000000006c6660d1, users=1
[Fri May 31 00:50:25 2024] [tempesta fw] new client socket: sk=0000000038642ad1, state=1
[Fri May 31 00:50:25 2024] [tempesta fw] client was found in tdb
[Fri May 31 00:50:25 2024] [tempesta fw] client 000000006c6660d1, users=2
[Fri May 31 00:50:25 2024] [tempesta fw] new client socket is accepted: sk=0000000038642ad1, conn=00000000862a4ec7, cli=000000006c6660d1
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=0, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] ---> ctx->queued_control_frames=1, max_queued_control_frames=10
[Fri May 31 00:50:25 2024] [tempesta fw] Warning: request dropped: incorrect X-Forwarded-For header: 127.0.0.2
You can see that just before sockbuf is full, the ping acks are mostly transmitted, so the limit is not triggered.
Check the attachment for package captures for what happens in detail:
test_ping_false_blocking.zip
don't forget to configure the wrieshark the pre-master-secret-log-file: secret.txt
.
from tempesta.
@RomanBelozerov @krizhanovsky @const-t
I have to point out another common problem in tempesta-test: when we check the messages in the dmesg log, we don't distinguish the boundaries of test runs, so warning message assertions can be true due to messages from the previous test run, which is why these test cases pass, even though the message didn't actually appear in the current test run. It's better to have an incremental reading of the dmesg log to ensure the correct message assertion.
self.assertTrue(
self.oops.find(
"Warning: Too many control frames in send queue, closing connection",
cond=lambda matches: len(matches) >= 0,
),
"An unexpected number of dmesg warnings",
)
from tempesta.
@kingluo
Yes, you were almost right. max_queued_control_frames
works correctly. The problem was in the tests.
The main problem in these tests - deproxy thread does not have time to open connection and the main thread calls the wait_for_connection_close
and restart
for client. As a result, we do not receive a warning in dmesg. I fixed tests here.
Also we have the problem with warning. You should add \n
to this warning to output the message to dmesg without delay.
I close issue because the problem was in tests and I fixed it.
from tempesta.
Sorry, I'm afraid this doesn't solve the problem.
As I mentioned above, response chunks for large responses are interleaved with control frame ACKs, so you cannot guarantee that enough chunks will be delivered before the control frame ACK to fill the deproxy client's sockbuf and ensure that the latter will block due to the TCP window becoming 0.
client.readable = lambda: True
only prevents asyncore from processing epoll read events, but does not prevent the linux kernel from reading data from the TCP socket. Control frame flooding is a slow read attack that relies on TCP-level read blocking.
Please look at my example above (the tcpdump file and log file) for a visual example of my clarification.
So, the correct approach is to use setsockopt()
to set a sockbuf small enough so that it is not large enough to hold all the control frame acknowledgments.
from tempesta.
Ok. I saw the tcpdump file and log file. But they is not correct because as I mentioned above, the tests did not work correctly. Please add \n
to warning in TempstaFW and check new version of the tests.
from tempesta.
I reworked the tests using socket.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1024)
, but it does not work without filling the buffer. So I use this way:
- set a small buffer for client; (it is needed for stability of tests. The different VMs have a different size of buffer);
- request a large data from backend and disable socket for reading. Tempesta can not send any data to the client until the client enable socket for reading;
- wait until the client buffer is full; (this is needed to avoid interleaving of data frames and ping ACKs);
- send number of attack frames (limit +10% for stability of test);
- wait for connection close and warning from dmesg;
In this case we check that max_queued_control_frames
limit works correctly. But we are not simulated attack
from tempesta.
Related Issues (20)
- HTTPtables: `log` action and actions grouping
- try to build a dependency cycle and ensure that Tempesta blocks the attempt HOT 1
- Tls errors under ping flood HOT 5
- Add static tracepoints
- Only the first location uses the default frang configuration. HOT 1
- `ERROR` in dmesg for slow read attack
- Config directive inheritance HOT 2
- Explicit HTTP/2 requests smuggling handling HOT 4
- tls_match_any_server_name not straightforward behavior HOT 1
- Control symbols from the request string in the configuration file passed to string variable as is. HOT 1
- Server failover configuration saving error. HOT 3
- Memory leak on test run HOT 1
- Rework the work queue from MPMC to MPSC
- Migrate to Ubuntu 24 LTS
- Headers processing optimization
- Move tls encryption from kernel code to Tempesta FW source code and adjuct TCP window, when we push skb to socket write queue for HTTP1 tls also HOT 2
- Zero 'sent bytes' in access log for non-empty responses HOT 1
- Integrate `checkpatch.pl` to our build/commit pipeline HOT 2
- Crash on malformed server response.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tempesta.