l4steam / linux Goto Github PK

When having BBR2 enabled by default or on a socket, before switching to another non-scalable congestion control via a socket option, then the non-scalable congestion control will still use ECT(1) instead of ECT(0). Solution id to clear the TCP_ECN_ECT_1 flag from the ecn_flags during release.

While waiting for #4, I thought I'd build the dpkg myself. Unfortunately, this fails per below on Ubuntu 20.10 (groovy):

root@ubuntu-groovy:/home/vagrant/linux# BUILD_NUMBER=${BUILD_NUMBER:-1} make VERBOSE=1 bindeb-pkg LOCALVERSION=-prague-${BUILD_NUMBER} KDEB_PKGVERSION=1
sh ./scripts/package/mkdebian
dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch)  -b -nc -uc
dpkg-buildpackage: warning: using a gain-root-command while being root
dpkg-buildpackage: info: source package linux-5.4.0-rc3-prague-
dpkg-buildpackage: info: source version 1
dpkg-buildpackage: info: source distribution groovy
dpkg-buildpackage: info: source changed by root <root@ubuntu-groovy>
dpkg-buildpackage: info: host architecture amd64
 dpkg-source --before-build .
 debian/rules build
make KERNELRELEASE=5.4.0-rc3-prague- ARCH=x86 	KBUILD_BUILD_VERSION=1 -f ./Makefile
  CALL    scripts/checksyscalls.sh
  CALL    scripts/atomic/check-atomics.sh
  DESCEND  objtool
  CHK     include/generated/compile.h
  CHK     kernel/kheaders_data.tar.xz
  LD      arch/x86/boot/compressed/vmlinux
ld: arch/x86/boot/compressed/pgtable_64.o:(.bss+0x0): multiple definition of `__force_order'; arch/x86/boot/compressed/kaslr_64.o:(.bss+0x0): first defined here
ld: arch/x86/boot/compressed/head_64.o: warning: relocation in read-only section `.head.text'
ld: warning: creating DT_TEXTREL in a PIE
make[5]: *** [arch/x86/boot/compressed/Makefile:118: arch/x86/boot/compressed/vmlinux] Error 1
make[4]: *** [arch/x86/boot/Makefile:112: arch/x86/boot/compressed/vmlinux] Error 2
make[3]: *** [arch/x86/Makefile:284: bzImage] Error 2
make[2]: *** [debian/rules:6: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
make[1]: *** [scripts/Makefile.package:83: bindeb-pkg] Error 2
make: *** [Makefile:1427: bindeb-pkg] Error 2

Hello all,

I'm trying to install and use the Linux kernel tree with L4S patches

To do so, I did those steps:

1: Cloning the repository on github

2 : Copying the code in the github in a ".sh" file

3: Executing the ".sh" file

No error, it means that all is wright. But when I try to enable tcp prague with the given command:
sysctl -w net.ipv4.tcp_congestion_control=prague, I have this error sysctl: setting key "net.ipv4.tcp_congestion_control": No such file or directory

I try to update grub and reboot my machine but I face the same issue.

Your help will be appreaciated.

Regards

Hi everyone,

I'm encountering an issue with obtaining a CE mark via a router. Below is the topology I'm utilizing for an experiment, where h1 and h2 serve as senders, h3 as the receiver, and there exists a bottleneck bandwidth of 10 mbps between routers r1 and r2, with a propagation delay of 10ms. Prague is being employed as a scalable congestion control from flow h1 to h3, while cubic serves as the classic congestion control from flow h2 to h3.

To facilitate the operation of Prague and dualpi2, I've incorporated the kernel tree from (https://github.com/L4STeam/linux) and installed iproute2 from (https://github.com/L4STeam/iproute2). Additionally, I'm conducting the experiment using Linux namespaces.

Here are the sysctl commands I've used to configure the experiment:

sysctl -w net.ipv4.tcp_congestion_control=prague (at h1)
sysctl -w net.ipv4.tcp_congestion_control=cubic (at h2)
sysctl -w net.ipv4.tcp_ecn=3 (enabled Accurate ECN at h1, r1, r2, and h3)
sysctl -w net.ipv4.tcp_ecn=1 (classic ECN at h2)
These are the parameters I've set for running the experiment:

limit: 10000
target: 15ms
tupdate: 16ms
alpha: 0.16
beta: 3.25
step.thresh: 1ms
step.in_packets: false
c_protection: 10%
ecn_mask: 1
coupling_factor: 2
drop_overload: true
drop_early: false
split_gso: true

I'm using Wireshark to capture packets, but I'm not observing any ECN marking whereas I can see ECT(1) for scalable traffic and ECT(0) for classic traffic. Any assistance in resolving this issue would be greatly appreciated.

Thank you.

Hi all,

I was doing some experiments in lab with L4S scenario and came across a strange behavior.

In few words, I noticed that in a TCP Prague stream, all generated ACKs (in any scenario, even in a low-load network with no congestion)

have the TCP ECN-Echo (ECE) flag set to 1, and not only in response to packets marked with the IP ECN flag "CE" as expected.

Here, a little bit of context.

I'm trying to test the L4S suite by sending TCP data between two linux hosts (act as client and server) using Prague/Cubic, with a forwarding AQM linux router equipped with the dualpi2 feature in between. My objective is to compare L4S/Prague against Cubic on a congested link.

Both hosts and the router utilize the modified kernel from this repository, and both the Prague and dualpi2 implementations I relied on are sourced from here.

We generate two data flows (with iPerf3) at 100 Mb/s from Server to Client, one flow is with Prague and the other is with Cubic Congestion control.

While performing the tests, a reasonable number of PRAGUE packets are marked with CE information by the router to warn the server of an incoming congestion, but unexpectedly all PRAGUE ACKs carry the TCP ECN-Echo (ECE) flag. I would expect to see this flag only on the ACKs in response to the CE marked packets.

This results into a wrong signaling of network congestion, which prevents PRAGUE from competing fairly with CUBIC: in fact the CUBIC average throughput occupies about 80% of the channel’s capacity, leaving to PRAGUE less than 20% of bandwidth.

The phenomeon can also be seen while generating traffic with low-load network and no congestions happening.

I might have, of course, made some mistakes in the system's configuration. I followed the instructions in repository's README.

All Prague-generated traffic is correctly marked as L4S capable (with ECT(1)), indicating that my setup should be correct.

Thank you in advance for your help, and apologize if I have overlooked something.

In the README.md, the "Installation (debian derivatives)" section mentions:

sudo dpkg --install debian_build/*

AFAICT there is no debian_build/ directory at this point, and instead this should be:

sudo dpkg --install *.deb

That *.deb form seems to work fine (installs correctly and boots fine, at least on Google Cloud).

Thanks!

When it tried to execute "sudo modprobe sch_dualpi2"

I encountered the following error

modprobe: FATAL: Module sch_dualpi2 not found in directory /lib/modules/5.15.0-48-generic

I was wondering how to resolve this issue. Thanks!

Greetings,

Apologize for my second issue: I am still trying to perform some fairness measurements with both L4S and classic flow. I now though migrated to a physical environment and I seem not to be able to achieve the correct configuration.

The setup is very simple: I have four servers (two senders and two receivers) exchanging two traffic flows through one server router. One client-server pair uses Prague as CC, while the other uses Cubic. All servers have the patched kernel provided in this repository branch.

Still, if I trigger a congestion by generating both the Prague and the Cubic flows (let's say the flows measure 100 Mbit/s each, and they come though a L2 switch both on the same router's input interface on a 1Gb Ethernet link; only a 100M link though is in place on the output interface towards the receivers) I see the L4S flow having higher delay, higher jitter and a smaller (and more variable) bandwidth share. The Prague share is 1/4 of the Cubic share.

I configured my L4S endpoints as follows:

• I set the CC as tcp Prague (sysctl -w net.ipv4.tcp_congestion_control=prague)
• I set the AccEcn, even if it's not necessary apparently (sysctl -w net.ipv4.tcp_ecn=3)
• I disabled the required offloading capabilities on the endpoints (sudo ethtool -K $NETIF tso off gso off gro off lro off)
• I configured the fair queue on the endpoints (sudo tc qdisc replace dev $NETIF root fq)

I configured my router server as follows:

• I set the dualpi2 on both interfaces (sudo tc qdisc replace dev $NETIF root dualpi2)
• I enabled forwarding through these interfaces to obtain the routing capabilities (sudo sysctl -w net.ipv4.ip_forward=1)

I then applied the fair queue and disabled the offloading capabilities on both my classic endpoints to keep the classic and l4s flows as similar as possible, but to no avail (even without these precautions the results remain roughly the same).

I am sure I am missing some important details in the setup, and I would really appreciate some help.

Regards,

Matteo

P.s.
I just want to point out that by looking at the packet traces everything seems fine: Prague carries the ECN=1, the dualpi2 marks packets with ECN=3, the AccEcn control signals on the ACE fields are coherent, and no losses occur in the Prague flow, when they do happen with Cubic... It just looks like Prague is underperforming for whatever reason. Furthermore, if I switch back to two Cubic flows I measure perfect share, equal delay and equal jitter, so it looks to me like there are no physical impairments on the testbed.

One of the major theoretical concerns with TCP Prague, with respect to deployability in the public Internet, is what happens when the bottleneck is a conventional single-queue AQM supporting ECN. While such an AQM can probably still control a DCTCP-style ECN response sufficiently well, the sustained marking rates required to do so would cause any conventional CC algo sharing that link to collapse to minimum cwnd.

To mitigate this, the "TCP Prague Requirements" draft mentions a need to detect such conventional AQMs and switch to a conventional ECN response. However, I do not see any mechanism to do so in this implementation (only a fallback to conventional TCP if AccECN isn't negotiated, which is a completely different concern). Nor have I seen a detailed description of how such single-queue AQM detection would work.

Is there a plan to address this shortcoming?

Hello all,

I tried to count the numbers of drops in your dualpi2 aqm located at this link : https://github.com/L4STeam/linux/blob/testing/net/sched/sch_dualpi2.c
For this I looked in the source code and I saw this parameter named cnt in this struct
struct { /* Deferred drop statistics */ u32 cnt; /* Packets dropped */ u32 len; /* Bytes dropped */ } deferred_drops;

For me this variable counts the total number of drop in both queues (classic and l4s) in the dualpi2 AQM
when a packet is dropped.

But what I remarked is that this variable that must count drop is incremented only one time ++q->deferred_drops.cnt; at line 596 of the dualpi2's source code in the dequeue function static struct sk_buff *dualpi2_qdisc_dequeue(struct Qdisc *sch).

What I don't understand is why the drops counters q->deferred_drops.cnt is incremented only one time ?
Packet dropping does not happens when there are packet to enqueue and queue limit is reached or drop_early is enable ?

If I mistake, can you show me in the source code where packet dropping happens , please?

Regards

There seems to be a bug in the L4S kernel "testing" branch where when the server is configured with sysctl net.ipv4.tcp_ecn=0/1/2/3, a TCP flow using prague all shows ECT(1) in wireshark.

How to set the ECT(0) and Not-ECT in prague?

Of course, in bbr2 or cubic, there are no problem, sysctl net.ipv4.tcp_ecn=0 means Not-ECT, sysctl net.ipv4.tcp_ecn=1 means ECT(0), sysctl net.ipv4.tcp_ecn=2 means Not-ECT( in this case, is there any problem?), sysctl net.ipv4.tcp_ecn=3 means ECT(1)

I want to set a flow to ECT(1) as low-latency traffic.
So I would like to know if I can selectively use TCP Prague as a congestion control method in socket programming, what parameters can be set, and how to set them.
Looking forward to your reply.

I am looking at the code for TCP Prague at
https://github.com/L4STeam/linux/blob/testing/net/ipv4/tcp_prague.c

One thing that puzzles me is that the congestion window (tp->snd_cwnd) is not reduced immediately when TCP_CA_CWR state is entered.
Instead, ca->cwnd_cnt is updated with the intended reduction (line 580)

The actual CWND reduction is done on the per-ACK processing (line 446). I get the impression that this will slow down the congestion window reduction and that the slow start can overshoot more ?

I realize that it is quite likely that I misunderstand the code here

It would be great if you could set up a GitHub Actions workflow that would build Debian packages, etc. of the kernel for easy experimentation.

(You probably want to do something like this for CI reasons anyway.)

Is it still needed to let BBR2 use ECT(0) in case of sysctl tcp_ecn=1/2 or can we line-up with TCP-Prague?

BBR2 relies on the internally hard-coded assumption that if a CC NEEDS_ECN and NEEDS_ACC_ECN, then it would WANT_ECT1. In fact only a scalable CCs "NEEDS"_ECT1, which in turn NEEDS_ACC_ECN and clearly NEEDS_ECN. Any other reasoning could even lead to having future non-scalable congestion controls that for some other reason NEEDS_ECN and NEEDS_ACC_ECN would also start using ECT1...

If the sender is not supporting ACC_ECN, BB2 (and even Prague) cannot use a classic CC (except maybe when Prague would be falling back to Reno if no ACC_ECN is supported).

I'd like to include TCP Prague as a candidate in a list of TCP optimizations I am working on performance comparisons for. My current testing setup is based on the somewhat recent kernel version 6.3, but haven't found a good way to get TCP Prague enabled for that kernel version.
Due to the way this repo is set up applying the commits on top of the Linux kernel in this repo doesn't really work. Is there an existing effort to get TCP Prague into kernel version 6?

Thank you in advance!

Till

Presently you calculate avg_psize = d_packets / acked_bytes. This will always be zero unless a very large stretched ack is involved, and appears to calculate the reciprocal of the value actually needed.

There seems to be a bug in the L4S kernel "testing" branch where when the server is configured with sysctl net.ipv4.tcp_ecn=0, a TCP server using cubic still uses RFC3168 ECN if the client offers AccECN.

In this case tcpdump traces show the server sending packets with code point ECT(0) and signaling its cwnd reductions with CWR.

In the test where I noticed this, the CMTS was not setting the CE code point, and the CWR signals were during fast recovery.

This is with the pre-built kernel linux-image-5.10.31-3cc3851880a1-prague-37_1_amd64.deb 👍
uname -a shows:

5.10.31-3cc3851880a1-prague-37 #1 SMP Tue Jun 29 07:47:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

proposed to init cwr_stamp with a value that is 10 seconds ago:

ca->cwr_stamp = tp->tcp_mstamp - 10000000; // init 10 seconds ago

In the tcp_prague.c code, in prague_init() it disables Prague if no ECN support has been negotiated:

    if (!tcp_ecn_mode_any(tp) &&
        sk->sk_state != TCP_LISTEN && sk->sk_state != TCP_CLOSE) {
            prague_release(sk);
            LOG(sk, "Switching to pure reno [ecn_status=%u,sk_state=%u]",
                tcp_ecn_mode_any(tp), sk->sk_state);
            inet_csk(sk)->icsk_ca_ops = &prague_reno;
            return;
    }

    tp->ecn_flags |= TCP_ECN_ECT_1;
    ...

However, AFAICT that means if RFC3168 support is negotiated then Prague CC stays enabled and enables ECT1.

Probably this should be checking for AccECN support instead:

    if (!tcp_ecn_mode_accecn(tp) &&
      ....

That would match more closely the logic in Ilpo's patch for bbr2.c:

git show 4b75165
commit 4b75165
Author: Ilpo Järvinen [email protected]
Date: Mon Jun 28 11:07:23 2021 +0300

l4s: make BBR v2 want ECT(1)

Signed-off-by: Ilpo Järvinen <[email protected]>

diff --git a/net/ipv4/tcp_bbr2.c b/net/ipv4/tcp_bbr2.c
index 5510adc92bbb4..2b4fc9abb1dbc 100644
--- a/net/ipv4/tcp_bbr2.c
+++ b/net/ipv4/tcp_bbr2.c
@@ -2471,6 +2471,8 @@ static void bbr2_init(struct sock *sk)
bbr->alpha_last_delivered_ce = 0;

    tp->fast_ack_mode = min_t(u32, 0x2U, bbr_fast_ack_mode);

•   if (tcp_ecn_mode_accecn(tp))
  

•           tp->ecn_flags |= TCP_ECN_ECT_1;
  

}

/* Core TCP stack informs us that the given skb was just marked lost. */

This issue is in the current tcp_prague.c from:

3cc3851 (tag: testing-build, l4steam/testing, l4s/testing) github workflows

7d7c8a9 (l4steam/tcp_prague, l4s/tcp_prague) tcp: Optionally pace IW