Comments (12)
Could you share your configuration? Actually would assume that a single BNG Blaster instance is able to handle much more sessions.
Are you testing with traffic streams per session? If yes, have you tried to assign the traffic streams to TX threads?
In general, I guess I can add a configurable session-id offset here, which will also change the MAC address because the last 6 bytes of MAC address are always equal to the session-id.
from bngblaster.
Thanks very much for the useful reply.
Not tried traffic-streams so far, so no streams
config. Only session-traffic
:
"interfaces": {
"qdisc-bypass": true,
"io-mode": "packet_mmap_raw",
"network": {
"interface": "ens160.2016",
"address": "172.31.19.5",
"gateway": "172.31.19.4",
"address-ipv6": "2001:db8:2016::2",
"gateway-ipv6": "2001:db8:2016::1"
},
"access": [
{
"interface": "ens224f1",
"type": "ipoe",
"qinq": "false",
"outer-vlan-min": 2016,
"outer-vlan-max": 2048,
"inner-vlan-min": 2,
"inner-vlan-max": 4094,
"ipv4": true,
"ipv6": true,
"i1_start": "500",
"i2_start": "1000",
"vlan-mode": "1:1"
}
]
},
"sessions": {
"count": 25000,
"session-time": 0,
"max-outstanding": 16000,
"start-rate": 1000,
"stop-rate": 1000
},
"access-line": {
"agent-remote-id": "ABCDEFGH{session-global}",
"agent-circuit-id": "0.0.0.0/0.0.0.0 eth 0:{session-global}"
},
"dhcp": {
"broadcast": true,
"enable": true
},
"dhcpv6": {
"enable": true,
"rapid-commit": true
},
"session-traffic": {
"ipv4-pps": 1,
"ipv6pd-pps": 1
}
from bngblaster.
You can try with the following configuration to see if performance becomes better.
{
"interfaces": {
"tx-interval": 1,
"rx-interval": 1,
"io-slots": 4096,
}
}
I expect also a mayor improvement as soon as we support DPDK. In general I plan to work on performance in the next weeks to be able to scale up to 10M PPS.
from bngblaster.
Thanks for the suggestions.
Adding the suggested config below and sending session-traffic does not seem to improve performance, unfortunately.
{
"interfaces": {
"tx-interval": 1,
"rx-interval": 1,
"io-slots": 4096,
}
}
With regards to the earlier comment about traffic-streams, the documentation seems to imply 200-300kpps is achievable with one thread, 1Mpps with multiple threads.
Trying this simple traffic-stream config seems to result in traffic loss earlier than when using session-traffic mostly due to TX send failed and at about 40kpps.
"access": [
{
"interface": "ens224f1",
"type": "ipoe",
"qinq": "false",
"outer-vlan-min": 2016,
"outer-vlan-max": 2048,
"inner-vlan-min": 2,
"inner-vlan-max": 4094,
"ipv4": true,
"ipv6": true,
"stream-group-id": 1,
"vlan-mode": "1:1"
}
]
},
"streams": [
{
"stream-group-id": 1,
"name": "S1v4",
"type": "ipv4",
"direction": "both",
"length": 256,
"threaded": true,
"thread-group": 1,
"pps": 1
},
{
"stream-group-id": 1,
"name": "S1v6",
"type": "ipv6pd",
"direction": "both",
"length": 256,
"threaded": true,
"thread-group": 2,
"pps": 1
}
Approx 10k dual-stack sessions with 1pps traffic:
Sessions 10000 (0 PPPoE / 10000 IPoE)
Established 9630 [########################################################## ]
Outstanding 0 [ ]
Terminated 0 [ ]
DHCPv4 10000/10000 [############################################################]
DHCPv6 9630/10000 [########################################################## ]
Setup Time 80904 ms
Setup Rate 119.03 CPS (MIN: 1.06 AVG: 51.98 MAX: 126.19)
Flapped 0
Traffic Flows Verified
Stream 37932/40000 [######################################################### ]
Network Interface ( ens160.2016 )
Tx Packets 7499901 | 19383 PPS 41867 Kbps
Rx Packets 7232919 | 19074 PPS 41198 Kbps
Tx Stream Packets 7499521 | 19382 PPS
Rx Stream Packets 7232539 | 19073 PPS 281346 Loss
Tx Multicast Packets 0 | 0 PPS
Access Interface ( ens224f1 )
Tx Packets 7805860 | 19153 PPS 42565 Kbps
Rx Packets 7391646 | 19357 PPS 42430 Kbps
Tx Stream Packets 7503485 | 19079 PPS
Rx Stream Packets 7299483 | 19356 PPS 211846 Loss
Rx Multicast Packets 0 | 0 PPS 0 Loss
Report:
Sessions PPPoE: 0 IPoE: 64000
Sessions established: 576/64000
DHCPv6 sessions established: 3977
Setup Time: 47102 ms
Setup Rate: 12.23 CPS (MIN: 0.51 AVG: 4.56 MAX: 12.23)
Flapped: 0
Network Interface ( ens160.2016 ):
TX: 83813 packets
RX: 79873 packets
TX Stream: 83804 packets
RX Stream: 78085 packets (0 loss)
TX Multicast: 0 packets
RX Drop Unknown: 0 packets
TX Encode Error: 0
RX Decode Error: 0 packets
TX Send Failed: 0
TX No Buffer: 0
TX Poll Kernel: 0
RX Poll Kernel: 27996
Access Interface ( ens224f1 ):
TX: 322816 packets
RX: 123150 packets
TX Stream: 83838 packets
RX Stream: 78019 packets (0 loss)
TX Poll Kernel: 0
RX Poll Kernel: 87046
Access Interface ( ens224f1 ):
TX: 14438634 packets
RX: 13655472 packets
TX Stream: 14002837 packets
RX Stream: 13504899 packets (484425 loss)
RX Multicast: 0 packets (0 loss)
RX Drop Unknown: 58 packets
TX Encode Error: 0 packets
RX Decode Error: 0 packets
TX Send Failed: 330793
TX No Buffer: 0
TX Poll Kernel: 0
RX Poll Kernel: 86355
Access Interface Protocol Packet Stats:
ARP TX: 48863 RX: 48862
PADI TX: 0 RX: 0
PADO TX: 0 RX: 0
PADR TX: 0 RX: 0
PADS TX: 0 RX: 0
PADT TX: 0 RX: 0
LCP TX: 0 RX: 0
PAP TX: 0 RX: 0
CHAP TX: 0 RX: 0
IPCP TX: 0 RX: 0
IP6CP TX: 0 RX: 0
IGMP TX: 0 RX: 0
ICMP TX: 0 RX: 0
DHCP TX: 177544 RX: 39975
DHCPv6 TX: 190424 RX: 38212
ICMPv6 TX: 18966 RX: 23268
IPv4 Fragmented RX: 0
Access Interface Protocol Timeout Stats:
LCP Echo Request: 0
LCP Request: 0
IPCP Request: 0
IP6CP Request: 0
PAP: 0
CHAP: 0
DHCP Request: 127554
DHCPv6 Request: 151842
ICMPv6 RS: 0
Traffic Streams:
Verified Traffic Flows: 37932/40000
First Sequence Number Received MIN: 1 MAX: 1
Flow Receive Packet Loss MIN: 2 MAX: 124
Flow Receive Delay (msec) MIN: 0.000 MAX: 66.478
The above was with two thread-groups, one for v4 and one for v6pd. I guess to scale it, the sessions need to be split across more thread-groups - not sure what the easiest way of doing that is. The idea is to verify forwarding for both protocol stacks for all the sessions. Or is the conclusion to wait for DPDK support?
from bngblaster.
A significant amount of loss is caused by TX Send Failed
which is currently not retried and therefore counted as a loss. I plan to add a retry queue to retry packets if send has failed to ensure that stats and loss stats are correct.
In the meantime, the number of send errors could be reduced by increasing the queue size of your interfaces.
If you execute ethtool -g
it shows you currently applied and max HW queue size.
$ sudo ethtool -g ens5f1
Ring parameters for ens5f1:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 512
RX Mini: 0
RX Jumbo: 0
TX: 512
Which can then be changed with:
sudo ethtool -G ens5f1 tx 4096 rx 4096
sudo ethtool -G ens5f1 tx 4096 rx 4096
You can even change the software queue size:
sudo ip link set txqueuelen 4096 dev pci1f2
sudo ip link set txqueuelen 4096 dev pci1f3
from bngblaster.
I can also see some improvement if the interfaces used by Blaster are configured without any family so that linux kernel is not handling those packets.
This can be also archived via netplan using the following configuration for each BNG Blaster interface.
network:
version: 2
renderer: networkd
ethernets:
eth1:
dhcp4: no
dhcp6: no
link-local: []
mtu: 9000
eth2:
dhcp4: no
dhcp6: no
link-local: []
mtu: 9000
https://rtbrick.github.io/bngblaster/interfaces.html
from bngblaster.
Could you even share the CPU model which is used?
from bngblaster.
Thanks, hadn't tried changing the software queue size but already had TX HW queue size at the max, and no protocol family configured on the interface.
$ sudo ethtool -g ens224f1
Ring parameters for ens224f1:
Pre-set maximums:
RX: 4078
RX Mini: 0
RX Jumbo: 0
TX: 4078
Current hardware settings:
RX: 4078
RX Mini: 0
RX Jumbo: 0
TX: 4078
$ ip addr show dev ens224f1
5: ens224f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 4096
link/ether 80:61:5f:11:ff:ff brd ff:ff:ff:ff:ff:ff
$
The suggestion to increase the software queue size improves things slightly but does not eliminate the need to run multiple instances.
It's running on a VM on an ESXi hypervisor with 16 Intel Xeon E5-2667 3.20GHz cores (32 threads).
~ vim-cmd hostsvc/hostsummary | grep cpuModel
cpuModel = "Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz",
~ esxcli hardware cpu global get
CPU Packages: 2
CPU Cores: 16
CPU Threads: 32
~ esxtop
2:03:35pm up 50 days 54 min, 1008 worlds, 14 VMs, 27 vCPUs; CPU load average: 0.19, 0.20, 0.19
PCPU USED(%): 5.6 6.8 4.7 8.7 6.1 5.7 5.3 6.8 6.1 4.2 7.4 2.5 7.1 3.8 5.4 6.8 4.0 0.1 0.1 110 0.7 0.5 0.5 0.3 0.4 0.5 0.7 0.3 0.4 0.2 0.2 0.0 NUMA: 5.8 7.5 AVG: 6.6
PCPU UTIL(%): 20 23 16 27 19 18 16 20 19 14 21 8.7 21 13 17 21 6.8 0.3 0.3 100 2.4 1.5 1.6 0.9 1.1 1.6 2.1 0.9 1.2 0.9 0.5 0.1 NUMA: 18 7.6 AVG: 13
CORE UTIL(%): 35 37 33 33 29 27 31 33 7.0 100 3.7 2.4 2.7 2.9 2.0 nan NUMA: 32 15 AVG: 23
ID GID NAME NWLD %USED %RUN %SYS %WAIT %VMWAIT %RDY %IDLE %OVRLP %CSTP %MLMTD %SWPWT
44663 44663 bngblaster 17 77.50 225.54 0.07 1489.35 0.00 3.03 598.63 0.33 0.00 0.00 0.00
from bngblaster.
I added a new configuration option mac-modifier
, which allows changing the third byte of the automatically generated access session MAC address. This should suit your use case.
Pull Request: #75
{
"interfaces": {
"mac-modifier": 1
}
}
Could you please verify this before merging into the dev
branch?
You can build manually from mac-modifier
branch or use the following dev package:
https://github.com/rtbrick/bngblaster/actions/runs/2165874892
from bngblaster.
Many thanks for adding the mac-modifier
config option.
It works for my use-case.
from bngblaster.
This issue is fixed in version 0.7.2
.
Please verify and close the issue if the solution is accepted!
from bngblaster.
mac-modifer
feature working fine in version 0.7.2
.
Many thanks for adding this.
from bngblaster.
Related Issues (20)
- UDP Checksum is shown zero in Subscriber Data Packets HOT 1
- LSPGEN - potential overflow HOT 1
- Add command to dynamically enable/disable PCAP
- Downstream traffic loses header information for one of the L2TP subscribers. HOT 3
- Command li-flows does not report downstream flows from BNGs running in HA mode. HOT 2
- Not seeing ICMP replies from the blaster when subscriber IP is pinged HOT 2
- Counts corresponding to tx/rx remain at zero for one of the flows and this affects the verified count. HOT 2
- TCP-FIN, TCP-FIN-ACK Messages handling in blaster.
- Traffic flows are associated with the member interfaces of non-active lag interface. HOT 1
- Support for packet-crafting
- After restarting the DUT, IPoE subscribers get re-established whereas PPPoE/L2TP subscribers do not. HOT 1
- While defining streams for NAT, instead of specifying each stream in blaster file, step can be introduced
- DHCPv4/v6 Server support for IPoE Subscribers in BNG Blaster
- Blaster crash observed after fibd restart at rbfs side HOT 2
- Support for PPP without authentication HOT 4
- BNG Bluster doesn't respond to the PADO message HOT 9
- null DHCP siaddr being used for renews and releases HOT 3
- Question for implementing rx_avg_delay_us in bbl_stream.c
- IS-IS Broadcast Support HOT 1
- BNGBlaster not sending IA_NA request for PPPoE HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bngblaster.