In order to scale test a BNG with a large number of dual-stack IPoE sessions, bngblast

Changing start source MAC address and DUID so that multiple instances of bngblaster can be run towards the same BNG about bngblaster HOT 12 CLOSED

rtbrick commented on September 6, 2024

Changing start source MAC address and DUID so that multiple instances of bngblaster can be run towards the same BNG

from bngblaster.

Comments (12)

GIC-de commented on September 6, 2024

Could you share your configuration? Actually would assume that a single BNG Blaster instance is able to handle much more sessions.

Are you testing with traffic streams per session? If yes, have you tried to assign the traffic streams to TX threads?

In general, I guess I can add a configurable session-id offset here, which will also change the MAC address because the last 6 bytes of MAC address are always equal to the session-id.

from bngblaster.

mivsvit commented on September 6, 2024

Thanks very much for the useful reply.
Not tried traffic-streams so far, so no streams config. Only session-traffic:

    "interfaces": {
        "qdisc-bypass": true,
        "io-mode": "packet_mmap_raw",
         "network": {
         "interface": "ens160.2016",
         "address": "172.31.19.5",
         "gateway": "172.31.19.4",
         "address-ipv6": "2001:db8:2016::2",
         "gateway-ipv6": "2001:db8:2016::1"
        },
        "access": [
        {
            "interface": "ens224f1",
            "type": "ipoe",
            "qinq": "false",
            "outer-vlan-min": 2016,
            "outer-vlan-max": 2048,
            "inner-vlan-min": 2,
            "inner-vlan-max": 4094,
            "ipv4": true,
            "ipv6": true,
            "i1_start": "500",
            "i2_start": "1000",
            "vlan-mode": "1:1"
        }
     ]
    },
        "sessions": {
        "count": 25000,
        "session-time": 0,
        "max-outstanding": 16000,
        "start-rate": 1000,
        "stop-rate": 1000
    },
        "access-line": {
        "agent-remote-id": "ABCDEFGH{session-global}",
        "agent-circuit-id": "0.0.0.0/0.0.0.0 eth 0:{session-global}"
    },
    "dhcp": {
        "broadcast": true,
        "enable": true
    },
    "dhcpv6": {
        "enable": true,
        "rapid-commit": true
    },
      "session-traffic": {
        "ipv4-pps": 1,
        "ipv6pd-pps": 1
    }

from bngblaster.

GIC-de commented on September 6, 2024

You can try with the following configuration to see if performance becomes better.

{
    "interfaces": {
        "tx-interval": 1,
        "rx-interval": 1,
        "io-slots": 4096,
    }
}

I expect also a mayor improvement as soon as we support DPDK. In general I plan to work on performance in the next weeks to be able to scale up to 10M PPS.

from bngblaster.

mivsvit commented on September 6, 2024

Thanks for the suggestions.

Adding the suggested config below and sending session-traffic does not seem to improve performance, unfortunately.

{
    "interfaces": {
        "tx-interval": 1,
        "rx-interval": 1,
        "io-slots": 4096,
    }
}

With regards to the earlier comment about traffic-streams, the documentation seems to imply 200-300kpps is achievable with one thread, 1Mpps with multiple threads.

Trying this simple traffic-stream config seems to result in traffic loss earlier than when using session-traffic mostly due to TX send failed and at about 40kpps.

        "access": [
        {
            "interface": "ens224f1",
            "type": "ipoe",
            "qinq": "false",
            "outer-vlan-min": 2016,
            "outer-vlan-max": 2048,
            "inner-vlan-min": 2,
            "inner-vlan-max": 4094,
            "ipv4": true,
            "ipv6": true,
             "stream-group-id": 1,
            "vlan-mode": "1:1"
        }
     ]
    },
        "streams": [
        {
            "stream-group-id": 1,
            "name": "S1v4",
            "type": "ipv4",
            "direction": "both",
            "length": 256,
            "threaded": true,
            "thread-group": 1,
            "pps": 1
        },
                {
            "stream-group-id": 1,
            "name": "S1v6",
            "type": "ipv6pd",
            "direction": "both",
            "length": 256,
            "threaded": true,
            "thread-group": 2,
            "pps": 1
        }

Approx 10k dual-stack sessions with 1pps traffic:

Sessions           10000 (0 PPPoE / 10000 IPoE)
  Established       9630 [##########################################################  ]
  Outstanding          0 [                                                            ]
  Terminated           0 [                                                            ]
  DHCPv4     10000/10000 [############################################################]
  DHCPv6      9630/10000 [##########################################################  ]
  Setup Time       80904 ms
  Setup Rate      119.03 CPS (MIN: 1.06 AVG: 51.98 MAX: 126.19)
  Flapped              0
  
Traffic Flows Verified
  Stream     37932/40000 [#########################################################   ]
  
  Network Interface ( ens160.2016 )
  Tx Packets                   7499901 |  19383 PPS      41867 Kbps
  Rx Packets                   7232919 |  19074 PPS      41198 Kbps
  Tx Stream Packets            7499521 |  19382 PPS
  Rx Stream Packets            7232539 |  19073 PPS     281346 Loss
  Tx Multicast Packets               0 |      0 PPS 
Access Interface ( ens224f1 )
  Tx Packets                   7805860 |  19153 PPS      42565 Kbps
  Rx Packets                   7391646 |  19357 PPS      42430 Kbps
  Tx Stream Packets            7503485 |  19079 PPS
  Rx Stream Packets            7299483 |  19356 PPS     211846 Loss
  Rx Multicast Packets               0 |      0 PPS          0 Loss

Report:

Sessions PPPoE: 0 IPoE: 64000
Sessions established: 576/64000
DHCPv6 sessions established: 3977
Setup Time: 47102 ms
Setup Rate: 12.23 CPS (MIN: 0.51 AVG: 4.56 MAX: 12.23)
Flapped: 0

Network Interface ( ens160.2016 ):
  TX:                     83813 packets
  RX:                     79873 packets
  TX Stream:              83804 packets
  RX Stream:              78085 packets (0 loss)
  TX Multicast:               0 packets
  RX Drop Unknown:            0 packets
  TX Encode Error:            0
  RX Decode Error:            0 packets
  TX Send Failed:             0
  TX No Buffer:               0
  TX Poll Kernel:             0
  RX Poll Kernel:         27996

Access Interface ( ens224f1 ):
  TX:                    322816 packets
  RX:                    123150 packets
  TX Stream:              83838 packets
  RX Stream:              78019 packets (0 loss)

  TX Poll Kernel:             0
  RX Poll Kernel:         87046

Access Interface ( ens224f1 ):
  TX:                  14438634 packets
  RX:                  13655472 packets
  TX Stream:           14002837 packets
  RX Stream:           13504899 packets (484425 loss)
  RX Multicast:               0 packets (0 loss)
  RX Drop Unknown:           58 packets
  TX Encode Error:            0 packets
  RX Decode Error:            0 packets
  TX Send Failed:        330793
  TX No Buffer:               0
  TX Poll Kernel:             0
  RX Poll Kernel:         86355

  Access Interface Protocol Packet Stats:
    ARP    TX:      48863 RX:      48862
    PADI   TX:          0 RX:          0
    PADO   TX:          0 RX:          0
    PADR   TX:          0 RX:          0
    PADS   TX:          0 RX:          0
    PADT   TX:          0 RX:          0
    LCP    TX:          0 RX:          0
    PAP    TX:          0 RX:          0
    CHAP   TX:          0 RX:          0
    IPCP   TX:          0 RX:          0
    IP6CP  TX:          0 RX:          0
    IGMP   TX:          0 RX:          0
    ICMP   TX:          0 RX:          0
    DHCP   TX:     177544 RX:      39975
    DHCPv6 TX:     190424 RX:      38212
    ICMPv6 TX:      18966 RX:      23268
    IPv4 Fragmented       RX:          0

  Access Interface Protocol Timeout Stats:
    LCP Echo Request:          0
    LCP Request:               0
    IPCP Request:              0
    IP6CP Request:             0
    PAP:                       0
    CHAP:                      0
    DHCP Request:         127554
    DHCPv6 Request:       151842
    ICMPv6 RS:                 0

Traffic Streams:
  Verified Traffic Flows: 37932/40000
  First Sequence Number Received  MIN:        1 MAX:        1
  Flow Receive Packet Loss        MIN:        2 MAX:      124
  Flow Receive Delay (msec)       MIN:    0.000 MAX:   66.478

The above was with two thread-groups, one for v4 and one for v6pd. I guess to scale it, the sessions need to be split across more thread-groups - not sure what the easiest way of doing that is. The idea is to verify forwarding for both protocol stacks for all the sessions. Or is the conclusion to wait for DPDK support?

from bngblaster.

GIC-de commented on September 6, 2024

A significant amount of loss is caused by TX Send Failed which is currently not retried and therefore counted as a loss. I plan to add a retry queue to retry packets if send has failed to ensure that stats and loss stats are correct.

In the meantime, the number of send errors could be reduced by increasing the queue size of your interfaces.

If you execute ethtool -g it shows you currently applied and max HW queue size.

$ sudo ethtool -g ens5f1
Ring parameters for ens5f1:
Pre-set maximums:
RX:                4096
RX Mini:        0
RX Jumbo:        0
TX:                4096
Current hardware settings:
RX:                512
RX Mini:        0
RX Jumbo:        0
TX:                512

Which can then be changed with:

sudo ethtool -G ens5f1 tx 4096 rx 4096
sudo ethtool -G ens5f1 tx 4096 rx 4096

You can even change the software queue size:

sudo ip link set txqueuelen 4096 dev pci1f2
sudo ip link set txqueuelen 4096 dev pci1f3

from bngblaster.

GIC-de commented on September 6, 2024

I can also see some improvement if the interfaces used by Blaster are configured without any family so that linux kernel is not handling those packets.

This can be also archived via netplan using the following configuration for each BNG Blaster interface.

network:
version: 2
renderer: networkd
ethernets:
    eth1:
    dhcp4: no
    dhcp6: no
    link-local: []
    mtu: 9000
    eth2:
    dhcp4: no
    dhcp6: no
    link-local: []
    mtu: 9000

https://rtbrick.github.io/bngblaster/interfaces.html

from bngblaster.

GIC-de commented on September 6, 2024

Could you even share the CPU model which is used?

from bngblaster.

mivsvit commented on September 6, 2024

Thanks, hadn't tried changing the software queue size but already had TX HW queue size at the max, and no protocol family configured on the interface.

$ sudo ethtool -g ens224f1
Ring parameters for ens224f1:
Pre-set maximums:
RX:             4078
RX Mini:        0
RX Jumbo:       0
TX:             4078
Current hardware settings:
RX:             4078
RX Mini:        0
RX Jumbo:       0
TX:             4078

$ ip addr show dev ens224f1
5: ens224f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 4096
    link/ether 80:61:5f:11:ff:ff brd ff:ff:ff:ff:ff:ff
$

The suggestion to increase the software queue size improves things slightly but does not eliminate the need to run multiple instances.

It's running on a VM on an ESXi hypervisor with 16 Intel Xeon E5-2667 3.20GHz cores (32 threads).

~ vim-cmd hostsvc/hostsummary | grep cpuModel
      cpuModel = "Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz",
~ esxcli hardware cpu global get
   CPU Packages: 2
   CPU Cores: 16
   CPU Threads: 32   
~    esxtop
 2:03:35pm up 50 days 54 min, 1008 worlds, 14 VMs, 27 vCPUs; CPU load average: 0.19, 0.20, 0.19
PCPU USED(%): 5.6 6.8 4.7 8.7 6.1 5.7 5.3 6.8 6.1 4.2 7.4 2.5 7.1 3.8 5.4 6.8 4.0 0.1 0.1 110 0.7 0.5 0.5 0.3 0.4 0.5 0.7 0.3 0.4 0.2 0.2 0.0 NUMA: 5.8 7.5 AVG: 6.6
PCPU UTIL(%):  20  23  16  27  19  18  16  20  19  14  21 8.7  21  13  17  21 6.8 0.3 0.3 100 2.4 1.5 1.6 0.9 1.1 1.6 2.1 0.9 1.2 0.9 0.5 0.1 NUMA:  18 7.6 AVG:  13
CORE UTIL(%):  35      37      33      33      29      27      31      33     7.0     100     3.7     2.4     2.7     2.9     2.0     nan     NUMA:  32  15 AVG:  23

        ID        GID NAME             NWLD   %USED    %RUN    %SYS   %WAIT %VMWAIT    %RDY   %IDLE  %OVRLP   %CSTP  %MLMTD  %SWPWT
     44663      44663 bngblaster        17   77.50  225.54    0.07 1489.35    0.00    3.03  598.63    0.33    0.00    0.00    0.00

from bngblaster.

GIC-de commented on September 6, 2024

I added a new configuration option mac-modifier, which allows changing the third byte of the automatically generated access session MAC address. This should suit your use case.

Pull Request: #75

{
    "interfaces": {
        "mac-modifier": 1
    }
}

Could you please verify this before merging into the dev branch?

You can build manually from mac-modifier branch or use the following dev package:
https://github.com/rtbrick/bngblaster/actions/runs/2165874892

from bngblaster.

mivsvit commented on September 6, 2024

Many thanks for adding the mac-modifier config option.

It works for my use-case.

from bngblaster.

GIC-de commented on September 6, 2024

This issue is fixed in version 0.7.2.

Please verify and close the issue if the solution is accepted!

from bngblaster.

mivsvit commented on September 6, 2024

mac-modifer feature working fine in version 0.7.2.
Many thanks for adding this.

from bngblaster.

Changing start source MAC address and DUID so that multiple instances of bngblaster can be run towards the same BNG about bngblaster HOT 12 CLOSED

Comments (12)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent