Hello,

I am using the following code to generate 25K packets per second on a single straight pipeline (no network interface involved).

//--------Program Start----------
rs :: RatedSource(LENGTH 64, RATE 25000, LIMIT 500000, STOP true)
        -> NumberPacket(OFFSET 0)
        -> UDPIP6Encap(2001:2001:2001:2001::1, 1234, 2001:2001:2001:2001::2, 1234)
        -> EtherEncap(0x86DD, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db)
        -> record :: RecordTimestamp()
        -> CheckNumberPacket(OFFSET 62, COUNT 500000)
        -> diff :: TimestampDiff(OFFSET 62, N 500000, RECORDER record)
        -> counter :: AverageCounter()
        -> Discard;
//--------Program End----------

DriverManager(wait,read diff.average, read counter.count, read counter.rate)

CPU utilization is around %50 during execution and following is the output.

diff.average:
2.146926
counter.count:
500000
counter.rate:
16257.5191026

So the reception rate is at 16Kpps while the generation rate is at 25Kpps, which I believe is due to the numbering and encapsulation elements. I was wondering whether I could place a ReplayUnqueue between EtherEncap and RecordTimestamp, and configure (is it even possible? There is no wiki for the element) it to release packets at a rate packets per second. Or is there any other element that I could use to achieve this?

Ubuntu 17.10 artful, Fastclick version 2.1, compiled with
./configure --enable-multithread --enable-intel-cpu --enable-user-multithread --enable-ip6

I am trying to pin elements related to packet generation to CPU core 7, while letting the packet sink related elements stay on CPU core 6. Following is the code

//------------Declarations-----------

OUT ::  Queue(500000) -> ToDevice(VETH-SRC_00);
IN  ::  FromDevice(VETH-SNK_00, METHOD LINUX);


//--------Program Start----------
rs :: RatedSource(LENGTH 64, RATE 50000, LIMIT 1000000, STOP true)
        -> numPack :: NumberPacket(OFFSET 0)
        -> u6Encap :: UDPIP6Encap(2001:2001:2001:2001::1, 1234, 64:ff9b::192.168.2.5, 1234)
        -> ethEncap :: EtherEncap(0x86DD, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db)
        -> record :: RecordTimestamp()
        -> OUT;

IN
        -> chnum :: CheckNumberPacket(OFFSET 62, COUNT 1000000)
        -> diff :: TimestampDiff(OFFSET 62, RECORDER record)
        -> counter :: AverageCounter()
        -> Discard;
//--------Program End----------

StaticThreadSched(rs 1);
StaticThreadSched(numPack 1);
StaticThreadSched(u6Encap 1)
StaticThreadSched(record 1);
StaticThreadSched(OUT 1);

DriverManager(wait,read diff.average, read counter.count, read counter.rate)

Starting the code with following arguments
click analyser.click -j 2 -a=6

It segfaults after couple of seconds. Following is the trace.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `click overr.click -j 2 -a=6'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  RecordTimestamp::get (i=130706, this=0x5623394f17f0) at ../elements/analysis/recordtimestamp.hh:102
102         Timestamp t = _timestamps[i];
[Current thread is 1 (Thread 0x7fe285325780 (LWP 12887))]
(gdb) bt
#0  RecordTimestamp::get (i=130706, this=0x5623394f17f0) at ../elements/analysis/recordtimestamp.hh:102
#1  TimestampDiff::smaction (p=0x5623394f9690, this=0x5623394e7930) at ../elements/analysis/timestampdiff.cc:219
#2  TimestampDiff::push (this=0x5623394e7930, p=0x5623394f9690) at ../elements/analysis/timestampdiff.cc:247
#3  0x000056233793b470 in Element::Port::push (p=<optimized out>, this=<optimized out>) at ../include/click/element.hh:718
#4  FromDevice::selected (this=0x5623394e3f60) at ../elements/userlevel/fromdevice.cc:522
#5  0x00005623379a35b2 in SelectSet::call_selected (mask=<optimized out>, fd=3, this=0x5623394e3af8) at ../lib/selectset.cc:367
#6  SelectSet::run_selects_poll (this=this@entry=0x5623394e3af8, thread=thread@entry=0x5623394e3a80) at ../lib/selectset.cc:474
#7  0x00005623379a3881 in SelectSet::run_selects (this=this@entry=0x5623394e3af8, thread=thread@entry=0x5623394e3a80) at ../lib/selectset.cc:581
#8  0x0000562337994f79 in RouterThread::run_os (this=0x5623394e3a80) at ../lib/routerthread.cc:476
#9  RouterThread::driver (this=0x5623394e3a80) at ../lib/routerthread.cc:645
#10 0x00005623378896cb in main (argc=<optimized out>, argv=<optimized out>) at click.cc:809
(gdb)

Then tried to pin both RecordTimestamp and TimestampDiff onto the same core and I got the following

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `click overr.click -j 2 -a=6'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  RecordTimestamp::get (i=261804, this=0x560692f6c7f0) at ../elements/analysis/recordtimestamp.hh:102
102         Timestamp t = _timestamps[i];
[Current thread is 1 (Thread 0x7f29ed9b8700 (LWP 13034))]
(gdb) bt
#0  RecordTimestamp::get (i=261804, this=0x560692f6c7f0) at ../elements/analysis/recordtimestamp.hh:102
#1  TimestampDiff::smaction (p=0x7f29e8000c00, this=0x560692f62930) at ../elements/analysis/timestampdiff.cc:219
#2  TimestampDiff::push (this=0x560692f62930, p=0x7f29e8000c00) at ../elements/analysis/timestampdiff.cc:247
#3  0x000056069163c470 in Element::Port::push (p=<optimized out>, this=<optimized out>) at ../include/click/element.hh:718
#4  FromDevice::selected (this=0x560692f5ef60) at ../elements/userlevel/fromdevice.cc:522
#5  0x00005606916a45b2 in SelectSet::call_selected (mask=<optimized out>, fd=3, this=0x560692f5ec48) at ../lib/selectset.cc:367
#6  SelectSet::run_selects_poll (this=this@entry=0x560692f5ec48, thread=thread@entry=0x560692f5ebd0) at ../lib/selectset.cc:474
#7  0x00005606916a4881 in SelectSet::run_selects (this=this@entry=0x560692f5ec48, thread=thread@entry=0x560692f5ebd0) at ../lib/selectset.cc:581
#8  0x0000560691695f79 in RouterThread::run_os (this=0x560692f5ebd0) at ../lib/routerthread.cc:476
#9  RouterThread::driver (this=0x560692f5ebd0) at ../lib/routerthread.cc:645
#10 0x00005606916543f9 in thread_driver (user_data=<optimized out>) at click.cc:483
#11 0x00007f29eee357fc in start_thread (arg=0x7f29ed9b8700) at pthread_create.c:465
#12 0x00007f29ee26fb5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Any ideas?

Results in a crash.

Does not occur without --enable-netmap-pool.

The simplest way to reproduce is to forward a packet received with FromNetmapDevice to KernelTap element. KernelTap calls Packet::push(4), which turns into Packet::expensive_push, since there is no headroom in a Netmap packet. Packet::expensive_push calls Packet::expensive_uniqueify which frees the packet and allocates a new one. After the push, KernelTap writes packet content to TAP and calls Packet::kill. Probably this is the moment, when the double free happens.

To reproduce you can use a veth pair, either patched or emulated.

Preparation:

insmod /usr/local/lib/netmap.ko
ip link add type veth
ip link set veth0 up
ip link set veth1 up

Then run the following configuration:

define($burst 32)

kt :: KernelTap(192.168.5.1/24);
kt -> Discard;
FromNetmapDevice("netmap:veth1", BURST $burst) -> Print("Received FromNetmapDevice, forwarded to KernelTap", ACTIVE 1) -> kt;

Click process will crash after a while, when kernel will send any packet to veth0:

root@localhost:/merged# click famtar/test/kill.click
[  136.367326] 966.520454 [1262] generic_netmap_attach     Emulated adapter for veth1 created (prev was           (null))
966.520336 nm_open [861] overriding ARG3 4096
[  136.455915] 966.609050 [ 441] generic_netmap_register   Emulated adapter for veth1 activated
[  136.457532] 966.610669 [ 486] generic_netmap_register   Emulated adapter: ring 'veth1 RX0' activated
[  136.459186] 966.612324 [ 493] generic_netmap_register   Emulated adapter: ring 'veth1 TX0' activated
[  136.460879] 966.614016 [ 818] tc_configure              ifp veth1 qdisc netmapemu parent 4294967295 handle 0
Allocated 4096 buffers from Netmap buffer pool
966.617968 nm_open [874] overriding ifname veth1 ringid 0x1000 flags 0x4
966.617997 nm_mmap [984] do not mmap, inherit from parent
Received FromNetmapDevice, forwarded to KernelTap:   70 | 33330000 00026631 82953e2d 86dd6000 00000010 3afffe80
expensive Packet::push; have 0 wanted 4
free(): invalid pointer
[  160.758624] 990.912471 [ 818] tc_configure              ifp veth1 qdisc pfifo parent 4294967295 handle 0
[  160.761353] 990.915204 [ 352] generic_netmap_unregister Emulated adapter: ring 'veth1 RX0' deactivated
[  160.763998] 990.917850 [ 359] generic_netmap_unregister Emulated adapter: ring 'veth1 TX0' deactivated
[  160.766231] 990.920084 [ 416] generic_netmap_unregister Emulated adapter for veth1 deactivated
[  160.768852] 990.922704 [1168] generic_netmap_dtor       Emulated netmap adapter for veth1 destroyed
Aborted

To induce a crash faster, you can set some IP address on veth0 and ping some random IP address in the same subnet. Kernel will send an ARP request to veth0, which received in Click from veth1 will crash it.

ip addr add 192.168.6.1/24 dev veth0
ping 192.168.6.2

As I already have mentioned, the same happens with patched veth module. The only difference is that kernel prints:

[ 8661.060906] 237.928916 [1056] netmap_obj_free           ouch, double free on buffer 53250

at the crash moment.

DPDK KNI

After classifying packets in Click, I would like to send packets to a linux app. Is there a way to do this without a Tun/Tap interface so that there aren't system calls?

Is there a way to send and receive packets on a KNI interface within a fastclick configuration?

Dear all, I have encountered an issue when compiling the metron branch. During compilation phase, it reports an error like this:

../lib/librte_parse_17.8.0/testpmd.h:381:13: error: use of enum 'dcb_queue_mapping_mode' without previous declaration

I'm using DPDK 17.08, and I can compile and run fastclick master branch without any problem. Could you give me some hints about this?

Best,
Tianzhu

After commit 98c74e6 there seems to be an issue that crashes run_task() of FromDPDKDevice.

Backtracking the issue with gdb outputs the following (not very informative though):

Thread 1 "click" received signal SIGSEGV, Segmentation fault.
0x0000000000960c2c in FromDPDKDevice::run_task(Task*) ()
(gdb) bt
0 0x0000000000960c2c in FromDPDKDevice::run_task(Task*) ()
1 0x0000000000a07419 in RouterThread::driver() ()
2 0x000000000048925a in main ()

I guess this is a memory issue related to the commit above.

FastClick is unable to attach to a host ring (using netmap:eth0^ interface name). It seems that it always attach to default hardware rings, no matter what modifier is given after interface name. In vanilla Click attaching to host rings works as expected. The same goes with pipes.

Receiving packets from host ring is useful when some control applications need to be run in userspace. In my case I run Quagga daemons on each router. They need to exchange routing protocol control packets. In vanilla Click I receive Quagga packets from host ring (with FromDevice(netmap:eth0^)) and inject them to ToDevice(netmap:eth0), providing connectivity between Quagga and physical interface. In FastClick it is impossible.

Some (admittedly dubious) IPFilter configurations result in an assertion error when the router is built.

I built FastClick with

./configure --prefix=/home/soldani/devel/fastclick/prefix RTE_SDK=$RTE_SDK \
    RTE_TARGET=$RTE_TARGET --enable-dpdk --enable-user-multithread --disable-linuxmodule

I try to run the following minimal testcase configuration:

fd :: InfiniteSource();
ipf :: IPFilter(
    drop tcp opt psh and ip proto 168,
    drop ip[2:2] 40,
    allow all
);
td :: Discard();

fd -> ipf -> td;

Loading this router with ./userlevel/click conf/testcase.click fails with

click: ../include/click/vector.hh:291: T& Vector<T>::operator[](Vector<T>::size_type)
[with T = int; Vector<T>::size_type = int]: Assertion `(unsigned) i < (unsigned) vm_.n_'
failed.

Here is a relevant extract of the GDB backtrace:

#3  0x00007ffff789e0a2 in __GI___assert_fail (assertion=assertion@entry=0x555555c27190 "(unsigned) i < (unsigned) vm_.n_", file=file@entry=0x555555c27140 "../include/click/vector.hh", 
    line=line@entry=291, 
    function=function@entry=0x555555c3a4a0 <_ZZN6VectorIiEixEiE19__PRETTY_FUNCTION__> "T& Vector<T>::operator[](Vector<T>::size_type) [with T = int; Vector<T>::size_type = int]")
    at assert.c:101
#4  0x0000555555845a73 in Vector<int>::operator[] (this=<optimized out>, this=<optimized out>, i=5) at ../include/click/vector.hh:290
#5  0x000055555584a7b7 in Vector<int>::operator[] (this=<optimized out>, this=<optimized out>, i=<optimized out>) at ../include/click/vector.hh:290
#6  Vector<int>::back (this=<optimized out>) at ../include/click/vector.hh:333
#7  Classification::Wordwise::DominatorOptimizer::calculate_dom (this=0x7fffffffdb20, state=5) at ../elements/standard/classification.cc:459
#8  0x000055555584aad8 in Classification::Wordwise::DominatorOptimizer::run (this=0x7fffffffdb20, state=5) at ../elements/standard/classification.cc:694
#9  0x000055555584ab5a in Classification::Wordwise::Program::optimize (this=0x5555563b2b00, 
    offset_map_begin=offset_map_begin@entry=0x555555c33a70 <IPFilter::parse_program(Classification::Wordwise::CompressedProgram&, Vector<String> const&, int, Element const*, ErrorHandler*)::offset_map>, offset_map_end=offset_map_end@entry=0x555555c33a78, last_offset=last_offset@entry=2147483647) at ../elements/standard/classification.cc:892
#10 0x0000555555828c7a in IPFilter::parse_program (zprog=..., conf=..., noutputs=1, context=0x5555563c06c0, errh=0x7fffffffde90) at ../include/click/vector.hh:290

It looks like the failing function is Vector<int>::back in

assert(_dom_start.size() - 1 == _domlist_start.back());

at elements/standard/classification.cc:459. In GDB, it appears OK relatively to indexing, with _domlist_start.vm_.n_ equal to 6, which corresponds to the index 5 that we see in backtrace #4. But if I add a printf in Vector::operator[], the argument seems to actually be -1 (4294967295 once cast to unsigned). I guess there is either an optimization problem, or some memory corruption going on.

If the ip proto is changed to 6 (for TCP, in line with inspecting the PSH flag), the error goes away. It also disappear if I remove either of the drop lines. On the other hand, this is just one example of strange interaction between rules that trigger this assertion. We use a randomly-generated rule set and this assertion error fires as soon as we used more than a few hundreds random rules.

Any idea what is causing this? While obviously not critical (we had no problem with simple 5tuple matching, and when using the right IP proto), this is still quite puzzling.

Hi,

click code launches as following

click --dpdk $EAL_PARAMS -- analyzerbkp.click

when the EAL is

export EAL_PARAMS="-l 4 --master-lcore 4 -n 1
-m 1024 --file-prefix=analyser --no-pci
--vdev=virtio_user0,path=/var/run/openvswitch/SRC,mac=00:00:92:00:00:01
--vdev=virtio_user1,path=/var/run/openvswitch/SNK,mac=00:00:92:00:00:02"

If I replace the argument 4 in the EAL with a core number on NUMA node 1, I get _ Could not allocate packet MBuf pools_. But as you can see in the following output, there are enough hugepages and memory.

Following are some info from the server

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 iommu=pt intel_iommu=on isolcpus=1-19"

root@analysis:/home/tdev# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9
node 0 size: 16091 MB
node 0 free: 5097 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19
node 1 size: 16105 MB
node 1 free: 7834 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

root@analysis:/home/tdev# cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages:         0 kB
Node 0 ShmemHugePages:        0 kB
Node 0 HugePages_Total:     8
Node 0 HugePages_Free:      4
Node 0 HugePages_Surp:      0
Node 1 AnonHugePages:         0 kB
Node 1 ShmemHugePages:        0 kB
Node 1 HugePages_Total:     8
Node 1 HugePages_Free:      5
Node 1 HugePages_Surp:      0

There are numerous misuses of Packet::uniqueify in many click Elements, specifically related to what happens when uniqueify fails to generate a WritablePacket. The semantic is that uniqueify returns NULL and the original packet loses a reference, that is should be assumed to be gone. In practice, uniqueify will only fail in this way if there are multiple refs already, which means the original Packet probably looks the same for a little while but will disappear mysteriously later. The result are strange bugs under low memory conditions or when allocating while atomic.

In several cases, e.g. fasttcpflows, the result of uniqueify is not checked and there is a comment that says "better not fail." In several cases, the original packet is used, pushed, or killed.

I plan to work on this at some point, but I'm not there yet. I took a quick survey by forcing Packet::uniqueify to always take the expensive path, disabling packet pooling, and causing expensive_uniqueify to fail 10% of the time. With valgrind's track-origins, code with this issue lights right up.

When I replay a trace with raw IP packets I get the following memory dump:

Thread 1 "click" received signal SIGABRT, Aborted.
0x00007ffff5ed0428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
No0 0x00007ffff5ed0428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
No1 0x00007ffff5ed202a in __GI_abort () at abort.c:89
No2 0x00007ffff5f127ea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff602bed8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
No3 0x00007ffff5f1b37a in malloc_printerr (ar_ptr=, ptr=, str=0x7ffff6028caf "free(): invalid pointer", action=3) at malloc.c:5006
No4 _int_free (av=, p=, have_lock=0) at malloc.c:3867
No5 0x00007ffff5f1f53c in __GI___libc_free (mem=) at malloc.c:2968
No6 0x000000000066ab2f in Packet::~Packet (this=0x15fbbb0, __in_chrg=) at ../lib/packet.cc:236
No7 WritablePacket::~WritablePacket (this=0x15fbbb0, __in_chrg=) at ../include/click/packet.hh:950
No8 WritablePacket::recycle (p=p@entry=0x15fbbb0) at ../lib/packet.cc:538
No9 0x00000000006019b6 in Packet::kill (this=0x15fbbb0) at ../include/click/packet.hh:1697
No10 EnsureDPDKBuffer::smaction (p=0x15fbbb0, this=0x15ea230) at ../elements/userlevel/ensuredpdkbuffer.cc:87
No11 EnsureDPDKBuffer::simple_action (this=0x15ea230, p=0x15fbbb0) at ../elements/userlevel/ensuredpdkbuffer.cc:106
No12 0x000000000067f9e5 in Element::push (this=0x15ea230, port=0, p=) at ../lib/element.cc:3061
No13 0x00000000006059f8 in Element::Port::push (p=, this=) at ../include/click/element.hh:714
No14 FromDump::run_task (this=0x15e9690) at ../elements/userlevel/fromdump.cc:605
No15 0x00000000006aff29 in Task::fire (this=0x15e9860) at ../include/click/task.hh:584
No16 RouterThread::run_tasks (ntasks=15, this=0x15698f0) at ../lib/routerthread.cc:392
No17 RouterThread::driver (this=0x15698f0) at ../lib/routerthread.cc:613
No18 0x00000000004911ca in main (argc=, argv=) at click.cc:809

To reproduce, use the trace /mnt/traces/caida/caida_939_to_941.pcap on rack7 and a simple configuration:
FromDump -> EnsureDPDKBuffer -> Discard;

MiddleClick (HSPR '18 + Tom Barbette thesis) proposes the context link. A link, noted as ~>, allows to automatically introduce a classifier that uses the element on the right hand side of the arrow to discover a rule. It would be great to have it in a more general way (not tied to the MiddleClick subsystem) in FastClick. E.g., one could have :
fd :: FromDevice;
fd ~> [1]ARPQuerier
fd ~> ARPResponder
fd ~> CheckIPHeader
Without the old same Classifier(). This is a streamlined default case.

When we run successfully after fastclick, radio buttons or check boxes will appear in the label content area can't normal triggers the click event
But the padding area can trigger the click event
pray someone to explain
thanks~

Hi Tom,

(I realize there are no issues yet, hope this is the approppriate place for bug reports/help)

I was dealing with a failed assertion raised by the Vector indexing operation. Here's the stack trace just before the failure:

#0  operator[] (i=<optimized out>, this=<optimized out>) at ../include/click/vector.hh:292
#1  thread_for_queue (queue=<optimized out>, this=<optimized out>) at ../elements/userlevel/queuedevice.hh:259
#2  QueueDevice::get_runnable_threads (this=0xaf6c60, bmk=...) at ../elements/userlevel/queuedevice.hh:153
#3  0x0000000000677859 in InputThreadVisitor::visit (this=0x7fffffffddd0, e=<optimized out>) at ../lib/element.cc:1676
#4  0x000000000068f6c1 in Router::visit (this=0xaf5770, first_element=first_element@entry=0xaff3f0, forward=forward@entry=false, 
    first_port=first_port@entry=-1, visitor=visitor@entry=0x7fffffffddd0) at ../lib/router.cc:926
#5  0x0000000000675a9e in Element::get_threads (this=this@entry=0xaff3f0, is_pull=is_pull@entry=false) at ../lib/element.cc:1700
#6  0x000000000064c32e in QueueDevice::initialize_tx (this=this@entry=0xaff3f0, errh=errh@entry=0x7fffffffdf10)
    at ../elements/userlevel/queuedevice.cc:99
#7  0x0000000000651cd7 in ToDPDKDevice::initialize (this=0xaff3f0, errh=0x7fffffffdf10) at ../elements/userlevel/todpdkdevice.cc:81
#8  0x0000000000692cb8 in Router::initialize (this=this@entry=0xaf5770, errh=errh@entry=0xa9e9c0) at ../lib/router.cc:1233
#9  0x000000000065b10e in parse_configuration (text=..., text_is_expr=text_is_expr@entry=false, hotswap=hotswap@entry=false, 
    errh=errh@entry=0xa9e9c0) at click.cc:392
#10 0x000000000065c4e6 in main (argc=<optimized out>, argv=<optimized out>) at click.cc:703

Where:

• At frames #7-#5, this is an instance of ToDPDKDevice
• At frame #2, instead, this is a FromDPDKDevice which (if I'm not mistaken) has not been initialized yet:

(gdb) frame 2
#2  QueueDevice::get_runnable_threads (this=0xaf6c60, bmk=...) at ../elements/userlevel/queuedevice.hh:153
153                                     bmk[thread_for_queue(i) - j] = 1;
(gdb) p this->class_name()
$21 = 0x72e058 "FromDPDKDevice"
(gdb) p this->_queue_to_thread
$22 = {vm_ = {l_ = 0x0, n_ = 0, capacity_ = 0}}

For the moment I've been able to fix this by forcing FromDPDKDevice to initialized earlier patching its configure_phase (which actually also influences initalization order):

int configure_phase() const {
    return CONFIGURE_PHASE_PRIVILEGED - 1;  // was CONFIGURE_PHASE_PRIVILEGED
}

Not sure whether this could have subtle drawbacks, would you suggest any other approach?

Some more context follows.

• Excerpt from the configuration:

  from0 :: FromDPDKDevice(0, MINQUEUES 2, MAXTHREADS 64, RSS_AGGREGATE yes);
  from1 :: FromDPDKDevice(1, MINQUEUES 2, MAXTHREADS 64, RSS_AGGREGATE yes);
  from0, from1 => ... => ToDPDKDevice(0), ToDPDKDevice(1);
  

  This appears to happen only when there is actually something in the middle, but I guess this depends on how Click chooses the order for configuration/initialization.

• configure line

  ./configure --enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --verbose CFLAGS="-g -Og" CXXFLAGS="-g -std=gnu++11 -Og" --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --with-netmap=no --enable-zerocopy --disable-dpdk-pools --enable-dpdk --disable-batch --with-numa=no
  

  It's almost like the one in the README except for -Og, --disable-batch, and --with-numa=no (those have other unrelated stories)

• uname -a

  Linux anslab1 3.16.0-50-generic #67~14.04.1-Ubuntu SMP Fri Oct 2 22:07:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

• gcc --version

  gcc (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4

• DPDK version: 2.2.0

Besides the issue, thanks for all the open source work.

Best regards,
Davide Kirchner

As one of the first step to try to get FastClick more open and properly maintained by the community, I'd like to fix the Kernel mode.

With the last patch, it is compiling on up to 4.4. However recent kernel crashes, I think it's due to the last changes by Eddie in the scheduling, among them the recent issues mentioned there : kohler/click#407 . @pallas @peterhurley did you experience this kind of problem with a recent merge?

In a more long-term, I think we can bring back Kernel Click in the front stage if, instead of using the rx_handler we used the xdp_handler. One can hijack this one to not use eBPF but eg a Click function. This allows to completely byoass the kernel stack functions, including the heavy sk_buff creation step - which is mostly useless for a Click program - and achieve speed comparable to what DPDK proposes, but in-kernel. Looking at the ixgbe driver, the XDP buffers are safe to keep as the ring bucket will be re-mapped to some new buffer in the DMA zone, so for this kind of driver we do not even need to remap or copy the buffers. If some people are still looking into Kernel Click, this would probably be the biggest performance improvement for Kernel Click in years.

Aside Clicky that can export the graph visualization in PDF, what other tools do you guys now about ?

I'll try to make a wiki page out of this question.

Hi again,
This time (I believe) I managed to fix a new issue, a possible patch follows.

The patch should be git am-able on top of your master. Also, both this patch and another verbose version with prints around are available at my fork (at the fix-todpdk-get_mbuf and fix-todpdk-get_mbuf-DEBUG branches respectively).

The weired thing is that, while explicitly trying to access the data in the wrong packet from click causes a segfault, it appears that when DPDK tried to forward it, it would pick an actual mbuf containing an old packet (causing the old packet to be sent again).

As usual, I am compiling against DPDK 2.2.0 (perhaps they changed some internals of the struct mbuf?), compiling with ./configure --enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --verbose CFLAGS="-g -O0" CXXFLAGS="-g -std=gnu++11 -O0" --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --with-netmap=no --enable-zerocopy --disable-dpdk-pools --enable-dpdk --disable-batch. I was running with -c 0x1 -n 1 and sending packets one at a time (funny thing, even the first packet incurred in the issue and it would replicate a packet from the previous session. This drove me mad). The conf file I was using:

FromDPDKDevice(0, MINQUEUES 2, MAXTHREADS 64, THREADOFFSET 0, NUMA no, RSS_AGGREGATE yes)
  -> CheckIPHeader(OFFSET 14, VERBOSE yes) // needed for some prints in the *-DEBUG branch
  -> ToDPDKDevice(0);

I realize using destructor_argument is not the cleanest thing in the world, but it seems reasonable to me that if a Packet was allocated from an mbuf, then _destructor_argument must be set, and I believe nothing in the path has a good reason to override it (while at times _destructor may get set to the no-op).

For the record, here is the output generated with the prints in the -DEBUG branch:

-- 24.0.32.69 > 1.2.3.4
dump mbuf at 0x0x7f68b55c5000, phys=0, buf_len=0
  pkt_len=0, ol_flags=0, nb_segs=0, in_port=0
dump mbuf at 0x0x7f68b55c4f80, phys=7f55c5000, buf_len=2176
  pkt_len=73, ol_flags=2, nb_segs=1, in_port=0
  segment at 0x0x7f68b55c4f80, data=0x0x7f68b55c5080, data_len=73
  Dump data at [0x7f68b55c5080], len=20
00000000: 54 75 D0 C4 4F FF EC F4 BB E6 1E 14 08 00 45 00 | Tu..O.........E.
00000010: 00 3B 00 01 |  |  |  |  |  |  |  |  |  |  |  |  | .;..
-- 24.0.32.69 > 1.2.3.4

From 539c14c2a62678df3494a816bba167785a845b44 Mon Sep 17 00:00:00 2001
From: Davide Kirchner <[email protected]>
Date: Tue, 23 Feb 2016 10:15:15 +0100
Subject: [PATCH] Fixing retrieval of original mbuf in ToDPDKDevice

---
 elements/userlevel/todpdkdevice.cc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/elements/userlevel/todpdkdevice.cc b/elements/userlevel/todpdkdevice.cc
index c9ce65e..9597398 100644
--- a/elements/userlevel/todpdkdevice.cc
+++ b/elements/userlevel/todpdkdevice.cc
@@ -128,7 +128,7 @@ inline struct rte_mbuf* get_mbuf(Packet* p, bool create=true) {
     if (likely(DPDKDevice::is_dpdk_packet(p) && !p->shared())) {
         /* If the packet is an unshared DPDK packet, we can send
          *  the mbuf as it to DPDK*/
-        mbuf = (struct rte_mbuf *) (p->buffer() - sizeof(struct rte_mbuf));
+        mbuf = (struct rte_mbuf *) p->destructor_argument();
         rte_pktmbuf_pkt_len(mbuf) = p->length();
         rte_pktmbuf_data_len(mbuf) = p->length();
         static_cast<WritablePacket*>(p)->set_buffer(0);
@@ -180,7 +180,7 @@ void ToDPDKDevice::flush_internal_queue(InternalQueue &iqueue) {
             // The sub_burst wraps around the ring
             sub_burst = _iqueue_size - iqueue.index;
         r = rte_eth_tx_burst(_port_id, queue_for_thread_begin(), &iqueue.pkts[iqueue.index],
-                             iqueue.nr_pending);
+                             sub_burst);

         iqueue.nr_pending -= r;
         iqueue.index += r;
-- 
1.9.1

EDIT: minor format change -- I've discovered how to colorize the diff

Hi,

I am trying to get started with DPDK & fastclick. DPDK 17.05 is installed. I also have openvswitch installed and configured to work with DPDK in the same machine, which works just fine.

Here is my simpletest.click

FromDPDKDevice(0, PROMISC true) -> CheckIPHeader(OFFSET 14) -> ToDPDKDevice(1); FromDPDKDevice(1, PROMISC true) -> CheckIPHeader(OFFSET 14) -> ToDPDKDevice(0);

Hugepages utilization

root@T-Dev:/home/tdev# cat /proc/meminfo | grep Huge
AnonHugePages: 6144 kB
ShmemHugePages: 0 kB
HugePages_Total: 1024
HugePages_Free: 1024
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

Running the click code and the error

root@T-Dev:/home/tdev/git/fastclick/thesistest# click --dpdk -c 0xf -n 4 -- simpletest.click
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL: probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL: probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:09.0 on NUMA socket -1
EAL: probe driver: 8086:100e net_e1000_em
Initializing DPDK
simpletest.click:2: While initializing ‘ToDPDKDevice@6 :: ToDPDKDevice’:
Could not allocate packet MBuf pools : error 12 (Cannot allocate memory)
Router could not be initialized!

Hugepages utilization after above error

root@T-Dev:/home/tdev/git/fastclick/thesistest# cat /proc/meminfo | grep Huge
AnonHugePages: 6144 kB
ShmemHugePages: 0 kB
HugePages_Total: 1024
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

Any ideas?
Thanks

Hi

As a continuation of #54, and the same approach as this, I am trying to attach my click process (two ports)
to two ports of ovs.

Following ovs guide, I I add the following two lines and create two ports in form of dpdkring.

ovs-vsctl add-port br0 dpdkr0 -- set Interface dpdkr0 type=dpdkr
ovs-vsctl add-port br0 dpdkr1 -- set Interface dpdkr1 type=dpdkr

As for click side,

I would appreciate if you could share sample commands/configuration related to this scenario.

Is there any point in printing batch mode warnings for elements which do not process packets?

click_driver@@ControlSocket won't be in batch mode because no element produces or sends batches to it.
click_driver@@ControlSocket@1 won't be in batch mode because no element produces or sends batches to it.

Can FastClick automatically detect such elements and do not print warnings for them?

I have the following packet generator/sink

//------------Declarations-----------
DPDK0_OUT ::  ToDPDKDevice(0);

//+++++DPDK1 Interface(Outside)+++++
DPDK1_IN :: FromDPDKDevice(1, PROMISC true);

//--------Program Start----------
//InfiniteSource(LENGTH 64, LIMIT 150000, STOP true)
RatedSource(LENGTH 64, RATE 7000, LIMIT 150000, STOP true)
        -> NumberPacket(OFFSET 0)
        -> UDPIP6Encap(2001:2001:2001:2001::1, 1234, 64:ff9b::192.168.2.5, 1234)
        -> EtherEncap(0x86DD, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db)
        -> record :: RecordTimestamp()
        -> DPDK0_OUT;

DPDK1_IN
        -> CheckNumberPacket(OFFSET 62, COUNT 150000)
        -> diff :: TimestampDiff(OFFSET 62, RECORDER record)
        -> counter :: AverageCounter()
        -> Discard;
//--------Program End----------

DriverManager(wait,read diff.average, read counter.count, read counter.rate)

When I run it, I get the following warning,

expensive Packet::push; have 12 wanted 14

Everything works just fine if I use InfiniteSource instead of RatedSource (comment out the RatedSource line and uncomment the InfiniteSource line).

Any ideas?

Before we jump full on board with FastClick I would like to get a better understanding of the main differences and any breaking changes between FastClick and Click. We only use userlevel Click, so my questions relate specifically to it, but others using kernel Click should feel free to ask here.

• If my understanding is correct, FastClick adopts a full-push model. Am I correct in assuming if we only use standard Click elements then push + pull processing is still 100% supported?

• We rely heavily on keeping either original packets or references (cloned) packets around indefinitely. I recall that I read somewhere that the DPDK elements don't allow this, is that correct? If we don't use DPDK I'm assuming it would still be acceptable?

• Are there any changes in the core Click scheduling and threading implementation? We use Click Tasks/Timers significantly in our elements and rely on Click's StaticThreadSched for thread allocation.

• FastClick's Batching sounds like something we could use as well. Is batching also only support in a full-push mode? Has any Click standard elements been converted to support batching natively?

• We make use of most Click template and utility functions (HashMap, Vector, String, List, Spinlock, atomics etc.). Are there any new or interesting utility functions in FastClick that you think could be of interest or use to vanilla Click developers?

• Any other gotchas or things we need to be aware of in moving from Click to FastClick?

Any comments or advice would be much appreciated.

There are basically 2 ways :

• Implement everything in the From/ToDPDKDevice elements. But that makes I/O elements very complex, with a lot of parameters to turn offloadings on and off, and code to support backup solution when hardware does not support the feature.
• Add feature-element like SetTimestamp to set the timestamp, SetTCPChecksum, .... And somehow if the timestamp is provided by the hardware, this element would be "deactivated" and the hardware one used.
  The second one is more "Click-like" because the logic is specified by elements. But how to make the element "discuss" to establish that the feature is supported by the upfront hardware?

Hello,

Prior to DPDK, I was instantiating click apps in linux namespaces and connecting the NSes to ovs switch running on the same machine using virtual ethernet (veth) pairs.

Now I am investigating whether the same approach is still viable for fast processing via DPDK or not. Some sources such asthis link made me think a little bit and I wanted to ask your opinions.

Thanks!

Hello,

I am creating boxplots out of TimeStampDiff values, where the whisker boundaries are the 5th and 95th percentiles. Is there any way to dump all the values between 0th-5th and 95th-100th percentiles so I can plot them as outliers?

Thanks.

Currently, click includes the core router, a set of elements, a non-std C++ template & utility library, clickfs, a userland/Linux/BSD compatibility layer, and code for mutating C-only kernel headers to work in C++. These could all be and probably all should be separate, reusable libraries.

Hi Tom,

Is there a plan to update FastClick and make it compatible with the latest DPDK version 16.11 (soon 17 is coming)?
I would like to test a Mellanox 100GbE board using FastClick.

Best regards,
Georgios

Hi

I writing an element performing stateful NAT64. Documented it as much as possible and willing to push to mainstream in future when the code gets more mature.
kthkaya@a7acbab

Still many things to consider such as map thread safety and port lease expiry, but at this point I am concerned with packet handling performance. Therefore I am generating traffic with single ip and port.

The packet handling functions are almost identical to the existing protocoltranslator64.cc and protocoltranslator46.cc elements. When I push one-way traffic only,

//------------Declarations-----------
Idle -> EnsureDPDKBuffer -> Discard;
NAT64 :: StatefulTranslator64();
//--------Program Start----------
InfiniteSource(LENGTH 24, LIMIT 250000, STOP true)
        -> NumberPacket(OFFSET 0)
        -> UDPIP6Encap(2001:2001:2001:2001::1, 1234, 64:ff9b::192.168.2.5, 1234)
        -> NAT64
        -> EtherEncap(0x0800, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db)
        -> record :: RecordTimestamp()
        -> CheckNumberPacket(OFFSET 42, COUNT 250000)
        -> diff :: TimestampDiff(OFFSET 42, RECORDER record)
        -> counter :: AverageCounter()
        -> Discard;
//--------Program End----------
Idle -> [1]NAT64[1] ->Discard;
DriverManager(wait,read diff.average, read counter.count, read counter.rate)

Results
diff.average:
0.006084
counter.count:
250000
counter.rate:
478011.472275

Despite the 2GB memory I pass via EAL, I get the following when I try to push more than 250K packets.
No more DPDK buffer available ! Try using DPDKInfo to allocate more. Segmentation fault (core dumped)

It is a disaster when I try the over the wire setup (again one-way).
The NAT

NAT64  :: StatefulTranslator64();

FromDPDKDevice(0, PROMISC true)
                -> Strip(14)
                -> NAT64
                -> EtherEncap(0x0800, 00:17:cb:02:c8:d1, 00:04:23:d1:5a:54)
//              -> Print(CONTENTS true, MAXLENGTH 1500)
                -> ToDPDKDevice(1);

FromDPDKDevice(1, PROMISC true)
                -> Strip(14)
                -> [1]NAT64[1]
                -> EtherEncap(0x86DD, 00:17:cb:02:c8:d1, 00:04:23:d1:5a:54)
                -> ToDPDKDevice(0);

Above code starts with the following warning

nat64test.click:10: While initializing 'FromDPDKDevice@6 :: FromDPDKDevice':
warning: Thread offset 0 will be ignored because the numa node has not enough cores.
nat64test.click:3: While initializing 'FromDPDKDevice@2 :: FromDPDKDevice':
warning: Thread offset 1 will be ignored because the numa node has not enough cores.
Initializing DPDK

//------------Declarations-----------
DPDK0_OUT ::  ToDPDKDevice(0);
DPDK1_IN :: FromDPDKDevice(1, PROMISC true);
//--------Program Start----------
InfiniteSource(LENGTH 64, LIMIT 100000, STOP true)
        -> NumberPacket(OFFSET 0)
        -> UDPIP6Encap(2001:2001:2001:2001::1, 1234, 64:ff9b::192.168.2.5, 1234)
        -> EtherEncap(0x86DD, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db)
        -> record :: RecordTimestamp()
        -> DPDK0_OUT;

DPDK1_IN
        -> CheckNumberPacket(OFFSET 42, COUNT 100000)
        -> diff :: TimestampDiff(OFFSET 42, RECORDER record)
        -> counter :: AverageCounter()
        -> Discard;
//--------Program End----------
DriverManager(wait,read diff.average, read counter.count, read counter.rate)

Results
diff.average:
463.398800505
counter.count:
31680
counter.rate:
633600

%70 of the packets are lost. I'd very much appreciate if you could have a quick look at the element code and see if I am missing something obvious that could cause large memory consumption/high drop rate.

Note that in none of the cases above could I assign more than a single core via EAL as explained in #71. I believe the numa node has not enough cores. warning has more to do with the packet drops.

Thanks!

Greetings.

Been running plain vanilla click elements within LXD/LXC containers for a bit now. Curious if anyone has successfully run FastClick within an LXC container w/DPDK installed on the host. If so, any pointers with respect to configuration would be greatly appreciated.

Thanks,

Mike B.

Hello. I'm trying to run click-devirtualize on a fairly basic config. After some tweaks to deal with DPDK, I can get the devirtualized code to compile without errors, but then I end up with a slew of undefined references:

$ click-devirtualize -d -u router-2if-shaped-sm16 > router-2if-shaped-sm16.dv
/usr/local/bin/click-buildtool makepackage -C /tmp/clicktmp25355/ -t userlevel -w -fno-access-control -I/usr/local/include/dpdk clickdv_8hmRp6kM7fFhsMbe7LtiMb clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.cc 1>&2

• make clickdv_8hmRp6kM7fFhsMbe7LtiMb.uo
  make -R CLICK_PACKAGE_MAKING=userlevel clickdv_8hmRp6kM7fFhsMbe7LtiMb.uo
  make[1]: Entering directory '/tmp/clicktmp25355'
  CXX clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.cc
  g++ -g -std=gnu++11 -O3 -Wno-pmf-conversions -fPIC -o clickdv_8hmRp6kM7fFhsMbe7LtiMb.uo clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.u.o
  /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crt1.o: In function _start': (.text+0x20): undefined reference to main'
  clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.u.o: In function Discard_a_aDiscard_a24::Discard_a_aDiscard_a24()': /tmp/clicktmp25355/clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.hh:161: undefined reference to Discard::Discard()'
  clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.u.o: In function SetTimestamp_a_aSetTimestamp_a26::SetTimestamp_a_aSetTimestamp_a26()': /tmp/clicktmp25355/clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.hh:172: undefined reference to SetTimestamp::SetTimestamp()'
  clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.u.o: In function FromDPDKDevice_a_ain0::FromDPDKDevice_a_ain0()': /tmp/clicktmp25355/clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.hh:11: undefined reference to FromDPDKDevice::FromDPDKDevice()'
  clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.u.o: In function Classifier_a_ac0::Classifier_a_ac0()': /tmp/clicktmp25355/clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.hh:22: undefined reference to Classifier::Classifier()'
  clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.u.o: In function ToDPDKDevice_a_aout0::ToDPDKDevice_a_aout0()': /tmp/clicktmp25355/clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.hh:34: undefined reference to ToDPDKDevice::ToDPDKDevice()'
  clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.u.o: In function ARPResponder_a_aresponder0::ARPResponder_a_aresponder0()': /tmp/clicktmp25355/clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.hh:45: undefined reference to ARPResponder::ARPResponder()'
  clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.u.o: In function ARPQuerier_a_aarp0::ARPQuerier_a_aarp0()': /tmp/clicktmp25355/clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.hh:60: undefined reference to ARPQuerier::ARPQuerier()'
  ...
  clickdv_8hmRp6kM7fFhsMbe7LtiMb.u.u.o:(.data.rel.ro._ZTV32SetTimestamp_a_aSetTimestamp_a26[_ZTV32SetTimestamp_a_aSetTimestamp_a26]+0xf0): undefined reference to `Element::llrpc(unsigned int, void*)'
  collect2: error: ld returned 1 exit status
  /usr/local/share/click/pkg-userlevel.mk:84: recipe for target 'clickdv_8hmRp6kM7fFhsMbe7LtiMb.uo' failed
  make[1]: *** [clickdv_8hmRp6kM7fFhsMbe7LtiMb.uo] Error 1
  make[1]: Leaving directory '/tmp/clicktmp25355'

Any suggestions on how to get a loadable module for userspace greatly appreciated.

Hi,

I have two machines with 4x10 Gbps NICs each, connected to each other as follows:

                  NIC 0    ---------------------- NIC 0
                  NIC 1    ---------------------- NIC 1
               Machine 01                      Machine 02
                  NIC 2    ---------------------- NIC 2
                  NIC 3    ---------------------- NIC 3

On machine 01 I use MoonGen to send/receive 64-byte packets to/from both NICs.
On machine 02, I use Fast-Click to forward the packets back:
FromDPDKDevice(0) -> ToDPDKDevice(1, IQUEUE 1024);
FromDPDKDevice(1) -> ToDPDKDevice(0, IQUEUE 1024);
FromDPDKDevice(2) -> ToDPDKDevice(3, IQUEUE 1024);
FromDPDKDevice(3) -> ToDPDKDevice(2, IQUEUE 1024);

All NICs belong to the same NUMA socket and can share 8 cores in this socket (Intel Xeon).
I start Fast-Click as follows: path/click -c ffff -n 4 -v -- my_example.click

The problem I see is that the throughput is not line rate (31.5 out of 40Gbps) when I inject packets of 60 bytes each. This is exactly what Figure 2 in the Fast-Click paper depicts.

If I use only one NIC from machine 01 to send/receive traffic, then throughput is maximum (14.88Mpps).
If I replace Fast-Click with a MoonGen application throughput is also maximum (40Gbps).

Is there any particular reason behind this degradation? Why Click is so much slower compared to MoonGen? Is it because of the virtua; function calls used by From/ToDPDKDevice elements?

I tried to configure the 2 elements by using more queues to read packets from (RSS). If I specify more than 4 queues (using MINQUEUES argument), the process crashes. How can I use 8 queues and 8 cores to scale?

Best regards,
Georgios

I'm getting a segfault that I managed to reproduce with a fresh-from-repo checkout.
It appears it only happens when I'm running on a thread that is on a different NUMA node from where the device is. Note that I'm compiling with --with-numa=no, otherways it would complain that it can't find a suitable thread. Nevertheless, in our application we require every thread (in either NUMA node) to be able to read from every interface (each thread will have a dedicated hardware queue), so for the moment I disabled explicit support. I realize this might be the issue, but I'm not sure how to check it or rule this out.

The simplest example where I managed to reproduce it is with a thread in node 1 reading from the NIC, which is located on node 0.

• configure: ./configure --enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --verbose CFLAGS="-g -O0" CXXFLAGS="-g -std=gnu++11 -O0" --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --with-netmap=no --enable-zerocopy --disable-dpdk-pools --enable-dpdk --disable-batch --with-numa=no
• run:
  • userlevel/click -c 0x3 -n 1 -- -f /tmp/test.click 2 cores, one per numa node -> crash A
  • userlevel/click -c 0x2 -n 1 -- -f /tmp/test.click 1 core on the other (wrt the NIC card) numa node -> crash B (somethat understandable to me)
  • userlevel/click -c 0x1 -n 1 -- -f /tmp/test.click, or -c 0x5 1 or 2 cores, all on the same numa node as the -> no crash, works smoothly
• conf: (the DiscardNoFree plays the role of a caching element in our architecture)
  FromDPDKDevice(0) -> Print('PKT') -> tee::Tee() -> ToDPDKDevice(0); tee[1] -> DiscardNoFree();
• Generate a packet (the crash only happens with certain packets, no idea why. In my case this packet always triggered it, but it might be platform-related?). Here I'm using python's scapy library, running (as root) on the other side of the cable (without dpdk drivers)
  `
  $ sudo python2

  import scapy.all as s
  s.sendp(s.Ether("Tu\xd0\xc4O\xff\xec\xf4\xbb\xe6\x1e\x14\x08\x00E\x00\x00;\x00\x01\x00\x00@\xfd\x03\xf7-\xe9C\xe0\x01\x02\x03\x04\x00\x00'\x00\x00'\x00\x171/aaa/bbb/ccc/000000013\x00\x03\x00\x040-\x07\xa8"), iface='em1') # replace em1 with suitable interface name
  `

Also, while digging into the segfault, I stumbled across the behaviour of ToDPDKDevice on shared packets, but I'll leave this for another issue :)

Some post-mortem for crash A:

PKT:   73 | 5475d0c4 4fffecf4 bbe61e14 08004500 003b0001 000040fd

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff598a700 (LWP 57721)]
0x0000000000654cd8 in rte_mempool_get_bulk (n=1, obj_table=0x7ffff5989628, mp=0x3e363c00000003)
    at /home/davide/dpdk-2.2.0/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1095
1095                                     !(mp->flags & MEMPOOL_F_SC_GET));
(gdb) bt
#0  0x0000000000654cd8 in rte_mempool_get_bulk (n=1, obj_table=0x7ffff5989628, mp=0x3e363c00000003)
    at /home/davide/dpdk-2.2.0/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1095
#1  rte_mempool_get (obj_p=0x7ffff5989628, mp=0x3e363c00000003) at /home/davide/dpdk-2.2.0/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1168
#2  __rte_mbuf_raw_alloc (mp=0x3e363c00000003) at /home/davide/dpdk-2.2.0/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1062
#3  0x0000000000655bff in rte_pktmbuf_alloc (mp=0x3e363c00000003) at /home/davide/dpdk-2.2.0/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1333
#4  0x0000000000655eab in DPDKDevice::get_pkt () at ../include/click/dpdkdevice.hh:51
#5  0x0000000000671b70 in get_mbuf (p=0x7ffff0000c30, create=true) at ../elements/userlevel/todpdkdevice.cc:135
#6  0x000000000067154f in ToDPDKDevice::push_packet (this=0xb3dd40, p=0x7ffff0000c30) at ../elements/userlevel/todpdkdevice.cc:217
#7  0x000000000058fca9 in BatchElement::push (this=0xb3dd40, port=0, p=0x7ffff0000c30) at ../include/click/batchelement.hh:196
#8  0x0000000000562bcf in Element::Port::push (this=0xb3dcb8, p=0x7ffff0000c30) at ../include/click/element.hh:649
#9  0x000000000060eac7 in Tee::push (this=0xb3dca0, p=0x7ffff0000b80) at ../elements/standard/tee.cc:45
#10 0x0000000000562bcf in Element::Port::push (this=0xb3dbf8, p=0x7ffff0000b80) at ../include/click/element.hh:649
#11 0x0000000000566e2c in Element::checked_output_push (this=0xb3dbe0, port=0, p=0x7ffff0000b80) at ../include/click/element.hh:734
#12 0x000000000069792f in BatchElement::push_packet (this=0xb3dbe0, port=0, p=0x7ffff0000b80) at ../lib/batchelement.cc:223
#13 0x000000000058fca9 in BatchElement::push (this=0xb3dbe0, port=0, p=0x7ffff0000b80) at ../include/click/batchelement.hh:196
#14 0x0000000000562bcf in Element::Port::push (this=0xb3d958, p=0x7ffff0000b80) at ../include/click/element.hh:649
#15 0x000000000065a25e in FromDPDKDevice::run_task (this=0xb3d940, t=0xb3e430) at ../elements/userlevel/fromdpdkdevice.cc:148
#16 0x00000000006a8394 in Task::fire (this=0xb3e430) at ../include/click/task.hh:635
#17 0x00000000006a86c8 in RouterThread::run_tasks (this=0xb3a500, ntasks=128) at ../lib/routerthread.cc:422
#18 0x00000000006a7eca in RouterThread::driver (this=0xb3a500) at ../lib/routerthread.cc:649
#19 0x000000000067b735 in thread_driver (user_data=0xb3a500) at click.cc:476
#20 0x00000000004705a5 in eal_thread_loop ()
#21 0x00007ffff7274182 in start_thread (arg=0x7ffff598a700) at pthread_create.c:312
#22 0x00007ffff6a8747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) frame 4
#4  0x0000000000655eab in DPDKDevice::get_pkt () at ../include/click/dpdkdevice.hh:51
51              return rte_pktmbuf_alloc(get_mpool(rte_socket_id()));
(gdb) l
46                      return true;
47              return false;
48          }
49
50          inline static rte_mbuf* get_pkt() {
51              return rte_pktmbuf_alloc(get_mpool(rte_socket_id()));
52          }
53
54          static void free_pkt(unsigned char *, size_t, void *pktmbuf);
55
(gdb) p rte_socket_id()
$1 = 1

Some post-mortem for crash B:

[...]
PKT:   73 | 5475d0c4 4fffecf4 bbe61e14 08004500 003b0001 000040fd

Program received signal SIGSEGV, Segmentation fault.
0x0000000000671b92 in get_mbuf (p=0xb3ae50, create=true) at ../elements/userlevel/todpdkdevice.cc:136
136                 memcpy((void*)rte_pktmbuf_mtod(mbuf, unsigned char *),p->data(),p->length());
(gdb) bt
#0  0x0000000000671b92 in get_mbuf (p=0xb3ae50, create=true) at ../elements/userlevel/todpdkdevice.cc:136
#1  0x000000000067154f in ToDPDKDevice::push_packet (this=0xb3da30, p=0xb3ae50) at ../elements/userlevel/todpdkdevice.cc:217
#2  0x000000000058fca9 in BatchElement::push (this=0xb3da30, port=0, p=0xb3ae50) at ../include/click/batchelement.hh:196
#3  0x0000000000562bcf in Element::Port::push (this=0xb3d9a8, p=0xb3ae50) at ../include/click/element.hh:649
#4  0x000000000060eac7 in Tee::push (this=0xb3d990, p=0xb3a660) at ../elements/standard/tee.cc:45
#5  0x0000000000562bcf in Element::Port::push (this=0xb3d8e8, p=0xb3a660) at ../include/click/element.hh:649
#6  0x0000000000566e2c in Element::checked_output_push (this=0xb3d8d0, port=0, p=0xb3a660) at ../include/click/element.hh:734
#7  0x000000000069792f in BatchElement::push_packet (this=0xb3d8d0, port=0, p=0xb3a660) at ../lib/batchelement.cc:223
#8  0x000000000058fca9 in BatchElement::push (this=0xb3d8d0, port=0, p=0xb3a660) at ../include/click/batchelement.hh:196
#9  0x0000000000562bcf in Element::Port::push (this=0xb3d6d8, p=0xb3a660) at ../include/click/element.hh:649
#10 0x000000000065a25e in FromDPDKDevice::run_task (this=0xb3d6c0, t=0xb3d630) at ../elements/userlevel/fromdpdkdevice.cc:148
#11 0x00000000006a8394 in Task::fire (this=0xb3d630) at ../include/click/task.hh:635
#12 0x00000000006a86c8 in RouterThread::run_tasks (this=0xb3a320, ntasks=128) at ../lib/routerthread.cc:422
#13 0x00000000006a7eca in RouterThread::driver (this=0xb3a320) at ../lib/routerthread.cc:649
#14 0x000000000067c619 in main (argc=3, argv=0x7fffffffe670) at click.cc:772
(gdb) frame 0
#0  0x0000000000671b92 in get_mbuf (p=0xb3ae50, create=true) at ../elements/userlevel/todpdkdevice.cc:136
136                 memcpy((void*)rte_pktmbuf_mtod(mbuf, unsigned char *),p->data(),p->length());
(gdb) l
131             rte_pktmbuf_data_len(mbuf) = p->length();
132             static_cast<WritablePacket*>(p)->set_buffer(0);
133         } else {
134             if (create) {
135                 mbuf = DPDKDevice::get_pkt();
136                 memcpy((void*)rte_pktmbuf_mtod(mbuf, unsigned char *),p->data(),p->length());
137                 rte_pktmbuf_pkt_len(mbuf) = p->length();
138                 rte_pktmbuf_data_len(mbuf) = p->length();
139             } else
140                 return NULL;
(gdb) p mbuf
$4 = (rte_mbuf *) 0x0

Linux Kernel just added a new AIO poll interface (see https://lwn.net/Articles/743714/)

The advantage is that one may submit a list of FD to poll on (like poll), but the answers can be consumed without any systemcall, it is basically a queue.

Question is: should we support the new poll?

For DPDK users, this does not apply.

For netmap users I actually wonder what could be leveraged from this API (if it is supported by Netmap?).
For PCAP users, well they are probably not considering speed. But anyway one scope for both cases I can see is :

• We register all fds like usual in the event part of the router driver loop.
• We process the one that directly returned (that will schedule some tasks), and at next turn we look at other events that are now available instead of re-polling.
• If a fd is rescheduled, we submit it at next turn too.

We would avoid some system calls. One win case is, if one device is busy, we do not re-poll for a second idle device at each turn (after processing some packets of the busy device) for nothing. Its fd is still in the AIO queue. While with the current scheme, one would process a burst of packet, poll, process another one, poll, ...
But then with Netmap we need to call poll to flush the packets anyway for the busy device, so it's for PCAP only. And XDP will not use poll either. So this would make PCAP/socket a little better, but who really cares?

Hello,

I am using UDPIP6ENCAP element in my packet generator code and I need to introduce a "unique flows per second" feature. I am thinking of a configuration parameter "fps" accepting an integer n, and implement a function that polls systime and increments the _saddr every 1/n seconds increase in systime.

If you think this is an acceptable approach, what would be the best way to call this increment function on its own thread so that it doesnt lie on the fastpath? And do you think UDPIP6Encap::configure(Vector<String> &conf, ErrorHandler *errh) is the appropriate place to call it?

Thanks!

Hi Guys,

Thanks for the software, i found very interesting to use this software for my project.

I want to integrate click in user space on netmap, we were successfull on compiling the Click for netmap. But we are not sure on how to generate the click configuration file to perform some basic level testing.

As in the README.md document its specified only in single line
An example configuration is : FromNetmapDevice(netmap:eth0) -> CheckIPHeader() -> ToNetmapDevice(netmap:eth1)
but when we open other Click configuration file in conf folder there were multiple lines.

Please help me on creating/procedure for generating Click configuration file for Netmap support.
Please share if any example configuration exists

Waiting for your prompt feedback

Regards,
Srinivas

Following is the single pipeline packet generator

Idle -> EnsureDPDKBuffer -> Discard;
InfiniteSource(LENGTH 10, LIMIT 10000000, STOP true)
        -> NumberPacket(OFFSET 0)
        -> UDPIP6Encap(2001:2001:2001:2001::1, 1234, 2001:2001:2001:2001::2, 1234)
        -> EtherEncap(0x86DD, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db)
        -> record :: RecordTimestamp()
        -> CheckNumberPacket(OFFSET 62, COUNT 10000000)
        -> diff :: TimestampDiff(OFFSET 62, RECORDER record)
        -> counter :: AverageCounter()
        -> Discard;

Yielding these results

diff.average:
0.0118919
counter.count:
10000000
counter.rate:
2661698.16343

Running on

Model name:          Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Stepping:            4
CPU MHz:             2800.103
CPU max MHz:         3600.0000
CPU min MHz:         1200.0000
BogoMIPS:            5600.20

No matter how many cores I pass, I get approximately 2.6Mpps as shown above. If I comment out the etherencap and udpencap, it is approx 3Mpps.

Is this expected?

It would be great to have some way to fix parameters at compile time. E.g., avoid run-time checks when those parameters are fixed, like

Element<STATICPARAM true>(PARAM 5, ...)

Btw, I converted the card from https://github.com/tbarbette/fastclick/projects/1 which appear to not be very great for discussions. Feel free to leave ideas there and convert them to issue when you want to start implementing them and/or want to start a discussion to make the idea more mature.

The easiest path is to modify the EXPORT_ELEMENT to allow some kind of alias, and then define, eg :

template class FromDPDKDevice<bool is_kni>

and then :

EXPORT_ELEMENT(FromDPDKDevice, FromDPDKDevice<false>)
EXPORT_ELEMENT(FromDPDKKNIDevice, FromDPDKDevice<true>)

This to avoid a costly branch in the fast path.

The longer road would be to have some tool like click-devirtualize that actually duplicate the source to compile a version of fromdpdkdevice.cc where all occurences of a normal parameter are replaced by the value of the config file. The advantage for the method is that it allows to replace any parameter to any value. Eg, with the template method if we want to allow int replacement, we need to declare a template for each possible int...

It appears that the last commits introduced a small issue with the default NDESC in ToDPDKDevice: if the value is not specified at configuration time, then QueueDevice::ndesc (which is an int) gets -1 (here), but then it is passed to DPDKDevice::add_tx_device (here), which takes an unsigned. Also related, in the ternary expression here and here the condition is always true (because it's unsigned).

Of course, this causes the call to rte_eth_tx_queue_setup to fail.
Trivial workaround is to manually specify NDESC when configuring ToDPDKDevice.

This might be useful for migrating existing XDP programs to FastClick and could simplify the way existing elements, e.g. IPClassifier, work.

I am running the following code

DPDK0_OUT ::  ToDPDKDevice(0, N_QUEUES 1);
DPDK1_IN :: FromDPDKDevice(1, PROMISC true);

InfiniteSource(LENGTH 64, LIMIT 15000000, STOP true)
        -> NumberPacket(OFFSET 0)
        -> UDPIP6Encap(2001:2001:2001:2001::1, 1234, 2001:2001:2001:2001::2, 1234)
        -> EtherEncap(0x86DD, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db)
        -> record :: RecordTimestamp()
        -> DPDK0_OUT;

DPDK1_IN
        -> CheckNumberPacket(OFFSET 62, COUNT 15000000)
        -> diff :: TimestampDiff(OFFSET 62, RECORDER record)
        -> counter :: AverageCounter()
        -> Discard;

The code runs if the DPDK eal param passes a single core (-l 1 --master-lcore 1). It throws the error in the title when multiple cores are passed (-l 1-5 --master-lcore 1) despite the fact that N_QUEUES 1 is defined.

Any ideas?

Thanks

Hi,

I am trying to run a traffic source (SRC) and sink (SNK) in separate docker containers on the same server hardware. My aim is to measure the packet rate and latency.

The RecordTimestamp element ecosystem seems to be fit for this purpose. However the TimeStampDiff, which shall reside in SNK, requires the instance of RecordTimestamp that provides the initial timestamp, which resides in the SRC.

Would it make sense to introduce a SYSTIME argument to RecordTimestamp and TimeStampDiff for cases like mine where both the SRC and the SNK share the same time source?

Thanks

While DPDK is adding support for AF_XDP (http://mails.dpdk.org/archives/dev/2018-February/091502.html) we should have a FromXDPDevice and ToXDPDevice element (or make it a sub-method of FromDevice if the push/pull syntax is identical) that allows doing high-speed without using DPDK.
The setup of DPDK stays a little bit complex, requires to bind some NICs, etc. This would allow somehow much higher-speed processing in user-level, without the complexity of DPDK.

This Perl script is fragile and should be replaced with something like http://coccinelle.lip6.fr/ or https://people.eecs.berkeley.edu/~necula/cil/ or something based on http://clang.llvm.org/

I emulate a virtual network by connecting several Click processes with netmap native (patched) veth pairs and From/ToNetmapDevice.

Now I want to rate limit ToNetmapDevice interfaces.

My first attempt was to use BandwidthRatedSplitter in order to maintain full push path. However, TCP rate control algorithms go crazy with that. When I limit an interface to 1 Gbps, rate of a TCP flow varies between 0 and 1.5 Gbps and iperf reports average 200 Mbps.

So I decided to use Queue -> BandwidthRatedUnqueue. This introduced a problem of empty runs and failed pulls because Click was repeatedly scheduling run_task. Using QuickNoteQueue instead Queue reduced this problem to an acceptable level. But what is more important, TCP rate was smooth and equal to limit, the same as I would limit interface bandwidth with tc command.

This was working fine as long as all Click processes were connected serially. When I emulated any non-linear network, I started to observe segfaults. They happen on packet allocation or freeing. Here are some examples:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  WritablePacket::pool_data_allocate () at ../lib/packet.cc:397
397     ../lib/packet.cc: No such file or directory.
(gdb) bt
#0  WritablePacket::pool_data_allocate () at ../lib/packet.cc:397
#1  WritablePacket::pool_allocate (tailroom=0, length=1514, headroom=0) at ../lib/packet.cc:418
#2  Packet::make (headroom=0, data=data@entry=0x0, length=1514, tailroom=tailroom@entry=0) at ../lib/packet.cc:719
#3  0x000056093ab363d0 in KernelTun::one_selected (p=@0x7ffebad4ace0: 0x0, now=<synthetic pointer>..., this=0x56093c363160) at ../elements/userlevel/kerneltun.cc:522
#4  KernelTun::selected (this=0x56093c363160, fd=<optimized out>) at ../elements/userlevel/kerneltun.cc:494
#5  0x000056093abb7022 in SelectSet::call_selected (mask=<optimized out>, fd=10, this=0x56093c34f8e8) at ../lib/selectset.cc:367
#6  SelectSet::run_selects_poll (this=this@entry=0x56093c34f8e8, thread=thread@entry=0x56093c34f870) at ../lib/selectset.cc:474
#7  0x000056093abb7221 in SelectSet::run_selects (this=this@entry=0x56093c34f8e8, thread=thread@entry=0x56093c34f870) at ../lib/selectset.cc:581
#8  0x000056093aba7309 in RouterThread::run_os (this=0x56093c34f870) at ../lib/routerthread.cc:476
#9  RouterThread::driver (this=0x56093c34f870) at ../lib/routerthread.cc:645
#10 0x000056093aa4a0ed in main (argc=<optimized out>, argv=<optimized out>) at click.cc:803

Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI___libc_free (mem=0x663a61633a38313a) at malloc.c:3103
3103    malloc.c: No such file or directory.
(gdb) bt
#0  __GI___libc_free (mem=0x663a61633a38313a) at malloc.c:3103
#1  0x000056539d7e17ee in WritablePacket::~WritablePacket (this=0x56539eac5dac, __in_chrg=<optimized out>) at ../include/click/packet.hh:937
#2  WritablePacket::recycle (p=0x56539eac5dac) at ../lib/packet.cc:546
#3  0x000056539d7b6575 in Packet::kill (this=<optimized out>) at ../include/click/packet.hh:1671
#4  KernelTun::one_selected (p=@0x7ffcb59a68f0: 0x56539eac5dac, now=<synthetic pointer>..., this=0x56539eaea2d0) at ../elements/userlevel/kerneltun.cc:560
#5  KernelTun::selected (this=0x56539eaea2d0, fd=<optimized out>) at ../elements/userlevel/kerneltun.cc:494
#6  0x000056539d837022 in SelectSet::call_selected (mask=<optimized out>, fd=13, this=0x56539eab08e8) at ../lib/selectset.cc:367
#7  SelectSet::run_selects_poll (this=this@entry=0x56539eab08e8, thread=thread@entry=0x56539eab0870) at ../lib/selectset.cc:474
#8  0x000056539d837221 in SelectSet::run_selects (this=this@entry=0x56539eab08e8, thread=thread@entry=0x56539eab0870) at ../lib/selectset.cc:581
#9  0x000056539d827309 in RouterThread::run_os (this=0x56539eab0870) at ../lib/routerthread.cc:476
#10 RouterThread::driver (this=0x56539eab0870) at ../lib/routerthread.cc:645
#11 0x000056539d6ca0ed in main (argc=<optimized out>, argv=<optimized out>) at click.cc:803

Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:332
332     ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:332
#1  0x000055f2562c7ff5 in _send_packet (cur=<synthetic pointer>: <optimized out>, slot=<optimized out>, txring=0x7f469628f000, p=<optimized out>) at ../elements/userlevel/tonetmapdevice.cc:373
#2  ToNetmapDevice::send_packets (txsync_on_empty=true, ask_sync=true, head=@0x55f258113748: 0x7f470000001f, this=0x55f258113120) at ../elements/userlevel/tonetmapdevice.cc:429
#3  ToNetmapDevice::push_batch (this=<optimized out>, port=0, b_head=0x7f470000001f) at ../elements/userlevel/tonetmapdevice.cc:207
#4  0x000055f2562518f6 in Element::Port::push_batch (batch=<optimized out>, this=<optimized out>) at ../include/click/element.hh:781
#5  BandwidthRatedUnqueue::run_task (this=0x55f258113c70) at ../elements/standard/bwratedunqueue.cc:48
#6  0x000055f2563253a9 in Task::fire (this=0x55f258113da0) at ../include/click/task.hh:584
#7  RouterThread::run_tasks (ntasks=124, this=0x55f2580cc870) at ../lib/routerthread.cc:392
#8  RouterThread::driver (this=0x55f2580cc870) at ../lib/routerthread.cc:613
#9  0x000055f2561c80ed in main (argc=<optimized out>, argv=<optimized out>) at click.cc:803

Most of them happen in KernelTun, but they are not specific to this element. When I replace it with From/ToDevice, the same happens inside it. These elements are the ones which allocate/deallocate packets mostly (since From/ToNetmapDevice only forward them). I use these elements for connecting with routing daemons.

With Pipeliner instead Queue -> BandwidthRatedUnqueue segfaults do not happen, but of course I can not rate limit it.

All the above applies to single thread runs (click -j 1).

With multiple threads, these same segfaults happen. In addition, Click crashes even in linear network after forwarding some number of packets with messages "No more netmap buffers" and "netmap_ring_reinit". This happen also with Pipeliner. So the only configurations working with multiple threads are full push paths without Queue and Pipeliner.

So there are at least two problems:

• segfaults when Queue is used and processes are connected non-serially (happen both with single and multiple threads)
• "No more netmap buffers" when Queue or Pipeliner is used, no matter how routers are connected (happens only with multiple threads)

Full push paths works fine, both with single and multi threads and any network topology.

So are these bugs? Or Queue elements are meant not to be used in netmap pool mode? If so, how rate limiting can be achieved? Should it be implemented in Pipeliner? But it still would only work with single thread.

using this click config:

// test3.click
// This simple test demonstrates how RoundRobinSched works as a round robin
// scheduler.
// Run with 'click test3.click'
rr :: RoundRobinSched;
TimedSource(0.2) -> Queue(20) -> Print(q1) -> [0]rr;
TimedSource(0.5) -> Queue(20) -> Print(q2) -> [1]rr;
rr -> TimedSink(0.1);

and fastclick configured like so:

./configure \
  --prefix=/usr/local \
  --enable-multithread \
  --disable-linuxmodule \
  --enable-intel-cpu \
  --disable-batch \
  --enable-user-multithread \
  --verbose \
  --disable-dynamic-linking \
  --enable-poll \
  --enable-bound-port-transfer \
  --with-netmap=no \
  --enable-zerocopy \
  --enable-dpdk \
  --enable-dpdk-pool \
  --disable-dpdk-packet \
  RTE_SDK=/usr/local/share/dpdk RTE_TARGET=x86_64-native-linuxapp-gcc \
  CFLAGS='-g -O3 -I/usr/local/include/dpdk' \
  CXXFLAGS='-g -O3 -I/usr/local/include/dpdk -std=gnu++11' \
  --enable-local

from git commit 419abd7 (Wed Jul 5 10:59:15 2017 +0200).

and run click test3.click:
I get a segfault:

Program received signal SIGSEGV, Segmentation fault.
0x00000000007ab1c9 in DPDKDevice::get_mpool (socket_id=4294967295) at ../lib/dpdkdevice.cc:128
128         return _pktmbuf_pools[socket_id];
(gdb) bt
#0  0x00000000007ab1c9 in DPDKDevice::get_mpool (socket_id=4294967295) at ../lib/dpdkdevice.cc:128
#1  0x000000000075b936 in DPDKDevice::get_pkt (numa_node=<optimized out>)
    at ../include/click/dpdkdevice.hh:217
#2  DPDKDevice::get_pkt () at ../include/click/dpdkdevice.hh:229
#3  Packet::alloc_data (tailroom=0, length=2176, headroom=0, this=0xe15940) at ../lib/packet.cc:605
#4  WritablePacket::pool_data_allocate () at ../lib/packet.cc:401
#5  0x000000000075c1fd in WritablePacket::pool_allocate (tailroom=0, length=69, headroom=128)
    at ../lib/packet.cc:418
#6  Packet::make (headroom=128, data=0xe15dcc, length=69, tailroom=tailroom@entry=0) at ../lib/packet.cc:717
#7  0x00000000004397a6 in TimedSource::configure (this=0xe11b00, conf=..., errh=0x7fffffffe2e0)
    at ../elements/standard/timedsource.cc:51
#8  0x0000000000798935 in Router::initialize (this=this@entry=0xe15ac0, errh=errh@entry=0xe0da90)
    at ../lib/router.cc:1158
#9  0x000000000074eed9 in parse_configuration (text=..., text_is_expr=text_is_expr@entry=false,
    hotswap=hotswap@entry=false, errh=errh@entry=0xe0da90) at click.cc:400
#10 0x0000000000466b9b in main (argc=<optimized out>, argv=<optimized out>) at click.cc:730

if I configure without any dpdk arguments:

./configure \
  --prefix=/usr/local \
  --enable-multithread \
  --disable-linuxmodule \
  --enable-intel-cpu \
  --disable-batch \
  --enable-user-multithread \
  --verbose \
  --disable-dynamic-linking \
  --enable-poll \
  --enable-bound-port-transfer \
  --with-netmap=no \
  --enable-zerocopy \
  CFLAGS='-g -O3 ' \
  CXXFLAGS='-g -O3 -std=gnu++11' \
  --enable-local

It works great:

dl3n-1 $ click test3.click
q1:   69 | 52616e64 6f6d2062 756c6c73 68697420 696e2061 20706163
q1:   69 | 52616e64 6f6d2062 756c6c73 68697420 696e2061 20706163

Hi,

I am looking for a way to measure the average rate of cpu cycles that a pipeline of batch supporting elements take. Is it doable with the *CycleCount family of elements? If so, can you point me to a testie/sample config?

Thanks

Hi all,
I try to run fast click with SRIOV-VF (virtual function) as follow

;================ ;==================
| NIC1-----------NIC1-VF1 |
| Traffic generator | | SUT l2fwd-click |
| NIC2----------NIC2-VF1 |
;================; ; =================

l2fwd.click
thread1:: FromDPDKDevice(0,MAXTHREADS 1, PROMISC true,BURST 32) -> ToDPDKDevice(1, BLOCKING true);
thread2:: FromDPDKDevice(1,MAXTHREADS 1, PROMISC true,BURST 32) -> ToDPDKDevice(0, BLOCKING true);
StaticThreadSched(thread1 0, thread2 1);

L2FWD-Click can received packets (I checked by insert Print(ok) element in the .click file above)
However, the packets never comeback to the Traffic generator, it seems at SUT, that packets never send out.
According to this link http://dpdk.org/doc/guides/rel_notes/known_issues.html#packets-are-not-sent-by-the-1-gbe-10-gbe-sr-iov-driver-when-the-source-mac-is-not-the-mac-assigned-to-the-vf-nic
I already update click ToDPDKDevice element to configure the Ethernet source address in each packet to match that of the VF NIC but no luck
Can you suggest me some direction?
P/S: The connection between two servers are back-back (no switch)
Thank and Best Regards
Lapd

Hi,

In the paper, "Fast Userspace Packet Processing" - you mention improvements done to netmap:

We realized that our NICs have a feature whereby the
status of a transmit packet ring can be mirrored in memory
at very low performance cost. We modified Netmap to exploit
this feature and also limited the release rate of packets
consumed by the NIC to only recover buffer for sent packets
once every interrupt, which released the PCIe bus of useless
informations and brought Netmap performances above
DPDK as we can see in figure 1

Where can this improved netmap code be found?

To be enquiried.

While inspecting the stack traces for #2, it appears that the issue arises when ToDPDKDevice tries to copy a packet's data, after the packet has been clone()d (i.e., when shared() is true).

I was wondering whether that copy is needed: given that (a) the packet cache is thread-private and (b) after the clone, the packet content is completely read-only, would it be possible to avoid this extra copy? In the standalone version we are using rte_pktmbuf_refcnt_update to prevent the mbuf from being released after sending, and I'd like to try and implement the same strategy.

While thinking about the possibilities we have to replicate that behaviour, I realized of that --(dis|en)able-dpdk-pools compilation flag: reading around the code it seems that enabling it causes a Packet to be just a casted mbuf , but I didn't understand what effects this has overall, nor why in the README you suggest to keep it disabled for best performance (nor where annotations are stored): would you mind elaborating a little?
I'm particularly interested in understanding whether it would be best to implement this dont-copy-even-if-cloned strategy to work with or without the flag enabled.

Thanks a lot,
Davide