Poor userspace performance with custom element about fastclick HOT 7 CLOSED

You should double the core assigned to the DUT application when using NUMA, because FastClick will avoid the cores that are not on the same NUMA node than the device. Or you can pass NUMA false to the FromDPDKDevice. Pass VERBOSE 1 to display the chosen assignation.

But in this case as you only have one side, the second FDD does nothing, so I guess it have one full core for itself anyway.

On the generator side, the memory in itself is one thing, but you have to pass DPDKInfo(X) to allocate more DPDK buffers. These are different things. Default is 65536, so if you allocate more than that, it won't go well as Replay keeps all of them in memory.

Please display counter.link_rate, I'm more familiar with that one.

For the code :
Don't use String in the fastpath. unparse_expanded prohibited. One reason is that it uses malloc/free, absolutely not made for high-speed.

Then simply launch perf top -a -g --cpu X --no-children and check the hit points. You can give the result here if you want. Flamegraphs are cool too. You may want to use those in a version recomplied with ./configure .... CXXFLAGS="-O1 -g"

You need state for a 4to6? What's your state management, a HashTable?

from fastclick.

I so much appreciate your feedback Tom!

You should double the core assigned to the DUT application when using NUMA, because FastClick will avoid the cores that are not on the same NUMA node than the device.

If only could I do that! Over-the-wire code mentioned above refuses to start with the failure #71 when I pass more than one core. And I am making sure that I am passing core numbers from the same NUMA node. "NUMA false" didnt change anything and "VERBOSE 1" didnt display anything.

But the following starts when two cores are passed via EAL!

FromDPDKDevice(0, PROMISC true, VERBOSE 1) -> ToDPDKDevice(1);
FromDPDKDevice(1, PROMISC true, VERBOSE 1) -> ToDPDKDevice(0);

But in this case as you only have one side, the second FDD does nothing, so I guess it have one full core for itself anyway.

Correct.

... DPDKInfo(X) to allocate more DPDK buffers. These are different things. Default is 65536, so if you allocate more than that, it won't go well as Replay keeps all of them in memory.

Thanks, I am not using Replay yet due to the weird behavior described in #68 when used with encapsulation elements.

Please display counter.link_rate, I'm more familiar with that one.

Will do

Don't use String in the fastpath. unparse_expanded prohibited.

A noob mistake of mine, thanks for pointing that out.

You need state for a 4to6? What's your state management, a HashTable?

Not for new connections, only to track if the incoming 4 belongs to an initiator 6. The following is a short explanation.
https://github.com/kthkaya/fastclick/blob/e37de1010c0c364796c9455d05fd5ca1520cb140/elements/ip6/statefultranslator64.hh#L20-L26
But at this stage, the traffic pattern pushed does not stress it, i.e. SA, SP, DA, DP is fixed for all packets.

from fastclick.

Hi Tom,

I followed your suggestions and removed string parsing. Now the code looks like this, https://github.com/kthkaya/fastclick/blob/e37de1010c0c364796c9455d05fd5ca1520cb140/elements/ip6/statefultranslator64.cc .

The improvement is great, however I have a very peculiar situation. No matter at what rate I send packets, the nat64 code segfaults with No more DPDK buffer available error exactly after 261376 packets. Note that this is still one-way traffic generation on single core.

Here is the trace

No more DPDK buffer available ! Try using DPDKInfo to allocate more.

Thread 1 "click" received signal SIGSEGV, Segmentation fault.
__memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:78
78      ../sysdeps/x86_64/multiarch/../memset.S: No such file or directory.
(gdb) bt
#0  __memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:78
#1  0x00000000008163b1 in memset (__len=<optimized out>, __ch=0, __dest=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/string3.h:90
#2  StatefulTranslator64::translate64 (this=this@entry=0x2991c20, p=p@entry=0x5580200,
    v6l3h=v6l3h@entry=0x7f9cf467ef8e, l4h=l4h@entry=0x7f9cf467efb6, addressAndPort=0x29869b0)
    at ../elements/ip6/statefultranslator64.cc:30
#3  0x0000000000816d4f in StatefulTranslator64::sixToFour (p=0x5580200, this=0x2991c20)
    at ../elements/ip6/statefultranslator64.cc:203
#4  StatefulTranslator64::push_batch (this=0x2991c20, port=<optimized out>, batch=0x5580200)
    at ../elements/ip6/statefultranslator64.cc:246
#5  0x00000000008744ef in Element::Port::push_batch (batch=0x5580200, this=<optimized out>)
    at ../include/click/element.hh:781
#6  BatchElement::output_push_batch (port=0, batch=0x5580200, this=0x2992700)
    at ../include/click/batchelement.hh:73
#7  FromDPDKDevice::run_task (this=<optimized out>, t=0x2991790) at ../elements/userlevel/fromdpdkdevice.cc:189
#8  0x00000000008e07fd in Task::fire (this=0x2991790) at ../include/click/task.hh:584
#9  RouterThread::run_tasks (ntasks=39, this=0x2982600) at ../lib/routerthread.cc:392
#10 RouterThread::driver (this=0x2982600) at ../lib/routerthread.cc:613
#11 0x0000000000470c99 in main (argc=<optimized out>, argv=<optimized out>) at click.cc:803
(gdb)

If I set the ratedsource limit to 250K, no issues. After packet src/snk completes succesfully, if I start another packet gen with 15K limit, nat64 segfaults. So the the total nr of packets the nat can process is 261376.

I believe the code is having difficulty with freeing buffers for later re-use. Can you point me to the right direction please?

from fastclick.

Clearly yes packets are leaking. I didn't catch the fact that it worked for some time.

Line 207 : return 0, may create leak. But I doubt it's the problem...

Maybe remove elements in your pipeline until you find the one that leaks, maybe it's not yours? You may also try with --disable-dpdk-pool once, it may help catch the leak. Still without replay right?

Your performance results are not to trust in this stage, because not reusing DPDK buffers hits the memory cache very much... That problem must be fixed...

from fastclick.

You seem to be leaking packet at line 203 and line 218 where you return the newly created translated packet but never kill the original Packet *p

from fastclick.

Also, in the non-translate case there is an arbitrary value returned by the function. It would actually be a nice addition to have a scope guard that took a Packet* and guaranteed that it would invoke kill prior to the end of scope.

from fastclick.

Thanks guys. I have been away for some time, will be circling back to this soon. Will let you know.

from fastclick.