Could not allocate packet MBuf pools on different NUMA node about fastclick HOT 13 CLOSED

I see what happens. I'm on it. Actually I'll solve #46 at the same time. I'll simply allocate pools for all sockets, instead of allocating it only for the sockets of the used devices. I guess in your case the segfault happens because an allocation/deallocation is done on a socket that was not expected. I'm not sure I had any gains from finding the nodes actually used and that may change at runtime so let's allocate for all available sockets, and maybe give an explicit option to change that...

from fastclick.

What elements do you have in your config? Only From/ToDPDKDevices?

from fastclick.

Yes.

FromDPDKDevice(0, PROMISC true) -> ToDPDKDevice(1);

Same thing happens with the recordtimestamp pipeline that I posted in the other issue.

from fastclick.

Actually, in this case you probably just need to ask for less MBUF with DPDKInfo. If you change -m (or remove it) does it work?

from fastclick.

Commit 98c74e6 should give more info about the problem and solves the one I spoke of. Reopen if it's still there ;)

from fastclick.

Could not allocate packet MBuf pools : error 12 (Cannot allocate memory)
Maybe try to allocate less buffers with DPDKInfo(X) or allocate more memory to DPDK by giving/increasing the -m parameter or allocate more hugepages.

The same config works when I pass a core in numa node 0.

from fastclick.

It may still allocate packets on the two nodes. If you set DPDKInfo(4096) does it work?

from fastclick.

No it doesnt work.
Another observation; when I run the code on a core on numa 0, the number of free hugepages drop by one (-m parameter passes 1024 and hugepages size is 1G so makes sense)
If I map a core on numa 1, the available hugepages on numa 0 go all the way down to 0 and I get the error above.

Just to make sure, following is my configure script

./configure \
--enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --enable-numa \
--verbose CFLAGS="-g -O3" CXXFLAGS="-g -std=gnu++11 -O3" --disable-dynamic-linking --enable-poll \
--enable-ip6 --enable-local --enable-bound-port-transfer --enable-dpdk --enable-batch --with-netmap=no \
--enable-zerocopy --enable-dpdk-pool --disable-dpdk-packet

Should I try with --enable-dpdk-packet maybe? Above script is mostly from the DPDK guide in the main page of the repo.

from fastclick.

No --enable-dpdk-packet allows to use DPDK packets instead of the Packet object. Won't matter in this problem. I'm afraid my timeslot on this problem is restricted these days. You can work around it as of now, right?

from fastclick.

Hi, I think in NUMA system, this problem can be seen as a bug.
Because, when we set the EAL parameters "--socket-mem 0,1024" for fastclick, we want fastclick just allocates exactly what we set.
However, I find that function "DPDKDevice::alloc_pktmbufs()" in "lib/dpdkdevice.cc " didn't check EAL parameters at all.
Then fastclick will fail to allocate memory because it will allocate memory on numa node 0 and we didn't allocate for it.

And there is one thing I can't understand. Why function "DPDKDevice::alloc_pktmbufs()" allocates 1 _pktmbuf_pools when fastclick run on NUMA node 0? But it allocates one _pktmbuf_pools on NUMA node 0 and one _pktmbuf_pools on NUMA node 1 when fastclick process run on NUMA node 1? What's the motivation to allocate memory like this way?
I think fastclick should allocate local memory preferentially for the sake of performance.

from fastclick.

It will always allocate buffer pools starting from ID zero. But they won't necessarily be used. If you have DPDKInfo(NB_SOCKET_MBUF 0, NB_SOCKET_MBUF 65536) then the socket 0 will have no memory.

Then by default FromDPDKDevice will use the buffer pool of the socket where the device is attached. Old elements that do not use the thread vector system defaults to allocating packets on pool 0, so we need it. For instance an element that generates ICMP error. It's hardly critical. Then when you use devices on two sockets, it's not sure, even if this ICMP error element was smarter, if the packet should be written on memory of the current socket, or if it should be written of memory of the remote socket, but where the device is attached. In any case, it won't hurt the cache to allocate memory for nothing, which is all that matters at high speed.

I understand it seems like you pass the information twice, but --socket-mem parameter is consumed by the DPDK EAL, it's not available to the application. Then even if we could get that information from the application, that parameter is a number of MB, not a number of packets. Some hashtables etc will consume memory, so we cannot allocate 1024M/buffer_size packets on each node.

I'm open to suggestions, though. We could find that if the app is only using cores and devices of socket 1, then there's no chance we'll actually need packets of socket 0. But in the end of the day it does not change anything.

from fastclick.

Hi, I have a requirement for high-performance network functions using fastclick with DPDK. I'd like to know whether fastclick supports multi-thread execution using DPDK. Because you said fastclick cannot read EAL arguments, how can I assign the cores for fastclick? And what will happen if I simultaneously let both DPDK (through EAL arguments) and fastclick use multiple threads?

from fastclick.

In DPDK applications, the EAL arguments are passed to DPDK. Most DPDK-based application have a parameter like "--dpdk" or "--dpdk-args" then followed by "--". Anything in between is passed to the EAL, like the -l or -c parameters to set the number of threads and affinity.

Of course, once passed to the DPDK EAL the application can query the amount of threads etc and work with it. This is what FastClick does. So be assured, what you pass in the --dpdk arguments is used by FastClick. The normal core parameters of FastClick are even deactivated when passing --dpdk (-j and -a will be ignored).

Basically if you pass -l 1,7,9,18 FastClick will have 4 threads, pinned to the 4 given cores.

Now, FastClick handles threading by itself in basic cases, but you can force things :

Using StaticThreadSched(element 3); will pin the element to core 18, as the 4th core in the list is 18 (and threads are indexed at 0). FromDPDKDevice has also arguments such as MAXTHREADS, MINTHREADS, NUMA and SCALE :)

from fastclick.