Comments (13)
I see what happens. I'm on it. Actually I'll solve #46 at the same time. I'll simply allocate pools for all sockets, instead of allocating it only for the sockets of the used devices. I guess in your case the segfault happens because an allocation/deallocation is done on a socket that was not expected. I'm not sure I had any gains from finding the nodes actually used and that may change at runtime so let's allocate for all available sockets, and maybe give an explicit option to change that...
from fastclick.
What elements do you have in your config? Only From/ToDPDKDevices?
from fastclick.
Yes.
FromDPDKDevice(0, PROMISC true) -> ToDPDKDevice(1);
Same thing happens with the recordtimestamp pipeline that I posted in the other issue.
from fastclick.
Actually, in this case you probably just need to ask for less MBUF with DPDKInfo. If you change -m (or remove it) does it work?
from fastclick.
Commit 98c74e6 should give more info about the problem and solves the one I spoke of. Reopen if it's still there ;)
from fastclick.
Could not allocate packet MBuf pools : error 12 (Cannot allocate memory)
Maybe try to allocate less buffers with DPDKInfo(X) or allocate more memory to DPDK by giving/increasing the -m parameter or allocate more hugepages.
The same config works when I pass a core in numa node 0.
from fastclick.
It may still allocate packets on the two nodes. If you set DPDKInfo(4096) does it work?
from fastclick.
No it doesnt work.
Another observation; when I run the code on a core on numa 0, the number of free hugepages drop by one (-m parameter passes 1024 and hugepages size is 1G so makes sense)
If I map a core on numa 1, the available hugepages on numa 0 go all the way down to 0 and I get the error above.
Just to make sure, following is my configure script
./configure \
--enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --enable-numa \
--verbose CFLAGS="-g -O3" CXXFLAGS="-g -std=gnu++11 -O3" --disable-dynamic-linking --enable-poll \
--enable-ip6 --enable-local --enable-bound-port-transfer --enable-dpdk --enable-batch --with-netmap=no \
--enable-zerocopy --enable-dpdk-pool --disable-dpdk-packet
Should I try with --enable-dpdk-packet maybe? Above script is mostly from the DPDK guide in the main page of the repo.
from fastclick.
No --enable-dpdk-packet allows to use DPDK packets instead of the Packet object. Won't matter in this problem. I'm afraid my timeslot on this problem is restricted these days. You can work around it as of now, right?
from fastclick.
Hi, I think in NUMA system, this problem can be seen as a bug.
Because, when we set the EAL parameters "--socket-mem 0,1024" for fastclick, we want fastclick just allocates exactly what we set.
However, I find that function "DPDKDevice::alloc_pktmbufs()" in "lib/dpdkdevice.cc " didn't check EAL parameters at all.
Then fastclick will fail to allocate memory because it will allocate memory on numa node 0 and we didn't allocate for it.
And there is one thing I can't understand. Why function "DPDKDevice::alloc_pktmbufs()" allocates 1 _pktmbuf_pools when fastclick run on NUMA node 0? But it allocates one _pktmbuf_pools on NUMA node 0 and one _pktmbuf_pools on NUMA node 1 when fastclick process run on NUMA node 1? What's the motivation to allocate memory like this way?
I think fastclick should allocate local memory preferentially for the sake of performance.
from fastclick.
It will always allocate buffer pools starting from ID zero. But they won't necessarily be used. If you have DPDKInfo(NB_SOCKET_MBUF 0, NB_SOCKET_MBUF 65536) then the socket 0 will have no memory.
Then by default FromDPDKDevice will use the buffer pool of the socket where the device is attached. Old elements that do not use the thread vector system defaults to allocating packets on pool 0, so we need it. For instance an element that generates ICMP error. It's hardly critical. Then when you use devices on two sockets, it's not sure, even if this ICMP error element was smarter, if the packet should be written on memory of the current socket, or if it should be written of memory of the remote socket, but where the device is attached. In any case, it won't hurt the cache to allocate memory for nothing, which is all that matters at high speed.
I understand it seems like you pass the information twice, but --socket-mem parameter is consumed by the DPDK EAL, it's not available to the application. Then even if we could get that information from the application, that parameter is a number of MB, not a number of packets. Some hashtables etc will consume memory, so we cannot allocate 1024M/buffer_size packets on each node.
I'm open to suggestions, though. We could find that if the app is only using cores and devices of socket 1, then there's no chance we'll actually need packets of socket 0. But in the end of the day it does not change anything.
from fastclick.
Hi, I have a requirement for high-performance network functions using fastclick with DPDK. I'd like to know whether fastclick supports multi-thread execution using DPDK. Because you said fastclick cannot read EAL arguments, how can I assign the cores for fastclick? And what will happen if I simultaneously let both DPDK (through EAL arguments) and fastclick use multiple threads?
from fastclick.
In DPDK applications, the EAL arguments are passed to DPDK. Most DPDK-based application have a parameter like "--dpdk" or "--dpdk-args" then followed by "--". Anything in between is passed to the EAL, like the -l or -c parameters to set the number of threads and affinity.
Of course, once passed to the DPDK EAL the application can query the amount of threads etc and work with it. This is what FastClick does. So be assured, what you pass in the --dpdk arguments is used by FastClick. The normal core parameters of FastClick are even deactivated when passing --dpdk (-j and -a will be ignored).
Basically if you pass -l 1,7,9,18 FastClick will have 4 threads, pinned to the 4 given cores.
Now, FastClick handles threading by itself in basic cases, but you can force things :
Using StaticThreadSched(element 3); will pin the element to core 18, as the 4th core in the list is 18 (and threads are indexed at 0). FromDPDKDevice has also arguments such as MAXTHREADS, MINTHREADS, NUMA and SCALE :)
from fastclick.
Related Issues (20)
- Create Element HOT 2
- Secondary DPDK with FastClick HOT 2
- Access useful_kcycles() HOT 3
- Input aggregation element HOT 1
- --with-netmap does not compile HOT 3
- FastClick can't identify DPDK PORT,whatever Integer or PCI address.
- port 0 Rx queue allocation failed: Cannot allocate memory HOT 7
- Illegal instruction (core dumped) when using ToDPDKDevice() HOT 3
- rte_mempool segmentation fault. HOT 8
- storeing data in packet HOT 2
- How to enable scattered mode in fastclick? HOT 7
- issue in make package element HOT 3
- How to use replayUnqueue or MultiReplayUnqueue to limit the tx rate? HOT 3
- Automatically update wiki doc with CI
- middleclick configuration HOT 1
- compile error about context HOT 1
- Allow loop in FromDump HOT 4
- Non-existent librairies when parsing llvm-config's output
- FromDPDKDevice(FLOW_RULES_FILE) only work with DPDK<=21.11 HOT 2
- About address of IIO LLC WAYS in Ice lake HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fastclick.