TimeStamp elements cause segfault in multi-threaded execution. about fastclick HOT 13 CLOSED

The description of N is

N — Size of the reservoir. Defaults to 0.

which is a bit ambiguous to me. How should I configure RecordTimestamp TimestampDiff to play along thread safe?

from fastclick.

Just a note on the use of StaticThreadSched. You typically only need to specify schedulable elements using StaticThreadSched, these include elements that implement Task such as FromXDevice, ToXDevice, Infinite/Random/RatedSource, Unqueue, ToXDevice etc. So using your example the following would probably be sufficient:

StaticThreadSched(rs 1);

Which would result in packet path from rs all the way up to OUT queue to run on thread 1, with FromDevice and ToDevice on default thread 0.

from fastclick.

Which would result in packet path from rs all the way up to OUT queue to run on thread 1, with FromDevice and ToDevice on default thread 0.

Wow good to know, thanks Beyers. I assume StaticThreadSched can only pin to a single thread. In that case, would it make sense creating multiple pipelines as following?

record :: RecordTimestamp();

rs :: RatedSource(LENGTH 64, RATE 100000, LIMIT 1000000, STOP true)
        -> numPack :: NumberPacket(OFFSET 0)
        -> u6Encap :: UDPIP6Encap(2001:2001:2001:2001::1, 1234, 64:ff9b::192.168.2.5, 1234)
        -> ethEncap :: EtherEncap(0x86DD, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db);

rs1 :: RatedSource(LENGTH 64, RATE 100000, LIMIT 1000000, STOP true)
        -> numPack :: NumberPacket(OFFSET 0)
        -> u6Encap :: UDPIP6Encap(2001:2001:2001:2001::1, 1234, 64:ff9b::192.168.2.5, 1234)
        -> ethEncap :: EtherEncap(0x86DD, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db);

rs2 :: RatedSource(LENGTH 64, RATE 100000, LIMIT 1000000, STOP true)
        -> numPack :: NumberPacket(OFFSET 0)
        -> u6Encap :: UDPIP6Encap(2001:2001:2001:2001::1, 1234, 64:ff9b::192.168.2.5, 1234)
        -> ethEncap :: EtherEncap(0x86DD, 00:04:23:D0:93:63, 00:17:cb:0d:f8:db);

rs-> record ->OUT;
rs2-> record ->OUT;
rs3-> record ->OUT;

If yes, how could I pin rs to Thread 1, rs1 to Thread 2 and rs2 to Thread 3? If I just do StaticThreadSched(rs 1), will the packet path all the way up to ethEncap be mapped?

PS: I understand that NumberPacket element should have a startFrom handler for each pipeline's packets to be uniquely numbered.

from fastclick.

For timestampdiff to be threadsafe, you must pass N, the maximum amount of packets it will receive. If you don't it will double its size from time to time, which is of course not thread safe.

For the StaticThreadSched, the element running the task only matters. So you must pin the RatedSource in this case, if your out is a fully "push" path. Eg if you use ToDPDKDevice. If not, it's your Queue that actually matters.

Then having 3 record -> OUT; is bad practice. Define the path once. Eg

rs -> record;
rs2 -> record;
rs3 -> record;
record -> OUT;

from fastclick.

For the StaticThreadSched, the element running the task only matters. So you must pin the RatedSource in this case, if your out is a fully "push" path. Eg if you use ToDPDKDevice. If not, it's your Queue that actually matters.

Queue does not run a task. In "normal" Click with something like fd::FromDevice(eth0) -> Queue -> td::ToDevice(eth1) you would need to schedule fd and td as they are running the tasks.

from fastclick.

Yes. You're right. I was thinking of the Pipeliner. Which is different.

from fastclick.

Thanks a lot gentlemen, passing "N packetcount" to TimeStampDiff solved the segfault and I can pin the pipeline from ratedsource to OUT as Beyers suggested. Imho "N — Size of the reservoir. Defaults to 0" description could be changed to someting more descriptive to an "outsider" to the element.

I am observing a peculiar latency increase when using multithreading, compared to single thread execution. In the following tests, RatedSource RATE 30000 LIMIT 600000 and TimeStampDiff N 600000

All on single thread/core, (no -j or -a flags)

diff.average:
1237.78100009
counter.count:
599845
counter.rate:
16222.111042

RatedSource is on thread/core 1, rest is on thread/core 0

diff.average:
1339.71605533
counter.count:
599807
counter.rate:
16345.2964901

I executed these tests several times and multi-threaded execution reports approximately 100 microseconds higher diff.average and I don't see any reason for that. Do you have any ideas?

PS: The cores are isolated during OS bootup via isolcpus. Click threads are the only ones running on them.

from fastclick.

Two cores, twice more queues. More latency. Try dividing NDESC by 2 and see if it's perfectly compensated. Try NDESC 128 if you don't know the default (hardware dependent).

from fastclick.

Oh you're using Linux stack. If RatedSource is on core 1 and ToDevice on core 0, then you pay cache misses and scheduling. So the queue is probably the problem.

from fastclick.

This paper explains the issue Tom mentions above in more detail: Flexible Control of Parallelism in a Multiprocessor PC Router

from fastclick.

Thanks guys, much appreciated!

from fastclick.

Just to add to that, typically you want to try and schedule FromDevice/ToDevice in pair in the typical packet flow. Example, router/switch with two NICs:

fd0::FromDevice(eth0) -> Queue -> td1::ToDevice(eth1);
fd1::FromDevice(eth1) -> Queue -> td0::ToDevice(eth0);

StaticThreadSched(fd0 0, td1 0, fd1 1, td0 1);

That way packet forward from eth0 to eth1 runs on the same thread, and inverse from eth1 to eth0 on another thread.

from fastclick.

Just to add to that, typically you want to try and schedule FromDevice/ToDevice in pair in the typical packet flow. Example, router/switch with two NICs:

fd0::FromDevice(eth0) -> Queue -> td1::ToDevice(eth1);
fd1::FromDevice(eth1) -> Queue -> td0::ToDevice(eth0);

StaticThreadSched(fd0 0, td1 0, fd1 1, td0 1);

That way packet forward from eth0 to eth1 runs on the same thread, and inverse from eth1 to eth0 on another thread.

Thanks, I guess that is what I achieved by scheduling the rs (in my first post) to T1 so everything from RatedSource all the way to OUT (interface eth0) are pinned to T1. I didn't do a specific pinning for the rest so that everything from IN (interface eth1) all the way to Discard are mapped to the default thread T0.

from fastclick.