How to ReplayUnqueue at RATE packets per second? about fastclick HOT 10 CLOSED

Add PushNull element after RatedSources. That will improve the performance.

from fastclick.

If CPU is not at 100% on any CPU it means the problem is not the cycles.
Counter use an EWMA, it may just lag a little. AverageCounter will give you the rate between the first and last packet.

from fastclick.

Tried with Counter too. I believe something is not right with the RatedSource element, or I don't properly understand its purpose. Following is a lean test of RatedSource.

//--------Program Start----------
rs :: RatedSource(LENGTH 64, RATE 25000, LIMIT 500000, STOP true)
        -> NumberPacket(OFFSET 0)
        -> record1 :: RecordTimestamp()
        -> CheckNumberPacket(OFFSET 0, COUNT 500000)
        -> diff1 :: TimestampDiff(OFFSET 0, N 500000, RECORDER record1)
        -> c1 :: Counter()
        -> Discard;

DriverManager(wait, read diff1.average, read c1.count, read c1.rate)
//--------Program End----------

Results

diff1.average:
2.172142
c1.count:
500000
c1.rate:
16722.656

After some exploration, I found the RatedUnqueue element, which I believe what I was looking for. And I crafted the following test with InfiniteSource and RatedUnqueue.

//--------Program Start----------
is :: InfiniteSource(LENGTH 64, LIMIT 525000, STOP true)
        -> Queue(25007)
        -> RatedUnqueue (RATE 25000)
        -> NumberPacket(OFFSET 0)
        -> record :: RecordTimestamp()
        -> CheckNumberPacket(OFFSET 0, COUNT 500000)
        -> diff1 :: TimestampDiff(OFFSET 0, N 500000, RECORDER record)
        -> c1 :: Counter()
        -> Discard;

DriverManager(wait, read diff1.average, read c1.count, read c1.rate)
//--------Program End----------

Results

diff1.average:
0.117046
c1.count:
500000
c1.rate:
25007.8125

I got the counter rate that I was looking for (a little higher than what I was expecting). If my understanding of RatedUnqueue and Counter.rate in the last experiment is correct, then something is wrong with the RatedSource. What do you think?

from fastclick.

RatedSource should work.
The solution you have is performance-wise horrible. And horrible. Maybe batching is the issue here? It would break the algorithm maybe. Can you try once RatedSource solution by compiling with --disable-batch ?

from fastclick.

Wow! It indeed was the issue. The CPU utilization now hit %100 and following is the result.

diff1.average:
0.002588
c1.count:
500000
c1.rate:
25007.8125

Perhaps it could be a good idea to pop a warning, or completely lock RatedSource when batching is not disabled?

from fastclick.

No we have to fix the issue :p
I'm not sure why it is at 100% though...

from fastclick.

Idk if it matters but fastclick is running in a Docker container in all those tests.

from fastclick.

Can you provide the full config?

from fastclick.

Dockerfile

FROM ubuntu:17.10

######################
#Initial Preparations#
######################

RUN echo "Initial Preparations ..."

RUN apt-get update && apt-get install -y \
        build-essential automake git gcc tar wget python-pip libcap-ng-dev libnuma-dev \
        gawk pciutils linux-headers-$(uname -a | awk '{print $3}') nano kmod apt-utils \
        iproute2 zlib1g-dev
RUN pip install -U pip six

#################
#Installing DPDK#
#################
RUN echo "Installing DPDK ..."

ENV INSTALL_PATH "/usr/src"
ENV RTE_SDK "$INSTALL_PATH/dpdk-17.11"
ENV RTE_TARGET "x86_64-native-linuxapp-gcc"
ENV DPDK_BUILD "$RTE_SDK/$RTE_TARGET"

RUN cd $INSTALL_PATH; \
wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz; \
tar xf dpdk-17.11.tar.xz; \
rm dpdk-17.11.tar.xz; \
cd $RTE_SDK; \
make install -j4 T=$RTE_TARGET DESTDIR=install

######################
#Installing fastclick#
######################
RUN echo "Installing fastclick ..."

ENV CLICK_DIR "$INSTALL_PATH/fastclick"
ENV GIT_URL "https://github.com/kthkaya/fastclick"

RUN cd $INSTALL_PATH; \
git clone $GIT_URL; \
cd $CLICK_DIR; \
./configure --enable-multithread --enable-intel-cpu --enable-user-multithread --enable-ip6; \
make && make install

######
#Done#
######
RUN echo "Done."
CMD ["/bin/bash"]

Build an image

Save above in a file named Dockerfile. In the same folder, run the following
docker build -t fastClick .

Instantiate a container and attach to it.

docker run -ti --name="trafGen" --privileged fastClick:latest

Run the packetgen Click code

Fastclick is in /usr/src/fastclick. Save the RatedSource generator code at my first post and run it.

from fastclick.

No activity, and it should be possible. The OP probably misconfigured.

from fastclick.