pkelsey / libuinet Goto Github PK

I've done some minor work to get libuinet to build under OS X again. It's not complete and I'm not particularly happy with how I turned off netmap, but libuinet does compile cleanly with these changes:

https://github.com/wxsBSD/libuinet/commit/0623d029b495f7656898592d1e5b0b203d2a910e

However, when building anything that uses libuinet fails with this:

cc -I/Users/wxs/Documents/dev/libuinet/bin/echo/../../lib/libev -g -O0  -I/Users/wxs/Documents/dev/libuinet/bin/echo/../../lib/libuinet/api_include  -o echo echo.o  -L/Users/wxs/Documents/dev/libuinet/bin/echo/../../lib/libuinet -luinet /Users/wxs/Documents/dev/libuinet/bin/echo/../../lib/libev/.libs/libev.a -lm -lpcap -lpthread -lssl
Undefined symbols for architecture x86_64:
  "__DYNAMIC", referenced from:
      _link_elf_init in libuinet.a(libuinet.ro)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [echo] Error 1

If you are interested in the changes mentioned above I'd be happy to submit them as a pull request or if you can point me in the right direction for solving this I'd be happy to come back if I can get any further with it.

I was measuring the performance of libuinet on a KVM VM (which uses virtio) with one core. I performed two tests -

1. I first ran a simple TCP sever which accepts the connections and close the connections. When I run the sample server, I get around 200k PPS and around 25K Connections Per Seconds.
   Next,
2. I modified my TCP server to make use of libuinet and when I run the program, I got around 40K PPS and 5K CPS.
   I am not sure why the performance get reduced when I run with libuinet. I am planning to run callgrind and see where in the libuinet we take time. But, I expected the libuinet will get me better performance.

I can't attach my simple TCP programs here. But, if you can drop me an email at [email protected], where I can share my sample programs.

Other thing is to get the 200K PPS (without libuinet), I made the sysctl changes as follows. Not sure if I need to modify some of your header files for the below mentioned sysctl changes. Can you please point me to where I need to modify.

fs.file-max = 5000000
net.core.netdev_max_backlog = 4000000
net.core.optmem_max = 10000000
net.core.rmem_default = 10000000
net.core.rmem_max = 3000000
net.core.somaxconn = 10000000
net.core.wmem_default = 10000000
net.core.wmem_max = 30000000
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_congestion_control = bic
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_max_syn_backlog = 65000
net.ipv4.tcp_max_tw_buckets = 6000000
net.ipv4.tcp_mem = 30000000 30000000 30000000
net.ipv4.tcp_rmem = 30000000 30000000 30000000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_wmem = 30000000 30000000 30000000
net.ipv4.tcp_early_retrans = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

net.ipv4.tcp_fin_timeout = 7

net.ipv4.tcp_slow_start_after_idle = 0

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_low_latency = 1

Please let me know what you think.

Thank you.

~Peter

Hi,

Just as a note: ether_sprintf() doesn't work. It uses the '%xD' type which prints out x bytes of data as hex values, delimited by an argument. eg, ("%6D", ether_addr, ":").

%xD seems to be a freebsd kernel printf extension?

peter:/UTCP/libuinet/lib/libuinet$ make NETMAP_INCLUDES=/home/peter/UTCP/netmap/sys/
cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -ggdb -O0 -Wall -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wno-pointer-sign -Wmissing-include-dirs -fdiagnostics-show-option -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -DUINET=1 -D_BSD_SOURCE -fstack-protector -I. -I/home/peter/UTCP/libuinet/lib/libuinet/api_include -I/home/peter/UTCP/netmap/sys/ -Werror uinet_if_netmap_host.c
In file included from /home/peter/UTCP/netmap/sys/net/netmap_user.h:70:0,
from uinet_if_netmap_host.c:61:
/usr/include/net/if.h:45:5: error: expected identifier before numeric constant
/usr/include/net/if.h:112:8: error: redefinition of ‘struct ifmap’
/usr/include/linux/if.h:137:8: note: originally defined here
/usr/include/net/if.h:127:8: error: redefinition of ‘struct ifreq’
/usr/include/linux/if.h:171:8: note: originally defined here
/usr/include/net/if.h:177:8: error: redefinition of ‘struct ifconf’
/usr/include/linux/if.h:220:8: note: originally defined here
uinet_if_netmap_host.c: In function ‘if_netmap_ethtool_set_flag’:
uinet_if_netmap_host.c:320:16: error: assignment from incompatible pointer type [-Werror]
uinet_if_netmap_host.c: In function ‘if_netmap_ethtool_set_discrete’:
uinet_if_netmap_host.c:353:16: error: assignment from incompatible pointer type [-Werror]
uinet_if_netmap_host.c: In function ‘if_netmap_set_promisc’:
uinet_if_netmap_host.c:454:19: error: ‘IFF_PROMISC’ undeclared (first use in this function)
uinet_if_netmap_host.c:454:19: note: each undeclared identifier is reported only once for each function it appears in
cc1: all warnings being treated as errors
make: *** [uinet_if_netmap_host.o] Error 1
peter:/UTCP/libuinet/lib/libuinet$

Hi Kelsey,

I am trying to use netmap+libuinet on a Linux machine.
I have compiled the system successfully and now I am trying to test it.
I have started the echo server with ./echo -v -i eth6 -l 192.168.10.1 -p 1111
It seems echo server runs succefully except link_elf_symbol: missing symbol hash table:

uinet starting: cpus=1, nmbclusters=131072
callwheelsize=524288
callwheelsize=524288
link_elf_lookup_symbol: missing symbol hash table
link_elf_lookup_symbol: missing symbol hash table
UINET multiprocessor subsystem configured with 1 CPUs
Timecounters tick every 10.000 msec
Creating interface netmap0, Promiscuous INET disabled, cdom=0
configstr is eth6
netmap0: Ethernet address: 00:1b:21:d7:ff:95
Adding address 192.168.10.1 to interface netmap0
Creating echo server at 192.168.10.1:1111 on interface netmap0
Listening on 192.168.10.1:1111
Bringing up interface netmap0
Creating interface thread for interface netmap0

I cannot ping or connect to echo server!
It seems the echo server is running under some container and is not accessible!
Do I need to configure a bridge/tunnel to access echo server?
Thank you.

Best regards,

Alim

cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -O -gdwarf-2 -Wall -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-inline -Wcast-qual -Wno-pointer-sign -Wmissing-include-dirs -fdiagnostics-show-option -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-incompatible-library-redeclaration -Wno-builtin-requires-header -DUINET=1 -DHAS_NATIVE_TLS -fstack-protector -D__FreeBSD__ -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-builtin -I/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/override_include -I/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/api_include -undef -imacros filtered_predefined_macros.h -nostdinc -I. -I/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys -I/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys/contrib/altq -I/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys/contrib/pf -I./machine_include -I./opt -Werror /usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys/i386/i386/in_cksum.c
/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys/i386/i386/in_cksum.c:508:15: error: variable has incomplete type 'union q_util'
union q_util q_util;
^
/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys/i386/i386/in_cksum.c:508:8: note: forward declaration of 'union q_util'
union q_util q_util;
^
/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys/i386/i386/in_cksum.c:509:15: error: variable has incomplete type 'union l_util'
union l_util l_util;
^
/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys/i386/i386/in_cksum.c:509:8: note: forward declaration of 'union l_util'
union l_util l_util;
^
/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys/i386/i386/in_cksum.c:526:14: error: implicit declaration of function 'in_cksumdata' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
sum += in_cksumdata(addr, mlen) << 8;
^
/usr/home/adrian/git/github/pkelsey/libuinet/lib/libuinet/../../sys/i386/i386/in_cksum.c:533:2: error: use of undeclared identifier 'REDUCE16'
REDUCE16;
^
4 errors generated.
Makefile:349: recipe for target 'in_cksum.o' failed
gmake: *** [in_cksum.o] Error 1
adrian@lucy-11i386:~/git/github/pkelsey/libuinet/lib/libuinet %

I am trying to benchmark user-space TCP stack (libuinet+netmap) on Linux.
I have successfuly built netmap including device drivers and libuinet with netmap.
I have tested netmap alone using pkt-gen utitlity and it works. However, I cannot
not connect to libuinet echo server. I start 'echo' server on a machine and then try
to connect to it from another machine. I have added an ARP entry for
the server IP address on the client machine. I use telnet/nc/iperf to connect to 'echo'
server but nothing works. Please note that 'echo' starts successfully and says listenning
on the socket.

It seems to me that netmap does not invoke the interface callback function to
deliver packets. Can any tell me why I cannot connect to 'echo' server?
Thank you very much.

Cheers,

Abdul Alim

Looks like the Linux port is out of sync. It's referencing a BSD file vnet.h which is included under sys/net/vnet.h, but that folder is not included in the Makefile. If I modify the Makefile, there are too many errors to list here. I've tried this on Centos 6(preferred) and Ubuntu 12.04.
-randy

the error is as following:
sys/netinet/tcp_output.c:1193:11: error: 'ipov' undeclared (first use in this function)

Linux VM 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2 (2019-08-28) x86_64 GNU/Linux

make[2]: Leaving directory '/root/sft/pbscan-master/123/libuinet/bin/multitool'
make[2]: Entering directory '/root/sft/pbscan-master/123/libuinet/bin/tproxy'
cc -I/root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libev -O -gdwarf-2 -I/root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libuinet/api_include -c -o tproxy.o tproxy.c
In file included from /root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libuinet/api_include/uinet_api.h:36,
from tproxy.c:36:
/root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libuinet/api_include/uinet_api_types.h:556:2: error: unknown type name 'uintptr_t'
uintptr_t ref;
^~~~~~~~~
make[2]: *** [: tproxy.o] Error 1
make[2]: Leaving directory '/root/sft/pbscan-master/123/libuinet/bin/tproxy'
make[1]: *** [Makefile:13: all] Error 2
make[1]: Leaving directory '/root/sft/pbscan-master/123/libuinet/bin'
make: *** [Makefile:10: all] Error 2

I found there are many error.
libhttp parser is also can't compile. I updated it with new version(I just copy .a file into folder and error disappeared) but libuinet show error!!!
/root/sft/pbscan-master/123/libuinet/lib/libuinet/../../sys/kern/vnode_if.src(451): Missing end-of-line ; in " IN struct task *task; ".
make: *** [Makefile:484: vnode_if.h] Error 1

libuinet_memstat
memstat_malloc.c:31:10: fatal error: sys/malloc.h: No such file or directory
#include <sys/malloc.h>
^~~~~~~~~~~~~~
compilation terminated.
make: *** [: memstat_malloc.o] Error 1

libuinetnv
In file included from /usr/include/x86_64-linux-gnu/sys/socket.h:33,
from msgio.c:37:
msgio.c: In function 'cred_send':
msgio.c:166:42: error: invalid application of 'sizeof' to incomplete type 'struct cmsgcred'
unsigned char credbuf[CMSG_SPACE(sizeof(struct cmsgcred))];
^~~~~~
msgio.c:193:35: error: invalid application of 'sizeof' to incomplete type 'struct cmsgcred'
cmsg->cmsg_len = CMSG_LEN(sizeof(struct cmsgcred));
^~~~~~
msgio.c:195:20: error: 'SCM_CREDS' undeclared (first use in this function); did you mean 'SOL_RDS'?
cmsg->cmsg_type = SCM_CREDS;
^~~~~~~~~
SOL_RDS
msgio.c:195:20: note: each undeclared identifier is reported only once for each function it appears in
In file included from /usr/include/x86_64-linux-gnu/sys/socket.h:33,
from msgio.c:37:
msgio.c: In function 'cred_recv':
msgio.c:206:42: error: invalid application of 'sizeof' to incomplete type 'struct cmsgcred'
unsigned char credbuf[CMSG_SPACE(sizeof(struct cmsgcred))];
^~~~~~
msgio.c:229:40: error: invalid application of 'sizeof' to incomplete type 'struct cmsgcred'
cmsg->cmsg_len != CMSG_LEN(sizeof(struct cmsgcred)) ||
^~~~~~
msgio.c:230:59: error: 'SCM_CREDS' undeclared (first use in this function); did you mean 'SOL_RDS'?
cmsg->cmsg_level != SOL_SOCKET || cmsg->cmsg_type != SCM_CREDS) {
^~~~~~~~~
SOL_RDS
msgio.c:234:38: error: dereferencing pointer to incomplete type 'struct cmsgcre '
bcopy(CMSG_DATA(cmsg), cred, sizeof(*cred));
^~~~~
make: *** [: msgio.o] Error 1

2019-09-20 15:07:31
I used #34 and #37
But none of them work.
Error changed to
make[2]: *** [: echo++.o] Error 1
make[2]: Leaving directory '/root/sft/pbscan-master/123/libuinet/bin/echo++'
make[2]: Entering directory '/root/sft/pbscan-master/123/libuinet/bin/passive'
cc -I/root/sft/pbscan-master/123/libuinet/bin/passive/../../lib/libev -I/root/sft/pbscan-master/123/libuinet/bin/passive/../../lib/libhttp_parser -DENABLE_EXTRACT -O -gdwarf-2 -I/root/sft/pbscan-master/123/libuinet/bin/passive/../../lib/libuinet/api_include -c -o passive.o passive.c
cc -I/root/sft/pbscan-master/123/libuinet/bin/passive/../../lib/libev -I/root/sft/pbscan-master/123/libuinet/bin/passive/../../lib/libhttp_parser -DENABLE_EXTRACT -O -gdwarf-2 -I/root/sft/pbscan-master/123/libuinet/bin/passive/../../lib/libuinet/api_include -o passive passive.o -L/root/sft/pbscan-master/123/libuinet/bin/passive/../../lib/libuinet -luinet -L/usr/local//lib/ /root/sft/pbscan-master/123/libuinet/bin/passive/../../lib/libev/.libs/libev.a -lm -lpcap -L/root/sft/pbscan-master/123/libuinet/bin/passive/../../lib/libhttp_parser -lhttp_parser -lz -lpthread -lrt -lcrypto
/usr/bin/ld: cannot find -luinet
collect2: error: ld returned 1 exit status
make[2]: *** [/root/sft/pbscan-master/123/libuinet/bin/passive/../../mk/prog.mk:56: passive] Error 1
make[2]: Leaving directory '/root/sft/pbscan-master/123/libuinet/bin/passive'
make[2]: Entering directory '/root/sft/pbscan-master/123/libuinet/bin/tproxy'
cc -I/root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libev -O -gdwarf-2 -I/root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libuinet/api_include -c -o tproxy.o tproxy.c
cc -I/root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libev -O -gdwarf-2 -I/root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libuinet/api_include -o tproxy tproxy.o -L/root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libuinet -luinet -L/usr/local//lib/ /root/sft/pbscan-master/123/libuinet/bin/tproxy/../../lib/libev/.libs/libev.a -lm -lpcap -lpthread -lrt -lcrypto
/usr/bin/ld: cannot find -luinet
collect2: error: ld returned 1 exit status
make[2]: *** [/root/sft/pbscan-master/123/libuinet/bin/tproxy/../../mk/prog.mk:56: tproxy] Error 1
make[2]: Leaving directory '/root/sft/pbscan-master/123/libuinet/bin/tproxy'
make[1]: *** [Makefile:13: all] Error 2
make[1]: Leaving directory '/root/sft/pbscan-master/123/libuinet/bin'
make: *** [Makefile:10: all] Error 2

I entered to libuinet folder under lib to make and it show
awk -f /root/sft/pbscan-master/123/libuinet/lib/libuinet/../../sys/tools/vnode_if.awk /root/sft/pbscan-master/123/libuinet/lib/libuinet/../../sys/kern/vnode_if.src -p
/root/sft/pbscan-master/123/libuinet/lib/libuinet/../../sys/kern/vnode_if.src(451): Missing end-of-line ; in " IN struct task *task; ".
make: *** [Makefile:391: vnode_if_newproto.h] Error 1

[adrian@build2 /beards/appliance/snapshots/nightlies/20140625_175135/34051ff/10.0-DEVELOPER/nas_source/blockd]$
#0 syncache_lookup (inc=0x7ffffebf5c30, schp=0x7ffffebf5bf8, m=0x0) at /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:546

546 /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c: No such file or directory.
in /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c
[New Thread 802c0c800 (LWP 100264/blockd)]
[New Thread 802c0c400 (LWP 100263/blockd)]
[New Thread 802c0c000 (LWP 100262/blockd)]
[New Thread 802c0bc00 (LWP 100261/blockd)]
[New Thread 802c0b800 (LWP 100260/blockd)]
[New Thread 802c0b400 (LWP 100259/blockd)]
[New Thread 802c0b000 (LWP 100258/blockd)]
[New Thread 802c0ac00 (LWP 100257/blockd)]
[New Thread 802c09000 (LWP 100256/nm_rx: netmap0)]
[New Thread 802c08c00 (LWP 100255/nm_tx: netmap0)]
[New Thread 802c08000 (LWP 100222/intr: newborn)]
[New Thread 802c07c00 (LWP 100221/intr: newborn)]
[New Thread 802c07800 (LWP 100220/intr: newborn)]
[New Thread 802c07400 (LWP 100219/thread taskq)]
[New Thread 802c07000 (LWP 100218/kqueue taskq)]
[New Thread 802c06c00 (LWP 100217/clock)]
[New Thread 802c06800 (LWP 100216/intr: newborn)]
New Thread 802c06400 (LWP 100203/blockd) bt
#0 syncache_lookup (inc=0x7ffffebf5c30, schp=0x7ffffebf5bf8, m=0x0) at /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:546
#1 0x0000000000480ce7 in syncache_unreach (inc=0x7ffffebf5c30, th=0x807dce03e, m=0x0) at /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:841
#2 0x000000000047f2eb in tcp_ctlinput (cmd=, sa=, vip=0x807dce02a) at /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_subr.c:1455
#3 0x000000000046e78f in icmp_input (m=, off=20) at /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/ip_icmp.c:492
#4 0x000000000046f4de in ip_input (m=0x8007e3500) at /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/ip_input.c:774
#5 0x0000000000450c2c in netisr_dispatch_src (proto=, source=, m=0x8007e3500) at /usr/nas_source/libuinet/lib/libuinet/../../sys/net/netisr.c:1013
#6 0x000000000044f13e in ether_nh_input (m=) at /usr/nas_source/libuinet/lib/libuinet/../../sys/net/if_ethersubr.c:951
#7 0x0000000000450c2c in netisr_dispatch_src (proto=, source=, m=0x8007e3500) at /usr/nas_source/libuinet/lib/libuinet/../../sys/net/netisr.c:1013
#8 0x000000000041aab1 in if_netmap_receive (arg=0x802efc6c0) at uinet_if_netmap.c:710
#9 0x000000000040e1fb in pthread_start_routine (arg=0x806832420) at uinet_host_interface.c:425
#10 0x000000080228f4a5 in pthread_create () from /lib/libthr.so.3
#11 0x0000000000000000 in ?? ()

Current language: auto; currently minimal
(gdb) frame 0
#0 syncache_lookup (inc=0x7ffffebf5c30, schp=0x7ffffebf5bf8, m=0x0) at /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:546

546 in /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c
(gdb) frame 1
#1 0x0000000000480ce7 in syncache_unreach (inc=0x7ffffebf5c30, th=0x807dce03e, m=0x0) at /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:841

841 in /usr/nas_source/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c
(gdb) print m
$1 = (struct mbuf *) 0x0
(gdb)

Hi,

Can an application which has multiple process make use of "libuinet" (the user space TCP stack) ?

For example, I am thinking if I can integrate Nginx (which works as revere proxy) with libuinet. As you might know, in Nginx we have master process which creates several worker process. The master process creates the listen socket (to receive the requests from the clients) and all the worker listens to that socket and the synchronization is achieved through the mutex. Once a worker process accepts a connection from the client, it talks to the backend webs servers directly. So, I am wondering if the the
libuinet can work with a multiprocess application or it only works with multi threaded applications?

Any pointers how it can work with multi process applications?

Thanks...Peter

Are there any instructions on how to compile this project successfully on Linux distribution (e.g. Ubuntu)?
It seems like there are failures around vnet and data types.

uinet_internal.h:33:22: fatal error: net/vnet.h: No such file or directory
uinet_api.h:119:61: error: unknown type name ‘u_int’

Please help out.

Compiling under Fedora 21, gets the following error and a few more like it:

g++ -I/home/ray/src/libuinet/bin/echo++/../../lib/libev -O -gdwarf-2  -I/home/ray/src/libuinet/bin/echo++/../../lib/libuinet/api_include   -c -o echo++.o echo++.cc
echo++.cc: In constructor ‘EchoServer::EchoServer(ev::dynamic_loop&, server_config*)’:
echo++.cc:232:72: error: invalid conversion from ‘int’ to ‘uinet_instance_t {aka uinet_instance*}’ [-fpermissive
error = uinet_socreate(UINET_PF_INET, &listener_, UINET_SOCK_STREAM, 0);

Currently ARP and ICMP messages, in nearly all cases, are ignored by libuinet instances. This is a holdover from the original libuinet applications in which ARP and ICMP processing were mostly of no interest. It's time to provide libuinet applications some controls for configuring ARP and ICMP handling behavior, especially now that users are interested in running libuinet in traditional mode (non-passive, non-promiscuous sockets) where regular ARP behavior and ping responses are desirable.

This issue is commonly encountered when trying to run bin/echo with a non-promiscuous listen. For example,

./echo -i <some_interface> -l 192.0.2.1 -p 1234

The above instance won't respond to pings, and to connect to it, one has to manually make an entry for 192.0.2.1 in the arp table of the client machine.

When doing VLAN interception, I've noticed that the noprotocol ifnet counter is increasing. What's going on?

[root@darkwatch2] /data/libuinet/sys/net# printf "stats_all\nquit\n" | talkblockd.sh | grep netmap
net.libuinet.netif_stats.netmap0.ifi_ipackets: 247614
net.libuinet.netif_stats.netmap0.ifi_ierrors: 0
net.libuinet.netif_stats.netmap0.ifi_opackets: 0
net.libuinet.netif_stats.netmap0.ifi_oerrors: 0
net.libuinet.netif_stats.netmap0.ifi_collisions: 0
net.libuinet.netif_stats.netmap0.ifi_ibytes: 144801528
net.libuinet.netif_stats.netmap0.ifi_obytes: 0
net.libuinet.netif_stats.netmap0.ifi_imcasts: 5748
net.libuinet.netif_stats.netmap0.ifi_omcasts: 0
net.libuinet.netif_stats.netmap0.ifi_iqdrops: 0
net.libuinet.netif_stats.netmap0.ifi_noproto: 247612
net.libuinet.netif_stats.netmap0.ifi_hwassist: 0
net.libuinet.netif_stats.netmap0.ifi_epoch: 1
net.libuinet.netif_stats.netmap0.ifi_icopies: 247614
net.libuinet.netif_stats.netmap0.ifi_izcopies: 0
net.libuinet.netif_stats.netmap0.ifi_ocopies: 0
net.libuinet.netif_stats.netmap0.ifi_ozcopies: 0
net.libuinet.netif_stats.netmap1.ifi_ipackets: 247614
net.libuinet.netif_stats.netmap1.ifi_ierrors: 0
net.libuinet.netif_stats.netmap1.ifi_opackets: 0
net.libuinet.netif_stats.netmap1.ifi_oerrors: 0
net.libuinet.netif_stats.netmap1.ifi_collisions: 0
net.libuinet.netif_stats.netmap1.ifi_ibytes: 144801528
net.libuinet.netif_stats.netmap1.ifi_obytes: 0
net.libuinet.netif_stats.netmap1.ifi_imcasts: 5748
net.libuinet.netif_stats.netmap1.ifi_omcasts: 0
net.libuinet.netif_stats.netmap1.ifi_iqdrops: 0
net.libuinet.netif_stats.netmap1.ifi_noproto: 247612
net.libuinet.netif_stats.netmap1.ifi_hwassist: 0
net.libuinet.netif_stats.netmap1.ifi_epoch: 1
net.libuinet.netif_stats.netmap1.ifi_icopies: 247614
net.libuinet.netif_stats.netmap1.ifi_izcopies: 0
net.libuinet.netif_stats.netmap1.ifi_ocopies: 0
net.libuinet.netif_stats.netmap1.ifi_ozcopies: 0
[root@darkwatch2] /data/libuinet/sys/net#

somebody know the problem?
My netmap doesn't install correctly?
command:
under /libuinet/bin/echo,execute ./echo -i eth2 -l 192.168.56.105 -p 8888
result:
uinet starting:cpus=1,nmbclusters=131072
cllwheelsize=524288
cllwheelsize=524288
UINET multiprocessor subsystem configured with 1 CPUs
WARNING:VIMAGE(virtualized network stack) is a highly experimental feature.
Timecounters tick every 10.000 msec
configstr is eth2
Segmentation fault
dmesg:
netmap_ioctl API mismatch for eth2 got 3 need 11
netmap_interp_ringid deprecated API,old ringid 0x5000->ringid 0 reg 4
netmap_do_regif lut ffffc900003a1000 bufs 37940 size 2048
generic_netmap_dtor Restored native NA(null)

Is there any successful case?Any suggestion is great.

I wonder how much work it will be to add support for gif and ipsec tunneling. I may be able to work on it and contribute the code if I can get some hints on how this should be done within the libuinet/netmap architectural framework.

After running for a while, a clean shutdown ends with a segfault:

Core was generated by `blockd'.
Program terminated with signal 11, Segmentation fault.
#0 uma_zfree_arg (zone=0x802e58d80, item=0x802505000, udata=0x0) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/vm/uma_core.c:2788

2788 bucket = cache->uc_freebucket;
(gdb) bt
#0 uma_zfree_arg (zone=0x802e58d80, item=0x802505000, udata=0x0) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/vm/uma_core.c:2788
#1 0x000000000049485c in zone_free_item (zone=0x802e57000, item=0x8024f5b00, udata=0x0, skip=, flags=0)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/vm/uma_core.c:2932

#2 0x0000000000493342 in bucket_drain (zone=0x802e57000, bucket=0x80278f418) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/vm/uma_core.c:660
#3 local_cache_drain (zone=0x802e57000, cache=) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/vm/uma_core.c:670
#4 uma_tls_destructor (arg=0x81c00f000) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/vm/uma_core.c:1653
#5 0x000000080229dd25 in ?? () from /lib/libthr.so.3
#6 0x00000008022a533e in ?? () from /lib/libthr.so.3
#7 0x00000008022a5289 in ?? () from /lib/libthr.so.3
#8 0x00000008022a50fb in pthread_exit () from /lib/libthr.so.3
#9 0x000000080229a4fd in ?? () from /lib/libthr.so.3
#10 0x0000000000000000 in ?? ()

(gdb) frame 0
#0 uma_zfree_arg (zone=0x802e58d80, item=0x802505000, udata=0x0) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/vm/uma_core.c:2788

2788 bucket = cache->uc_freebucket;
(gdb) print cache
$1 =
(gdb) print bucket
$2 =
(gdb) print zone
$3 = (uma_zone_t) 0x802e58d80
(gdb) print *zone
$4 = {uz_name = 0x4b0ff5 "mbuf_cluster", uz_lock = 0x802e598d0, uz_link = {le_next = 0x0, le_prev = 0x802e59908}, uz_full_bucket = {lh_first = 0x0}, uz_free_bucket = {lh_first = 0x0}, uz_kegs = {
lh_first = 0x802e58db8}, uz_klink = {kl_link = {le_next = 0x0, le_prev = 0x802e58db0}, kl_keg = 0x802e598c0}, uz_slab = 0x4963b0 <zone_fetch_slab>, uz_ctor = 0x432540 <mb_ctor_clust>,
uz_dtor = 0x432620 <mb_dtor_clust>, uz_init = 0x0, uz_fini = 0x0, uz_flags = 9224, uz_size = 2048, uz_allocs = 245, uz_frees = 0, uz_fails = 0, uz_sleeps = 0, uz_fills = 0, uz_count = 24,
uz_cacheidx = 2}
(gdb) print uma_tls_key
$5 = 3
(gdb) print uhi_tls_get(uma_tls_key)
You can't do that without a process to debug.
(gdb) frame 4
#4 uma_tls_destructor (arg=0x81c00f000) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/vm/uma_core.c:1653

1653 local_cache_drain(tls->ut_caches[i].uc_zone, &tls->ut_caches[i]);
(gdb)

I'll recompile with -O0 and retry but I bet it's NULL.

Can the libuinet provide an API for "sendfile". The FreeBSD used by libuinet has the support for sendfile, so it will be good to help sendfile implemented in libuinet.

http://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2

I would like to use Vegas congestion control in libuinet. I have added cc_vegas.c and h_ertt.c to the libuinet Makefile and managed to compile it. However, the vegas module fails to load and the e_rtt khelp module reports some error as well.

uinet starting: cpus=1, nmbclusters=131072
callwheelsize=524288
callwheelsize=524288
link_elf_lookup_symbol: missing symbol hash table
link_elf_lookup_symbol: missing symbol hash table
UINET multiprocessor subsystem configured with 1 CPUs
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
Timecounters tick every 10.000 msec
vegas_mod_init: h_ertt module not found
module_register_init: MOD_LOAD (vegas, 0x485581, 0x6c78a0) error 2
khelp_init: Failed to add hooks for helper "ertt" (0x6c7ac0) to vnet 0xb85440.

This is my first time working with the FreeBSD kernel code but it looks like the modules are being loaded in the wrong order. The VNET should be initialized first, e_rtt loaded followed by vegas.

Any pointers to fix the issue or at least help me understand what would need to be fixed?

Hi!

I have an odd problem in local testing - it seems hardware TCP checksums aren't making it up to the TCP stack in libuinet and it's doing software checksumming. Is this known?

[root@foo] ~# dtrace -n 'profile-1997 /arg1/ { @[execname, ustack()] = count(); } END { trunc(@, 3); }'
dtrace: description 'profile-1997 ' matched 2 probes
^C
CPU ID FUNCTION:NAME
15 2 :END

blockd
libc.so.7memcpy+0x20 blockdnetisr_dispatch_src+0x6c
blockdif_netmap_receive+0x332 blockdpthread_start_routine+0xdb
libthr.so.30x8022954a5 166 blockd blockdether_nh_input+0x85
blockdnetisr_dispatch_src+0x6c blockdif_netmap_receive+0x332
blockdpthread_start_routine+0xdb libthr.so.30x8022954a5
349
blockd
blockdin_cksumdata+0xdf blockdin_cksum_skip+0x69
blockdtcp_input+0xc1 blockdip_input+0x8e
blockdnetisr_dispatch_src+0x6c blockdether_nh_input+0x61c
blockdnetisr_dispatch_src+0x6c blockdif_netmap_receive+0x332
blockdpthread_start_routine+0xdb libthr.so.30x8022954a5
835

Hi,

Is it possible to get the list of Supported/Unsupported features by libuinet?

Thanks,
-Hemal

connscale.c:775: error: 'UINET_SYNF_REJECT' undeclared (first use in this function)

UINET_SYNF_REJECT isn't defined anywhere

I encountered this undefined reference to `__start_set_vnet' when I compiling libuinet on ubuntu. I am wondering where this global variable __start_set_vnet is defined.

I get some errors on compiling the library :

error 1
----------------------------------------------------------------------------->
uinet_if_pcap_host.c:377:1: error: static declaration of ‘pcap_dump_open_append’ follows non-static declaration
pcap_dump_open_append(pcap_t *p, const char *fname)
^~~~~~~~~~~~~~~~~~~~~
In file included from uinet_if_pcap_host.c:39:0:
/usr/include/pcap/pcap.h:423:25: note: previous declaration of ‘pcap_dump_open_append’ was here
PCAP_API pcap_dumper_t *pcap_dump_open_append(pcap_t *, const char *);
^~~~~~~~~~~~~~~~~~~~~
Makefile:432: recipe for target 'uinet_if_pcap_host.o' failed

<---------------------------------------------------------------------------------------------------------------------

error 2
-------------------------->
http_parser.c: In function ‘http_parser_parse_url’:
http_parser.c:2168:18: error: this statement may fall through [-Werror=implicit-fallthrough=]
found_at = 1;
~~~~~~~~~^~~
http_parser.c:2171:7: note: here
case s_req_server:
^~~~
cc1: all warnings being treated as errors
Makefile:79: recipe for target 'libhttp_parser.o' failed
make[2]: *** [libhttp_parser.o] Error 1
<---------------------------------------
error 3
-------------------------->
/opt/scanner/libuinet/bin/echo++/../../lib/libev/ev++.h:381:46: warning: dynamic exception specifications are deprecated in C++11 [-Wdeprecated]
default_loop (unsigned int flags = AUTO) throw (bad_loop)
^~~~~
echo++.cc: In function ‘int main(int, char**)’:
echo++.cc:453:27: error: invalid conversion from ‘int’ to ‘uinet_global_cfg*’ [-fpermissive]
uinet_init(1, 1281024, 0);
^
echo++.cc:453:19: error: invalid conversion from ‘int’ to ‘uinet_instance_cfg’ [-fpermissive]
uinet_init(1, 1281024, 0);
~~~^~~~~
echo++.cc:453:27: error: too many arguments to function ‘int uinet_init(uinet_global_cfg, uinet_instance_cfg*)’
uinet_init(1, 128*1024, 0);
^
In file included from echo++.cc:36:0:
/opt/scanner/libuinet/bin/echo++/../../lib/libuinet/api_include/uinet_api.h:51:7: note: declared here
int uinet_init(struct uinet_global_cfg cfg, struct uinet_instance_cfg inst_cfg);
^~~~~~~~~~
echo++.cc:457:124: error: cannot convert ‘uinet_iftype_t’ to ‘uinet_if_cfg’ for argument ‘2’ to ‘int uinet_ifcreate(uinet_instance_t, uinet_if_cfg, uinet_if**)’
t(), UINET_IFTYPE_NETMAP, interfaces[i].ifname, interfaces[i].ifname, 0, NULL);
^
: recipe for target 'echo++.o' failed

I am trying to make some changes to congestion control in libuinet. I need to track the current time (or up time). I have tried using the nanotime/microtime and nanouptime/microuptime functions in sys/time.h but the timespec/timeval struct returned always has tv_sec=1. It does not change.

A simple test to replicate this issue would be to add the following lines at the top of the cc_ack_received function in sys/netinet/tcp_input.c:

struct timespec ts;
getnanotime(&ts);
printf("Time= %ld\n", ts.tv_sec);

I tried to trace the function calls in the sys/ folder as well as the uinet_kern_*.c files but I can't figure out why the tv_sec value is always equal to 1.

Trying to build libuinet under a fairly recent FreeBSD current (11.0-CURRENT FreeBSD 11.0-CURRENT #23 r269088) fails with the following:

cc -c -O2 -pipe -fno-strict-aliasing  -std=c99 -ggdb -O0 -Wall -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wno-pointer-sign -Wmissing-include-dirs -fdiagnostics-show-option -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-incompatible-library-redeclaration -Wno-builtin-requires-header  -DUINET=1   -fstack-protector -I. -I/usr/home/wxs/dev/libuinet/lib/libuinet/api_include  -Werror  uinet_host_interface.c
In file included from uinet_host_interface.c:82:
In file included from /usr/home/wxs/dev/libuinet/lib/libuinet/api_include/uinet_api.h:36:
/usr/home/wxs/dev/libuinet/lib/libuinet/api_include/uinet_api_types.h:449:11: error: 
      expected ';' at end of declaration list
        uint64_t        ifi_epoch;              /* uptime at attach or s...
                        ^
/usr/include/net/if.h:107:30: note: expanded from macro 'ifi_epoch'
#define ifi_epoch       __ifi_epoch.tt
                                   ^
1 error generated.
gmake: *** [uinet_host_interface.o] Error 1
wxs@ack libuinet % 

After the passive-to-active merge, there's a problem with the TCP code now sending TCP segments during session establishment.

It looks like the INP_PASSIVE flag isn't set early enough on in the connection lifecycle for tcp_output() to not output things.

(gdb) break if_netmap_start
Breakpoint 4 at 0x422fca: file uinet_if_netmap.c, line 524.
(gdb) cont
Continuing.
[New Thread 802c09c00 (LWP 101148 nm_rx: netmap0)]
[Switching to Thread 802c09c00 (LWP 101148 nm_rx: netmap0)]

Breakpoint 4, if_netmap_start (ifp=0x802ef0300) at uinet_if_netmap.c:524
524 struct if_netmap_softc *sc = ifp->if_softc;
(gdb) bt
#0 if_netmap_start (ifp=0x802ef0300) at uinet_if_netmap.c:524
#1 0x0000000000455602 in if_start (ifp=0x802ef0300, ifp=0x802ef0300) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/if.c:3401
#2 if_transmit (ifp=0x802ef0300, m=) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/if.c:3413
#3 0x0000000000456d57 in ether_output_frame (ifp=, m=0x800724300, m=0x800724300, m=0x800724300, m=0x800724300, m=0x800724300)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/if_ethersubr.c:513
#4 ether_output (ifp=0x802ef0300, m=0x800724300, dst=, ro=) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/if_ethersubr.c:484
#5 0x000000000047bcab in ip_output (m=0x800724300, opt=, flags=, imo=, inp=)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/ip_output.c:699
#6 0x000000000048538d in tcp_output (tp=0x800735418) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_output.c:1254
#7 0x0000000000482016 in tcp_do_segment (m=, th=, so=, tp=, drop_hdrlen=, tlen=0, iptos=,

ti_locked=<error reading variable: Cannot access memory at address 0x1>, no_unlock=1) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_input.c:3088
#8 0x000000000048adbd in syncache_passive_client_socket (lso=0x800728c20, m=, sc=, to=)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:1276
#9 syncache_socket (sc=, lso=, sc=, lso=, m=, to=)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:1311
#10 syncache_expand (inc=, to=, th=, lsop=0x7ffffe7f3d40, m=0x800724500)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:1737
#11 0x000000000047fdcb in tcp_input (m=, off0=<error reading variable: Cannot access memory at address 0x14>)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_input.c:1151
#12 0x00000000004787ae in ip_input (m=0x800724500) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/ip_input.c:774
#13 0x0000000000459c7c in netisr_dispatch_src (proto=, source=, m=0x800724500) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/netisr.c:1013
#14 0x000000000045819e in ether_input_internal (ifp=, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500,

m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500,
m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500,
m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500,
m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500,
m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500,
m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500, m=0x800724500,
m=0x800724500, m=0x800724500, m=0x800724500) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/if_ethersubr.c:986
#15 ether_nh_input (m=) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/if_ethersubr.c:998
#16 0x0000000000459c7c in netisr_dispatch_src (proto=, source=, m=0x800724500) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/netisr.c:1013
#17 0x0000000000423882 in if_netmap_receive (arg=0x806832180) at uinet_if_netmap.c:839
#18 0x00000000004968eb in pthread_start_routine (arg=0x8068314c0) at uinet_host_interface.c:537
#19 0x00000008022984f5 in ?? () from /lib/libthr.so.3
#20 0x0000000000000000 in ?? ()

(gdb)

and then;

It seems that:

the listen sockets have SO_PASSIVE set
when sonewconn_passive_client() is called, it clears SO_PASSIVE on the new socket, so it doesn't inherit it
but then when it attaches via tcp_attach(), it ends up at in_pcballoc() -> in_passive_inpcb_init(), which finds SO_PASSIVE isn't set, and thus doesn't mark the inpcb as passive and thus we see transmits.
Here's what's going down:

(gdb) bt
#0 in_passive_inpcb_init (inp=0x80072f640, warning: Unmapped DWARF Register #-2 encountered.

flags=<error reading variable: Unable to access DWARF register number 0xfffffffe>)
at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/in_passive.c:60
#1 0x0000000000471350 in in_pcballoc (so=0x800728308, pcbinfo=0x7042b8 ) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/in_pcb.c:352
#2 0x000000000048e64a in tcp_attach (so=0x800728308) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_usrreq.c:1648
#3 tcp_usr_attach (so=0x800728308, proto=, td=) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_usrreq.c:140
#4 0x000000000044affb in sonewconn_passive_client (head=0x800728c20, connstatus=4) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/kern/uipc_socket.c:590
#5 0x000000000048a34b in syncache_passive_client_socket (lso=0x800728c20, m=, sc=, to=)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:1043
#6 syncache_socket (sc=, lso=, sc=, lso=, m=, to=)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:1311
#7 syncache_expand (inc=, to=0x7ffffe7f3d18, th=, lsop=0x7ffffe7f3d40, m=0x80071f800)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:1737
#8 0x000000000047fdcb in tcp_input (m=, off0=<error reading variable: Cannot access memory at address 0x14>)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_input.c:1151
#9 0x00000000004787ae in ip_input (m=0x80071f800) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/ip_input.c:774
#10 0x0000000000459c7c in netisr_dispatch_src (proto=, source=, m=0x80071f800) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/netisr.c:1013
#11 0x000000000045819e in ether_input_internal (ifp=, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800,

m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800,
m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800,
m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800,
m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800,
m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800,
m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800, m=0x80071f800,
m=0x80071f800, m=0x80071f800, m=0x80071f800) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/if_ethersubr.c:986
#12 ether_nh_input (m=) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/if_ethersubr.c:998
#13 0x0000000000459c7c in netisr_dispatch_src (proto=, source=, m=0x80071f800) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/net/netisr.c:1013
#14 0x0000000000423882 in if_netmap_receive (arg=0x806832180) at uinet_if_netmap.c:839
#15 0x00000000004968eb in pthread_start_routine (arg=0x8068314c0) at uinet_host_interface.c:537
#16 0x00000008022984f5 in ?? () from /lib/libthr.so.3
#17 0x0000000000000000 in ?? ()

(gdb) frame 2
#2 0x000000000048e64a in tcp_attach (so=0x800728308) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_usrreq.c:1648

1648 error = in_pcballoc(so, &V_tcbinfo);
(gdb) frame 1
#1 0x0000000000471350 in in_pcballoc (so=0x800728308, pcbinfo=0x7042b8 ) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/in_pcb.c:352

352 error = in_passive_inpcb_init(inp, M_NOWAIT);
(gdb) frame 3
#3 tcp_usr_attach (so=0x800728308, proto=, td=) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_usrreq.c:140

140 error = tcp_attach(so);
(gdb) frame 4
#4 0x000000000044affb in sonewconn_passive_client (head=0x800728c20, connstatus=4) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/kern/uipc_socket.c:590

590 (*so->so_proto->pr_usrreqs->pru_attach)(so, 0, NULL)) {
(gdb) frame 5
#5 0x000000000048a34b in syncache_passive_client_socket (lso=0x800728c20, m=, sc=, to=)

at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_syncache.c:1043
1043 so = sonewconn_passive_client(lso, SS_ISCONNECTING);
(gdb) frame 0
#0 in_passive_inpcb_init (inp=0x80072f640, warning: Unmapped DWARF Register #-2 encountered.

flags=<error reading variable: Unable to access DWARF register number 0xfffffffe>)
at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/in_passive.c:60
60 if (inp->inp_socket->so_options & SO_PASSIVE)
(gdb) print inp->inp_socket->so_options
$1 = 66048

Trying to build libuinet under a fairly recent FreeBSD current (11.0-CURRENT FreeBSD 11.0-CURRENT #23 r269088) fails with the following:

cc -c -O2 -pipe -fno-strict-aliasing  -std=c99 -ggdb -O0 -Wall -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wno-pointer-sign -Wmissing-include-dirs -fdiagnostics-show-option -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-incompatible-library-redeclaration -Wno-builtin-requires-header  -DUINET=1   -fstack-protector -I. -I/usr/home/wxs/dev/libuinet/lib/libuinet/api_include  -Werror  uinet_host_interface.c
In file included from uinet_host_interface.c:82:
In file included from /usr/home/wxs/dev/libuinet/lib/libuinet/api_include/uinet_api.h:36:
/usr/home/wxs/dev/libuinet/lib/libuinet/api_include/uinet_api_types.h:449:11: error: 
      expected ';' at end of declaration list
        uint64_t        ifi_epoch;              /* uptime at attach or s...
                        ^
/usr/include/net/if.h:107:30: note: expanded from macro 'ifi_epoch'
#define ifi_epoch       __ifi_epoch.tt
                                   ^
1 error generated.
gmake: *** [uinet_host_interface.o] Error 1
wxs@ack libuinet % 

Can the libuinet expose an API for accept4. I think accept4 is supported in the FreeBSD version which libuinet uses. If an API could be provided, it would be great help for the application to use that.

http://www.freebsd.org/cgi/man.cgi?query=accept&sektion=2

uinet shutting down from signal handler (signal 1)
Shutting down all uinet instances...
netmap3 (netmap0:3): Stopping rx thread
netmap3 (netmap0:3): Stopping tx thread
netmap3 (netmap0:3): Interface threads stopped
Freed UMA keg was not empty (18 items). Lost 2 pages of memory.
Freed UMA keg was not empty (168 items). Lost 1 pages of memory.
Freed UMA keg was not empty (466 items). Lost 129 pages of memory.
Freed UMA keg was not empty (529 items). Lost 408 pages of memory.
hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
netmap2 (netmap0:2): Stopping rx thread
netmap2 (netmap0:2): Stopping tx thread

load: 3.28 cmd: blockd 23808 [suspended] 2857.94r 6699.35u 2496.73s 33% 318072k
load: 3.28 cmd: blockd 23808 [suspended] 2858.55r 6699.35u 2496.88s 30% 335036k
load: 3.28 cmd: blockd 23808 [suspended] 2858.70r 6699.35u 2496.92s 30% 339512k
load: 3.28 cmd: blockd 23808 [suspended] 2858.98r 6699.35u 2496.98s 30% 347204k
load: 3.28 cmd: blockd 23808 [suspended] 2859.16r 6699.35u 2497.04s 27% 352280k
load: 3.28 cmd: blockd 23808 [suspended] 2859.45r 6699.35u 2497.09s 27% 357268k
load: 3.28 cmd: blockd 23808 [suspended] 2859.61r 6699.35u 2497.12s 27% 357812k
Segmentation fault (core dumped)
root@test3:/home/adrian/git/gitlab/ahc/blockd #
root@test3:/home/adrian/git/gitlab/ahc/blockd #
root@test3:/home/adrian/git/gitlab/ahc/blockd # gdb78 ./blockd
GNU gdb (GDB) 7.8 [GDB v7.8 for FreeBSD]
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd10.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./blockd...done.
(gdb) quit
root@test3:/home/adrian/git/gitlab/ahc/blockd # gdb78 ./blockd ./blockd.core
GNU gdb (GDB) 7.8 [GDB v7.8 for FreeBSD]
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd10.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./blockd...done.
[New process 100773]
[New process 100802]
[New process 100801]
[New process 100800]
[New process 100799]
[New process 100798]
[New process 100797]
[New process 100796]
[New process 100795]
[New process 100794]
[New process 100793]
[New process 100792]
[New process 100791]
[New process 100788]
[New process 100787]
[New process 100786]
[New process 100785]
[New process 100784]
[New process 100783]
[New process 100782]
[New process 100781]
[New process 100780]
[New process 100779]
[New process 100778]
[New process 100777]
[New process 100776]
[New process 100775]
[New process 100774]
[New process 100233]
[New Thread 802c10800 (LWP 100802)]
[New Thread 802c10400 (LWP 100801)]
[New Thread 802c10000 (LWP 100800)]
[New Thread 802c0fc00 (LWP 100799)]
[New Thread 802c0f800 (LWP 100798)]
[New Thread 802c0f400 (LWP 100797)]
[New Thread 802c0f000 (LWP 100796)]
[New Thread 802c0ec00 (LWP 100795)]
[New Thread 802c0e800 (LWP 100794)]
[New Thread 802c0e400 (LWP 100793)]
[New Thread 802c0e000 (LWP 100792)]
[New Thread 802c0dc00 (LWP 100791)]
[New Thread 802c0d000 (LWP 100788)]
[New Thread 802c0cc00 (LWP 100787)]
[New Thread 802c0c800 (LWP 100786)]
[New Thread 802c0c400 (LWP 100785)]
[New Thread 802c0c000 (LWP 100784)]
[New Thread 802c0bc00 (LWP 100783)]
[New Thread 802c0b800 (LWP 100782)]
[New Thread 802c09400 (LWP 100781)]
[New Thread 802c09000 (LWP 100780)]
[New Thread 802c08400 (LWP 100779)]
[New Thread 802c08000 (LWP 100778)]
[New Thread 802c07c00 (LWP 100777)]
[New Thread 802c07800 (LWP 100776)]
[New Thread 802c07400 (LWP 100775)]
[New Thread 802c07000 (LWP 100774)]
[New Thread 802c06c00 (LWP 100773)]
[New Thread 802c06400 (LWP 100233)]
Core was generated by `blockd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3

(gdb) bt
#0 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
#1 0x000000000049ab99 in tcp_timer_keep (xtp=0x81d9a5830) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_timer.c:294
#2 0x0000000000422ac0 in softclock (arg=0x7184a0 <cc_cpu>) at uinet_kern_timeout.c:428
#3 0x000000000041ed91 in uinet_hardclock () at uinet_kern_clock.c:56
#4 0x0000000000423446 in timer_intr (arg=) at uinet_kern_timeout.c:844
#5 0x00000000004a5ee2 in pthread_start_routine (arg=0x802c3d3e0) at uinet_host_interface.c:580
#6 0x00000008013a04f5 in ?? () from /lib/libthr.so.3
#7 0x0000000000000000 in ?? ()

Backtrace stopped: Cannot access memory at address 0x7fffffbfe000
(gdb) info threads
Id Target Id Frame
58 Thread 802c06400 (LWP 100233) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3

• 57 Thread 802c06c00 (LWP 100773) 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
  56 Thread 802c07000 (LWP 100774) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  55 Thread 802c07400 (LWP 100775) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  54 Thread 802c07800 (LWP 100776) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  53 Thread 802c07c00 (LWP 100777) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  52 Thread 802c08000 (LWP 100778) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  51 Thread 802c08400 (LWP 100779) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  50 Thread 802c09000 (LWP 100780) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  49 Thread 802c09400 (LWP 100781) 0x000000080226a238 in read () from /lib/libc.so.7
  48 Thread 802c0b800 (LWP 100782) 0x00000008021f2d98 in accept () from /lib/libc.so.7
  47 Thread 802c0bc00 (LWP 100783) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  46 Thread 802c0c000 (LWP 100784) 0x000000080226a1da in ioctl () from /lib/libc.so.7
  45 Thread 802c0c400 (LWP 100785) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  44 Thread 802c0c800 (LWP 100786) 0x00000008021fa97a in poll () from /lib/libc.so.7
  43 Thread 802c0cc00 (LWP 100787) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  42 Thread 802c0d000 (LWP 100788) 0x00000008021d842e in free () from /lib/libc.so.7
  41 Thread 802c0dc00 (LWP 100791) 0x00000008013ab8cc in ?? () from /lib/libthr.so.3
  40 Thread 802c0e000 (LWP 100792) 0x0000000802192048 in _sigwait () from /lib/libc.so.7
  39 Thread 802c0e400 (LWP 100793) 0x00000008021f2d98 in accept () from /lib/libc.so.7
  38 Thread 802c0e800 (LWP 100794) 0x000000080226a238 in read () from /lib/libc.so.7
  37 Thread 802c0ec00 (LWP 100795) 0x0000000802247efa in nanosleep () from /lib/libc.so.7
  36 Thread 802c0f000 (LWP 100796) 0x000000080226a23a in read () from /lib/libc.so.7
  35 Thread 802c0f400 (LWP 100797) 0x0000000802247efa in nanosleep () from /lib/libc.so.7
  34 Thread 802c0f800 (LWP 100798) 0x000000080224968a in kevent () from /lib/libc.so.7
  33 Thread 802c0fc00 (LWP 100799) 0x0000000802247efa in nanosleep () from /lib/libc.so.7
  32 Thread 802c10000 (LWP 100800) 0x000000080224968a in kevent () from /lib/libc.so.7
  31 Thread 802c10400 (LWP 100801) 0x0000000802247efa in nanosleep () from /lib/libc.so.7
  30 Thread 802c10800 (LWP 100802) 0x000000080224968a in kevent () from /lib/libc.so.7
• 1 Thread 802c06c00 (LWP 100773) 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
  (gdb) frame 0
  #0 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
  (gdb) frame 1
  #1 0x000000000049ab99 in tcp_timer_keep (xtp=0x81d9a5830) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_timer.c:294
  294 INP_INFO_WLOCK(&V_tcbinfo);
  (gdb) bt
  #0 0x00000008013a5f9a in pthread_mutex_lock () from /lib/libthr.so.3
  #1 0x000000000049ab99 in tcp_timer_keep (xtp=0x81d9a5830) at /home/adrian/git/github/erikarn/libuinet/lib/libuinet/../../sys/netinet/tcp_timer.c:294
  #2 0x0000000000422ac0 in softclock (arg=0x7184a0 <cc_cpu>) at uinet_kern_timeout.c:428
  #3 0x000000000041ed91 in uinet_hardclock () at uinet_kern_clock.c:56
  #4 0x0000000000423446 in timer_intr (arg=) at uinet_kern_timeout.c:844
  #5 0x00000000004a5ee2 in pthread_start_routine (arg=0x802c3d3e0) at uinet_host_interface.c:580
  #6 0x00000008013a04f5 in ?? () from /lib/libthr.so.3
  #7 0x0000000000000000 in ?? ()
  Backtrace stopped: Cannot access memory at address 0x7fffffbfe000
  (gdb)

.. it looks like the timer thread hasn't been stopped but the VNET state has been freed.