tempesta-tech / tempesta Goto Github PK

View Code? Open in Web Editor NEW

603.0 49.0 104.0 20.21 MB

All-in-one solution for high performance web content delivery and advanced protection against DDoS and web attacks

Home Page: https://tempesta-tech.com/

License: GNU General Public License v2.0

C++ 0.66% C 95.15% Makefile 0.60% Shell 0.86% Smarty 0.05% Perl 0.17% Assembly 2.52%

web-application-firewall linux-kernel high-performance http-accelerator load-balancer database tls http2 security bots

tempesta's Introduction

Tempesta FW

What it is?

Tempesta FW is an all-in-one open-source solution for high performance web content delivery and advanced protection against DDoS and web attacks. This is a drop-in-replacement for the whole web server frontend infrastructure: an HTTPS load balancer, a web accelerator, a DDoS mitigation system, and a web application firewall (WAF).

Tempesta FW is the first and only hybrid of a Web accelerator and a multi-layer firewall. This unique architecture provides seamless integration with the Linux iptables or nftables.

Tempesta FW services up to 1.8M HTTP requests per second on the cheapest hardware, which is x3 faster than Nginx or HAProxy. Tempesta TLS is about 40-80% faster than Nginx/OpenSSL and provides up to x4 lower latency.

Demo

Watch the Tempesta FW demo in the Security Weekly show - Fast And Secure Web.

How it works?

Tempesta FW is built into Linux TCP/IP stack for better and more stable performance characteristics in comparison with TCP servers on top of common Socket API or even DPDK or other kernel bypass technology.

We do our best to keep the kernel modifications as small as possible. Current patch is just about 3,100 lines.

Current state

We're in alpha state for now. The alpha is available by:

source code
installation script (binary packages)

The master branch is a development (and unstable) branch for contributers and early testers only. Use release-0.7 branch for a stable version.

Installation and Configuration

Please see our Wiki for following topics:

We appreciate support of our technical partner

tempesta's People

Contributors

Stargazers

Watchers

Forkers

tzuryby hisdo shengxinking sunshinehuang zeus911 cloudxtreme pentesting artiya4u travislee89 twdow jyizheng chixsh woylaski nekulin guozi058 w796933 cynron orz-- vankoven huawenyu olivierh59500 csuhawk jijicanyu gsdu8g9 randall2030 devlo umesh160185 javarange fortitudepub bartolootrit keshonok vladtcvs mailanetworks octaplexsys msliusarev nunb inkdou linus5 tigerjibo safe3 craig johnlu0907 bhyvex xiongzx cih-y2k sburn alex1528 duzhanyuan tiancheng-luo leelie clodfisher weliu kenser paulyc estella beike2020 moneytech fastlyos starliiit jkteam007 varsptz yangboyd eshwaranad rope93 xdlhzdh y0d4a feiyunwill icodein shgurbanov roelvdp kolabszar viracall serge-kaimin blockspacer linecode huangyingting fatihusta sinsixx byko3y zlzhangv007 ttaym egida rock59 0xsojalsec const-t mikemuzzy 5l1v3r1 ajunlonglive millken lihongguang ciari samermassoud cvlabsio dmpetroff hmthanh zmkva hau99 shadow-cipher ykargin jsat97

tempesta's Issues

Load balancing

Following sched modules must be implemented:

round-robin
by list of statically specified URL, Host or backed IP addresses (i.e. each backend server must have it's own list of URLs and/or Host values (or IP addresses instead of DNS names) by which are sent to it). This is very similar to location based load balancing of Nginx;
by hash calculated on URI, Host HTTP header or client IP

See Nginx load balancing as an example.

The system must ensure that only one shceduler module can be loaded by a user.

Tempesta must add X-Forwarded-For HTTP header with the client IP address to forwarded to backed server request.

Failovering over backend nodes also must be implemented. See HAProxy

[RFC7234] Cache warnings

Tempesta MUST attach 110 warning header on sending stale responses.

Depends on #513 (cache entry freshness calculation).

sk->sk_destruct() is not being called when connection is closed

Scenario:

start nginx
start Tempesta FW
wget http://localhost - should pass and give you index.html
stop nginx
wget http://locslhost - crashes

That happens because when the connection to nginx is closed, no sk->sk_destruct() is called.

We put tfw_destroy_server() to sk->sk_destruct(), so when the connection is closed, the server is not deleted from the load-balancing scheduler as it should be, and you get a NULL pointer dereference when you try to acess its connection.

Generally, the problem is wider than the fact that tfw_destroy_server() is not invoked.
Original socket destructor (usually inet_release()) is not invoked as well, so we likely have some memory leak here.

A notable thing that when you call sock_release(), the sk->sk_destruct() is invoked as it should be.
We need to understand why sk->sk_destruct() is not invoked when a TCP connection is closed, but is invoked when you do sock_release().

`sk_destruct` is overwritten in tfw_connection_new()

The sk_destruct is set to inet_sock_destruct() by inet_create() and we overwrite it in tfw_connection_new(), so the old one is not invoked when a socket is destroyed. That leads to a memory leak because the inet_sock_destruct() releases some memory.

We should save the old destructor and call it manually from our custom destructor.

Poor hash function calculation

Current implementation of hash function based on crc32 in tempesta_fw/hash.c is poor. Firstly, it's 32bit while we need for Tempesta DB's HTrie full 64bit hash value. Secondly, it breaks user-space MMX code and must call kernel_fpu_begin()/kernel_fpu_end() like crc_pcl does it.

The function must be reworked or replaced by native crc_pcl.

connect to a backend server specified in the tempesta.h by default

At this point the Tempesta FW doesn't connect to any backends when the module is loaded.
Instead, it waits for a value to be written to /proc/sys/net/tempesta/backend.

So we need to change it, the Tempesta FW should connect to a backend specified by DEF_BACKEND_ADDR/DEF_BACKEND_PORT defined in tempesta.h.

crash on routing

It seems egress skb has invalid output hook (initially set to ip_rt_bug).

    [tempesta]   request parsed: len=277 parsed=277 msg_len=277 res=0
    [tempesta]   GFSM return code 0
    [tempesta]   added X-Forwarded-For header: X-Forwarded-For: 172.16.0.1^M

    [sync_socket]   ss_send:89 entail skb=ffff88007b0b5580 data_len=0 len=306
    [sync_socket]   ss_send:99 sk=ffff880039e278c0 is_queue_empty=0 tcp_send_head(sk)=ffff88007b0b5580 sk->sk_state=1
    ------------[ cut here ]------------
    WARNING: at net/ipv4/route.c:1097 ip_rt_bug+0x2f/0x80()
    Modules linked in: tfw_sched_dummy(O) tempesta_fw(O) tempesta_db(O) sync_socket(O) nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables xfs exportfs ppdev i2c_piix4 microcode parport_pc parport serio_raw i2c_core pcspkr dm_mirror dm_region_hash dm_log dm_mod uinput btrfs xor zlib_deflate raid6_pq libcrc32c ata_generic pata_acpi e1000 ata_piix floppy ipv6 autofs4
    CPU: 1 PID: 0 Comm: swapper/1 Tainted: G           O 3.10.10+ #29
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
     0000000000000009 ffff88007fd03678 ffffffff814cbf7f ffff88007fd036b0
     ffffffff8103c091 ffff880038b9ce88 ffff880039e278c0 ffff880078ece94e
     ffff880039e27bb8 0000000000000000 ffff88007fd036c0 ffffffff8103c16a
    Call Trace:
     <IRQ>  [<ffffffff814cbf7f>] dump_stack+0x19/0x1b
     [<ffffffff8103c091>] warn_slowpath_common+0x61/0x80
     [<ffffffff8103c16a>] warn_slowpath_null+0x1a/0x20
     [<ffffffff8143584f>] ip_rt_bug+0x2f/0x80
     [<ffffffff8143f6b5>] ip_local_out+0x25/0x30
     [<ffffffff8143fa0b>] ip_queue_xmit+0x14b/0x3f0
     [<ffffffff8145714d>] tcp_transmit_skb+0x45d/0x8c0
     [<ffffffff81457744>] tcp_write_xmit+0x194/0xb90
     [<ffffffff8145835e>] __tcp_push_pending_frames+0x2e/0xc0
     [<ffffffff81447250>] tcp_push+0x70/0x90
     [<ffffffffa043810d>] ss_send+0x9d/0x140 [sync_socket]
     [<ffffffffa044fdf3>] tfw_connection_send_srv+0xc4/0xcb [tempesta_fw]
     [<ffffffffa04536bf>] tfw_http_req_cache_cb+0x62/0x64 [tempesta_fw]
     [<ffffffffa044be9d>] tfw_cache_req_process+0x3e/0x140 [tempesta_fw]
     [<ffffffffa045365d>] ? tfw_http_adjust_resp+0x18/0x18 [tempesta_fw]
     [<ffffffffa0453982>] tfw_http_req_process+0x2c1/0x330 [tempesta_fw]
     [<ffffffffa0453d67>] tfw_http_msg_process+0x48/0x63 [tempesta_fw]
     [<ffffffffa0450f7b>] tfw_gfsm_dispatch+0x46/0x48 [tempesta_fw]
     [<ffffffffa044ffc3>] tfw_connection_recv+0x8d/0x8f [tempesta_fw]
     [<ffffffffa0438273>] ss_tcp_process_proto_skb+0x63/0x90 [sync_socket]
     [<ffffffffa043841e>] ss_tcp_process_skb+0x17e/0x190 [sync_socket]
     [<ffffffffa0438ca5>] ss_tcp_data_ready+0x115/0x264 [sync_socket]
     [<ffffffff81453dde>] tcp_rcv_established+0x3ce/0x8a0
     [<ffffffff8145e924>] tcp_v4_do_rcv+0x1b4/0x470
     [<ffffffff8145ff5e>] tcp_v4_rcv+0x68e/0x7e0
     [<ffffffff8143a4e0>] ? ip_rcv_finish+0x320/0x320
     [<ffffffff814318c4>] ? nf_hook_slow+0x74/0x130
     [<ffffffff8143a4e0>] ? ip_rcv_finish+0x320/0x320
     [<ffffffff8143a5b0>] ip_local_deliver_finish+0xd0/0x210
     [<ffffffff8143a8c8>] ip_local_deliver+0x88/0x90
     [<ffffffff8143a238>] ip_rcv_finish+0x78/0x320
     [<ffffffff8143ab3d>] ip_rcv+0x26d/0x390
     [<ffffffff81403996>] __netif_receive_skb_core+0x5e6/0x820
     [<ffffffff81460100>] ? tcp4_gro_receive+0x30/0x130
     [<ffffffff81403bed>] __netif_receive_skb+0x1d/0x60
     [<ffffffff81403c5d>] netif_receive_skb+0x2d/0x90
     [<ffffffff81404600>] napi_gro_receive+0x80/0xb0
     [<ffffffffa009c4c3>] e1000_clean_rx_irq+0x2b3/0x580 [e1000]
     [<ffffffff81071e18>] ? sched_clock_cpu+0xa8/0x100
     [<ffffffff8106d500>] ? check_preempt_curr+0x50/0xa0
     [<ffffffffa009e9d5>] e1000_clean+0x255/0x8c0 [e1000]
     [<ffffffff8108ce03>] ? ktime_get+0x43/0xc0
     [<ffffffff8102516d>] ? lapic_next_event+0x1d/0x30
     [<ffffffff81403f99>] net_rx_action+0x139/0x220
     [<ffffffff81044480>] __do_softirq+0x100/0x250
     [<ffffffff81044725>] irq_exit+0x95/0xa0
     [<ffffffff814dacf6>] do_IRQ+0x56/0xc0
     [<ffffffff814d15aa>] common_interrupt+0x6a/0x6a
     <EOI>  [<ffffffff8100a101>] ? default_idle+0x21/0xb0
     [<ffffffff8100a92e>] arch_cpu_idle+0x1e/0x30
     [<ffffffff8108b1fe>] cpu_startup_entry+0xce/0x270
     [<ffffffff81092ba5>] ? clockevents_register_device+0xb5/0x120
     [<ffffffff814beda2>] start_secondary+0x247/0x249
    ---[ end trace c35bbeb7bc462623 ]---

Kernel panic when freeing a Tempesta message

This appears to be a case of SKB's double-free - freeing an SKB that had been freed already.

Loading Tempesta kernel modules...
[    8.219962] [tempesta] Initializing Tempesta FW kernel module...
[    8.220504] [tempesta] init: cfg_if
[    8.220800] [tempesta] init: http
[    8.221066] [tempesta] init: server
[    8.221363] [tempesta] init: client
[    8.221650] [tempesta] init: session
[    8.221944] [tempesta] init: connection
[    8.222268] [tempesta]   register cfg: cache
[    8.222613] [tempesta]   register cfg: sock_backend
[    8.223006] [tempesta]   register cfg: sock_frontend
[    8.238490] [tempesta] Registering new scheduler: dummy
Starting Tempesta...
[    8.240987] [tempesta] got state via sysctl: start
[    8.241387] [tempesta]   reading configuration file...
[    8.241800] [tempesta]   reading file: /home/natsys-lab/tfw-run/tempesta/etc/tempesta_fw.conf
[    8.242864] [tempesta]   file size: 2224 bytes
[    8.243236] [tempesta]   read by offset: 0
[    8.243717] [tempesta]   read by offset: 2224
[    8.244194] [tempesta] starting all modules...
[    8.244559] [tempesta]   parsing configuration and pushing it to modules...
[    8.245134] [tempesta]   spec handle: 'listen'
[    8.245616] [tempesta]   created front-end socket: sk=ffff88003cc2e040
[    8.246197] [tempesta]   use default entry: 'cache off;'
[    8.246758] [tempesta]   spec handle: 'cache'
[    8.247231] [tempesta]   use default entry: 'cache_size 262144;'
[    8.247975] [tempesta]   spec handle: 'cache_size'
[    8.248682] [tempesta]   use default entry: 'cache_dir /opt/tempesta/cache;'
[    8.249596] [tempesta]   spec handle: 'cache_dir'
[    8.250104] [tempesta]   use default entry: 'backend 127.0.0.1:8080;'
[    8.250533] [tempesta]   spec handle: 'backend'
[    8.250833] [tempesta]   starting modules...
[    8.251111] [tempesta]   mod_start(): cache
[    8.251383] [tempesta]   mod_start(): sock_backend
[    8.251695] [tempesta]   Starting thread: tfw_bconnd
[    8.252101] [tempesta]   mod_start(): sock_frontend
[    8.252432] [tempesta]   start listening on socket: sk=ffff88003cc2e040
[    8.252886] [tempesta] modules are started
net.tempesta.state = start
done
done
Starting nginx: nginx.
[   12.253724] [tempesta] Connected to backend: 127.0.0.1:8080
[   25.326849] [tempesta]   New client socket ffff88003d4af2c0 (state=1)
[   25.327669] [tempesta]   Link new msg ffff88003d411040 with connection ffff88003d660590
[   25.328621] [tempesta]   Add skb ffff88003d6b7da8 to message ffff88003d411040
[   25.329299] [tempesta]   Received 109 client data bytes (GET / HTTP/1.1
[   25.329299] User-Agent: Wget/1.13.4 (linux-gnu)
[   25.329299] Accept: */*
[   25.329299] Host: 127.0.0.1
[   25.329299] Connection: Keep-Alive
[   25.329299] 
[   25.329299] ) on socket (conn=ffff88003d660590)
[   25.331721] [tempesta]   enter FSM at state 0
[   25.332140] [tempesta]   parser: Req_0(0:0): c=0x47(G), r=-2
[   25.332666] [tempesta]   parser: Req_Method(1:0): c=0x47(G), r=-2
[   25.333231] [tempesta]   parser: Req_MUSpace(2:0): c=0x2f(/), r=-2
[   25.333804] [tempesta]   parser: Req_UriAbsPath(6:0): c=0x20( ), r=-2
[   25.334404] [tempesta]   parser: Req_HttpVer(7:0): c=0x48(H), r=-2
[   25.334976] [tempesta]   parser: Req_EoL(15:0): c=0xd(.), r=-2
[   25.335526] [tempesta]   parser: Req_EoL(15:0): c=0xa(.), r=-2
[   25.336075] [tempesta]   parser: Req_Hdr(16:0): c=0x55(U), r=-2
[   25.336626] [tempesta]   parser: Req_HdrOther(92:0): c=0x55(U), r=-2
[   25.337213] [tempesta]   store header w/ ptr=ffff88003dbdb0b0 len=35 flags=0 id=3
[   25.337900] [tempesta]   parser: Req_Hdr(16:0): c=0x41(A), r=-2
[   25.338351] [tempesta]   parser: Req_HdrOther(92:0): c=0x41(A), r=-2
[   25.338768] [tempesta]   store header w/ ptr=ffff88003dbdb0d5 len=11 flags=0 id=4
[   25.339256] [tempesta]   parser: Req_Hdr(16:0): c=0x48(H), r=-2
[   25.339644] [tempesta]   parser: RGen_LWS(10000:21): c=0x20( ), r=-2
[   25.340067] [tempesta]   parser: RGen_LWS(10000:21): c=0x31(1), r=-2
[   25.340486] [tempesta]   enter FSM at state 21
[   25.340780] [tempesta]   parser: Req_HdrHostV(21:0): c=0x31(1), r=-2
[   25.341196] [tempesta]   enter FSM at state 1
[   25.341484] [tempesta]   parser: Req_I_H(1:1): c=0x31(1), r=-2
[   25.341867] [tempesta]   parser: Req_I_H(1:1): c=0x32(2), r=-2
[   25.342253] [tempesta]   parser: Req_I_H(1:1): c=0x37(7), r=-2
[   25.342636] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[   25.343020] [tempesta]   parser: Req_I_H(1:1): c=0x30(0), r=-2
[   25.343402] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[   25.343783] [tempesta]   parser: Req_I_H(1:1): c=0x30(0), r=-2
[   25.344176] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[   25.344561] [tempesta]   parser: Req_I_H(1:1): c=0x31(1), r=-2
[   25.344944] [tempesta]   parser: Req_I_H(1:1): c=0xd(.), r=-2
[   25.345322] [tempesta]   parser: Req_I_H_EoL(3:1): c=0xd(.), r=-2
[   25.345722] [tempesta]   parser: Req_I_H_EoL(3:1): c=0xa(.), r=-2
[   25.346124] [tempesta]   parse header __req_parse_host: ret=11 plen=9 id=1
[   25.346573] [tempesta]   store header w/ ptr=ffff88003dbdb0e2 len=15 flags=0 id=1
[   25.347059] [tempesta]   parser: Req_Hdr(16:0): c=0x43(C), r=-2
[   25.347447] [tempesta]   parser: Req_HdrC(22:0): c=0x6f(o), r=-2
[   25.347841] [tempesta]   parser: Req_HdrConnection(44:0): c=0x3a(:), r=-2
[   25.348292] [tempesta]   parser: RGen_LWS(10000:45): c=0x20( ), r=-2
[   25.348709] [tempesta]   parser: RGen_LWS(10000:45): c=0x4b(K), r=-2
[   25.349126] [tempesta]   enter FSM at state 45
[   25.349420] [tempesta]   parser: Req_HdrConnectionV(45:0): c=0x4b(K), r=-2
[   25.349869] [tempesta]   enter FSM at state 1
[   25.350160] [tempesta]   parser: I_Conn(1:1): c=0x4b(K), r=-2
[   25.350538] [tempesta]   parser: I_EoT(6:1): c=0xd(.), r=0
[   25.350898] [tempesta]   parser: I_EoL(7:1): c=0xd(.), r=0
[   25.351259] [tempesta]   parser: I_EoL(7:1): c=0xa(.), r=0
[   25.351619] [tempesta]   parser: Connection parsed: flags 0x2
[   25.351997] [tempesta]   parse header __parse_connection: ret=12 plen=10 id=0
[   25.352474] [tempesta]   store header w/ ptr=ffff88003dbdb0f3 len=22 flags=0 id=0
[   25.352965] [tempesta]   parser: Req_Hdr(16:0): c=0xd(.), r=-2
[   25.353349] [tempesta]   parser: Req_HdrDone(93:0): c=0xa(.), r=-2
[   25.353755] [tempesta]   parse msg body: flags=0x2 content_length=0
[   25.354167] [tempesta]   request parsed: len=109 parsed=109 msg_len=109 res=0
[   25.354632] [tempesta]   GFSM return code 0
[   25.354910] [tempesta]   added X-Forwarded-For header: X-Forwarded-For: 127.0.0.1
[   25.354910] 
[   25.355497] 
[   25.355602] =============================================
[   25.355955] [ INFO: possible recursive locking detected ]
[   25.356312] 3.10.10-tempesta #7 Tainted: G           O
[   25.356470] ---------------------------------------------
[   25.356470] wget/1954 is trying to acquire lock:
[   25.356470]  (slock-AF_INET/1){+.-...}, at: [<ffffffffa00002c8>] ss_send+0x28/0x160 [sync_socket]
[   25.356470] 
[   25.356470] but task is already holding lock:
[   25.356470]  (slock-AF_INET/1){+.-...}, at: [<ffffffff813433bc>] tcp_v4_rcv+0x40c/0x8b0
[   25.356470] 
[   25.356470] other info that might help us debug this:
[   25.356470]  Possible unsafe locking scenario:
[   25.356470] 
[   25.356470]        CPU0
[   25.356470]        ----
[   25.356470]   lock(slock-AF_INET/1);
[   25.356470]   lock(slock-AF_INET/1);
[   25.356470] 
[   25.356470]  *** DEADLOCK ***
[   25.356470] 
[   25.356470]  May be due to missing lock nesting notation
[   25.356470] 
[   25.356470] 5 locks held by wget/1954:
[   25.356470]  #0:  (sk_lock-AF_INET){+.+.+.}, at: [<ffffffff8132f107>] tcp_sendmsg+0x27/0xc00
[   25.356470]  #1:  (rcu_read_lock){.+.+..}, at: [<ffffffff81323670>] ip_queue_xmit+0x0/0x4a0
[   25.356470]  #2:  (rcu_read_lock){.+.+..}, at: [<ffffffff812e6463>] __netif_receive_skb_core+0x63/0x590
[   25.356470]  #3:  (rcu_read_lock){.+.+..}, at: [<ffffffff8131d750>] ip_local_deliver_finish+0x40/0x150
[   25.356470]  #4:  (slock-AF_INET/1){+.-...}, at: [<ffffffff813433bc>] tcp_v4_rcv+0x40c/0x8b0
[   25.356470] 
[   25.356470] stack backtrace:
[   25.356470] CPU: 0 PID: 1954 Comm: wget Tainted: G           O 3.10.10-tempesta #7
[   25.356470]  ffff88003d645080 ffff88003fc038c0 ffffffff813bf35d ffff88003fc03930
[   25.356470]  ffffffff81073b96 ffff88003d6b7da8 ffff88003fc03938 ffffffffa0014169
[   25.356470]  ffff880000000000 00000b00584a0250 ffffffff00000000 ffffffff816f5cc0
[   25.356470] Call Trace:
[   25.356470]  <IRQ>  [<ffffffff813bf35d>] dump_stack+0x19/0x1b
[   25.356470]  [<ffffffff81073b96>] __lock_acquire+0xfa6/0x1240
[   25.356470]  [<ffffffffa0014169>] ? __hdr_add+0x96/0x17a [tempesta_fw]
[   25.356470]  [<ffffffff81074658>] lock_acquire+0x98/0xd0
[   25.356470]  [<ffffffffa00002c8>] ? ss_send+0x28/0x160 [sync_socket]
[   25.356470]  [<ffffffff813c30ca>] _raw_spin_lock_nested+0x3a/0x50
[   25.356470]  [<ffffffffa00002c8>] ? ss_send+0x28/0x160 [sync_socket]
[   25.356470]  [<ffffffffa00002c8>] ss_send+0x28/0x160 [sync_socket]
[   25.356470]  [<ffffffffa00493d1>] ? tfw_sched_get_srv+0x49/0x4b [tempesta_fw]
[   25.356470]  [<ffffffffa0011c50>] tfw_connection_send_srv+0xbf/0xc6 [tempesta_fw]
[   25.356470]  [<ffffffffa0015360>] tfw_http_req_cache_cb+0x5d/0x5f [tempesta_fw]
[   25.356470]  [<ffffffffa000ddec>] tfw_cache_req_process+0x39/0x163 [tempesta_fw]
[   25.356470]  [<ffffffffa0015303>] ? tfw_http_adjust_resp+0x13/0x13 [tempesta_fw]
[   25.356470]  [<ffffffffa001561e>] tfw_http_req_process+0x2bc/0x32b [tempesta_fw]
[   25.356470]  [<ffffffffa00159f9>] tfw_http_msg_process+0x43/0x5e [tempesta_fw]
[   25.356470]  [<ffffffffa0012d0b>] tfw_gfsm_dispatch+0x41/0x43 [tempesta_fw]
[   25.356470]  [<ffffffffa0011e11>] tfw_connection_recv+0x88/0x8a [tempesta_fw]
[   25.356470]  [<ffffffffa000005b>] ss_tcp_process_proto_skb+0x5b/0x90 [sync_socket]
[   25.356470]  [<ffffffffa000020f>] ss_tcp_process_skb+0x17f/0x190 [sync_socket]
[   25.356470]  [<ffffffffa0000d8e>] ss_tcp_data_ready+0x10e/0x25e [sync_socket]
[   25.356470]  [<ffffffff8133425d>] tcp_data_queue+0x4dd/0xd10
[   25.356470]  [<ffffffff81337660>] tcp_rcv_established+0x560/0x620
[   25.356470]  [<ffffffff8134183c>] tcp_v4_do_rcv+0x1dc/0x4e0
[   25.356470]  [<ffffffff811eea42>] ? do_raw_spin_lock+0x82/0x140
[   25.356470]  [<ffffffff813433e7>] tcp_v4_rcv+0x437/0x8b0
[   25.356470]  [<ffffffff8131d7c5>] ip_local_deliver_finish+0xb5/0x150
[   25.356470]  [<ffffffff8131d750>] ? ip_local_deliver_finish+0x40/0x150
[   25.356470]  [<ffffffff8131df70>] ip_local_deliver+0x50/0x90
[   25.356470]  [<ffffffff8131dcde>] ip_rcv_finish+0x47e/0x580
[   25.356470]  [<ffffffff8131e26c>] ip_rcv+0x2bc/0x340
[   25.356470]  [<ffffffff812e6869>] __netif_receive_skb_core+0x469/0x590
[   25.356470]  [<ffffffff812e6463>] ? __netif_receive_skb_core+0x63/0x590
[   25.356470]  [<ffffffff8107258e>] ? mark_held_locks+0x10e/0x130
[   25.356470]  [<ffffffff812e69df>] __netif_receive_skb+0x4f/0x70
[   25.356470]  [<ffffffff812e855f>] process_backlog+0x7f/0x200
[   25.356470]  [<ffffffff812e8783>] net_rx_action+0xa3/0x1f0
[   25.356470]  [<ffffffff81036550>] __do_softirq+0xe0/0x1d0
[   25.356470]  [<ffffffff813c5afc>] call_softirq+0x1c/0x26
[   25.356470]  <EOI>  [<ffffffff81003ef5>] do_softirq+0x65/0x100
[   25.356470]  [<ffffffff81322db4>] ? ip_finish_output+0x3f4/0x500
[   25.356470]  [<ffffffff81035dc0>] local_bh_enable+0xd0/0x120
[   25.356470]  [<ffffffff81322db4>] ip_finish_output+0x3f4/0x500
[   25.356470]  [<ffffffff81322c10>] ? ip_finish_output+0x250/0x500
[   25.356470]  [<ffffffff81323e35>] ip_output+0x95/0x110
[   25.356470]  [<ffffffff813233ee>] ip_local_out+0x5e/0x90
[   25.356470]  [<ffffffff813239ff>] ip_queue_xmit+0x38f/0x4a0
[   25.356470]  [<ffffffff81323670>] ? ip_build_and_send_pkt+0x250/0x250
[   25.356470]  [<ffffffff8133a7e2>] tcp_transmit_skb+0x792/0x820
[   25.356470]  [<ffffffff8133b1dd>] tcp_write_xmit+0x96d/0xb40
[   25.356470]  [<ffffffff8133b61d>] __tcp_push_pending_frames+0x2d/0xd0
[   25.356470]  [<ffffffff8132b171>] tcp_push+0x81/0x90
[   25.356470]  [<ffffffff8132fb03>] tcp_sendmsg+0xa23/0xc00
[   25.356470]  [<ffffffff813563ff>] inet_sendmsg+0xdf/0x100
[   25.356470]  [<ffffffff81356320>] ? inet_sendpage+0x130/0x130
[   25.356470]  [<ffffffff812d1232>] sock_aio_write+0x142/0x160
[   25.356470]  [<ffffffff810f9cc4>] ? core_sys_select+0x34/0x3c0
[   25.356470]  [<ffffffff810e6d61>] do_sync_write+0x81/0xa0
[   25.356470]  [<ffffffff810e75c5>] vfs_write+0xe5/0x1b0
[   25.356470]  [<ffffffff810e7b87>] SyS_write+0x47/0x90
[   25.356470]  [<ffffffff813c4729>] system_call_fastpath+0x16/0x1b
[   25.391764] [sync_socket]   ss_send:89 entail skb=ffff88003d6b7da8 data_len=0 len=137
[   25.392290] [sync_socket]   ss_send:99 sk=ffff88003d71b280 is_queue_empty=0 tcp_send_head(sk)=ffff88003d6b7da8 sk->sk_state=1
[   25.393662] [tempesta]   Link new msg ffff88003cc69040 with connection ffff88003d660548
[   25.394205] [tempesta]   Add skb ffff88003d6b7ba8 to message ffff88003cc69040
[   25.394677] [tempesta]   received 215 server data bytes (HTTP/1.1 200 OK
[   25.394677] Server: nginx/1.2.1
[   25.394677] Date: Thu, 19 Feb 2015 11:24:07 GMT
[   25.394677] Content-Type: text/html
[   25.394677] Content-Length: 151
[   25.394677] Last-Modified: Mon, 04 Oct 2004 15:04:06 GMT
[   25.394677] Connection: keep-alive
[   25.394677] Accept-Ranges: bytes
[   25.394677] 
[   25.394677] ) on socket (conn=ffff88003d660548)
[   25.397258] [tempesta]   enter FSM at state 0
[   25.397552] [tempesta]   parser: Resp_0(0:0): c=0x48(H), r=-2
[   25.397932] [tempesta]   parser: Resp_HttpVer(2:0): c=0x48(H), r=-2
[   25.398351] [tempesta]   parser: Resp_StatusCode(11:0): c=0x32(2), r=-2
[   25.398784] [tempesta]   parser: Resp_ReasonPhrase(12:1): c=0x20( ), r=-2
[   25.399231] [tempesta]   parser: Resp_Hdr(13:1): c=0x53(S), r=-2
[   25.399625] [tempesta]   parser: Resp_HdrOther(87:1): c=0x65(e), r=-2
[   25.400055] [tempesta]   parser: Resp_Hdr(13:1): c=0x44(D), r=-2
[   25.400452] [tempesta]   parser: Resp_HdrOther(87:1): c=0x61(a), r=-2
[   25.400878] [tempesta]   parser: Resp_Hdr(13:1): c=0x43(C), r=-2
[   25.401275] [tempesta]   parser: Resp_HdrC(14:1): c=0x6f(o), r=-2
[   25.401676] [tempesta]   parser: Resp_HdrCo(28:1): c=0x6e(n), r=-2
[   25.402087] [tempesta]   parser: Resp_HdrCon(29:1): c=0x74(t), r=-2
[   25.402501] [tempesta]   parser: Resp_HdrCont(38:1): c=0x65(e), r=-2
[   25.402920] [tempesta]   parser: Resp_HdrConte(39:1): c=0x6e(n), r=-2
[   25.403344] [tempesta]   parser: Resp_HdrConten(40:1): c=0x74(t), r=-2
[   25.403774] [tempesta]   parser: Resp_HdrContent(41:1): c=0x2d(-), r=-2
[   25.404219] [tempesta]   parser: Resp_HdrContent_(42:1): c=0x54(T), r=-2
[   25.404662] [tempesta]   parser: Resp_HdrOther(87:1): c=0x79(y), r=-2
[   25.405087] [tempesta]   parser: Resp_Hdr(13:1): c=0x43(C), r=-2
[   25.405483] [tempesta]   parser: Resp_HdrC(14:1): c=0x6f(o), r=-2
[   25.405886] [tempesta]   parser: RGen_LWS(10000:49): c=0x20( ), r=-2
[   25.406307] [tempesta]   parser: RGen_LWS(10000:49): c=0x31(1), r=-2
[   25.406724] [tempesta]   enter FSM at state 49
[   25.407019] [tempesta]   parser: Resp_HdrContent_LengthV(49:0): c=0x31(1), r=-2
[   25.407498] [tempesta]   enter FSM at state 3
[   25.407789] [tempesta]   parser: I_ContLen(3:3): c=0x31(1), r=-2
[   25.408192] [tempesta]   parser: I_EoL(7:3): c=0xd(.), r=-2
[   25.408563] [tempesta]   parser: I_EoL(7:3): c=0xa(.), r=-2
[   25.408932] [tempesta]   parse header __parse_content_length: ret=5 plen=3 id=3
[   25.409413] [tempesta]   store header w/ ptr=ffff88003cc69548 len=19 flags=1 id=3
[   25.409906] [tempesta]   parser: Resp_Hdr(13:0): c=0x4c(L), r=-2
[   25.410305] [tempesta]   parser: Resp_HdrOther(87:0): c=0x61(a), r=-2
[   25.410726] [tempesta]   parser: Resp_Hdr(13:0): c=0x43(C), r=-2
[   25.411116] [tempesta]   parser: Resp_HdrC(14:0): c=0x6f(o), r=-2
[   25.411511] [tempesta]   parser: Resp_HdrConnection(36:0): c=0x3a(:), r=-2
[   25.411949] [tempesta]   parser: RGen_LWS(10000:37): c=0x20( ), r=-2
[   25.412363] [tempesta]   parser: RGen_LWS(10000:37): c=0x6b(k), r=-2
[   25.412770] [tempesta]   enter FSM at state 37
[   25.413057] [tempesta]   parser: Resp_HdrConnectionV(37:0): c=0x6b(k), r=-2
[   25.413501] [tempesta]   enter FSM at state 1
[   25.413782] [tempesta]   parser: I_Conn(1:1): c=0x6b(k), r=-2
[   25.414154] [tempesta]   parser: I_EoT(6:1): c=0xd(.), r=0
[   25.414506] [tempesta]   parser: I_EoL(7:1): c=0xd(.), r=0
[   25.414858] [tempesta]   parser: I_EoL(7:1): c=0xa(.), r=0
[   25.415209] [tempesta]   parser: Connection parsed: flags 0x2
[   25.415577] [tempesta]   parse header __parse_connection: ret=12 plen=10 id=0
[   25.416038] [tempesta]   store header w/ ptr=ffff88003cc69588 len=22 flags=1 id=0
[   25.416533] [tempesta]   parser: Resp_Hdr(13:0): c=0x41(A), r=-2
[   25.416948] [tempesta]   parser: Resp_HdrOther(87:0): c=0x63(c), r=-2
[   25.417377] [tempesta]   parser: Resp_Hdr(13:0): c=0xd(.), r=-2
[   25.417770] [tempesta]   parser: Resp_HdrDone(88:0): c=0xa(.), r=-2
[   25.418188] [tempesta]   parse msg body: flags=0x2 content_length=151
[   25.418614] [tempesta]   response parsed: len=215 parsed=215 res=-1
[   25.419029] [tempesta]   postpone skb ffff88003d6b7ba8
[   25.419369] [tempesta]   Add skb ffff88003d6b7ba8 to message ffff88003cc69040
[   25.419840] [tempesta]   received 151 server data bytes (<html>
[   25.419840] <head>
[   25.419840] <title>Welcome to nginx!</title>
[   25.419840] </head>
[   25.419840] <body bgcolor="white" text="black">
[   25.419840] <center><h1>Welcome to nginx!</h1></center>
[   25.419840] </body>
[   25.419840] </html>
[   25.419840] ) on socket (conn=ffff88003d660548)
[   25.421971] [tempesta]   enter FSM at state 89
[   25.422272] [tempesta]   parser: Resp_Body(89:0): c=0x3c(<), r=-2
[   25.422677] [tempesta]   read body: to_read=151
[   25.422979] [tempesta]   parser: Resp_BodyReadChunk(92:0): c=0x3c(<), r=-2
[   25.423431] [tempesta]   response parsed: len=151 parsed=151 res=0
[   25.423842] [sync_socket]   ss_send:89 entail skb=ffff88003d6b7ba8 data_len=151 len=366
[   25.424372] [sync_socket]   ss_send:99 sk=ffff88003d4af2c0 is_queue_empty=0 tcp_send_head(sk)=ffff88003d6b7ba8 sk->sk_state=1
[   25.425121] [tempesta]   Free msg: ffff88003d411040
[   25.425436] [tempesta]   free skb ffff88003d6b7da8: truesize=2304 sk=ffff88003d4af2c0, destructor=ffffffff812d2ea0 users=1
[   25.426182] [tempesta]   Free msg: ffff88003cc69040
[   25.426493] [tempesta]   free skb ffff88003d6b7ba8: truesize=2455 sk=ffff88003d71b280, destructor=ffffffff812d2ea0 users=1
[   25.428753] general protection fault: 0000 [#1] SMP 
[   25.429189] Modules linked in: tfw_sched_dummy(O) tempesta_fw(O) tempesta_db(O) sync_socket(O)
[   25.429800] CPU: 0 PID: 1854 Comm: nginx Tainted: G           O 3.10.10-tempesta #7
[   25.430309] task: ffff88003d53b100 ti: ffff88003cc8e000 task.ti: ffff88003cc8e000
[   25.430802] RIP: 0010:[<ffffffff81336257>]  [<ffffffff81336257>] tcp_ack+0x3f7/0xf10
[   25.431322] RSP: 0018:ffff88003fc03bd8  EFLAGS: 00010296
[   25.431675] RAX: 000000006b6b6b6b RBX: ffff88003d4af2c0 RCX: 000000000000006b
[   25.432154] RDX: 00000000f166e383 RSI: 6b6b6b6b6b6b6b6b RDI: 0000000000000001
[   25.432628] RBP: ffff88003fc03c58 R08: 00000000ffffffff R09: 0000000000000000
[   25.433099] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88003d6b7ba8
[   25.433569] R13: 00000000f166e383 R14: 0000000000004502 R15: 00000000f166e215
[   25.434040] FS:  00007fdd100e0700(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[   25.434578] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   25.434958] CR2: 00007faf7f1a2d2c CR3: 000000003d51e000 CR4: 00000000000006b0
[   25.435432] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   25.435904] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   25.436381] Stack:
[   25.436521]  ffff88003fc03cf8 0000000081366a59 fffef3dc00000001 0000000000000000
[   25.437043]  ffffffff81366a59 00000001810727cd ffff88003d4af4a0 00000000ffffffff
[   25.437567]  0000000000000000 0000000000000001 0000000100000000 ffff88003d4af2c0
[   25.438093] Call Trace:
[   25.438259]  <IRQ> 
[   25.438392]  [<ffffffff81366a59>] ? ipt_do_table+0x689/0x6b0
[   25.438800]  [<ffffffff813375e5>] tcp_rcv_established+0x4e5/0x620
[   25.439203]  [<ffffffff8134183c>] tcp_v4_do_rcv+0x1dc/0x4e0
[   25.439577]  [<ffffffff811eea42>] ? do_raw_spin_lock+0x82/0x140
[   25.439969]  [<ffffffff813433e7>] tcp_v4_rcv+0x437/0x8b0
[   25.440330]  [<ffffffff8131d7c5>] ip_local_deliver_finish+0xb5/0x150
[   25.440751]  [<ffffffff8131d750>] ? ip_local_deliver_finish+0x40/0x150
[   25.441180]  [<ffffffff8131df70>] ip_local_deliver+0x50/0x90
[   25.441558]  [<ffffffff8131dcde>] ip_rcv_finish+0x47e/0x580
[   25.441928]  [<ffffffff8131e26c>] ip_rcv+0x2bc/0x340
[   25.442260]  [<ffffffff812e6869>] __netif_receive_skb_core+0x469/0x590
[   25.442692]  [<ffffffff812e6463>] ? __netif_receive_skb_core+0x63/0x590
[   25.443129]  [<ffffffff812e69df>] __netif_receive_skb+0x4f/0x70
[   25.443522]  [<ffffffff812e855f>] process_backlog+0x7f/0x200
[   25.443899]  [<ffffffff812e8783>] net_rx_action+0xa3/0x1f0
[   25.444269]  [<ffffffff81036550>] __do_softirq+0xe0/0x1d0
[   25.444631]  [<ffffffff813c5afc>] call_softirq+0x1c/0x26
[   25.444985]  <EOI> 
[   25.445118]  [<ffffffff81003ef5>] do_softirq+0x65/0x100
[   25.445497]  [<ffffffff812d54f8>] ? release_sock+0x1b8/0x210
[   25.445873]  [<ffffffff810362d0>] local_bh_enable_ip+0xd0/0x120
[   25.446270]  [<ffffffff813c31ab>] _raw_spin_unlock_bh+0x3b/0x40
[   25.446664]  [<ffffffff812d54f8>] release_sock+0x1b8/0x210
[   25.447028]  [<ffffffff8132ea38>] tcp_setsockopt+0x7b8/0x800
[   25.447404]  [<ffffffff812d329f>] sock_common_setsockopt+0xf/0x20
[   25.447808]  [<ffffffff812d254e>] SyS_setsockopt+0x8e/0xd0
[   25.448179]  [<ffffffff813c4729>] system_call_fastpath+0x16/0x1b
[   25.448581] Code: 1f 44 00 00 8b 93 80 06 00 00 41 0f b6 4c 24 4d 41 8b 84 24 c8 00 00 00 41 3b 54 24 44 0f 89 b1 00 00 00 49 8b b4 24 d0 00 00 00 <44> 0f b7 5c 06 04 66 41 83 fb 01 0f 84 76 02 00 00 41 8b 44 24 
[   25.450396] RIP  [<ffffffff81336257>] tcp_ack+0x3f7/0xf10
[   25.450767]  RSP <ffff88003fc03bd8>
[   25.451002] ---[ end trace 05a09eb893958b3b ]---
[   25.451311] Kernel panic - not syncing: Fatal exception in interrupt

[configuration] Connection refused on module loading

If backend server is started on IP or address different from the hardcoded default, then tempesta_fw.ko can't be loaded:

# SYNC_SOCKET=/root/sync_socket/ ./tempesta.sh start
Starting Tempesta...
Error: could not insert module /root/tempesta_fw/tempesta_fw.ko: Connection refused
ERROR: cannot load tempesta module
# dmesg|tail
[tempesta] Start Tempesta HTTP
[tempesta] Open 1 backend sockets
[tempesta] ERROR: Can't connect to back-end server 172.16.0.1:8080 (-111)
[tempesta] Close 0 backend sockets

Expected behavior: Tempesta starts normally, processes incomming connecitons and requests, but drop all the requests with appropriate log errors if there is no any specified backend server.

Oops on HTTP parser error

If HTTP parser returns TFW_BLOCK (bad HTTP message), then we get Oops:

    [tempesta]   parser: Req_HdrDone(77:0): c=0xa(.)
    [tempesta]   request parsed: len=280 parsed=280 res=-2
    [tempesta]   Block bad HTTP request
    [sync_socket] Warning: can't process app data on socket ffff88003d152700
    [tempesta]   Close socket ffff88003d152700, conn=ffff88003d26e648
    ------------[ cut here ]------------
    WARNING: at net/core/stream.c:201 sk_stream_kill_queues+0xf5/0x11d()
    Modules linked in: tempesta_fw(O) tempesta_db(O) sync_socket(O) i2c_piix4 floppy
    CPU: 1 PID: 11 Comm: ksoftirqd/1 Tainted: G           O 3.10.10+ #7
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
     ffffffff8280d1d1 ffff88003dcd9828 ffffffff824d2d9c ffff88003dcd9868
     ffffffff82028e25 ffff88003d152700 0000000000000000 ffff88003d152700
     ffff88003d152858 0000000000000005 0000000000000000 ffff88003dcd9878
    Call Trace:
     [<ffffffff824d2d9c>] dump_stack+0x19/0x1b
     [<ffffffff82028e25>] warn_slowpath_common+0x62/0x7b
     [<ffffffff82028e53>] warn_slowpath_null+0x15/0x17
     [<ffffffff823b6559>] sk_stream_kill_queues+0xf5/0x11d
     [<ffffffff823f72ba>] inet_csk_destroy_sock+0x7c/0x107
     [<ffffffff823fa4e5>] tcp_done+0x74/0x79
     [<ffffffff8240c61f>] tcp_time_wait+0x1de/0x1ed
     [<ffffffff824038b8>] tcp_rcv_state_process+0x89a/0x9e2
     [<ffffffff824094e0>] tcp_v4_do_rcv+0x2a0/0x2e0

cfg: wrong position is printed in log messages in case of parser error

Scenario:

Take the default configuration file that consists of comments only.
Type few random characters (e.g. sdafasdg) in the middle of the file.
Start Tempesta FW.

Something like this appears in the log:

[   10.728074] [tempesta] ERROR: syntax error
[   10.728561] [tempesta] ERROR: configuration parsing error:
[   10.728561] 
[   10.728561] #   cache_size 1048576; # 1 GiB
[   10.728561] #
[   10.728561] # Default:
[   10.728561] #   cache_size 262144;  # 256 MiB
[   10.728561] 
[   10.728561] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It shows the wrong position in the configuration file.
That happens because the parser tries to read more tokens and eventually reaches EOF without meeting a ; character which is a syntax error.
The position where the parser stops is the end of the buffer, so it shows some contents around this position which is not what you expect to see.

Generally this is not about EOF, the tokenizer always eats comments implicitly, so you always see a wrong position in the log message if there are comments after it.
The default configuration file contains a lot of comments, so this is an issue.

TODO:

Try to display an actual position where an error happens. Perhaps need to show position of the previous token, not the current one. Or a position where the current entry started, but where the parser terminated.
Add more info: show file/line and expected/got clauses.

http_parser: bad handling of fragmented HTTP messages

Scenario:

git checkout http-parser-frag
make test

Result: a failed test, e.g.:

[   35.639269] tfw_test: EXPECT_TFWSTR_EQ(h_connection, s_connection) => false
[   35.639971] tfw_test: expected: 'Connection: Keep-Alive'
[   35.640542] tfw_test: got chunk: len=13, data='Connection: K'
[   35.641125] tfw_test: got chunk: len=10, data='eep-Alive
'
[   35.641683] tfw_test: /home/wq/natsys-lab/tempesta/tempesta_fw/t/unit/test_http_parser.c:277: error: 
[   35.642620] tfw_test: FAIL:
[   35.642920] tfw_test:   test__http_parser__fills_hdr_tbl()
[   35.643476] tfw_test:   these strings (see above) do not match

It fails because there is an extra \r character in the second chunk of the parsed header. The len should be 9, but it is actually 10.

The error is caused by this code:
https://github.com/natsys/tempesta/blob/http-parser-frag/tempesta_fw/http_parser.c#L849

        TRY_STR_LAMBDA("keep-alive", {
            if (msg->flags & TFW_HTTP_CONN_CLOSE)
                return CSTR_NEQ;
            msg->flags |= TFW_HTTP_CONN_KA;
            __FSM_I_MOVE_str(I_EoT, "keep-alive");
        });

Basically it is executed in the following scenario:
0. The header line is split into two chunks: "Connection: K" and "eep-Alive\r\n".

The parser FSM reads "Connection: " and the control reaches the TRY_STR_LAMBDA() at the position of "K".
TRY_STR_LAMBDA() reads K, but that is not enough, so it returns CSTR_POSTPONE and the FSM exits.
The next chunk is passed to the FSM, and TRY_STR_LAMBDA() reads "eep-alive". The string matches and it executes the code passed to the macro.
The __FSM_I_MOVE_str() moves the FSM to 10 steps forward from the current position, but not from a position where the TRY_STR_LAMBDA() was executed before. The position is changed since then, now it is the start of the second chunk ("eep-alive\r\n"), hence the extra "\r" character is added to the header.

The issue is not just about this particular header. It is about the whole parser. The TRY_STR()/TRY_STR_LAMBDA() is almost always used in this buggy way. This approach should be revised and fixed.

Beside the TRY_STR_LAMBDA(), there likely are other problems with the fragmentation exist in the parser, but now tests can't detect them. They will appear in future after the fix.

Duplicate HTTP headers processing

Currently we don't handle duplicate headers.

TFW_STR_COMPOUND2 is designated to handle duplicate headers each can consist from many real packet fragments (TFW_STR_COMPUND).

Thus stuff should be handled in __store_header() and functions like tfw_http_set_hdr_connection() (see comments for the functions).

Currently we handle only headers which are very rarely duplicated, but duplicate headers are frequent in real life.

Sticky cookies

Need to implement analog of Nginx sticky module (https://github.com/lusis/nginx-sticky-module) for users identification. However, the module must be able to generate cryptographically safe cookie value depending on the user IP and User-Agent header value, so the server classification logic can identify a user by the cookie value as well as forward his or her requests to the right backend.

The functionality to add HTTP headers to passing requests is broken

Currently this is used in tfw_http_add_forwarded_for() to add an X-Forwarded-For: header to HTTP requests that Tempesta receives, parses, and then passes to a backend server.

One reason is the use of constructs like vaddr = kmap_atomic(skb_frag_page(frag));, where something like frag_addr = skb_frag_address(frag); should be used instead. That's not the only reason for broken behaviour though.

Tempesta/keepalived failovering

Need to test VIP migration between two Tempesta instances as a process of dynamic failovering using keepalived.

As a result of the test, new Wiki for High availability installations must be created.

Handling values only for special headers

Need to fix comment as discussed in eae3608#commitcomment-8805941

Oops on transmit HTTP request to backend server

Not always reproducable.

    [tempesta]   parser: Req_Hdr(16:0): c=0xd(.), r=-2
    [tempesta]   parser: Req_HdrDone(77:0): c=0xa(.), r=-2
    [tempesta]   request parsed: len=280 parsed=280 res=0
    [sync_socket]   ss_send:93 is_queue_empty=0 tcp_send_head(sk)=ffff88003d5358c0 sk->sk_state=1
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
    IP: [<ffffffff82401cfa>] tcp_ack+0x506/0xb94
    PGD 3cd41067 PUD 3d17c067 PMD 0 
    Oops: 0002 [#1] SMP 
    Modules linked in: tempesta_fw(O) tempesta_db(O) sync_socket(O) i2c_piix4 floppy
    CPU: 1 PID: 11 Comm: ksoftirqd/1 Tainted: G           O 3.10.10+ #7
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
    task: ffff88003dcd6300 ti: ffff88003dcd8000 task.ti: ffff88003dcd8000
    RIP: 0010:[<ffffffff82401cfa>]  [<ffffffff82401cfa>] tcp_ack+0x506/0xb94
    RSP: 0018:ffff88003dcd99e8  EFLAGS: 00010246
    RAX: 0000000000000000 RBX: 0000000000000204 RCX: ffff88003d5358c0
    RDX: 0000000000000000 RSI: 0000000000000200 RDI: 00000000e889a0a2
    RBP: ffff88003dcd9aa8 R08: 0000000024e92d00 R09: 0000000000000001
    R10: 0000000000000001 R11: 000000000000ea9f R12: 000000003c6113bc
    R13: 0000000000000001 R14: 0000000000000000 R15: ffff880037b66080
    FS:  0000000000000000(0000) GS:ffff88003fd00000(0000) knlGS:0000000000000000
    CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
    CR2: 0000000000000008 CR3: 000000003d129000 CR4: 00000000000006e0
    DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    Stack:
     ffff88003dcd9a68 ffffffff82405559 0000000000000020 0000000000000000
     ffff880037b66188 0000000000000000 0000000000000000 0000000100000000
     0000000100000000 0000000000000000 000000000000817c 0000000000000001
    Call Trace:
     [<ffffffff82405559>] ? tcp_transmit_skb+0x6a7/0x70c
     [<ffffffff82407606>] ? tcp_send_ack+0xdb/0xe0
     [<ffffffff82402c73>] tcp_rcv_established+0x602/0x6ed
     [<ffffffff824092f9>] tcp_v4_do_rcv+0xb9/0x2e0
     [<ffffffff821e2c78>] ? security_sock_rcv_skb+0x11/0x13
     [<ffffffff823cae16>] ? sk_filter+0x31/0xa1
     [<ffffffff8240ba85>] tcp_v4_rcv+0x394/0x6f0

Remove unnecessary exports used in only in tests

Remove unnecessary symbol exports. See b022041#commitcomment-8848675 for reference and check all other places of exported symbols.

Performance test

Hello!

Thank you for a good tool! Do you have any performance tests? Could it work under 1-3 mpps load?

configuration: failure on listener address reconfiguration

etc/tempesta_fw.conf says that the default listen value is listen 80 and that If only PORT is given, then the address 0.0.0.0 (but not [::1]) is used.. However, this isn't true: if I start Tempesta with default config, then I see

    tcp        0      0 127.0.0.1:80            0.0.0.0:*               LISTEN      -

in netstat output.

Now I set listen 80; explicitly and execute ./tempesta.sh restart:

     [root@localhost tempesta]# ./tempesta.sh restart
     Stopping Tempesta...
     net.tempesta.state = stop
     Starting Tempesta...
     sysctl: setting key "net.tempesta.state": Invalid argument
     net.tempesta.state = start
     done
     done

Dmesg log:

    [ 9319.871284] [tdb] Start Tempesta DB
    [ 9319.880333] [tempesta] Initializing Tempesta FW kernel module...
    [ 9319.880794] [tempesta] init: cfg_if
    [ 9319.881029] [tempesta] init: http
    [ 9319.881221] [tempesta] init: server
    [ 9319.881437] [tempesta] init: client
    [ 9319.881657] [tempesta] init: session
    [ 9319.881878] [tempesta] init: connection
    [ 9319.882151] [tempesta]   register cfg: cache
    [ 9319.882458] [tempesta]   register cfg: sock_backend
    [ 9319.882787] [tempesta]   register cfg: sock_frontend
    [ 9319.884608] [tempesta] Registering new scheduler: dummy
    [ 9319.886167] [tempesta] got state via sysctl: start
    [ 9319.886855] [tempesta]   reading configuration file...
    [ 9319.887363] [tempesta]   reading file: /root/tempesta/etc/tempesta_fw.conf
    [ 9319.888033] [tempesta]   file size: 2204 bytes
    [ 9319.888459] [tempesta]   read by offset: 0
    [ 9319.888873] [tempesta]   read by offset: 2204
    [ 9319.889297] [tempesta] starting all modules...
    [ 9319.889720] [tempesta]   parsing configuration and pushing it to modules...
    [ 9319.890391] [tempesta]   use default entry: 'cache off;'
    [ 9319.890897] [tempesta]   spec handle: 'cache'
    [ 9319.891322] [tempesta]   use default entry: 'cache_size 262144;'
    [ 9319.891883] [tempesta]   spec handle: 'cache_size'
    [ 9319.892349] [tempesta]   use default entry: 'cache_dir /opt/tempesta/cache;'
    [ 9319.893015] [tempesta]   spec handle: 'cache_dir'
    [ 9319.893457] [tempesta]   use default entry: 'backend 127.0.0.1:8080;'
    [ 9319.894108] [tempesta]   spec handle: 'backend'
    [ 9319.894543] [tempesta]   use default entry: 'listen 127.0.0.1:80;'
    [ 9319.895130] [tempesta]   spec handle: 'listen'
    [ 9319.895571] [tempesta]   created front-end socket: sk=ffff880030dfd7c0
    [ 9319.896536] [tempesta]   starting modules...
    [ 9319.896954] [tempesta]   mod_start(): cache
    [ 9319.897367] [tempesta]   mod_start(): sock_backend
    [ 9319.897858] [tempesta]   Starting thread: tfw_bconnd
    [ 9319.898362] [tempesta]   mod_start(): sock_frontend
    [ 9319.898829] [tempesta]   start listening on socket: sk=ffff880030dfd7c0
    [ 9319.899466] [tempesta] modules are started
    [ 9340.766809] [tempesta] got state via sysctl: stop
    [ 9340.767376] [tempesta] stopping all modules...
    [ 9340.767847] [tempesta]   stopping modules...
    [ 9340.768268] [tempesta]   mod_stop(): sock_frontend
    [ 9340.768719] [tempesta]   release front-end socket: sk=ffff880030dfd7c0
    [ 9340.769348] [tempesta]   mod_stop(): sock_backend
    [ 9340.769792] [tempesta]   Stopping thread: tfw_bconnd
    [ 9340.770334] [tempesta] tfw_bconnd: stopping
    [ 9340.770764] [tempesta]   spec cleanup: 'backend'
    [ 9340.771230] [tempesta]   mod_stop(): cache
    [ 9340.771624] [tempesta]   spec cleanup: 'cache_dir'
    [ 9340.773728] [tempesta] Un-registering scheduler: dummy
    [ 9340.775833] [tempesta] exiting...
    [ 9340.776244] [tempesta]   stopping and unregistering all cfg modules
    [ 9340.776830] [tempesta]   unregister cfg: sock_frontend
    [ 9340.777331] [tempesta]   unregister cfg: sock_backend
    [ 9340.777807] [tempesta]   unregister cfg: cache
    [ 9340.779368] [tdb] Shutdown Tempesta DB
    [ 9651.546177] [tdb] Start Tempesta DB
    [ 9651.556668] [tempesta] Initializing Tempesta FW kernel module...
    [ 9651.557322] [tempesta] init: cfg_if
    [ 9651.557682] [tempesta] init: http
    [ 9651.558019] [tempesta] init: server
    [ 9651.558376] [tempesta] init: client
    [ 9651.558726] [tempesta] init: session
    [ 9651.559094] [tempesta] init: connection
    [ 9651.559482] [tempesta]   register cfg: cache
    [ 9651.559890] [tempesta]   register cfg: sock_backend
    [ 9651.560376] [tempesta]   register cfg: sock_frontend
    [ 9651.562353] [tempesta] Registering new scheduler: dummy
    [ 9651.564137] [tempesta] got state via sysctl: start
    [ 9651.564706] [tempesta]   reading configuration file...
    [ 9651.565210] [tempesta]   reading file: /root/tempesta/etc/tempesta_fw.conf
    [ 9651.565859] [tempesta]   file size: 2204 bytes
    [ 9651.566314] [tempesta]   read by offset: 0
    [ 9651.566717] [tempesta]   read by offset: 2204
    [ 9651.567147] [tempesta] starting all modules...
    [ 9651.568507] [tempesta]   parsing configuration and pushing it to modules...
    [ 9651.569268] [tempesta]   use default entry: 'cache off;'
    [ 9651.569779] [tempesta]   spec handle: 'cache'
    [ 9651.570212] [tempesta]   use default entry: 'cache_size 262144;'
    [ 9651.570779] [tempesta]   spec handle: 'cache_size'
    [ 9651.571248] [tempesta]   use default entry: 'cache_dir /opt/tempesta/cache;'
    [ 9651.571905] [tempesta]   spec handle: 'cache_dir'
    [ 9651.572366] [tempesta]   use default entry: 'backend 127.0.0.1:8080;'
    [ 9651.572967] [tempesta]   spec handle: 'backend'
    [ 9651.573418] [tempesta]   use default entry: 'listen 127.0.0.1:80;'
    [ 9651.573993] [tempesta]   spec handle: 'listen'
    [ 9651.574481] [tempesta]   created front-end socket: sk=ffff880024091800
    [ 9651.575104] [tempesta]   starting modules...
    [ 9651.575517] [tempesta]   mod_start(): cache
    [ 9651.575916] [tempesta]   mod_start(): sock_backend
    [ 9651.576388] [tempesta]   Starting thread: tfw_bconnd
    [ 9651.576900] [tempesta]   mod_start(): sock_frontend
    [ 9651.577389] [tempesta]   start listening on socket: sk=ffff880024091800
    [ 9651.578021] [tempesta] modules are started
    [ 9652.578236] [tempesta] Connected to backend: 127.0.0.1:8080
    [ 9704.000739] [sync_socket]   Close socket ffff8800313da840 (account=1)
    [ 9704.001850] [tempesta]   Close socket ffff8800313da840, conn=ffff8800333f0438
    [ 9704.002728] [tempesta]   Free msg:           (null)
    [ 9704.003051] [tempesta]   Free connection: ffff8800333f0438
    [ 9704.003681] [sync_socket]   free rcv skb ffff880024080e88
    [ 9883.037803] [tempesta] got state via sysctl: stop
    [ 9883.038370] [tempesta] stopping all modules...
    [ 9883.038869] [tempesta]   stopping modules...
    [ 9883.039304] [tempesta]   mod_stop(): sock_frontend
    [ 9883.039808] [tempesta]   release front-end socket: sk=ffff880024091800
    [ 9883.040452] [tempesta]   mod_stop(): sock_backend
    [ 9883.040902] [tempesta]   Stopping thread: tfw_bconnd
    [ 9883.041342] [tempesta] tfw_bconnd: stopping
    [ 9883.041835] ------------[ cut here ]------------
    [ 9883.042257] kernel BUG at /root/tempesta/tempesta_fw/server.c:39!
    [ 9883.042332] invalid opcode: 0000 [#1] SMP
    [ 9883.042332] Modules linked in: tfw_sched_dummy(O) tempesta_fw(O) tempesta_db(O) sync_socket(O) nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables xfs exportfs ppdev parport_pc microcode serio_raw parport pcspkr i2c_piix4 i2c_core dm_mirror dm_region_hash dm_log dm_mod uinput btrfs xor zlib_deflate raid6_pq libcrc32c ata_generic pata_acpi e1000 ata_piix floppy ipv6 autofs4 [last unloaded: sync_socket]
    [ 9883.042332] CPU: 0 PID: 10966 Comm: tfw_bconnd Tainted: G           O 3.10.10+ #29
    [ 9883.042332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
    [ 9883.042332] task: ffff88007aef17f0 ti: ffff880024f18000 task.ti: ffff880024f18000
    [ 9883.042332] RIP: 0010:[<ffffffffa0512ab8>]  [<ffffffffa0512ab8>] tfw_destroy_server+0x30/0xdf [tempesta_fw]
    [ 9883.042332] RSP: 0018:ffff880024f19d48  EFLAGS: 00010202
    [ 9883.042332] RAX: 0000000000000001 RBX: ffff8800313da840 RCX: 0000000000000020
    [ 9883.042332] RDX: ffffffffffffffe0 RSI: ffffffffffffffff RDI: ffff8800313da840
    [ 9883.042332] RBP: ffff880024f19d60 R08: 0000000000000000 R09: 0000000000000219
    [ 9883.042332] R10: 0000000000000050 R11: 0000000000000002 R12: ffff8800313da9c0
    [ 9883.042332] R13: 0000000000000007 R14: ffff8800313da8b0 R15: 0000000000000000
    [ 9883.042332] FS:  0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
    [ 9883.042332] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
    [ 9883.042332] CR2: 00007fabf8587320 CR3: 0000000024056000 CR4: 00000000000006f0
    [ 9883.042332] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [ 9883.042332] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    [ 9883.042332] Stack:
    [ 9883.042332]  ffff8800313da840 0000000000000000 ffff880024f19d80 ffff880024f19d88
    [ 9883.042332]  ffffffff813f0f8f ffff8800313da840 ffff8800313da9c0 0000000000000007
    [ 9883.042332]  ffff880024f19d98 ffffffff813f1101 ffff880024f19dc8 ffffffff814492a1
    [ 9883.042332] Call Trace:
    [ 9883.042332]  [<ffffffff813f0f8f>] __sk_free+0x1f/0x170
    [ 9883.042332]  [<ffffffff813f1101>] sk_free+0x21/0x30
    [ 9883.042332]  [<ffffffff814492a1>] tcp_close+0x141/0x3f0
    [ 9883.042332]  [<ffffffffa05134a4>] ? release_backend_socks+0x60/0x60 [tempesta_fw]
    [ 9883.042332]  [<ffffffff81470c57>] inet_release+0x77/0x80
    [ 9883.042332]  [<ffffffff813eb81f>] sock_release+0x1f/0x80
    [ 9883.042332]  [<ffffffffa0513483>] release_backend_socks+0x3f/0x60 [tempesta_fw]
    [ 9883.042332]  [<ffffffffa05136b9>] tfw_bconnd+0x215/0x21c [tempesta_fw]
    [ 9883.042332]  [<ffffffff81061020>] ? wake_up_bit+0x30/0x30
    [ 9883.042332]  [<ffffffff81060350>] kthread+0xc0/0xd0
    [ 9883.042332]  [<ffffffff81060290>] ? insert_kthread_work+0x40/0x40
    [ 9883.042332]  [<ffffffff814d939c>] ret_from_fork+0x7c/0xb0
    [ 9883.042332]  [<ffffffff81060290>] ? insert_kthread_work+0x40/0x40
    [ 9883.042332] Code: 55 48 89 e5 48 83 ec 18 48 89 7d e8 48 8b 45 e8 48 8b 80 20 02 00 00 48 89 45 f0 48 83 7d f0 00 0f 94 c0 0f b6 c0 48 85 c0 74 02 <0f> 0b 48 8b 45 f0 48 8b 40 18 48 89 45 f8 48 83 7d f8 00 0f 94
    [ 9883.042332] RIP  [<ffffffffa0512ab8>] tfw_destroy_server+0x30/0xdf [tempesta_fw]
    [ 9883.042332]  RSP <ffff880024f19d48>
    [ 9883.072752] ---[ end trace 1c7611e0e034c871 ]---
    [ 9883.073572] [tempesta]   spec cleanup: 'backend'
    [ 9883.074283] [tempesta]   mod_stop(): cache
    [ 9883.075035] [tempesta]   spec cleanup: 'cache_dir'
    [ 9883.077420] [tempesta] got state via sysctl: start
    [ 9883.078193] [tempesta]   reading configuration file...
    [ 9883.079141] [tempesta]   reading file: /root/tempesta/etc/tempesta_fw.conf
    [ 9883.079850] [tempesta]   file size: 2215 bytes
    [ 9883.080341] [tempesta]   read by offset: 0
    [ 9883.080875] [tempesta]   read by offset: 2215
    [ 9883.081325] [tempesta] starting all modules...
    [ 9883.081817] [tempesta]   parsing configuration and pushing it to modules...
    [ 9883.082494] [tempesta]   spec handle: 'listen'
    [ 9883.083073] [tempesta] ERROR: Can't parse IP address: '80'
    [ 9883.083696] [tempesta] ERROR: can't parse IP address: '80'
    [ 9883.084272] [tempesta] ERROR: configuration handler returned error: -22
    [ 9883.084992] [tempesta] ERROR: configuration parsing error:

    #   cache_size 1048576; # 1 GiB
    #
    # Default:
    #   cache_size 262144;  # 256 MiB

    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    [ 9883.087344] [tempesta] ERROR: can't parse configuration data
    [ 9883.087953] [tempesta] ERROR: failed to start modules

Slow HTTP prevention

req_con_limit/frang must be extended by detection and prevention Slow HTTP attack.

Server backend connection data is not cleared when the connection socket is closed

After a period of inactivity the socket that serves the server backend connection is closed. However the related Tempesta connection data is not cleared. That leads to a stale pointer to the connection data that no longer exists. Consequently, we get a kernel crash on first attempt to use that connection data, which happens on the very next http request.

Loading Tempesta kernel modules...
[   71.978352] [tempesta] Initializing Tempesta FW kernel module...
[   71.978949] [tempesta] init: cfg_if
[   71.979298] [tempesta] init: http
[   71.979616] [tempesta] init: server
[   71.979985] [tempesta] init: client
[   71.980347] [tempesta] init: session
[   71.980686] [tempesta] init: connection
[   71.981044] [tempesta]   register cfg: cache
[   71.981434] [tempesta]   register cfg: sock_backend
[   71.981877] [tempesta]   register cfg: sock_frontend
[   71.999827] [tempesta] Registering new scheduler: dummy
Starting Tempesta...
[   72.002657] [tempesta] got state via sysctl: start
[   72.003130] [tempesta]   reading configuration file...
[   72.003621] [tempesta]   reading file: /home/natsys-lab/tfw-run/tempesta/etc/tempesta_fw.conf
[   72.005235] [tempesta]   file size: 2204 bytes
[   72.005675] [tempesta]   read by offset: 0
[   72.006181] [tempesta]   read by offset: 2204
[   72.006668] [tempesta] starting all modules...
[   72.007093] [tempesta]   parsing configuration and pushing it to modules...
[   72.007762] [tempesta]   use default entry: 'cache off;'
[   72.008312] [tempesta]   spec handle: 'cache'
[   72.008739] [tempesta]   use default entry: 'cache_size 262144;'
[   72.009378] [tempesta]   spec handle: 'cache_size'
[   72.009983] [tempesta]   use default entry: 'cache_dir /opt/tempesta/cache;'
[   72.010878] [tempesta]   spec handle: 'cache_dir'
[   72.011444] [tempesta]   use default entry: 'backend 127.0.0.1:8080;'
[   72.012077] [tempesta]   spec handle: 'backend'
[   72.012509] [tempesta]   use default entry: 'listen 127.0.0.1:80;'
[   72.013088] [tempesta]   spec handle: 'listen'
[   72.013633] [tempesta]   created front-end socket: sk=ffff88003d7a0040
[   72.014246] [tempesta]   starting modules...
[   72.014653] [tempesta]   mod_start(): cache
[   72.015050] [tempesta]   mod_start(): sock_backend
[   72.015501] [tempesta]   Starting thread: tfw_bconnd
[   72.016035] [tempesta]   mod_start(): sock_frontend
[   72.016366] [tempesta]   start listening on socket: sk=ffff88003d7a0040
[   72.016817] [tempesta] modules are started
net.tempesta.state = start
done
done
Starting nginx: nginx.
[   76.016515] [tempesta] Connected to backend: 127.0.0.1:8080
[  136.077004] [sync_socket]   Close socket ffff88003c813280 (account=1)
[  136.079175] [tempesta]   Close socket ffff88003c813280, conn=ffff88003d55e548
[  136.081542] [tempesta]   Free msg:           (null)
[  136.083030] [tempesta]   Free connection: ffff88003d55e548
[  136.084197] [sync_socket]   free rcv skb ffff88003d638da8
[  275.983382] [tempesta]   New client socket ffff88003c812980 (state=1)
[  275.984126] [tempesta]   Link new msg ffff88003d5e8040 with connection ffff88003c881548
[  275.984758] [tempesta]   Add skb ffff88003d4e0de8 to message ffff88003d5e8040
[  275.985324] [tempesta]   Received 109 client data bytes (GET / HTTP/1.1
[  275.985324] User-Agent: Wget/1.13.4 (linux-gnu)
[  275.985324] Accept: */*
[  275.985324] Host: 127.0.0.1
[  275.985324] Connection: Keep-Alive
[  275.985324] 
[  275.985324] ) on socket (conn=ffff88003c881548)
[  275.987351] [tempesta]   enter FSM at state 0
[  275.987696] [tempesta]   parser: Req_0(0:0): c=0x47(G), r=-2
[  275.988163] [tempesta]   parser: Req_Method(1:0): c=0x47(G), r=-2
[  275.988643] [tempesta]   parser: Req_MUSpace(2:0): c=0x2f(/), r=-2
[  275.989130] [tempesta]   parser: Req_UriAbsPath(6:0): c=0x20( ), r=-2
[  275.989633] [tempesta]   parser: Req_HttpVer(7:0): c=0x48(H), r=-2
[  275.990116] [tempesta]   parser: Req_EoL(15:0): c=0xd(.), r=-2
[  275.990574] [tempesta]   parser: Req_EoL(15:0): c=0xa(.), r=-2
[  275.991030] [tempesta]   parser: Req_Hdr(16:0): c=0x55(U), r=-2
[  275.991497] [tempesta]   parser: Req_HdrOther(92:0): c=0x55(U), r=-2
[  275.992019] [tempesta]   store header w/ ptr=ffff88003d51b8b0 len=35 flags=0 id=3
[  275.992513] [tempesta]   parser: Req_Hdr(16:0): c=0x41(A), r=-2
[  275.992905] [tempesta]   parser: Req_HdrOther(92:0): c=0x41(A), r=-2
[  275.993324] [tempesta]   store header w/ ptr=ffff88003d51b8d5 len=11 flags=0 id=4
[  275.993817] [tempesta]   parser: Req_Hdr(16:0): c=0x48(H), r=-2
[  275.994211] [tempesta]   parser: RGen_LWS(10000:21): c=0x20( ), r=-2
[  275.994630] [tempesta]   parser: RGen_LWS(10000:21): c=0x31(1), r=-2
[  275.995049] [tempesta]   enter FSM at state 21
[  275.995346] [tempesta]   parser: Req_HdrHostV(21:0): c=0x31(1), r=-2
[  275.995765] [tempesta]   enter FSM at state 1
[  275.996064] [tempesta]   parser: Req_I_H(1:1): c=0x31(1), r=-2
[  275.996450] [tempesta]   parser: Req_I_H(1:1): c=0x32(2), r=-2
[  275.996836] [tempesta]   parser: Req_I_H(1:1): c=0x37(7), r=-2
[  275.997224] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[  275.997610] [tempesta]   parser: Req_I_H(1:1): c=0x30(0), r=-2
[  275.997996] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[  275.998383] [tempesta]   parser: Req_I_H(1:1): c=0x30(0), r=-2
[  275.998768] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[  275.999154] [tempesta]   parser: Req_I_H(1:1): c=0x31(1), r=-2
[  275.999542] [tempesta]   parser: Req_I_H(1:1): c=0xd(.), r=-2
[  275.999922] [tempesta]   parser: Req_I_H_EoL(3:1): c=0xd(.), r=-2
[  276.000337] [tempesta]   parser: Req_I_H_EoL(3:1): c=0xa(.), r=-2
[  276.000740] [tempesta]   parse header __req_parse_host: ret=11 plen=9 id=1
[  276.001193] [tempesta]   store header w/ ptr=ffff88003d51b8e2 len=15 flags=0 id=1
[  276.001684] [tempesta]   parser: Req_Hdr(16:0): c=0x43(C), r=-2
[  276.002074] [tempesta]   parser: Req_HdrC(22:0): c=0x6f(o), r=-2
[  276.002474] [tempesta]   parser: Req_HdrConnection(44:0): c=0x3a(:), r=-2
[  276.002923] [tempesta]   parser: RGen_LWS(10000:45): c=0x20( ), r=-2
[  276.003344] [tempesta]   parser: RGen_LWS(10000:45): c=0x4b(K), r=-2
[  276.003763] [tempesta]   enter FSM at state 45
[  276.004070] [tempesta]   parser: Req_HdrConnectionV(45:0): c=0x4b(K), r=-2
[  276.004523] [tempesta]   enter FSM at state 1
[  276.004813] [tempesta]   parser: I_Conn(1:1): c=0x4b(K), r=-2
[  276.005275] [tempesta]   parser: I_EoT(6:1): c=0xd(.), r=0
[  276.005692] [tempesta]   parser: I_EoL(7:1): c=0xd(.), r=0
[  276.006058] [tempesta]   parser: I_EoL(7:1): c=0xa(.), r=0
[  276.006422] [tempesta]   parser: Connection parsed: flags 0x2
[  276.006805] [tempesta]   parse header __parse_connection: ret=12 plen=10 id=0
[  276.007301] [tempesta]   store header w/ ptr=ffff88003d51b8f3 len=22 flags=0 id=0
[  276.007799] [tempesta]   parser: Req_Hdr(16:0): c=0xd(.), r=-2
[  276.008206] [tempesta]   parser: Req_HdrDone(93:0): c=0xa(.), r=-2
[  276.008617] [tempesta]   parse msg body: flags=0x2 content_length=0
[  276.009035] [tempesta]   request parsed: len=109 parsed=109 msg_len=109 res=0
[  276.009507] [tempesta]   GFSM return code 0
[  276.009791] [tempesta]   added X-Forwarded-For header: X-Forwarded-For: 127.0.0.1
[  276.009791] 
[  276.010393] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
[  276.010929] IP: [<ffffffffa0011bf8>] tfw_connection_send_srv+0x67/0xc6 [tempesta_fw]
[  276.011460] PGD 3dbaf067 PUD 3dbe8067 PMD 0 
[  276.011775] Oops: 0000 [#1] SMP 
[  276.012030] Modules linked in: tfw_sched_dummy(O) tempesta_fw(O) tempesta_db(O) sync_socket(O)
[  276.012200] CPU: 0 PID: 2287 Comm: wget Tainted: G           O 3.10.10-tempesta #7
[  276.012200] task: ffff88003d5890c0 ti: ffff88003c8de000 task.ti: ffff88003c8de000
[  276.012200] RIP: 0010:[<ffffffffa0011bf8>]  [<ffffffffa0011bf8>] tfw_connection_send_srv+0x67/0xc6 [tempesta_fw]
[  276.012200] RSP: 0018:ffff88003fc039e8  EFLAGS: 00010286
[  276.012200] RAX: 0000000000000000 RBX: ffff88003d4e0de8 RCX: 0000000000000000
[  276.012200] RDX: ffff88003c880400 RSI: 0000000000000200 RDI: ffff88003d58a490
[  276.012200] RBP: ffff88003fc03a00 R08: 0000000000000000 R09: 0000000000000000
[  276.012200] R10: 0000000000000001 R11: ffff88003fc035c6 R12: ffff88003d51b8a0
[  276.012200] R13: 000000000000006d R14: ffff88003c812980 R15: 0000000000000000
[  276.012200] FS:  00007f239b86f700(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[  276.012200] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  276.012200] CR2: 0000000000000020 CR3: 000000003d75a000 CR4: 00000000000006b0
[  276.012200] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  276.012200] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  276.012200] Stack:
[  276.012200]  ffff88003d5e8040 ffff88003d58a490 0000000000000000 ffff88003fc03a30
[  276.012200]  ffffffffa0015360 ffff88003d58a490 0000000000000000 ffff88003d5e8040
[  276.012200]  ffff88003d58a490 ffff88003fc03a78 ffffffffa000ddec ffff88003d58a490
[  276.012200] Call Trace:
[  276.012200]  <IRQ> 
[  276.012200]  [<ffffffffa0015360>] tfw_http_req_cache_cb+0x5d/0x5f [tempesta_fw]
[  276.012200]  [<ffffffffa000ddec>] tfw_cache_req_process+0x39/0x163 [tempesta_fw]
[  276.012200]  [<ffffffffa0015303>] ? tfw_http_adjust_resp+0x13/0x13 [tempesta_fw]
[  276.012200]  [<ffffffffa001561e>] tfw_http_req_process+0x2bc/0x32b [tempesta_fw]
[  276.012200]  [<ffffffffa00159f9>] tfw_http_msg_process+0x43/0x5e [tempesta_fw]
[  276.012200]  [<ffffffffa0012d0b>] tfw_gfsm_dispatch+0x41/0x43 [tempesta_fw]
[  276.012200]  [<ffffffffa0011e11>] tfw_connection_recv+0x88/0x8a [tempesta_fw]
[  276.012200]  [<ffffffffa000005b>] ss_tcp_process_proto_skb+0x5b/0x90 [sync_socket]
[  276.012200]  [<ffffffffa000020f>] ss_tcp_process_skb+0x17f/0x190 [sync_socket]
[  276.012200]  [<ffffffffa0000d8e>] ss_tcp_data_ready+0x10e/0x25e [sync_socket]
[  276.012200]  [<ffffffff8133425d>] tcp_data_queue+0x4dd/0xd10
[  276.012200]  [<ffffffff81337660>] tcp_rcv_established+0x560/0x620
[  276.012200]  [<ffffffff8134183c>] tcp_v4_do_rcv+0x1dc/0x4e0
[  276.012200]  [<ffffffff811eea42>] ? do_raw_spin_lock+0x82/0x140
[  276.012200]  [<ffffffff813433e7>] tcp_v4_rcv+0x437/0x8b0
[  276.012200]  [<ffffffff8131d7c5>] ip_local_deliver_finish+0xb5/0x150
[  276.012200]  [<ffffffff8131d750>] ? ip_local_deliver_finish+0x40/0x150
[  276.012200]  [<ffffffff8131df70>] ip_local_deliver+0x50/0x90
[  276.012200]  [<ffffffff8131dcde>] ip_rcv_finish+0x47e/0x580
[  276.012200]  [<ffffffff8131e26c>] ip_rcv+0x2bc/0x340
[  276.012200]  [<ffffffff812e6869>] __netif_receive_skb_core+0x469/0x590
[  276.012200]  [<ffffffff812e6463>] ? __netif_receive_skb_core+0x63/0x590
[  276.012200]  [<ffffffff8107258e>] ? mark_held_locks+0x10e/0x130
[  276.012200]  [<ffffffff812e69df>] __netif_receive_skb+0x4f/0x70
[  276.012200]  [<ffffffff812e855f>] process_backlog+0x7f/0x200
[  276.012200]  [<ffffffff812e8783>] net_rx_action+0xa3/0x1f0
[  276.012200]  [<ffffffff81036550>] __do_softirq+0xe0/0x1d0
[  276.012200]  [<ffffffff813c5afc>] call_softirq+0x1c/0x26
[  276.012200]  <EOI> 
[  276.012200]  [<ffffffff81003ef5>] do_softirq+0x65/0x100
[  276.012200]  [<ffffffff81322db4>] ? ip_finish_output+0x3f4/0x500
[  276.012200]  [<ffffffff81035dc0>] local_bh_enable+0xd0/0x120
[  276.012200]  [<ffffffff81322db4>] ip_finish_output+0x3f4/0x500
[  276.012200]  [<ffffffff81322c10>] ? ip_finish_output+0x250/0x500
[  276.012200]  [<ffffffff81323e35>] ip_output+0x95/0x110
[  276.012200]  [<ffffffff813233ee>] ip_local_out+0x5e/0x90
[  276.012200]  [<ffffffff813239ff>] ip_queue_xmit+0x38f/0x4a0
[  276.012200]  [<ffffffff81323670>] ? ip_build_and_send_pkt+0x250/0x250
[  276.012200]  [<ffffffff8133a7e2>] tcp_transmit_skb+0x792/0x820
[  276.012200]  [<ffffffff8133b1dd>] tcp_write_xmit+0x96d/0xb40
[  276.012200]  [<ffffffff8133b61d>] __tcp_push_pending_frames+0x2d/0xd0
[  276.012200]  [<ffffffff8132b171>] tcp_push+0x81/0x90
[  276.012200]  [<ffffffff8132fb03>] tcp_sendmsg+0xa23/0xc00
[  276.012200]  [<ffffffff813563ff>] inet_sendmsg+0xdf/0x100
[  276.012200]  [<ffffffff81356320>] ? inet_sendpage+0x130/0x130
[  276.012200]  [<ffffffff812d1232>] sock_aio_write+0x142/0x160
[  276.012200]  [<ffffffff810f9cc4>] ? core_sys_select+0x34/0x3c0
[  276.012200]  [<ffffffff810e6d61>] do_sync_write+0x81/0xa0
[  276.012200]  [<ffffffff810e75c5>] vfs_write+0xe5/0x1b0
[  276.012200]  [<ffffffff810e7b87>] SyS_write+0x47/0x90
[  276.012200]  [<ffffffff813c4729>] system_call_fastpath+0x16/0x1b
[  276.012200] Code: a0 b8 00 00 00 00 e8 36 a1 3a e1 b8 ff ff ff ff eb 76 48 8b 45 f0 be 00 02 00 00 48 89 c7 e8 73 fb ff ff 48 89 45 f8 48 8b 45 f8 <48> 8b 40 20 48 85 c0 0f 95 c0 0f b6 c0 48 85 c0 74 19 48 8b 45 
[  276.012200] RIP  [<ffffffffa0011bf8>] tfw_connection_send_srv+0x67/0xc6 [tempesta_fw]
[  276.012200]  RSP <ffff88003fc039e8>
[  276.012200] CR2: 0000000000000020
[  276.044634] ---[ end trace ccbde25cbefb1ae1 ]---
[  276.044946] Kernel panic - not syncing: Fatal exception in interrupt

cfg: hot configuration reloading

We need to change some configuration options while Tempesta FW is running.

For example, we want to add a new back-end server or tune DoS protection settings, but in the same time we don't want to stop serving client connections even for a short time.

Therefore:

The reloading feature should be added to the Configuration Framework. It should listen a "reload" event via sysctl, and then re-parse the configuration file and push the data to Tempesta moduels without stopping the whole system.
Perhaps we would not like to support the feature for all configuration options. Often it is better to have immutable configuration data in runtime and don't care about synchronization. Hence, the feature must be optional, and only some of configuration options will support it.
All existing code that requires the feature should be updated. At least, the following three modules should be updated:
- 'sock_backend' - need to change list of back-end servers without stopping the sytem
- 'sched_http' - need to change load-balancing rules
- 'req_conn_limit' - need to be able to tune limits on a running system

replace hash function in tfw_sched_hash.c

Replace the hash function in the tfw_sched_hash.c as soon as it will be ready (currently djb2 hash function is used).

A Connection could not be closed before it is scheduled

A NULL pointer dereference occurs when you connect to the Tempesta FW and close the TCP connection (send FIN from client) when the HTTP request is not yet parsed.

Scenario:

start Tempesta FW
wget http://remote-machine-address
Actually my wget hangs after this, so I'm just hitting Ctrl+C and it sends FIN to the Tempesta server.

Result:

[   43.851433] [tempesta]   New client socket ffff88003d704940 (state=1)
[   43.852547] [tempesta]   Link new msg ffff88003d564040 with connection ffff88003db99438
[   43.853760] [tempesta]   Add skb ffff88003db150c0 to message ffff88003d564040
[   43.854845] [tempesta]   Received 62 client data bytes (GET / HTTP/1.1
[   43.854845] User-Agent: Wget/1.13.4 (linux-gnu)
[   43.854845] Accept: *) on socket (conn=ffff88003db99438)
[   43.855206] [tempesta]   enter FSM at state 0
[   43.855206] [tempesta]   parser: Req_0(0:0): c=0x47(G), r=-2
[   43.855206] [tempesta]   parser: Req_Method(1:0): c=0x47(G), r=-2
[   43.855206] [tempesta]   parser: Req_MUSpace(2:0): c=0x2f(/), r=-2
[   43.855206] [tempesta]   parser: Req_UriAbsPath(6:0): c=0x20( ), r=-2
[   43.855206] [tempesta]   parser: Req_HttpVer(7:0): c=0x48(H), r=-2
[   43.855206] [tempesta]   parser: Req_EoL(15:0): c=0xd(.), r=-2
[   43.855206] [tempesta]   parser: Req_EoL(15:0): c=0xa(.), r=-2
[   43.855206] [tempesta]   parser: Req_Hdr(16:0): c=0x55(U), r=-2
[   43.855206] [tempesta]   parser: Req_HdrOther(76:0): c=0x55(U), r=-2
[   43.855206] [tempesta]   store header w/ ptr=ffff88003d738a92 len=34 flags=0
[   43.855206] [tempesta]   parser: Req_Hdr(16:0): c=0x41(A), r=-2
[   43.855206] [tempesta]   parser: Req_HdrOther(76:0): c=0x41(A), r=-2
[   43.855206] [tempesta]   store header w/ ptr=ffff88003d738ab7 len=62 flags=0
[   43.855206] [tempesta]   request parsed: len=62 parsed=116 res=-1
[   43.855206] [tempesta]   postpone skb ffff88003db150c0
[   43.855206] [tempesta]   Add skb ffff88003db150c0 to message ffff88003d564040
[   43.855206] [tempesta]   Received 51 client data bytes () on socket (conn=ffff88003db99438)
[   43.855206] [tempesta]   enter FSM at state 76
[   43.855206] [tempesta]   parser: Req_HdrOther(76:0): c=0x0(.), r=-2
[   43.855206] [tempesta]   store header w/ ptr=          (null) len=51 flags=0
[   43.855206] [tempesta]   request parsed: len=51 parsed=52 res=-1
[   43.855206] [tempesta]   postpone skb ffff88003db150c0
[   51.543472] [sync_socket]   Close socket ffff88003d704940 (account=1)
[   51.544509] [tempesta]   Close socket ffff88003d704940, conn=ffff88003db99438
[   51.545598] [tempesta]   free skb ffff88003db150c0: truesize=4864 sk=ffff88003d704940, destructor=ffffffff812e0f60 users=1
[   51.547410] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[   51.547410] IP: [<ffffffffa000ab73>] tfw_connection_close+0xa3/0x100 [tempesta_fw]
[   51.547410] PGD 3d891067 PUD 3d890067 PMD 0 
[   51.547410] Oops: 0000 [#1] SMP 
[   51.547410] Modules linked in: tfw_sched_rr(O) tempesta_fw(O) tempesta_db(O) sync_socket(O)
[   51.547410] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O 3.10.10-tempesta #2
[   51.547410] task: ffffffff815674c0 ti: ffffffff81556000 task.ti: ffffffff81556000
[   51.547410] RIP: 0010:[<ffffffffa000ab73>]  [<ffffffffa000ab73>] tfw_connection_close+0xa3/0x100 [tempesta_fw]
[   51.547410] RSP: 0018:ffff88003fc03ae0  EFLAGS: 00010202
[   51.547410] RAX: 0000000000000000 RBX: ffff88003db99438 RCX: 0000000000000000
[   51.547410] RDX: 0000000000000000 RSI: ffffffff81567a78 RDI: 0000000000000282
[   51.547410] RBP: ffff88003fc03af0 R08: 0000000000000001 R09: 0000000000000000
[   51.547410] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88003d704940
[   51.547410] R13: 0000000000000062 R14: ffff88003d738e00 R15: ffff88003d738e62
[   51.547410] FS:  0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[   51.547410] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   51.547410] CR2: 0000000000000008 CR3: 000000003d88d000 CR4: 00000000000006b0
[   51.547410] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   51.547410] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   51.547410] Stack:
[   51.547410]  ffff88003d704940 ffff88003d705118 ffff88003fc03b28 ffffffffa000053c
[   51.547410]  ffffffff81577600 ffff88003d704940 ffff88003d705118 0000000000000062
[   51.547410]  ffff88003d738e00 ffff88003fc03b48 ffffffffa0000bd8 ffff88003d704940
[   51.547410] Call Trace:
[   51.547410]  <IRQ> 
[   51.547410]  [<ffffffffa000053c>] ss_do_close+0x6c/0x5f0 [sync_socket]
[   51.547410]  [<ffffffffa0000bd8>] ss_tcp_state_change+0xc8/0x160 [sync_socket]
[   51.547410]  [<ffffffff81340779>] tcp_fin+0x159/0x1d0
[   51.547410]  [<ffffffff81341bf8>] tcp_data_queue+0x568/0xd90
[   51.547410]  [<ffffffff81344ed7>] tcp_rcv_established+0x267/0x670
[   51.547410]  [<ffffffff813504ec>] tcp_v4_do_rcv+0x19c/0x660
[   51.547410]  [<ffffffff811f6eb3>] ? do_raw_spin_lock+0x53/0x160
[   51.547410]  [<ffffffff8135195a>] tcp_v4_rcv+0x6aa/0x840
[   51.547410]  [<ffffffff8132a726>] ip_local_deliver_finish+0x86/0x140
[   51.547410]  [<ffffffff8132a6bd>] ? ip_local_deliver_finish+0x1d/0x140
[   51.547410]  [<ffffffff8132af32>] ip_local_deliver+0x42/0x80
[   51.547410]  [<ffffffff8132a88b>] ip_rcv_finish+0xab/0x5e0
[   51.547410]  [<ffffffff8132b176>] ip_rcv+0x206/0x330
[   51.547410]  [<ffffffff812f3d13>] __netif_receive_skb_core+0x1b3/0x660
[   51.547410]  [<ffffffff812f3ba7>] ? __netif_receive_skb_core+0x47/0x660
[   51.547410]  [<ffffffff812f41d1>] __netif_receive_skb+0x11/0x70
[   51.547410]  [<ffffffff812f4c2e>] netif_receive_skb+0x1e/0x110
[   51.547410]  [<ffffffff812e8520>] ? __netdev_alloc_skb+0x80/0xd0
[   51.547410]  [<ffffffff8127c068>] virtnet_poll+0x538/0x7b0
[   51.547410]  [<ffffffff812f5f7a>] net_rx_action+0x15a/0x210
[   51.547410]  [<ffffffff81038c8a>] __do_softirq+0x12a/0x230
[   51.547410]  [<ffffffff81038fa5>] irq_exit+0xd5/0xf0
[   51.547410]  [<ffffffff81003ede>] do_IRQ+0x4e/0xb0
[   51.547410]  [<ffffffff813ce232>] common_interrupt+0x72/0x72
[   51.547410]  <EOI> 
[   51.547410]  [<ffffffff810240f6>] ? native_safe_halt+0x6/0x10
[   51.547410]  [<ffffffff8107646d>] ? trace_hardirqs_on+0xd/0x10
[   51.547410]  [<ffffffff8100ae98>] default_idle+0x18/0x30
[   51.547410]  [<ffffffff8100b7aa>] arch_cpu_idle+0xa/0x10
[   51.547410]  [<ffffffff810698e0>] cpu_startup_entry+0x130/0x230
[   51.547410]  [<ffffffff813c3d3a>] rest_init+0xca/0xd0
[   51.547410]  [<ffffffff815dfc8a>] start_kernel+0x356/0x363
[   51.547410]  [<ffffffff815df120>] ? early_idt_handlers+0x120/0x120
[   51.547410]  [<ffffffff815df444>] x86_64_start_reservations+0x2a/0x2c
[   51.547410]  [<ffffffff815df535>] x86_64_start_kernel+0xef/0xfc
[   51.547410] Code: 00 48 8b 3d b0 4e 02 00 48 89 de e8 78 2e 0d e1 31 c0 49 c7 84 24 90 03 00 00 00 00 00 00 5b 41 5c 5d c3 0f 1f 44 00 00 48 8b 00 <48> 8b 40 08 48 8b 80 90 03 00 00 eb b8 48 89 fa 48 89 d9 48 c7 
[   51.547410] RIP  [<ffffffffa000ab73>] tfw_connection_close+0xa3/0x100 [tempesta_fw]
[   51.547410]  RSP <ffff88003fc03ae0>
[   51.547410] CR2: 0000000000000008
[   51.547410] ---[ end trace 2e62b0630e43dc00 ]---
[   51.547410] Kernel panic - not syncing: Fatal exception in interrupt

Actually the NULL pointer happens in an inlined function, so the full stack trace would look like:

tfw_sess_conn()
tfw_connection_peer()
tfw_connection_free()
tfw_connection_close()
ss_do_close()
ss_tcp_state_change()
tcp_fin()
...

The tfw_connection_peer() tries to find a server connection for the given client connection. But the problem is there is no server yet associated with the client because the message is not scheduled yet at this point (HTTP parser is not finished yet). So the sess->srv is NULL.

The server is obtained only to make peer_conn->sess = NULL; in tfw_connection_free(). If no session is yet established, then the field should already be NULL, so I think it may be fixed by simple NULL checks here.

NULL pointer dereference in skb_dequeue() in tfw_http_msg_free()

scenario:

compile and run Tempesta as always
do wget 192.168.1.246:80 (send HTTP GET from a remote machine)
the wget hangs, hit Ctrl+C, so the ouptut looks like this:

wget 192.168.1.246   
--2014-09-10 19:14:48--  http://192.168.1.246/
Connecting to 192.168.1.246:80... connected.
HTTP request sent, awaiting response... ^C

And then the kernel prints the Oops message listed below.

Starting Tempesta...
[tempesta] Start Tempesta
[tempesta] Open 1 backend sockets
[tempesta] ERROR: Can't connect to back-end server 127.0.0.1:8080 (-111)
[tempesta] Open 1 listening sockets
[tempesta]   Created listening socket ffff88003e3ce840
[tempesta] Close 0 backend sockets
[tempesta] Open 2 backend sockets
[tempesta] ERROR: Can't connect to back-end server 127.0.0.1:8080 (-111)
[tempesta] ERROR: Can't connect to back-end server 127.0.0.1:8081 (-111)
net.tempesta.backend = 127.0.0.1:8080 127.0.0.1:8081
done
[tempesta]   New client socket ffff88003e3ce000 (state=1)
[tempesta]   Link new msg ffff88003def3040 with connection ffff88003df88688
[tempesta]   received 62 client data bytes (GET / HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: *) on socket (conn=ffff88003df88688)
[tempesta]   enter FSM at state 0
[tempesta]   parser: Req_0(0:0): c=0x47(G)
[tempesta]   parser: Req_Method(1:0): c=0x47(G)
[tempesta]   parser: Req_MUSpace(2:0): c=0x2f(/)
[tempesta]   parser: Req_UriAbsPath(6:0): c=0x20( )
[tempesta]   parser: Req_HttpVer(7:0): c=0x48(H)
[tempesta]   parser: Req_EoL(15:0): c=0xd(.)
[tempesta]   parser: Req_EoL(15:0): c=0xa(.)
[tempesta]   parser: Req_Hdr(16:0): c=0x55(U)
[tempesta]   parser: Req_HdrOther(76:0): c=0x55(U)
[tempesta]   store header w/ ptr=ffff88003d7a0a92 len=34 flags=0
[tempesta]   parser: Req_Hdr(16:0): c=0x41(A)
[tempesta]   parser: Req_HdrOther(76:0): c=0x41(A)
[tempesta]   store header w/ ptr=ffff88003d7a0ab7 len=62 flags=0
[tempesta]   request parsed: len=62 parsed=116 res=-1
[tempesta]   postpone skb ffff88003df88cc0
[tempesta]   received 51 client data bytes () on socket (conn=ffff88003df88688)
[tempesta]   enter FSM at state 76
[tempesta]   parser: Req_HdrOther(76:0): c=0x0(.)
[tempesta]   store header w/ ptr=          (null) len=51 flags=0
[tempesta]   request parsed: len=51 parsed=52 res=-1
[tempesta]   postpone skb ffff88003df88cc0
[sync_socket]   Close socket ffff88003e3ce000 (account=1)
[tempesta]   Close socket ffff88003e3ce000, conn=ffff88003df88688
[tempesta]   free skb ffff88003df88cc0: truesize=4864 sk=ffff88003e3ce000, destructor=ffffffff81294b18
BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff8129a338>] skb_dequeue+0x4b/0x63
PGD 3e350067 PUD 3e34f067 PMD 0 
Oops: 0002 [#1] SMP 

Entering kdb (current=0xffffffff815074c0, pid 0) on processor 0 Oops: (null)
due to oops @ 0xffffffff8129a338
�dCPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O 3.10.10-tempesta #62
�dtask: ffffffff815074c0 ti: ffffffff814f6000 task.ti: ffffffff814f6000
�dRIP: 0010:[<ffffffff8129a338>]  [<ffffffff8129a338>] skb_dequeue+0x4b/0x63
�dRSP: 0018:ffff88003fc03b28  EFLAGS: 00010002
�dRAX: 0000000000000286 RBX: ffff88003df88cc0 RCX: ffff88000660006c
�dRDX: 0000000000000000 RSI: 0000000000000286 RDI: ffff88003def32d4
�dRBP: ffff88003fc03b40 R08: fffffffffffffffe R09: ffffffff8154e4e0
�dR10: ffff88003e198860 R11: ffffffff81723dd8 R12: ffff88003def32c0
�dR13: ffff88003def32d4 R14: 0000000000000000 R15: 0000000000002f94
�dFS:  0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
�dCS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
�dCR2: 0000000000000000 CR3: 000000003e34d000 CR4: 00000000000006b0
�dDR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
�dDR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
�dStack:
 ffff88003df88cc0 ffff88003def3040 ffff88003d7a0c62 ffff88003fc03b60
 ffffffffa0019502 ffff88003e3ce000 ffff88003df88688 ffff88003fc03b70
 ffffffffa0018f4a ffff88003fc03b90 ffffffffa0017841 ffff88003e3ce000
�dCall Trace:
�d <IRQ> �d [<ffffffffa0019502>] tfw_http_msg_free+0x2e/0x84 [tempesta_fw]
�d [<ffffffffa0018f4a>] tfw_http_conn_destruct+0xd/0xf [tempesta_fw]
�d [<ffffffffa0017841>] tfw_connection_close+0x5b/0x7f [tempesta_fw]
�d [<ffffffffa00004d9>] ss_do_close+0x6a/0x4f5 [sync_socket]
�d [<ffffffffa0000ca6>] ss_tcp_state_change+0xcd/0xd8 [sync_socket]
�d [<ffffffff812e791e>] tcp_fin+0x12c/0x191
�d [<ffffffff812e8c0a>] tcp_data_queue+0x221/0x406
�d [<ffffffff812eb478>] tcp_rcv_established+0x483/0x503
�d [<ffffffff812f49dc>] tcp_v4_do_rcv+0xd1/0x25a
�d [<ffffffff812f6092>] tcp_v4_rcv+0x299/0x508
�d [<ffffffff812c45cc>] ? nf_hook_slow+0x6a/0x114
�d [<ffffffff812d47c4>] ? inet_del_offload+0x36/0x36

The error happens on the second iteration of the while loop in the tfw_http_msg_free(). I know that because the TFW_DBG("free skb") is printed to the console.

It seems on the first iteration we are doing the kfree_skb(skb) in the context of processing this same skb (from the callback) and then on the second iteration we are referencing TfwHttpMsg which is stored inside the free'd skb.

Possible recursive locking

This message is issued by the kernel when an SKB is moved from the receive queue to the send queue as we do in the Synchronous Sockets paradigm.

[   16.946924] =============================================
[   16.946924] [ INFO: possible recursive locking detected ]
[   16.946924] 3.10.10-tempesta #8 Tainted: G           O
[   16.946924] ---------------------------------------------
[   16.946924] swapper/0/0 is trying to acquire lock:
[   16.946924]  (slock-AF_INET/1){+.-...}, at: [<ffffffffa00002a8>] ss_send+0x28/0x160 [sync_socket]
[   16.946924] 
[   16.946924] but task is already holding lock:
[   16.946924]  (slock-AF_INET/1){+.-...}, at: [<ffffffff813433bc>] tcp_v4_rcv+0x40c/0x8b0
[   16.946924] 
[   16.946924] other info that might help us debug this:
[   16.946924]  Possible unsafe locking scenario:
[   16.946924] 
[   16.946924]        CPU0
[   16.946924]        ----
[   16.946924]   lock(slock-AF_INET/1);
[   16.946924]   lock(slock-AF_INET/1);
[   16.946924] 
[   16.946924]  *** DEADLOCK ***
[   16.946924] 
[   16.946924]  May be due to missing lock nesting notation
[   16.946924] 
[   16.946924] 3 locks held by swapper/0/0:
[   16.946924]  #0:  (rcu_read_lock){.+.+..}, at: [<ffffffff812e6463>] __netif_receive_skb_core+0x63/0x590
[   16.946924]  #1:  (rcu_read_lock){.+.+..}, at: [<ffffffff8131d750>] ip_local_deliver_finish+0x40/0x150
[   16.946924]  #2:  (slock-AF_INET/1){+.-...}, at: [<ffffffff813433bc>] tcp_v4_rcv+0x40c/0x8b0
[   16.946924] 
[   16.946924] stack backtrace:
[   16.946924] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O 3.10.10-tempesta #8
[   16.946924]  ffffffff8155d440 ffff88003fc03828 ffffffff813bf35d ffff88003fc03898
[   16.946924]  ffffffff81073b96 ffffffff81072787 0000000000000007 00000000ffffffff
[   16.946924]  ffff880000000000 00000b005849e24f ffffffff00000000 ffffffff816f6640
[   16.946924] Call Trace:
[   16.946924]  <IRQ>  [<ffffffff813bf35d>] dump_stack+0x19/0x1b
[   16.946924]  [<ffffffff81073b96>] __lock_acquire+0xfa6/0x1240
[   16.946924]  [<ffffffff81072787>] ? trace_hardirqs_on_caller+0x1d7/0x210
[   16.946924]  [<ffffffff81074658>] lock_acquire+0x98/0xd0
[   16.946924]  [<ffffffffa00002a8>] ? ss_send+0x28/0x160 [sync_socket]
[   16.946924]  [<ffffffff813c30ca>] _raw_spin_lock_nested+0x3a/0x50
[   16.946924]  [<ffffffffa00002a8>] ? ss_send+0x28/0x160 [sync_socket]
[   16.946924]  [<ffffffffa00002a8>] ss_send+0x28/0x160 [sync_socket]
[   16.946924]  [<ffffffffa0049413>] ? tfw_sched_get_srv+0x49/0x4b [tempesta_fw]
[   16.946924]  [<ffffffffa0011c8d>] tfw_connection_send_srv+0xbf/0xc6 [tempesta_fw]
[   16.946924]  [<ffffffffa00153a2>] tfw_http_req_cache_cb+0x5d/0x5f [tempesta_fw]
[   16.946924]  [<ffffffffa000ddec>] tfw_cache_req_process+0x39/0x163 [tempesta_fw]
[   16.946924]  [<ffffffffa0015345>] ? tfw_http_adjust_resp+0x13/0x13 [tempesta_fw]
[   16.946924]  [<ffffffffa0015660>] tfw_http_req_process+0x2bc/0x32b [tempesta_fw]
[   16.946924]  [<ffffffffa0015a3b>] tfw_http_msg_process+0x43/0x5e [tempesta_fw]
[   16.946924]  [<ffffffffa0012d67>] tfw_gfsm_dispatch+0x41/0x43 [tempesta_fw]
[   16.946924]  [<ffffffffa0011e4e>] tfw_connection_recv+0x88/0x8a [tempesta_fw]
[   16.946924]  [<ffffffffa000005b>] ss_tcp_process_proto_skb+0x5b/0x90 [sync_socket]
[   16.946924]  [<ffffffffa0000167>] ss_tcp_process_skb+0xd7/0x170 [sync_socket]
[   16.946924]  [<ffffffffa0000eae>] ss_tcp_data_ready+0x10e/0x25e [sync_socket]
[   16.946924]  [<ffffffff8133425d>] tcp_data_queue+0x4dd/0xd10
[   16.946924]  [<ffffffff81337660>] tcp_rcv_established+0x560/0x620
[   16.946924]  [<ffffffff8134183c>] tcp_v4_do_rcv+0x1dc/0x4e0
[   16.946924]  [<ffffffff811eea42>] ? do_raw_spin_lock+0x82/0x140
[   16.946924]  [<ffffffff813433e7>] tcp_v4_rcv+0x437/0x8b0
[   16.946924]  [<ffffffff8131d7c5>] ip_local_deliver_finish+0xb5/0x150
[   16.946924]  [<ffffffff8131d750>] ? ip_local_deliver_finish+0x40/0x150
[   16.946924]  [<ffffffff8131df70>] ip_local_deliver+0x50/0x90
[   16.946924]  [<ffffffff8131dcde>] ip_rcv_finish+0x47e/0x580
[   16.946924]  [<ffffffff8131e26c>] ip_rcv+0x2bc/0x340
[   16.946924]  [<ffffffff812e6869>] __netif_receive_skb_core+0x469/0x590
[   16.946924]  [<ffffffff812e6463>] ? __netif_receive_skb_core+0x63/0x590
[   16.946924]  [<ffffffff812e69df>] __netif_receive_skb+0x4f/0x70
[   16.946924]  [<ffffffff812e7543>] netif_receive_skb+0xb3/0x140
[   16.946924]  [<ffffffff812d9be9>] ? __netdev_alloc_skb+0x49/0xb0
[   16.946924]  [<ffffffff81270fd4>] virtnet_poll+0x624/0x770
[   16.946924]  [<ffffffff812e8783>] net_rx_action+0xa3/0x1f0
[   16.946924]  [<ffffffff81036550>] __do_softirq+0xe0/0x1d0
[   16.946924]  [<ffffffff810367c9>] irq_exit+0x69/0xc0
[   16.946924]  [<ffffffff81003c5d>] do_IRQ+0x9d/0xc0
[   16.946924]  [<ffffffff813c3df2>] common_interrupt+0x72/0x72
[   16.946924]  <EOI>  [<ffffffff81023626>] ? native_safe_halt+0x6/0x10
[   16.946924]  [<ffffffff8100a8dd>] default_idle+0x1d/0x30
[   16.946924]  [<ffffffff8100b26a>] arch_cpu_idle+0xa/0x10
[   16.946924]  [<ffffffff810664ac>] cpu_startup_entry+0x15c/0x200
[   16.946924]  [<ffffffff813b79ad>] rest_init+0xcd/0xe0
[   16.946924]  [<ffffffff813b78e0>] ? csum_partial_copy_generic+0x170/0x170
[   16.946924]  [<ffffffff815d1da0>] start_kernel+0x34d/0x358
[   16.946924]  [<ffffffff815d185c>] ? repair_env_string+0x5e/0x5e
[   16.946924]  [<ffffffff815d1120>] ? early_idt_handlers+0x120/0x120
[   16.946924]  [<ffffffff815d155b>] x86_64_start_reservations+0x2a/0x2c
[   16.946924]  [<ffffffff815d164c>] x86_64_start_kernel+0xef/0xfc

NULL pointer dereference in tfw_connection_new()

scenario:

compile and run tempesta as usual
wget http://192.168.1.246:80 (access the VM from the host system)

Hardly reproducible, happens not often, perhaps 1 out of 10 times.
I've never seen that when doing wget localhost, I only saw that when accessed the VM from the host system, but because of the non-deterministic occurrence I can't say that it is an invariant.

No idea why that may happen.

The only thing which is dereferenced in the tfw_connection_new_upcall() is the struct sock *sk parameter. But has to be not NULL because it is already dereferenced in the parent function: ss_tcp_state_change(). Perhaps GCC performs some function inlining and the error is in some child function.

Also, I'm working on a multi-processor VM, so perhaps the socket is actually destroyed and NULL'ed while the callback is executing, but I doubt that because I saw locking surrounding sk_state_change() calls.

Starting Tempesta...
[tempesta] Start Tempesta
[tempesta] Open 1 backend sockets
[tempesta] ERROR: Can't connect to back-end server 127.0.0.1:8080 (-111)
[tempesta] Open 1 listening sockets
[tempesta]   Created listening socket ffff88003e3fe840
[tempesta] Close 0 backend sockets
[tempesta] Open 2 backend sockets
[tempesta] ERROR: Can't connect to back-end server 127.0.0.1:8080 (-111)
[tempesta] ERROR: Can't connect to back-end server 127.0.0.1:8081 (-111)
net.tempesta.backend = 127.0.0.1:8080 127.0.0.1:8081
done
BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffffa0017a69>] tfw_connection_new+0xa0/0xb5 [tempesta_fw]
PGD 3e34e067 PUD 3e34d067 PMD 0 
Oops: 0000 [#1] SMP 

Entering kdb (current=0xffffffff815074c0, pid 0) on processor 0 Oops: (null)
due to oops @ 0xffffffffa0017a69
�dCPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O 3.10.10-tempesta #62
�dtask: ffffffff815074c0 ti: ffffffff814f6000 task.ti: ffffffff814f6000
�dRIP: 0010:[<ffffffffa0017a69>]  [<ffffffffa0017a69>] tfw_connection_new+0xa0/0xb5 [tempesta_fw]
�dRSP: 0018:ffff88003fc03bd0  EFLAGS: 00010202
�dRAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000008020
�dRDX: 0000000000000001 RSI: 0000000000000038 RDI: ffff88003bc69798
�dRBP: ffff88003fc03be8 R08: ffff88003d5490a8 R09: ffff88003bc69798
�dR10: ffff88003bc23750 R11: ffff88003bc23628 R12: ffff88003e3744c0
�dR13: ffffffffa00176e7 R14: 0000000000000000 R15: 0000000000007193
�dFS:  0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
�dCS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
�dCR2: 0000000000000000 CR3: 000000003e342000 CR4: 00000000000006b0
�dDR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
�dDR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
�dStack:
 ffff88003e3744c0 ffff88003d6b6c00 ffff88003bc28a62 ffff88003fc03c00
 ffffffffa0017add ffff88003e3744c0 ffff88003fc03c28 ffffffffa0000c1b
 ffff88003e3744c0 ffff88003bc696c0 ffff88003bc28a62 ffff88003fc03c60

Stack traceback for pid 0
0xffffffff815074c0        0        0  1    0   R  0xffffffff81507800 *swapper/0
 ffff88003fc03bd0 0000000000000018 ffff88003e3744c0 ffff88003d6b6c00
 ffff88003bc28a62 ffff88003fc03c00 ffffffffa0017add ffff88003e3744c0
 ffff88003fc03c28 ffffffffa0000c1b ffff88003e3744c0 ffff88003bc696c0
Call Trace:
 <IRQ>  [<ffffffffa0017add>] ? tfw_connection_new_upcall+0x5f/0xa0 [tempesta_fw]
 [<ffffffffa0000c1b>] ? ss_tcp_state_change+0x42/0xd8 [sync_socket]
 [<ffffffff812ebeda>] ? tcp_rcv_state_process+0x260/0x5c8
 [<ffffffff812f6bd4>] ? tcp_child_process+0x3b/0x102
 [<ffffffff812f4ab6>] ? tcp_v4_do_rcv+0x1ab/0x25a
 [<ffffffff812f6092>] ? tcp_v4_rcv+0x299/0x508
 [<ffffffff812d4823>] ? ip_local_deliver_finish+0x5f/0xc6
 [<ffffffff812d4cd8>] ? ip_local_deliver+0x82/0x90
 [<ffffffff812d4b53>] ? ip_rcv_finish+0x2c9/0x2df
 [<ffffffff812d4fac>] ? ip_rcv+0x2c6/0x2fb
 [<ffffffff812a5ad5>] ? __netif_receive_skb_core+0x414/0x454
 [<ffffffff812a5b84>] ? __netif_receive_skb+0x6f/0x72
 [<ffffffff812a643f>] ? netif_receive_skb+0x87/0x8e
 [<ffffffff81235cc1>] ? receive_buf+0x2da/0x2fb
 [<ffffffff81237380>] ? virtnet_poll+0x40/0x10d
 [<ffffffff812a6e5e>] ? net_rx_action+0xb0/0x1df

Many HTTP requests are blocked by Tempesta

When an SKB consists of multiple parts/fragments Tempesta processes it fragment by fragment, passing each fragment separately to the HTTP parser. The HTTP parser keeps track of the current state of HTTP message parsing. However the code that passes the fragments to the parser is broken. The parser does not receive the complete HTTP message and reports a parsing error, which leads to be message being blocked.

Loading Tempesta kernel modules...
[   13.930348] [tempesta] Initializing Tempesta FW kernel module...
[   13.930891] [tempesta] init: cfg_if
[   13.931200] [tempesta] init: http
[   13.931484] [tempesta] init: server
[   13.931797] [tempesta] init: client
[   13.932138] [tempesta] init: session
[   13.932437] [tempesta] init: connection
[   13.932762] [tempesta]   register cfg: cache
[   13.933103] [tempesta]   register cfg: sock_backend
[   13.933496] [tempesta]   register cfg: sock_frontend
[   13.951327] [tempesta] Registering new scheduler: dummy
Starting Tempesta...
[   13.954693] [tempesta] got state via sysctl: start
[   13.955261] [tempesta]   reading configuration file...
[   13.955846] [tempesta]   reading file: /home/natsys-lab/tfw-run/tempesta/etc/tempesta_fw.conf
[   13.957494] [tempesta]   file size: 2204 bytes
[   13.958008] [tempesta]   read by offset: 0
[   13.960110] [tempesta]   read by offset: 2204
[   13.960690] [tempesta] starting all modules...
[   13.961180] [tempesta]   parsing configuration and pushing it to modules...
[   13.961993] [tempesta]   use default entry: 'cache off;'
[   13.962671] [tempesta]   spec handle: 'cache'
[   13.963279] [tempesta]   use default entry: 'cache_size 262144;'
[   13.964238] [tempesta]   spec handle: 'cache_size'
[   13.964785] [tempesta]   use default entry: 'cache_dir /opt/tempesta/cache;'
[   13.965509] [tempesta]   spec handle: 'cache_dir'
[   13.965986] [tempesta]   use default entry: 'backend 127.0.0.1:8080;'
[   13.966636] [tempesta]   spec handle: 'backend'
[   13.967107] [tempesta]   use default entry: 'listen 0.0.0.0:80;'
[   13.967718] [tempesta]   spec handle: 'listen'
[   13.968259] [tempesta]   created front-end socket: sk=ffff88003cc0c040
[   13.968752] [tempesta]   starting modules...
[   13.969073] [tempesta]   mod_start(): cache
[   13.969388] [tempesta]   mod_start(): sock_backend
[   13.969749] [tempesta]   Starting thread: tfw_bconnd
[   13.970169] [tempesta]   mod_start(): sock_frontend
[   13.970539] [tempesta]   start listening on socket: sk=ffff88003cc0c040
[   13.971042] [tempesta] modules are started
net.tempesta.state = start
done
done
Starting nginx: nginx.
[   17.973654] [tempesta] Connected to backend: 127.0.0.1:8080
[   22.517164] [tempesta]   New client socket ffff88003daf2080 (state=1)
[   22.519380] [tempesta]   Link new msg ffff88003da7e040 with connection ffff88003d923500
[   22.521026] [tempesta]   Add skb ffff88003db603c0 to message ffff88003da7e040
[   22.521026] [tempesta]   Received 62 client data bytes (GET / HTTP/1.1
[   22.521026] User-Agent: Wget/1.15 (linux-gnu)
[   22.521026] Accept: */*) on socket (conn=ffff88003d923500)
[   22.521026] [tempesta]   enter FSM at state 0
[   22.521026] [tempesta]   parser: Req_0(0:0): c=0x47(G), r=-2
[   22.521026] [tempesta]   parser: Req_Method(1:0): c=0x47(G), r=-2
[   22.521026] [tempesta]   parser: Req_MUSpace(2:0): c=0x2f(/), r=-2
[   22.521026] [tempesta]   parser: Req_UriAbsPath(6:0): c=0x20( ), r=-2
[   22.521026] [tempesta]   parser: Req_HttpVer(7:0): c=0x48(H), r=-2
[   22.521026] [tempesta]   parser: Req_EoL(15:0): c=0xd(.), r=-2
[   22.521026] [tempesta]   parser: Req_EoL(15:0): c=0xa(.), r=-2
[   22.521026] [tempesta]   parser: Req_Hdr(16:0): c=0x55(U), r=-2
[   22.521026] [tempesta]   parser: Req_HdrOther(92:0): c=0x55(U), r=-2
[   22.521026] [tempesta]   store header w/ ptr=ffff88003ccdaa92 len=33 flags=0 id=3
[   22.521026] [tempesta]   parser: Req_Hdr(16:0): c=0x41(A), r=-2
[   22.521026] [tempesta]   parser: Req_HdrOther(92:0): c=0x41(A), r=-2
[   22.521026] [tempesta]   store header w/ ptr=ffff88003ccdaab5 len=11 flags=0 id=4
[   22.521026] [tempesta]   request parsed: len=62 parsed=62 msg_len=62 res=-1
[   22.521026] [tempesta]   postpone skb ffff88003db603c0
[   22.521026] [tempesta]   Add skb ffff88003db603c0 to message ffff88003da7e040
[   22.521026] [tempesta]   Received 50 client data bytes () on socket (conn=ffff88003d923500)
[   22.521026] [tempesta]   enter FSM at state 92
[   22.521026] [tempesta]   parser: Req_HdrOther(92:0): c=0x1(.), r=-2
[   22.521026] [tempesta]   store header w/ ptr=          (null) len=40 flags=0 id=4
[   22.521026] [tempesta]   parser: Req_Hdr(16:0): c=0x1(.), r=-2
[   22.521026] [tempesta]   request parsed: len=50 parsed=0 msg_len=0 res=-2
[   22.521026] [tempesta]   Block bad HTTP request
[   22.521026] [sync_socket] Warning: can't process app data on socket ffff88003daf2080
[   22.521026] [sync_socket]   Close socket ffff88003daf2080 (account=1)
[   22.521026] [tempesta]   Close socket ffff88003daf2080, conn=ffff88003d923500
[   22.521026] [tempesta]   Free msg: ffff88003da7e040
[   22.521026] [tempesta]   free skb ffff88003db603c0: truesize=4864 sk=ffff88003daf2080, destructor=ffffffff812d2ea0 users=1
[   22.521026] [tempesta]   Free connection: ffff88003d923500
[   22.521026] [tempesta]   Free session: ffff88003da3f490
[   23.539579] [tempesta]   New client socket ffff88003db7d2c0 (state=1)
[   23.541371] [tempesta]   Link new msg ffff88003e344040 with connection ffff88003d923500
[   23.543410] [tempesta]   Add skb ffff88003db600c0 to message ffff88003e344040
[   23.543471] [tempesta]   Received 62 client data bytes (GET / HTTP/1.1
[   23.543471] User-Agent: Wget/1.15 (linux-gnu)
[   23.543471] Accept: */*) on socket (conn=ffff88003d923500)
[   23.543471] [tempesta]   enter FSM at state 0
[   23.543471] [tempesta]   parser: Req_0(0:0): c=0x47(G), r=-2
[   23.543471] [tempesta]   parser: Req_Method(1:0): c=0x47(G), r=-2
[   23.543471] [tempesta]   parser: Req_MUSpace(2:0): c=0x2f(/), r=-2
[   23.543471] [tempesta]   parser: Req_UriAbsPath(6:0): c=0x20( ), r=-2
[   23.543471] [tempesta]   parser: Req_HttpVer(7:0): c=0x48(H), r=-2
[   23.543471] [tempesta]   parser: Req_EoL(15:0): c=0xd(.), r=-2
[   23.543471] [tempesta]   parser: Req_EoL(15:0): c=0xa(.), r=-2
[   23.543471] [tempesta]   parser: Req_Hdr(16:0): c=0x55(U), r=-2
[   23.543471] [tempesta]   parser: Req_HdrOther(92:0): c=0x55(U), r=-2
[   23.543471] [tempesta]   store header w/ ptr=ffff88003ccdb292 len=33 flags=0 id=3
[   23.543471] [tempesta]   parser: Req_Hdr(16:0): c=0x41(A), r=-2
[   23.543471] [tempesta]   parser: Req_HdrOther(92:0): c=0x41(A), r=-2
[   23.543471] [tempesta]   store header w/ ptr=ffff88003ccdb2b5 len=11 flags=0 id=4
[   23.543471] [tempesta]   request parsed: len=62 parsed=62 msg_len=62 res=-1
[   23.543471] [tempesta]   postpone skb ffff88003db600c0
[   23.543471] [tempesta]   Add skb ffff88003db600c0 to message ffff88003e344040
[   23.543471] [tempesta]   Received 50 client data bytes () on socket (conn=ffff88003d923500)
[   23.543471] [tempesta]   enter FSM at state 92
[   23.543471] [tempesta]   parser: Req_HdrOther(92:0): c=0x1(.), r=-2
[   23.543471] [tempesta]   store header w/ ptr=          (null) len=40 flags=0 id=4
[   23.543471] [tempesta]   parser: Req_Hdr(16:0): c=0x1(.), r=-2
[   23.543471] [tempesta]   request parsed: len=50 parsed=0 msg_len=0 res=-2
[   23.543471] [tempesta]   Block bad HTTP request
[   23.543471] [sync_socket] Warning: can't process app data on socket ffff88003db7d2c0
[   23.543471] [sync_socket]   Close socket ffff88003db7d2c0 (account=1)
[   23.543471] [tempesta]   Close socket ffff88003db7d2c0, conn=ffff88003d923500
[   23.543471] [tempesta]   Free msg: ffff88003e344040
[   23.543471] [tempesta]   free skb ffff88003db600c0: truesize=4864 sk=ffff88003db7d2c0, destructor=ffffffff812d2ea0 users=1
[   23.543471] [tempesta]   Free connection: ffff88003d923500
[   23.543471] [tempesta]   Free session: ffff88003da3f490
[   78.033968] [sync_socket]   Close socket ffff88003daf3280 (account=1)
[   78.036185] [tempesta]   Close socket ffff88003daf3280, conn=ffff88003d923548
[   78.038443] [tempesta]   Free msg:           (null)
[   78.039954] [tempesta]   Free connection: ffff88003d923548
[   78.041824] [sync_socket]   free rcv skb ffff88003d88eda8

Requests scheduling to massive farm of backend servers

Currently TFW_SCHED_MAX_SERVERS is defined as 64 backend server at maximum, which is not enough for virtualized environments (virtual hosting or clouds), so it should be extended at least to 64K and all scheduling algorithms also must be updated accordingly to process such number of backend servers.

We still do not expect too many servers per one site group among which client requests are scheduled, but we expect a lot of independent sites.

CRUCIAL NOTE: it's quite atypical to have 64K servers in the same server group. Virtualized environment means many small small sites behind Tempesta FW, i.e. the sense of the issue is HTTP scheduler, which must schedule a request among thousands of server groups. In this case most of server groups can have only one server. Meantime there could be really powerful installations with hundreds upstream servers. Thus 2-tier schedulers (ration, hash etc.) still must have dynamic arrays for connections and servers, but probably we don't need to introduce special data structures able to efficiently handle thousands of servers in the same server group.

lb: backend server connection cannot be restored instantly after closing

NOTE:

The master branch doesn't have the feature and the bug can't be reproduced there.
Don't touch it until the failover branch is merged.

Scenario:

start Tempesta FW
start a backend server (e.g. service nginx start)
wait until the connection is established
restart the backend server

Expected:
The connection should be re-established (may be checked via netstat and a corresponding log message appears).

Got:
The connection is not re-established.
When the backend server is stopped (but not yet started) there is a log message:

IPv4: Attempt to release TCP socket in state 1
[sync_socket] process error on socket %p

Basic investigation shown the following scenario of how that happens:

The connection is closed, and Tempesta catches that and tries to re-connect to the server (which is offline now).
kernel_connect() actually succeeds and returns an alive socked without any error (but should fail because the server is offline and no connection is possible).
Tempesta takes the socket and thinks that the connection is re-established, so the background thread that establishes connections wents into sleep until a disconnect occurs.
The socket returned by kernel_connect() dies shortly after (asynchronously), but the error happens during handling this event (the IPv4: Attempt to release TCP socket in state 1 message).
No disconnect callbacks invoked (because of the error) and Tempesta doesn't detect the disconnect and doesn't even try to re-establish the connection. The socket seems to be not released properly.

If you add a delay (i.e. msleep()) between catching the disconnect event and kernel_connect() then the problem doesn't occur (the kernel_connect() returns an error as it should).

Oops on shutdown

tempesta.sh stop leads to Oops in dmesg:

    [tempesta] Shutdown Tempesta
    [tempesta] Close 1 listening sockets
    [tempesta] Close 1 backend sockets
    [tempesta]   Destroy server socket ffff88003d8e2740
    kmem_cache_destroy tfw_conn_cache: Slab cache still has objects
    CPU: 1 PID: 4475 Comm: rmmod Tainted: G           O 3.10.10+ #7
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
     0000000000000000 ffff88003d573ea8 ffffffff824d2d9c ffff88003d573ec8
     ffffffff8209b75f ffff88003d718200 ffffffffa003de90 ffff88003d573ed8
     ffffffffa0023ae3 ffff88003d573ee8 ffffffffa00377c7 ffff88003d573f78
    Call Trace:
     [<ffffffff824d2d9c>] dump_stack+0x19/0x1b
     [<ffffffff8209b75f>] kmem_cache_destroy+0xd8/0xea
     [<ffffffffa0023ae3>] tfw_connection_exit+0x2b/0x30 [tempesta_fw]
     [<ffffffffa00377c7>] tfw_exit+0x20/0x3b [tempesta_fw]
     [<ffffffff82065cd6>] SyS_delete_module+0x19b/0x215
     [<ffffffff8202103f>] ? do_page_fault+0x9/0xe
     [<ffffffff824d7f12>] system_call_fastpath+0x16/0x1b

Tempesta FW configuration framework

1. Kernel-space storage.

We need a place to store run-time configuration variables like "enable/disable caching" or "cache size" or " "list of back-end servers".

There are two general approaches to do that:
a. A static C structure, and a member for each setting.
b. A dyanamic data structure like a hash table or a tree where we can access settings by string keys.

Generally we would like to use the first approach because it is significantly faster and we likely will read the configuration from performance-critical routines executed from softirq context.
But it has a disadvantage: it is harder to split it to modules (relatively to the second approach).
It would be nicer to have a "private" configuration for each module in the system. With such global structure we have to join all of them statically. We need to discuss how to handle that.

Also the structure must be protected with RCU instead of a lock (see #16).

2. User-space interface.

We need to somehow update the kernel configuration from user-space.

Possible approaches:

a. /proc/sys
A lot of boilerplate code for sysctl tables, their registration and I/O handlers.
Serialization and parsing from strings in the kernel space.
The mess in /proc/sys is often considered as deprecated, so not very welcomed by the community.

b. configfs
Even more boilerplate code comparing to the sysctl (you have to maintain classes and objects and their attributes).
Serialization and parsing is done in the kernel space as well.

c. copy the whole structure via ioctl to the kernel space
Almost no boilerplate code in the kernel space.
Bad granularity (can change only whole structure, not a single value).
Need to create a parser for our configuration files.
Generally not welcomed by the open-source community.

We need to discuss which approach is the best for our needs.

[Cache] Storing Responses to Authenticated Requests

Currently the cache don't consider authorization headers in our caching, but RFC 2616, chapter 14.8 says:

  When a shared cache (see section 13.7) receives a request
  containing an Authorization field, it MUST NOT return the
  corresponding response as a reply to any other request, unless one
  of the following specific exceptions holds:

  1. If the response includes the "s-maxage" cache-control
     directive, the cache MAY use that response in replying to a
     subsequent request. But (if the specified maximum age has
     passed) a proxy cache MUST first revalidate it with the origin
     server, using the request-headers from the new request to allow
     the origin server to authenticate the new request. (This is the
     defined behavior for s-maxage.) If the response includes "s-
     maxage=0", the proxy MUST always revalidate it before re-using
     it.

  2. If the response includes the "must-revalidate" cache-control
     directive, the cache MAY use that response in replying to a
     subsequent request. But if the response is stale, all caches
     MUST first revalidate it with the origin server, using the
     request-headers from the new request to allow the origin server
     to authenticate the new request.

  3. If the response includes the "public" cache-control directive,
     it MAY be returned in reply to any subsequent request.

Oops after kernel module loading

Happens shortly after module loading. Occurs not often, perhaps 1 out of 10 or 20 times.

No idea why that happens or how to debug it.

Starting Tempesta...
[tempesta] Start Tempesta
[tempesta] Open 1 backend sockets
[tempesta] ERROR: Can't connect to back-end server 127.0.0.1:8080 (-111)
[tempesta] Open 1 listening sockets
[tempesta]   Created listening socket ffff88003d7bf840
[tempesta] Close 0 backend sockets
[tempesta] Open 2 backend sockets
[tempesta] ERROR: Can't connect to back-end server 127.0.0.1:8080 (-111)
[tempesta] ERROR: Can't connect to back-end server 127.0.0.1:8081 (-111)
net.tempesta.backend = 127.0.0.1:8080 127.0.0.1:8081
done

Entering kdb (current=0xffff88003d583840, pid 1917) on processor 0 Oops: (null)
due to oops @ 0xffffffff811bb6e8
�dCPU: 0 PID: 1917 Comm: bash Tainted: G           O 3.10.10-tempesta #62
�dtask: ffff88003d583840 ti: ffff88003be16000 task.ti: ffff88003be16000
�dRIP: 0010:[<ffffffff811bb6e8>]  [<ffffffff811bb6e8>] lookup_object+0x11/0x36
�dRSP: 0018:ffff88003be17c50  EFLAGS: 00010006
�dRAX: f000ff53f000ff53 RBX: ffff88003bc51078 RCX: 002dffe98ebfbc51
�dRDX: 000000000000003c RSI: ffffffff816e3790 RDI: ffff88003bc51078
�dRBP: ffff88003be17c50 R08: 0000000000000000 R09: 0000000000000021
�dR10: 0000000000000000 R11: ffff88003bc51020 R12: ffffffff81515140
�dR13: ffffffff816e3790 R14: ffffffff816e3798 R15: 0000000000000286
�dFS:  0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
�dCS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
�dCR2: 00007f44db8a6ee0 CR3: 000000003d4c2000 CR4: 00000000000006b0
�dDR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
�dDR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
�dStack:
 ffff88003be17cb8 ffffffff811bc0f1 0000000000000000 0000000000000000
 0000000000000005 ffff88003bc51078 ffffffff81515140 ffff88003be17cb8
 ffff88003bc51078 ffffffff810c19d1 00000000ffffffff ffff88003bc51ae8
�dCall Trace:
�d [<ffffffff811bc0f1>] debug_object_activate+0x76/0x12e
�d [<ffffffff810c19d1>] ? __kmem_cache_free+0x31/0x31
�d [<ffffffff8108444f>] __call_rcu+0x68/0x1f4
�d [<ffffffff810845f6>] call_rcu_sched+0x1b/0x1d
�d [<ffffffff810c1a23>] kmem_cache_free+0x3a/0x4c
�d [<ffffffff810b834f>] __put_anon_vma+0x7b/0x80
�d [<ffffffff810b847e>] unlink_anon_vmas+0x12a/0x15d
�d [<ffffffff810b1dab>] ? __remove_shared_vm_struct+0x4d/0x52
�d [<ffffffff810ab7b3>] free_pgtables+0x78/0xde
�d [<ffffffff810b3a33>] exit_mmap+0xb4/0x137
�d [<ffffffff8102d88a>] mmput+0x52/0xca
�d [<ffffffff81032c48>] exit_mm+0x108/0x115
�d [<ffffffff810502cd>] ? hrtimer_try_to_cancel+0x87/0x93
�d [<ffffffff81033e22>] do_exit+0x273/0x396
�d [<ffffffff81034939>] do_group_exit+0xed/0xed
�d [<ffffffff81034948>] SyS_exit_group+0xf/0xf
�d [<ffffffff8135e229>] system_call_fastpath+0x16/0x1b
�dCode: 3e e7 ff 49 89 5d 00 4d 89 65 08 4d 89 2c 24 4c 89 6b 08 5b 41 5c 41 5d 5d c3 55 48 89 e5 48 8b 06 ba 00 00 00 00 eb 0c 83 c2 01 <48> 39 78 18 74 1d 48 8b 00 48 85 c0 75 ef 3b 15 c0 5b 39 00 7e

same skb is added to the msg's skb_list multiple times in ss_tcp_process_skb()

The ss_tcp_process_skb()/ss_tcp_process_proto_skb() iterate over sk_buff's fragments and make several upcalls for each fragment.

There are two problems:

put_skb_to_msg() is called for each fragment. The function enqueues the given skb to the msg's skb_list, so the same skb is actually added to the list multiple times (for each fragment of the skb).
On SS_POSTPONE, the postpone_skb() is called after put_skb_to_msg(). Both functions add the skb to the msg's skb_list, so the same skb is added twice when the message is postponed.

Confusing Kconfig kernel configuration module

Linux kernel has a built-in security model called DAC. It's always active and used. Beside that, Linux supports different security models. Only one security model can be active at any given time. The default security model is specified in the Linux configuration file.

Tempesta kernel configuration file needs to present Tempesta as an another security model that can be set up as a default one.

a better design for run-tinme configuration system?

Recently Alexander Krizhanovsky and I had a discussion and he has stated doubts that sysctl is a good approach to configure our kernel modules since it causes a lot of boilerplate code if we have a lot of variables. I've agreed with him that it would be simpler to parse a configuration file into a C structure in userspace and then copy the stcurture into the kernel and reference it from there.

I like this approach because it is simple, but it is only a half of the problem. I am very concerned about other half of the problem since it caused a lot of pain in several projects I've seen so far.

The problem is: you need to syncrhonize access to the global configuration. It is accessed very often, so you need to do a lot of locking stuff and it is easy to make a mistake there.

Currently there is a semaphore in the tfw_cfg and you have to lock it every time you read or write the configuration. With callback-oriented stuff like synchronous sockets it is very easy to achieve a deadlock condition when you acquire the lock recursively in a callback.

Let me illustrate the problem. Consider this pseudo-code example:

int close_callback()
{
        down_read(&tfw_cfg.mtx);
        if (tfw_cfg.feature_is_enabled) {
               clean_up_feature_stuff();
        }
        up_read(&tfw_cfg.mtx);
}

void close_connection()
{
        down_read(&tfw_cfg.mtx);
        if (tfw_cfg.feature_is_enabled) {
                ss_hooks_register(&{ .connection_drop =  close_callback });
                ss_close();
        }
        up_read(&tfw_cfg.mtx);
}

The tfw_cfg.feature_is_enabled checked in both close_connection() and close_callback() and it is protected with a semaphore. But the problem is you can't (and would not like to) predict the context in which the close_callback() is called. Actually the callback may be called from the context of close_connection() and then it turns out that the semaphore is loced recursively and you get the deadlock since we have no recursive locks.

This kind of stuff is very likelly since you definitelly would like to turn features on and off trough this configuration structure and also it is very hard to debug since you have all this locking stuff and not very determined context.

I don't know a simple solution for the problem other than encapsulating the structure and providing a bunch of functions that will copy fields of the structure and won't allow you to execute any other code in critical sections.

We need to discuss it. Perhaps someone may suggest a better solution?

A lot of code will depend on the run-time configuration in future, so I think that this is an important issue and we need to solve early while cost of the change is relatively low.

Full/per-vhost dynamic (re)configuration: RESTful and/or gRPC API

Introduction

Current file-based reconfiguration is a huge pain:

it's hard to add/update single server group or vhost if there are already thousands of server groups and/or vhosts - we have to traverse them all to understand what differs. File configuration update always means full configuration update and we need to understand how it differs from the previous configuration.
There is no good solution for a configuration update - if a part of live reconfiguration update fails, then it's hard to rollback previous changes.
File configuration isn't suitable for a distributed environment: apply the same configuration change to all the nodes or configure a node remotely. If we update a configuration remotely (e.g. via REST API), then we have to update the configuration file somehow and it's painful to keep comments and formatting.
Automatic configuration generation and validation would be a problem - the configuration language should have a grammar and allow schema-less configuration updates like Envoy.

At this point, it's clear that the whole configuration, maybe except quite a few properties (e.g. path to TDB files) necessary to start Tempesta, must be dynamically reconfigurable. Rgardles whether a particular option is dynamically reconfigurable or not, the whole configuration must be stored in TDB persistent tables, so depends on #516. No current file configuration is required.

Thus we have to move to TDB-handled configuration from the file.

Reconfiguration

We need to provide RESTful or JSON-RPC API for the configuraion updates. The API must be implemented by a rest module which processes the HTTP request and fetches the recofiguration string which is dispatched (as part of parsed HTTP message via kernel-user space transport, #77) to a user space daemon cfg_upd.

The rest module must provide strong authentication of all the ingress requests. Initially it could be done based on ingress physical interface and IP addresses checking specified in the configuration.

The cfg_upd daemon parses JSON string and calls libtdb API to update appropriate table in TDB. The updates must be done in transactional context (#516) to be able to rollback the change on any problem. TDB must also implement events notification (callbacks), also required fro #516. TL programs must be compiled also in the user space using a separate parser. The daemon can also make any compilation of user friendly configuration into low-level representation (see for example #1105).

Different TDB tables must be used for the configuration handling. At least separate tables for vhosts and server groups are required since these are the place for multiple configuration entities. Moreover, currently server groups are stored in sg_hash - a perfect candidate for a migration to TDB HTrie.

The control flow for the configuration update is following (everything works in process context of cfg_upd):

cfg_upd starts a transaction and execute all the updates on all the tables
cfg_upd commits transaction and TDB calls trx_prepare hooks on table updates
trx_prepare hooks call routines of all the modules owning updated configuration tables and the modules do all necessary preparatin (e.g. memory allocations) and validation of the changes. They do not affect current configuration.
Next trx_commit hook is called in all trx_prepare returned 0 and all the modules apply changes to their configuration.
If some of the trx_prepare hooks fails, then trx_rollback hooks are called which free all allicated resources.

The user space configuration functionality must be implemented in a separate library libcfg, whcih is called by cfg_upd.

Basically, it seems we don't need transactions from Tempesta DB and just can update all the entries in RCU faschion. But the reconfiguration process on it's own really needs something like 2PC protocol (described at the above).

libcfg

The main configuration parsing and TDB actions (using libtdb) must be done in libcfg. The library is used by the configuration daemon cfg_upd and CLI tool tfw_cfg.

The configuration parsing is done in user space, se nft also should be called from the table to tread multi-layer firewalling rules (see #862 (comment)).

Examples

Possible request to load new filter rules could look like

    POST /rest/filter?load HTTP/1.1
    Host: tempesa
    Content-Type: application/x-www-form-urlencoded
    Content-Length: N

    block=IP%3D1%2E1%2E1%2E1%26URL%7Efoo&block=URL%3Dfoo.com

This request adds two blocking rules:

block clients with IP=1.1.1.1 (IP%3D1%2E1%2E1%2E1) and URL containing (~) foo;
block all requests with URL equal (=) to foo.com

PUT, DELETE and other methods also should be used.

TL (#102) programs must be stored in separate table, probably depending on their context (e.g. req or resp processing contexts).

    GET /rest/stats?perf HTTP/1.1
    Host: tempesa

System statistics must be returned by Tempesta FW in JSON document.

Interoperability

Migration from the file

Sometimes file configuration might be handy and it makes sense to move to the new configuraion scheme smoothly. So tempesta.sh (using a helper script) should

parse the configuration file
remove options supporting the new interface
run Tempesta with minimal configuration
executes the script to translate the config file to JSON and execute the REST API for configuration
run tfw_cfg tool to update the configuration via libcfg

File visibility

tfw_cfg should implement more user friendly configuration interface than tdbq working with raw TDB data representation. In particular, it must implement show option which print the whole configuration with all necessary identation is single text representation.

It also must show current active configuration (#862 (comment)).

Initialization

Current tfw_init() logic remains the same, but tfw_start() abandons tfw_cfg_parse() - instead of reading and parisng the file, Tempesta reads TDB configuration tables and execute trx_prepare and trx_commit hooks on all modules and tables. Next, either if we loaded full configuration or all the tables are empty, Tempesta waits for TDB events (callback, #516) on configuration table. On the initial run, when all the tables are empty, tfw_cfg loads configuration from the configuration file, JSON, or interactive user commands.

Revision of configuration consistency

There are claims regarding consistency of current configuration options, e.g. #862 (comment) :

Historically there wasn't vhost and location directives and srv_group directive included some options that actually describe web service behaviour. This options are about HTTP processing features:

server_forward_retries
server_forward_timeout
server_retry_nonidempotent

While others describe the group of servers as single load-balaned array of server connections and 'low-level' connection and load balancer properties:

server_connect_retries
server_queue_size
sched
sticky_sessions

It seems reasonable to move options from the first group to vhost settings.

Need to review all the configuration directives and stabiles them. Future additions must extend but not change configuration format. Tempesta's upgrade to the newer version should be possible without any issues.

`.htaccess`

tdbfs can provide directory based per-vhost configuration with full user/group access validation (thanks to OS filesystem inteface), so we can provide configuration overriding for non-priveledged users, which is available in Apache HTTPD only at the moment and which seems the most influential reason for people to stay with the server.

References

lb: backend traffic should not be classified/filtered

Currently, the classification/filtering functions are called for all sockets owned by the Tempesta FW.

However, they should be called only for client sockets. There is not much sense in filtering the internal backend traffic. Usually we can trust the internal network and would not like to inspect its traffic.

HTTP normalization

We need at least minimal HTTP requests normalization (like ngx_http_parse_complex_uri() from Nginx does it). The normalization must be implemented as a part of current HTTP FSM (to avoid double processing as in Nginx) and write normalized fields to appropriate part of TfwHttpReq.

Normalization depending on back-end server personality also must be done, however this is very customizable and expensive logic, so it should be possible to switch the functionality off. Thus it must be implemented in http_norm.h as plugable HTTP FSM states.

Basically, we shouldn't do the normalization if Cache-Control: no-transform is presented.

Depends on #902. Linked with #1207.

Recursive locking of tfw_cfg.mtx in tfw_open_backend_sockets()

The tfw_open_backend_sockets() calls tfw_close_backend_sockets(). Both functions do down_read(&tfw_cfg.mtx), so the rw-lock is down'ed recursively. Theoretically this should not cause a deadlock since multiple reads may be acquired simultaneously on an rw-semaphore, but generally this is not a good approach and in future we may have a bug here if someone will try to acquire the lock for writing (instead of reading) recursively.

[ INFO: possible recursive locking detected ]
3.10.10-tempesta #54 Tainted: G O

insmod/2444 is trying to acquire lock:
(&tfw_cfg.mtx){.+.+.+}, at: [] tfw_close_backend_sockets+0x10/0x80 [tempesta_fw]

but task is already holding lock:
(&tfw_cfg.mtx){.+.+.+}, at: [] tfw_open_backend_sockets+0x2c/0x2a0 [tempesta_fw]

lb: a server backend connection is lost after a period of inactivity

After a period of inactivity the connection to a server backend is closed and never restored. Tempesta attempts to get the server connection to pass data to it, and fails.

Please note that to expose this problem you need a patch to #52

Loading Tempesta kernel modules...
[   73.747534] [tempesta] Initializing Tempesta FW kernel module...
[   73.748171] [tempesta] init: cfg_if
[   73.748528] [tempesta] init: http
[   73.748855] [tempesta] init: server
[   73.749217] [tempesta] init: client
[   73.749567] [tempesta] init: session
[   73.749931] [tempesta] init: connection
[   73.750316] [tempesta]   register cfg: cache
[   73.750732] [tempesta]   register cfg: sock_backend
[   73.751205] [tempesta]   register cfg: sock_frontend
[   73.771215] [tempesta] Registering new scheduler: dummy
Starting Tempesta...
[   73.773393] [tempesta] got state via sysctl: start
[   73.773743] [tempesta]   reading configuration file...
[   73.774098] [tempesta]   reading file: /home/natsys-lab/tfw-run/tempesta/etc/tempesta_fw.conf
[   73.775119] [tempesta]   file size: 2204 bytes
[   73.775435] [tempesta]   read by offset: 0
[   73.775781] [tempesta]   read by offset: 2204
[   73.776141] [tempesta] starting all modules...
[   73.776449] [tempesta]   parsing configuration and pushing it to modules...
[   73.776932] [tempesta]   use default entry: 'cache off;'
[   73.777297] [tempesta]   spec handle: 'cache'
[   73.777597] [tempesta]   use default entry: 'cache_size 262144;'
[   73.778017] [tempesta]   spec handle: 'cache_size'
[   73.778347] [tempesta]   use default entry: 'cache_dir /opt/tempesta/cache;'
[   73.778826] [tempesta]   spec handle: 'cache_dir'
[   73.779148] [tempesta]   use default entry: 'backend 127.0.0.1:8080;'
[   73.779587] [tempesta]   spec handle: 'backend'
[   73.779898] [tempesta]   use default entry: 'listen 127.0.0.1:80;'
[   73.780357] [tempesta]   spec handle: 'listen'
[   73.780748] [tempesta]   created front-end socket: sk=ffff88003cb56040
[   73.781195] [tempesta]   starting modules...
[   73.781489] [tempesta]   mod_start(): cache
[   73.781786] [tempesta]   mod_start(): sock_backend
[   73.782117] [tempesta]   Starting thread: tfw_bconnd
[   73.782494] [tempesta]   mod_start(): sock_frontend
[   73.782829] [tempesta]   start listening on socket: sk=ffff88003cb56040
[   73.783289] [tempesta] modules are started
net.tempesta.state = start
done
done
Starting nginx: nginx.
[   77.784478] [tempesta] Connected to backend: 127.0.0.1:8080
[  137.844823] [sync_socket]   received FIN, do active close
[  137.846675] [sync_socket]   Close socket ffff88003cb89280 (account=1)
[  137.848803] [tempesta]   Close socket ffff88003cb89280, conn=ffff88003c8cb548
[  137.849989] [tempesta]   Free msg:           (null)
[  137.850304] [tempesta]   Destroy server socket ffff88003cb89280
[  137.850718] [tempesta]   Free connection: ffff88003c8cb548
[  369.738262] [tempesta]   New client socket ffff88003cb88080 (state=1)
[  369.738968] [tempesta]   Link new msg ffff88003d972040 with connection ffff88003cb91548
[  369.739514] [tempesta]   Add skb ffff88003d872e28 to message ffff88003d972040
[  369.739996] [tempesta]   Received 109 client data bytes (GET / HTTP/1.1
[  369.739996] User-Agent: Wget/1.13.4 (linux-gnu)
[  369.739996] Accept: */*
[  369.739996] Host: 127.0.0.1
[  369.739996] Connection: Keep-Alive
[  369.739996] 
[  369.739996] ) on socket (conn=ffff88003cb91548)
[  369.741821] [tempesta]   enter FSM at state 0
[  369.742174] [tempesta]   parser: Req_0(0:0): c=0x47(G), r=-2
[  369.742627] [tempesta]   parser: Req_Method(1:0): c=0x47(G), r=-2
[  369.743118] [tempesta]   parser: Req_MUSpace(2:0): c=0x2f(/), r=-2
[  369.743619] [tempesta]   parser: Req_UriAbsPath(6:0): c=0x20( ), r=-2
[  369.744142] [tempesta]   parser: Req_HttpVer(7:0): c=0x48(H), r=-2
[  369.744645] [tempesta]   parser: Req_EoL(15:0): c=0xd(.), r=-2
[  369.745122] [tempesta]   parser: Req_EoL(15:0): c=0xa(.), r=-2
[  369.745600] [tempesta]   parser: Req_Hdr(16:0): c=0x55(U), r=-2
[  369.746091] [tempesta]   parser: Req_HdrOther(92:0): c=0x55(U), r=-2
[  369.746621] [tempesta]   store header w/ ptr=ffff88003c8908b0 len=35 flags=0 id=3
[  369.747230] [tempesta]   parser: Req_Hdr(16:0): c=0x41(A), r=-2
[  369.747718] [tempesta]   parser: Req_HdrOther(92:0): c=0x41(A), r=-2
[  369.748247] [tempesta]   store header w/ ptr=ffff88003c8908d5 len=11 flags=0 id=4
[  369.748849] [tempesta]   parser: Req_Hdr(16:0): c=0x48(H), r=-2
[  369.749324] [tempesta]   parser: RGen_LWS(10000:21): c=0x20( ), r=-2
[  369.749824] [tempesta]   parser: RGen_LWS(10000:21): c=0x31(1), r=-2
[  369.750265] [tempesta]   enter FSM at state 21
[  369.750571] [tempesta]   parser: Req_HdrHostV(21:0): c=0x31(1), r=-2
[  369.751002] [tempesta]   enter FSM at state 1
[  369.751298] [tempesta]   parser: Req_I_H(1:1): c=0x31(1), r=-2
[  369.751697] [tempesta]   parser: Req_I_H(1:1): c=0x32(2), r=-2
[  369.752099] [tempesta]   parser: Req_I_H(1:1): c=0x37(7), r=-2
[  369.752500] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[  369.752895] [tempesta]   parser: Req_I_H(1:1): c=0x30(0), r=-2
[  369.753290] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[  369.753690] [tempesta]   parser: Req_I_H(1:1): c=0x30(0), r=-2
[  369.754087] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[  369.754482] [tempesta]   parser: Req_I_H(1:1): c=0x31(1), r=-2
[  369.754877] [tempesta]   parser: Req_I_H(1:1): c=0xd(.), r=-2
[  369.755266] [tempesta]   parser: Req_I_H_EoL(3:1): c=0xd(.), r=-2
[  369.755679] [tempesta]   parser: Req_I_H_EoL(3:1): c=0xa(.), r=-2
[  369.756097] [tempesta]   parse header __req_parse_host: ret=11 plen=9 id=1
[  369.756569] [tempesta]   store header w/ ptr=ffff88003c8908e2 len=15 flags=0 id=1
[  369.757075] [tempesta]   parser: Req_Hdr(16:0): c=0x43(C), r=-2
[  369.757477] [tempesta]   parser: Req_HdrC(22:0): c=0x6f(o), r=-2
[  369.757892] [tempesta]   parser: Req_HdrConnection(44:0): c=0x3a(:), r=-2
[  369.758357] [tempesta]   parser: RGen_LWS(10000:45): c=0x20( ), r=-2
[  369.758784] [tempesta]   parser: RGen_LWS(10000:45): c=0x4b(K), r=-2
[  369.759214] [tempesta]   enter FSM at state 45
[  369.759522] [tempesta]   parser: Req_HdrConnectionV(45:0): c=0x4b(K), r=-2
[  369.759986] [tempesta]   enter FSM at state 1
[  369.760288] [tempesta]   parser: I_Conn(1:1): c=0x4b(K), r=-2
[  369.760684] [tempesta]   parser: I_EoT(6:1): c=0xd(.), r=0
[  369.761058] [tempesta]   parser: I_EoL(7:1): c=0xd(.), r=0
[  369.761430] [tempesta]   parser: I_EoL(7:1): c=0xa(.), r=0
[  369.761815] [tempesta]   parser: Connection parsed: flags 0x2
[  369.762205] [tempesta]   parse header __parse_connection: ret=12 plen=10 id=0
[  369.762686] [tempesta]   store header w/ ptr=ffff88003c8908f3 len=22 flags=0 id=0
[  369.763189] [tempesta]   parser: Req_Hdr(16:0): c=0xd(.), r=-2
[  369.763590] [tempesta]   parser: Req_HdrDone(93:0): c=0xa(.), r=-2
[  369.764021] [tempesta]   parse msg body: flags=0x2 content_length=0
[  369.764446] [tempesta]   request parsed: len=109 parsed=109 msg_len=109 res=0
[  369.764925] [tempesta]   GFSM return code 0
[  369.765215] [tempesta] ERROR: Can't get an appropriate server for a session[tempesta] ERROR: Cannot schedule message, msg=ffff88003d972040 clnt=ffff88003c8cd3d8
[  383.140878] [sync_socket]   received FIN, do active close
[  383.141745] [sync_socket]   Close socket ffff88003cb88080 (account=1)
[  383.142724] [tempesta]   Close socket ffff88003cb88080, conn=ffff88003cb91548
[  383.143826] [tempesta]   Free msg:           (null)
[  383.144557] [tempesta]   Destroy client socket ffff88003cb88080
[  383.145487] [tempesta]   Free connection: ffff88003cb91548
[  383.146324] [tempesta]   Free session: ffff88003d949490
[  383.147109] [tempesta]   Free msg: ffff88003d972040
[  383.147816] [tempesta]   free skb ffff88003d872e28: truesize=2304 sk=ffff88003cb88080, destructor=ffffffff812d2ea0 users=2

Broken socket accounting

When a socket is freed the kernel issues several warnings regarding socket accounting. These warning tell that from the kernel's pint of view some resources were not freed at the time the socket is destroyed.

It's not that Tempesta doesn't free those resources. However it appears that Tempesta is careless with resource accounting when moving an SKB from the receive queue to the send queue.

Loading Tempesta kernel modules...
[   10.038782] [tempesta] Initializing Tempesta FW kernel module...
[   10.039460] [tempesta] init: cfg_if
[   10.039847] [tempesta] init: http
[   10.040255] [tempesta] init: server
[   10.040655] [tempesta] init: client
[   10.041044] [tempesta] init: session
[   10.041439] [tempesta] init: connection
[   10.041860] [tempesta]   register cfg: cache
[   10.042343] [tempesta]   register cfg: sock_backend
[   10.042895] [tempesta]   register cfg: sock_frontend
[   10.057398] [tempesta] Registering new scheduler: dummy
Starting Tempesta...
[   10.059534] [tempesta] got state via sysctl: start
[   10.059930] [tempesta]   reading configuration file...
[   10.060348] [tempesta]   reading file: /home/natsys-lab/tfw-run/tempesta/etc/tempesta_fw.conf
[   10.061421] [tempesta]   file size: 2250 bytes
[   10.061775] [tempesta]   read by offset: 0
[   10.062137] [tempesta]   read by offset: 2250
[   10.062509] [tempesta] starting all modules...
[   10.062858] [tempesta]   parsing configuration and pushing it to modules...
[   10.063396] [tempesta]   spec handle: 'backend'
[   10.063748] [tempesta]   spec handle: 'listen'
[   10.064195] [tempesta]   created front-end socket: sk=ffff88003cc76040
[   10.064697] [tempesta]   use default entry: 'cache off;'
[   10.065099] [tempesta]   spec handle: 'cache'
[   10.065431] [tempesta]   use default entry: 'cache_size 262144;'
[   10.065880] [tempesta]   spec handle: 'cache_size'
[   10.066246] [tempesta]   use default entry: 'cache_dir /opt/tempesta/cache;'
[   10.066774] [tempesta]   spec handle: 'cache_dir'
[   10.067133] [tempesta]   starting modules...
[   10.067452] [tempesta]   mod_start(): cache
[   10.067767] [tempesta]   mod_start(): sock_backend
[   10.068139] [tempesta]   Starting thread: tfw_bconnd
[   10.068547] [tempesta]   mod_start(): sock_frontend
[   10.068919] [tempesta]   start listening on socket: sk=ffff88003cc76040
[   10.069429] [tempesta] modules are started
net.tempesta.state = start
done
done
Starting nginx: nginx.
[   14.069328] [tempesta] Connected to backend: 192.168.10.1:8080
[   16.943182] [sync_socket]   state_change: TCP_ESTABLISHED
[   16.944378] [tempesta]   New client socket ffff88003cd14980 (state=1)
[   16.945755] [tempesta]   Link new msg ffff88003d712040 with connection ffff88003d9c5500
[   16.946924] [tempesta]   Add skb ffff88003db7b5c0 to message ffff88003d712040
[   16.946924] [tempesta]   Received 62 client data bytes (GET / HTTP/1.1
[   16.946924] User-Agent: Wget/1.15 (linux-gnu)
[   16.946924] Accept: */*) on socket (conn=ffff88003d9c5500)
[   16.946924] [tempesta]   enter FSM at state 0
[   16.946924] [tempesta]   parser: Req_0(0:0): c=0x47(G), r=-2
[   16.946924] [tempesta]   parser: Req_Method(1:0): c=0x47(G), r=-2
[   16.946924] [tempesta]   parser: Req_MUSpace(2:0): c=0x2f(/), r=-2
[   16.946924] [tempesta]   parser: Req_UriAbsPath(6:0): c=0x20( ), r=-2
[   16.946924] [tempesta]   parser: Req_HttpVer(7:0): c=0x48(H), r=-2
[   16.946924] [tempesta]   parser: Req_EoL(15:0): c=0xd(.), r=-2
[   16.946924] [tempesta]   parser: Req_EoL(15:0): c=0xa(.), r=-2
[   16.946924] [tempesta]   parser: Req_Hdr(16:0): c=0x55(U), r=-2
[   16.946924] [tempesta]   parser: Req_HdrOther(92:0): c=0x55(U), r=-2
[   16.946924] [tempesta]   store header w/ ptr=ffff88003cce1892 len=33 flags=0 id=3
[   16.946924] [tempesta]   parser: Req_Hdr(16:0): c=0x41(A), r=-2
[   16.946924] [tempesta]   parser: Req_HdrOther(92:0): c=0x41(A), r=-2
[   16.946924] [tempesta]   store header w/ ptr=ffff88003cce18b5 len=11 flags=0 id=4
[   16.946924] [tempesta]   request parsed: len=62 parsed=62 msg_len=62 res=-1
[   16.946924] [tempesta]   postpone skb ffff88003db7b5c0
[   16.946924] [tempesta]   Add skb ffff88003db7b5c0 to message ffff88003d712040
[   16.946924] [tempesta]   Received 50 client data bytes (
[   16.946924] Host: 192.168.10.230
[   16.946924] Connection: Keep-Alive
[   16.946924] 
[   16.946924] ) on socket (conn=ffff88003d9c5500)
[   16.946924] [tempesta]   enter FSM at state 92
[   16.946924] [tempesta]   parser: Req_HdrOther(92:0): c=0xd(.), r=-2
[   16.946924] [tempesta]   store header w/ ptr=          (null) len=1 flags=0 id=4
[   16.946924] [tempesta]   parser: Req_Hdr(16:0): c=0x48(H), r=-2
[   16.946924] [tempesta]   parser: RGen_LWS(10000:21): c=0x20( ), r=-2
[   16.946924] [tempesta]   parser: RGen_LWS(10000:21): c=0x31(1), r=-2
[   16.946924] [tempesta]   enter FSM at state 21
[   16.946924] [tempesta]   parser: Req_HdrHostV(21:0): c=0x31(1), r=-2
[   16.946924] [tempesta]   enter FSM at state 1
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x31(1), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x39(9), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x32(2), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x31(1), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x36(6), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x38(8), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x31(1), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x30(0), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x2e(.), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x32(2), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x33(3), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0x30(0), r=-2
[   16.946924] [tempesta]   parser: Req_I_H(1:1): c=0xd(.), r=-2
[   16.946924] [tempesta]   parser: Req_I_H_EoL(3:1): c=0xd(.), r=-2
[   16.946924] [tempesta]   parser: Req_I_H_EoL(3:1): c=0xa(.), r=-2
[   16.946924] [tempesta]   parse header __req_parse_host: ret=16 plen=14 id=1
[   16.946924] [tempesta]   store header w/ ptr=ffff88003e33b08e len=20 flags=0 id=1
[   16.946924] [tempesta]   parser: Req_Hdr(16:0): c=0x43(C), r=-2
[   16.946924] [tempesta]   parser: Req_HdrC(22:0): c=0x6f(o), r=-2
[   16.946924] [tempesta]   parser: Req_HdrConnection(44:0): c=0x3a(:), r=-2
[   16.946924] [tempesta]   parser: RGen_LWS(10000:45): c=0x20( ), r=-2
[   16.946924] [tempesta]   parser: RGen_LWS(10000:45): c=0x4b(K), r=-2
[   16.946924] [tempesta]   enter FSM at state 45
[   16.946924] [tempesta]   parser: Req_HdrConnectionV(45:0): c=0x4b(K), r=-2
[   16.946924] [tempesta]   enter FSM at state 1
[   16.946924] [tempesta]   parser: I_Conn(1:1): c=0x4b(K), r=-2
[   16.946924] [tempesta]   parser: I_EoT(6:1): c=0xd(.), r=0
[   16.946924] [tempesta]   parser: I_EoL(7:1): c=0xd(.), r=0
[   16.946924] [tempesta]   parser: I_EoL(7:1): c=0xa(.), r=0
[   16.946924] [tempesta]   parser: Connection parsed: flags 0x2
[   16.946924] [tempesta]   parse header __parse_connection: ret=12 plen=10 id=0
[   16.946924] [tempesta]   store header w/ ptr=ffff88003e33b0a4 len=22 flags=0 id=0
[   16.946924] [tempesta]   parser: Req_Hdr(16:0): c=0xd(.), r=-2
[   16.946924] [tempesta]   parser: Req_HdrDone(93:0): c=0xa(.), r=-2
[   16.946924] [tempesta]   parse msg body: flags=0x2 content_length=0
[   16.946924] [tempesta]   request parsed: len=50 parsed=50 msg_len=50 res=0
[   16.946924] [tempesta]   GFSM return code 0
[   16.946924] 
[   16.946924] =============================================
[   16.946924] [ INFO: possible recursive locking detected ]
[   16.946924] 3.10.10-tempesta #8 Tainted: G           O
[   16.946924] ---------------------------------------------
[   16.946924] swapper/0/0 is trying to acquire lock:
[   16.946924]  (slock-AF_INET/1){+.-...}, at: [<ffffffffa00002a8>] ss_send+0x28/0x160 [sync_socket]
[   16.946924] 
[   16.946924] but task is already holding lock:
[   16.946924]  (slock-AF_INET/1){+.-...}, at: [<ffffffff813433bc>] tcp_v4_rcv+0x40c/0x8b0
[   16.946924] 
[   16.946924] other info that might help us debug this:
[   16.946924]  Possible unsafe locking scenario:
[   16.946924] 
[   16.946924]        CPU0
[   16.946924]        ----
[   16.946924]   lock(slock-AF_INET/1);
[   16.946924]   lock(slock-AF_INET/1);
[   16.946924] 
[   16.946924]  *** DEADLOCK ***
[   16.946924] 
[   16.946924]  May be due to missing lock nesting notation
[   16.946924] 
[   16.946924] 3 locks held by swapper/0/0:
[   16.946924]  #0:  (rcu_read_lock){.+.+..}, at: [<ffffffff812e6463>] __netif_receive_skb_core+0x63/0x590
[   16.946924]  #1:  (rcu_read_lock){.+.+..}, at: [<ffffffff8131d750>] ip_local_deliver_finish+0x40/0x150
[   16.946924]  #2:  (slock-AF_INET/1){+.-...}, at: [<ffffffff813433bc>] tcp_v4_rcv+0x40c/0x8b0
[   16.946924] 
[   16.946924] stack backtrace:
[   16.946924] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O 3.10.10-tempesta #8
[   16.946924]  ffffffff8155d440 ffff88003fc03828 ffffffff813bf35d ffff88003fc03898
[   16.946924]  ffffffff81073b96 ffffffff81072787 0000000000000007 00000000ffffffff
[   16.946924]  ffff880000000000 00000b005849e24f ffffffff00000000 ffffffff816f6640
[   16.946924] Call Trace:
[   16.946924]  <IRQ>  [<ffffffff813bf35d>] dump_stack+0x19/0x1b
[   16.946924]  [<ffffffff81073b96>] __lock_acquire+0xfa6/0x1240
[   16.946924]  [<ffffffff81072787>] ? trace_hardirqs_on_caller+0x1d7/0x210
[   16.946924]  [<ffffffff81074658>] lock_acquire+0x98/0xd0
[   16.946924]  [<ffffffffa00002a8>] ? ss_send+0x28/0x160 [sync_socket]
[   16.946924]  [<ffffffff813c30ca>] _raw_spin_lock_nested+0x3a/0x50
[   16.946924]  [<ffffffffa00002a8>] ? ss_send+0x28/0x160 [sync_socket]
[   16.946924]  [<ffffffffa00002a8>] ss_send+0x28/0x160 [sync_socket]
[   16.946924]  [<ffffffffa0049413>] ? tfw_sched_get_srv+0x49/0x4b [tempesta_fw]
[   16.946924]  [<ffffffffa0011c8d>] tfw_connection_send_srv+0xbf/0xc6 [tempesta_fw]
[   16.946924]  [<ffffffffa00153a2>] tfw_http_req_cache_cb+0x5d/0x5f [tempesta_fw]
[   16.946924]  [<ffffffffa000ddec>] tfw_cache_req_process+0x39/0x163 [tempesta_fw]
[   16.946924]  [<ffffffffa0015345>] ? tfw_http_adjust_resp+0x13/0x13 [tempesta_fw]
[   16.946924]  [<ffffffffa0015660>] tfw_http_req_process+0x2bc/0x32b [tempesta_fw]
[   16.946924]  [<ffffffffa0015a3b>] tfw_http_msg_process+0x43/0x5e [tempesta_fw]
[   16.946924]  [<ffffffffa0012d67>] tfw_gfsm_dispatch+0x41/0x43 [tempesta_fw]
[   16.946924]  [<ffffffffa0011e4e>] tfw_connection_recv+0x88/0x8a [tempesta_fw]
[   16.946924]  [<ffffffffa000005b>] ss_tcp_process_proto_skb+0x5b/0x90 [sync_socket]
[   16.946924]  [<ffffffffa0000167>] ss_tcp_process_skb+0xd7/0x170 [sync_socket]
[   16.946924]  [<ffffffffa0000eae>] ss_tcp_data_ready+0x10e/0x25e [sync_socket]
[   16.946924]  [<ffffffff8133425d>] tcp_data_queue+0x4dd/0xd10
[   16.946924]  [<ffffffff81337660>] tcp_rcv_established+0x560/0x620
[   16.946924]  [<ffffffff8134183c>] tcp_v4_do_rcv+0x1dc/0x4e0
[   16.946924]  [<ffffffff811eea42>] ? do_raw_spin_lock+0x82/0x140
[   16.946924]  [<ffffffff813433e7>] tcp_v4_rcv+0x437/0x8b0
[   16.946924]  [<ffffffff8131d7c5>] ip_local_deliver_finish+0xb5/0x150
[   16.946924]  [<ffffffff8131d750>] ? ip_local_deliver_finish+0x40/0x150
[   16.946924]  [<ffffffff8131df70>] ip_local_deliver+0x50/0x90
[   16.946924]  [<ffffffff8131dcde>] ip_rcv_finish+0x47e/0x580
[   16.946924]  [<ffffffff8131e26c>] ip_rcv+0x2bc/0x340
[   16.946924]  [<ffffffff812e6869>] __netif_receive_skb_core+0x469/0x590
[   16.946924]  [<ffffffff812e6463>] ? __netif_receive_skb_core+0x63/0x590
[   16.946924]  [<ffffffff812e69df>] __netif_receive_skb+0x4f/0x70
[   16.946924]  [<ffffffff812e7543>] netif_receive_skb+0xb3/0x140
[   16.946924]  [<ffffffff812d9be9>] ? __netdev_alloc_skb+0x49/0xb0
[   16.946924]  [<ffffffff81270fd4>] virtnet_poll+0x624/0x770
[   16.946924]  [<ffffffff812e8783>] net_rx_action+0xa3/0x1f0
[   16.946924]  [<ffffffff81036550>] __do_softirq+0xe0/0x1d0
[   16.946924]  [<ffffffff810367c9>] irq_exit+0x69/0xc0
[   16.946924]  [<ffffffff81003c5d>] do_IRQ+0x9d/0xc0
[   16.946924]  [<ffffffff813c3df2>] common_interrupt+0x72/0x72
[   16.946924]  <EOI>  [<ffffffff81023626>] ? native_safe_halt+0x6/0x10
[   16.946924]  [<ffffffff8100a8dd>] default_idle+0x1d/0x30
[   16.946924]  [<ffffffff8100b26a>] arch_cpu_idle+0xa/0x10
[   16.946924]  [<ffffffff810664ac>] cpu_startup_entry+0x15c/0x200
[   16.946924]  [<ffffffff813b79ad>] rest_init+0xcd/0xe0
[   16.946924]  [<ffffffff813b78e0>] ? csum_partial_copy_generic+0x170/0x170
[   16.946924]  [<ffffffff815d1da0>] start_kernel+0x34d/0x358
[   16.946924]  [<ffffffff815d185c>] ? repair_env_string+0x5e/0x5e
[   16.946924]  [<ffffffff815d1120>] ? early_idt_handlers+0x120/0x120
[   16.946924]  [<ffffffff815d155b>] x86_64_start_reservations+0x2a/0x2c
[   16.946924]  [<ffffffff815d164c>] x86_64_start_kernel+0xef/0xfc
[   16.946924] [sync_socket]   ss_send:89 entail skb=ffff88003db7b5c0 data_len=50 len=112
[   16.946924] [sync_socket]   ss_send:99 sk=ffff88003cd15280 is_queue_empty=0 tcp_send_head(sk)=ffff88003db7b5c0 sk->sk_state=1
[   17.025136] [tempesta]   Link new msg ffff88003d505040 with connection ffff88003d9c5548
[   17.025741] [tempesta]   Add skb ffff88003db7b9c0 to message ffff88003d505040
[   17.026278] [tempesta]   received 62 server data bytes (HTTP/1.1 200 OK
[   17.026278] Server: nginx/1.4.6 (Ubuntu)
[   17.026278] Date: Tue, 24 F) on socket (conn=ffff88003d9c5548)
[   17.027497] [tempesta]   enter FSM at state 0
[   17.027824] [tempesta]   parser: Resp_0(0:0): c=0x48(H), r=-2
[   17.028264] [tempesta]   parser: Resp_HttpVer(2:0): c=0x48(H), r=-2
[   17.028732] [tempesta]   parser: Resp_StatusCode(11:0): c=0x32(2), r=-2
[   17.029124] [tempesta]   parser: Resp_ReasonPhrase(12:1): c=0x20( ), r=-2
[   17.029124] [tempesta]   parser: Resp_Hdr(13:1): c=0x53(S), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrOther(87:1): c=0x65(e), r=-2
[   17.029124] [tempesta]   parser: Resp_Hdr(13:1): c=0x44(D), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrOther(87:1): c=0x61(a), r=-2
[   17.029124] [tempesta]   response parsed: len=62 parsed=62 res=-1
[   17.029124] [tempesta]   postpone skb ffff88003db7b9c0
[   17.029124] [tempesta]   Add skb ffff88003db7b9c0 to message ffff88003d505040
[   17.029124] [tempesta]   received 796 server data bytes (eb 2015 10:29:05 GMT
[   17.029124] Content-Type: text/html
[   17.029124] Content-Length: 612
[   17.029124] Last-Modified: Tue, 04 Mar 2014 11:46:45 GMT
[   17.029124] Connection: keep-alive
[   17.029124] ETag: "5315bd25-264"
[   17.029124] Accept-Ranges: bytes
[   17.029124] 
[   17.029124] <!DOCTYPE html>
[   17.029124] <html>
[   17.029124] <head>
[   17.029124] <title>Welcome to nginx!</title>
[   17.029124] <style>
[   17.029124]     body {
[   17.029124]         width: 35em;
[   17.029124]         margin: 0 auto;
[   17.029124]         font-family: Tahoma, Verdana, Arial, sans-serif;
[   17.029124]     }
[   17.029124] </style>
[   17.029124] </head>
[   17.029124] <body>
[   17.029124] <h1>Welcome to nginx!</h1>
[   17.029124] <p>If you see this page, the nginx web server is successfully installed and
[   17.029124] working. Further configuration is required.</p>
[   17.029124] 
[   17.029124] [tempesta]   enter FSM at state 87
[   17.029124] [tempesta]   parser: Resp_HdrOther(87:1): c=0x65(e), r=-2
[   17.029124] [tempesta]   parser: Resp_Hdr(13:1): c=0x43(C), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrC(14:1): c=0x6f(o), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrCo(28:1): c=0x6e(n), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrCon(29:1): c=0x74(t), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrCont(38:1): c=0x65(e), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrConte(39:1): c=0x6e(n), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrConten(40:1): c=0x74(t), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrContent(41:1): c=0x2d(-), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrContent_(42:1): c=0x54(T), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrOther(87:1): c=0x79(y), r=-2
[   17.029124] [tempesta]   parser: Resp_Hdr(13:1): c=0x43(C), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrC(14:1): c=0x6f(o), r=-2
[   17.029124] [tempesta]   parser: RGen_LWS(10000:49): c=0x20( ), r=-2
[   17.029124] [tempesta]   parser: RGen_LWS(10000:49): c=0x36(6), r=-2
[   17.029124] [tempesta]   enter FSM at state 49
[   17.029124] [tempesta]   parser: Resp_HdrContent_LengthV(49:0): c=0x36(6), r=-2
[   17.029124] [tempesta]   enter FSM at state 3
[   17.029124] [tempesta]   parser: I_ContLen(3:3): c=0x36(6), r=-2
[   17.029124] [tempesta]   parser: I_EoL(7:3): c=0xd(.), r=-2
[   17.029124] [tempesta]   parser: I_EoL(7:3): c=0xa(.), r=-2
[   17.029124] [tempesta]   parse header __parse_content_length: ret=5 plen=3 id=3
[   17.029124] [tempesta]   store header w/ ptr=ffff88003d505548 len=19 flags=1 id=3
[   17.029124] [tempesta]   parser: Resp_Hdr(13:0): c=0x4c(L), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrOther(87:0): c=0x61(a), r=-2
[   17.029124] [tempesta]   parser: Resp_Hdr(13:0): c=0x43(C), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrC(14:0): c=0x6f(o), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrConnection(36:0): c=0x3a(:), r=-2
[   17.029124] [tempesta]   parser: RGen_LWS(10000:37): c=0x20( ), r=-2
[   17.029124] [tempesta]   parser: RGen_LWS(10000:37): c=0x6b(k), r=-2
[   17.029124] [tempesta]   enter FSM at state 37
[   17.029124] [tempesta]   parser: Resp_HdrConnectionV(37:0): c=0x6b(k), r=-2
[   17.029124] [tempesta]   enter FSM at state 1
[   17.029124] [tempesta]   parser: I_Conn(1:1): c=0x6b(k), r=-2
[   17.029124] [tempesta]   parser: I_EoT(6:1): c=0xd(.), r=0
[   17.029124] [tempesta]   parser: I_EoL(7:1): c=0xd(.), r=0
[   17.029124] [tempesta]   parser: I_EoL(7:1): c=0xa(.), r=0
[   17.029124] [tempesta]   parser: Connection parsed: flags 0x2
[   17.029124] [tempesta]   parse header __parse_connection: ret=12 plen=10 id=0
[   17.029124] [tempesta]   store header w/ ptr=ffff88003d505588 len=22 flags=1 id=0
[   17.029124] [tempesta]   parser: Resp_Hdr(13:0): c=0x45(E), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrE(50:0): c=0x54(T), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrOther(87:0): c=0x61(a), r=-2
[   17.029124] [tempesta]   parser: Resp_Hdr(13:0): c=0x41(A), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrOther(87:0): c=0x63(c), r=-2
[   17.029124] [tempesta]   parser: Resp_Hdr(13:0): c=0xd(.), r=-2
[   17.029124] [tempesta]   parser: Resp_HdrDone(88:0): c=0xa(.), r=-2
[   17.029124] [tempesta]   parse msg body: flags=0x2 content_length=612
[   17.029124] [tempesta]   parser: Resp_Body(89:0): c=0x3c(<), r=-2
[   17.029124] [tempesta]   read body: to_read=612
[   17.029124] [tempesta]   parser: Resp_BodyReadChunk(92:0): c=0x3c(<), r=-2
[   17.029124] [tempesta]   response parsed: len=796 parsed=796 res=0
[   17.029124] [sync_socket]   ss_send:89 entail skb=ffff88003db7b9c0 data_len=796 len=858
[   17.029124] [sync_socket]   ss_send:99 sk=ffff88003cd14980 is_queue_empty=0 tcp_send_head(sk)=ffff88003db7b9c0 sk->sk_state=1
[   17.029124] [tempesta]   Free msg: ffff88003d712040
[   17.029124] [tempesta]   free skb ffff88003db7b5c0: truesize=4864 sk=ffff88003cd14980, destructor=ffffffff812d2ea0 users=2
[   17.029124] [tempesta]   Free msg: ffff88003d505040
[   17.029124] [tempesta]   free skb ffff88003db7b9c0: truesize=4864 sk=ffff88003cd15280, destructor=ffffffff812d2ea0 users=2
[   17.029124] [sync_socket]   state_change: TCP_CLOSE_WAIT
[   17.029124] [sync_socket]   received FIN, do active close
[   17.029124] [sync_socket]   Close socket ffff88003cd14980 (account=1)
[   17.029124] [tempesta]   Close socket ffff88003cd14980, conn=ffff88003d9c5500
[   17.029124] [tempesta]   Free msg:           (null)
[   17.029124] [tempesta]   Free connection: ffff88003d9c5500
[   17.029124] [tempesta]   Detach from peer: ffff88003d9c5548
[   17.029124] [tempesta]   Free session: ffff88003d518490
[   17.071291] ------------[ cut here ]------------
[   17.071649] WARNING: at net/core/stream.c:201 sk_stream_kill_queues+0x12b/0x160()
[   17.072198] Modules linked in: tfw_sched_dummy(O) tempesta_fw(O) tempesta_db(O) sync_socket(O)
[   17.072879] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O 3.10.10-tempesta #8
[   17.073450]  0000000000000009 ffff88003fc03ae0 ffffffff813bf35d ffff88003fc03b18
[   17.074033]  ffffffff8102ee8c ffff88003cd14980 ffff88003cd14be8 ffff88003cce2262
[   17.074613]  0000000000000000 ffff88003cce225a ffff88003fc03b28 ffffffff8102ef65
[   17.075165] Call Trace:
[   17.075280]  <IRQ>  [<ffffffff813bf35d>] dump_stack+0x19/0x1b
[   17.075280]  [<ffffffff8102ee8c>] warn_slowpath_common+0x5c/0x80
[   17.075280]  [<ffffffff8102ef65>] warn_slowpath_null+0x15/0x20
[   17.075280]  [<ffffffff812df3ab>] sk_stream_kill_queues+0x12b/0x160
[   17.075280]  [<ffffffff8132a793>] inet_csk_destroy_sock+0x83/0xe0
[   17.075280]  [<ffffffff8132ba3d>] tcp_done+0x7d/0x90
[   17.075280]  [<ffffffff813384b0>] tcp_rcv_state_process+0xa80/0xc50
[   17.075280]  [<ffffffff81341a4b>] tcp_v4_do_rcv+0x3eb/0x4e0
[   17.075280]  [<ffffffff811eea42>] ? do_raw_spin_lock+0x82/0x140
[   17.075280]  [<ffffffff813433e7>] tcp_v4_rcv+0x437/0x8b0
[   17.075280]  [<ffffffff8131d7c5>] ip_local_deliver_finish+0xb5/0x150
[   17.075280]  [<ffffffff8131d750>] ? ip_local_deliver_finish+0x40/0x150
[   17.075280]  [<ffffffff8131df70>] ip_local_deliver+0x50/0x90
[   17.075280]  [<ffffffff8131dcde>] ip_rcv_finish+0x47e/0x580
[   17.075280]  [<ffffffff8131e26c>] ip_rcv+0x2bc/0x340
[   17.075280]  [<ffffffff812e6869>] __netif_receive_skb_core+0x469/0x590
[   17.075280]  [<ffffffff812e6463>] ? __netif_receive_skb_core+0x63/0x590
[   17.075280]  [<ffffffff812e69df>] __netif_receive_skb+0x4f/0x70
[   17.075280]  [<ffffffff812e7543>] netif_receive_skb+0xb3/0x140
[   17.075280]  [<ffffffff812d9be9>] ? __netdev_alloc_skb+0x49/0xb0
[   17.075280]  [<ffffffff81270fd4>] virtnet_poll+0x624/0x770
[   17.075280]  [<ffffffff812e8783>] net_rx_action+0xa3/0x1f0
[   17.075280]  [<ffffffff81036550>] __do_softirq+0xe0/0x1d0
[   17.075280]  [<ffffffff810367c9>] irq_exit+0x69/0xc0
[   17.075280]  [<ffffffff81003c5d>] do_IRQ+0x9d/0xc0
[   17.075280]  [<ffffffff813c3df2>] common_interrupt+0x72/0x72
[   17.075280]  <EOI>  [<ffffffff8100a8c9>] ? default_idle+0x9/0x30
[   17.075280]  [<ffffffff81023626>] ? native_safe_halt+0x6/0x10
[   17.075280]  [<ffffffff8100a8dd>] default_idle+0x1d/0x30
[   17.075280]  [<ffffffff8100b26a>] arch_cpu_idle+0xa/0x10
[   17.075280]  [<ffffffff810664ac>] cpu_startup_entry+0x15c/0x200
[   17.075280]  [<ffffffff813b79ad>] rest_init+0xcd/0xe0
[   17.075280]  [<ffffffff813b78e0>] ? csum_partial_copy_generic+0x170/0x170
[   17.075280]  [<ffffffff815d1da0>] start_kernel+0x34d/0x358
[   17.075280]  [<ffffffff815d185c>] ? repair_env_string+0x5e/0x5e
[   17.075280]  [<ffffffff815d1120>] ? early_idt_handlers+0x120/0x120
[   17.075280]  [<ffffffff815d155b>] x86_64_start_reservations+0x2a/0x2c
[   17.075280]  [<ffffffff815d164c>] x86_64_start_kernel+0xef/0xfc
[   17.075280] ---[ end trace b5f21e053c44a402 ]---

Oops after backend socket failure

Test scenario:

srv$ nc -l 127.0.0.1 8080    
srv$ insmod tempesta_http.ko
srv$ sleep 1 && pkill nc         # backend socket closed

client$ nc6 srv 80
hello
^C

Server hangs with following dmesg:

[tempesta] Start Tempesta HTTP
[tempesta] Open 1 backend sockets
[tempesta] Created back-end connection ffff880037ca2140
[tempesta] Added new server ffff88003d4a1448
[tempesta] Open 1 listening sockets
[tempesta] Created listening socket ffff88003d4a2180
[tempesta] Connection closed on socket ffff880037ca2140
[tempesta] Destroy socket ffff880037ca2140
[tempesta] New client socket ffff88003c3118c0 (state=1)
[tempesta] received 6 client data bytes (hello
) on socket (st=ffff88003c36e408)
[tempesta] ERROR: Can't get appropriate server for a session
[tempesta] Warning: Can't allocate new session
BUG: unable to handle kernel paging request at ffffeba400000000
IP: [] free_block+0xa0/0x170
PGD 0
Oops: 0000 [#1] SMP
Modules linked in: tempesta_http(O) ipv6 ppdev pcspkr e1000 i2c_piix4 i2c_core parport_pc parport floppy pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mperf]
CPU 1
Pid: 25, comm: kworker/1:1 Tainted: G O 3.6.0-rc7 #4 Bochs Bochs
RIP: 0010:[] [] free_block+0xa0/0x170
RSP: 0018:ffff88003d82bd10 EFLAGS: 00010086
RAX: ffffeba400000000 RBX: ffff88003c6e8d80 RCX: 0000000000000000
RDX: 0000003c00000000 RSI: ffff88003c00d018 RDI: 0000000000000000
RBP: ffff88003d82bd60 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000001
R13: ffff88003c00d018 R14: 0000000000000000 R15: ffffea0000000000
FS: 00007f905d1ce700(0000) GS:ffff88003fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffeba400000000 CR3: 0000000037f9f000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:1 (pid: 25, threadinfo ffff88003d82a000, task ffff88003d822a80)
Stack:
0000000000000000 ffff88003e006240 0000000000000000 0000000000000000
0000000000000001 ffff88003c00d000 ffff88003d6a7f40 ffff88003c6e8d80
0000000000000001 ffff88003c00d018 ffff88003d82bdb0 ffffffff8114cb07
Call Trace:
[] drain_array+0xc7/0x100
[] cache_reap+0x82/0x210
[] ? __schedule+0x3b2/0x6c0
[] ? drain_freelist+0xb0/0xb0
[] process_one_work+0x173/0x4b0
[] worker_thread+0x12e/0x320
[] ? manage_workers+0x180/0x180
[] kthread+0x9e/0xb0
[] kernel_thread_helper+0x4/0x10
[] ? kthread_freezable_should_stop+0x70/0x70
[] ? gs_change+0xb/0xb
Code: fe 49 89 4d 00 48 89 cf 48 89 4d c0 e8 aa a3 ee ff 48 c1 e8 0c 48 8b 4d c0 48 8d 14 c5 00 00 00 00 48 c1 e0 06 48 29 d0 4c 01 f8 <48> 8b 10 66 85 d2 0f 88 b5 00 00 00 48 8b 75 c8 48 8b 53 78 48
RIP [] free_block+0xa0/0x170
RSP

Web UI

Web UI must be implemented as set of html/javasctipt files on top of TDB Web content tables. RESTful API (#67) is the back-end for the UI. It must be able to manage cluster of Tempesta FW nodes. The cluster configuration must be stored in some replicated database to provide admin node redundancy (BTW the admin node and Tempesta node can be on the same machine).

fix defects introduced by commit 1cd52cf

Violation of encapsulation in tfw_session_free().
The session-related code should not know about HTTP internals.
Need to add a generic destructor method to TfwMsg, set it up in tfw_http_msg_create() and call the destructor from tfw_session_free().
tfw_http_msg_free() should be NULL-safe like kfree()
A TfwSession is bound with a client in tfw_connection_recv(), so at any point when tfw_sess_conn() is called the cli should not be NULL, so it is better to plce BUG() there instead of returning NULL when cli is NULL.

See comments for: 1cd52cf

tdbfs

We need Tempesta DB filesystem interface such that Web cache and content files can be commonly written to file system directory, but preprocessed and stored in TDB for immediate transfer.

The file system is just an interface for TDB. Each table should have fs option which allows/prohibits its mounting and defines view of the mounted table. Basically, all table records should be shown as files in the same directory. The record key should be used as file name, while the file should contain the rest of record fields in binary (as is) form.

lb: backend connections hang in TIME_WAIT state after closing

Scenario:

start a backend server (e.g. nc -lp 8080)
start Tempesta FW
wait until the connection is established
stop the backend server
wait for fin_timeout (/proc/sys/net/ipv4/tcp_fin_timeout)
check that the TCP connection is released (netstat -an | grep TIME_WAIT)

Got:
tcp 0 0 127.0.0.1:23112 127.0.0.1:8080 TIME_WAIT
The connection TCP hangs in the TIME_WAIT state even when the tcp_fin_timeout is expired.

Expected:
The closed TCP connection should be released immediately without handing in the TIME_WAIT state.
Or at least, it should be released after tcp_fin_timeout.

The TIME_WAIT means that either FIN is not received or it is not handled correclty.
Since we run everything locally, the FIN is likely received, so likely it is something wrong with FIN handling.

tempesta-tech / tempesta Goto Github PK

tempesta's Introduction

Tempesta FW

What it is?

Demo

How it works?

Current state

Installation and Configuration

We appreciate support of our technical partner

tempesta's People

Contributors

Stargazers

Watchers

Forkers

tempesta's Issues

Introduction

Reconfiguration

libcfg

Examples

Interoperability

Migration from the file

File visibility

Initialization

Revision of configuration consistency

.htaccess

References

Recommend Projects

Recommend Topics

Recommend Org

`.htaccess`