Automatically mitigate layer 7 DDoS attacks.

• cd /opt
• git clone https://github.com/makhomed/autofilter.git autofilter

Also you need to install unbound, dnspython and netaddr:

# yum install unbound
# vim /etc/unbound/unbound.conf
        interface: 127.0.0.1
        do-ip6: no
# systemctl enable unbound
# systemctl start unbound

# pip install dnspython
# pip install netaddr

• cd /opt/autofilter
• git pull

• vim /opt/autofilter/autofilter.conf
• write to config something like this:

limit UA none
limit RU 3000
limit PL 3000

Configuration file allow comments, from symbol # to end of line.

Configuration file has only two directives: limit and block.

limit directive has syntax: limit <entity> <threshold>.

<entity> can be ALL, or country code, for example, UA or RU or CN. Also <entity> can be IP address, ipv4 or ipv6 or IP network in CIDR notation.

<threshold> is load threshold after which specific ip address will be blocked. <threshold> is integer number or special value none.

Load measured as one-minute sum of load weight generated by each request from each ip. For example, one request to static resource or to redirect page measured as weight 1, one POST request has weight 30, one request to resource with args has weight 20, and one other request has weight 10.

If some specific ip generates load above <threshold> - this ip will be blocked as bot.

Search engine bots from Google, Yandex and Bing are detected automatically and will be never blocked:

def is_whitelisted_search_engine(self, domain):
    # https://support.google.com/webmasters/answer/80553?hl=en
    if domain.endswith(".googlebot.com.") or domain.endswith(".google.com."):
        return True
    # https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.xml
    elif domain.endswith(".yandex.com.") or domain.endswith(".yandex.net.") or domain.endswith(".yandex.ru."):
        return True
    # https://www.bing.com/webmaster/help/how-to-verify-bingbot-3905dc26
    elif domain.endswith(".search.msn.com."):
        return True
    else:
        return False

By default limit ALL 600 if other value not specified in config.

block directive has syntax: block <entity> <time>.

<entity> can be ALL, or country code, for example, UA or RU or CN. Also <entity> can be IP address, ipv4 or ipv6.

<time> is block time in hours or in days, suffix h or d is required. For example: 24h or 3d.

By default block ALL 24h if other value not specified in config.

nginx global configuration context:

worker_shutdown_timeout 60s;

nginx configuration in context http if CloudFlare used:

set_real_ip_from   103.21.244.0/22;
set_real_ip_from   103.22.200.0/22;
set_real_ip_from   103.31.4.0/22;
set_real_ip_from   104.16.0.0/12;
set_real_ip_from   108.162.192.0/18;
set_real_ip_from   131.0.72.0/22;
set_real_ip_from   141.101.64.0/18;
set_real_ip_from   162.158.0.0/15;
set_real_ip_from   172.64.0.0/13;
set_real_ip_from   173.245.48.0/20;
set_real_ip_from   188.114.96.0/20;
set_real_ip_from   190.93.240.0/20;
set_real_ip_from   197.234.240.0/22;
set_real_ip_from   198.41.128.0/17;
# WARNING!
# get actual list of networks from
# https://www.cloudflare.com/ips-v4

real_ip_header     CF-Connecting-IP;

geo $bot {
    default 0;
    include /opt/autofilter/var/bot.conf;
}

map $bot $loggable {
    0 1;
    1 0;
}

log_format frontend '$time_iso8601\t$http_cf_ipcountry\t$remote_addr\t$scheme\t$host\t$request_method\t'
                    '"$request_uri"\t$status\t$body_bytes_sent\t"$http_referer"\t"$http_user_agent"\t$http_cf_ray';

access_log /var/log/nginx/access.log frontend if=$loggable;

nginx configuration in context http if CloudFlare not used, but used nginx-geo:

geo $geoip_country_code {
    default XX;
    include /etc/nginx/geo/geoip_country_code.conf;
}

geo $bot {
    default 0;
    include /opt/autofilter/var/bot.conf;
}

map $bot $loggable {
    0 1;
    1 0;
}

log_format frontend '$time_iso8601\t$geoip_country_code\t$remote_addr\t$scheme\t$host\t$request_method\t'
                    '"$request_uri"\t$status\t$body_bytes_sent\t"$http_referer"\t"$http_user_agent"';

access_log /var/log/nginx/access.log frontend if=$loggable;

nginx configuration in context server:

if ( $bot ) { return 429; }

autofilter can be started in differend modes: daemon, top, ext, bot, errors, size, size-top.

daemon mode intended for using autofilter as systemd service, in this mode autofilter continuously monitor nginx access.log file and automatically block detected bots.

top mode display top load generated from users by ip address, and display which ip will be blocked in daemon mode.

ext mode display all unknown non-static extensions occurred in access.log file.

bot mode display all blocked bots.

errors mode display all 5xx errors occurred in access.log file.

size mode display page size frequency, this is useful for tuning proxy_buffers and fastcgi_buffers.

size-top mode display up to 20 urls with frequency count for each page size.

Create configuration file /opt/autofilter/autofilter.conf and define limits. After it create systemd service, for example, in file /etc/systemd/system/autofilter.service:

[Unit]
Description=autofilter
After=unbound.service

[Service]
ExecStart=/opt/autofilter/autofilter daemon
Restart=always

[Install]
WantedBy=multi-user.target

After this you need to start service:

• systemctl daemon-reload
• systemctl enable autofilter
• systemctl start autofilter
• systemctl status autofilter

If all ok you will see what service is enabled and running.

If CloudFlare not used - you can use ipset to block bots at ip level.

Create configuration file /etc/sysconfig/modules/ip_set.modules with content:

#/bin/bash

/sbin/modprobe ip_set

/usr/sbin/ipset create ddos hash:ip hashsize 16384 maxelem 262144 timeout 86400

mark it executable with command chmod +x /etc/sysconfig/modules/ip_set.modules and run this script to create ipset named ddos.

Also you need to remove firewalld, install iptables-services package via command yum install iptables-services and enable iptables service via command systemctl enable iptables. After this you need to edit configuration file /etc/sysconfig/iptables and add line -A INPUT -m set --match-set ddos src -j DROP to *filer table. After this - start or restart iptables service via command systemctl start iptables or systemctl restart iptables.

Now you can add any single IP to ddos table via command ipset add ddos 11.22.33.44 and this ip 11.22.33.44 will be blocked via iptables to 24 hours.

Create configuration file /etc/cron.d/autofilter-ban with content:

RANDOM_DELAY=45

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

 0 * * * * root /opt/autofilter/autofilter tor-ban

30 * * * * root /opt/autofilter/autofilter bot-ban

Command /opt/autofilter/autofilter tor-ban will block via ipset all tor exit nodes from list https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1

Command /opt/autofilter/autofilter bot-ban will block via ipset all bots from file /opt/autofilter/var/bot.conf.