molu8bits / modsecurity-parser Goto Github PK

modsecurity audit log analyser and parser

License: Apache License 2.0

Python 89.60% Dockerfile 1.41% Shell 3.46% Makefile 5.53%

modsecurity molu8bits analyser parser analyzer graphs modsecurity3-log modsecurity-audit-logs modsecurity-parser modsec-audit

modsecurity-parser's Introduction

modsecurity audit log parser, analyser and chart maker

TL;DR

Get the overview of security incidents reported by modsecurity module from modsec_audit.log file.

2023.05.03 update

fix showruleid #24
CI github actions
address vulnerabilities

2023.01.01 update

renamed to modsecurity_parser
fix for timezone with miliseconds
linting, testing added
requirements vulnerabilities fixed

2020.09.20 update

added support for logs from timezone "UTC-..."
updated plotting to matplotlib.3.1
added dockerhub autobuild
added requirements.txt

2019.04.17 update

added support for Modsecurity3 log (Nginx/Apache)
added feature to read Modsecurity log in JSON format

Description

modsecurity parser is a python program to read https://www.modsecurity.org/ modsec_audit.log, transform read events into more human and machine readable formats (xlsx/json) and make basic charts.

Functionality list:

JSON output file with formatting conformed to JSON logging added into Modsecurity 2.9
XLSX output file which can be analysed further with desktop tools
PNG file with some basic charts - Timeline nonblocked vs intercepted events, TOP10 IP source address, TOP20 Rule IDs hit, TOP10 Attacks intercepted

Graph analysis examples

Installation

Software needs at least Python 3.8.10 with additional libraries:

pandas 1.1.3
Pillow 9.2.0
matplotlib 3.3.2
numpy 1.22.4
openpyxl 2.4.2

Install them with command

pip3 install -r requirements.txt

Basic usage

python3 modsecurity_parser.py -f /home/user/logs/modsec_audit.log

for that case results will be recorded into subdirectory "modsec_output" where the log to analyse is placed.

More options

python3 modsecurity_parser.py -h

Filters INCLUDE and EXCLUDE are available for IP source addresses.

--exclude option ( e.g. "--exclude 192.168.0.1 10.0.0.1") just skips events with given IP source addresses

--include (e.g. "--include 10.0.5.6") take precedence over EXCLUDE. INCLUDE process only events with given IP source addresses.

--jsononeperline - option recommended for big number of events where e.g. produced JSON is supposed to be read by other SIEM tool. Uses the very same format as modsecurity software when type of logging is set to "JSON".

Processing Modsecurity3 log

--version3 (e.g. "modsecurity_parser.py -f modsec_audit.log --version3"

Processing Modsecurity log in JSON format:

--jsonaudit (e.g. "modsecurity_parser.py -f modsec_audit.log --jsonaudit"

Limitations

The biggest tested modsec_audit.log was 1GB size with around 70000 records. It took more or less 5 minutes on an 8 years old workstation and memory usage temporarily raised to 2GB of RAM
modsec_audit.log were taken from Apache web servers with locale set to en-US. Software can except some errors if datatime format is different in the audited log. Adjust LOG_TIMESTAMP_FORMAT and LOG_TIMESTAMP_FORMAT_SHORT accordingly
To process more than 90000 events just adjust MAXEVENTS
Tested with modsec_audit.log from version 2.8/2.9/3.0. Anyway Modsecurity3 for some cases produces empty H section and not all information is available to be properly presented in all graphs

run via Docker

Create a subfolder (e.g. "modseclogs") and put into some modsecurity audit logs (by default modsec_audit.log name is processed only). Output files will be created inside of ${subfolder}/modsec_output

Run command

docker run --rm -ti --mount type=bind,source="$(pwd)"/modseclogs,target=/opt/mounted molu8bits/modsecurity-parser:latest

Get some more docker options:

docker run --rm -ti -e HELP=Yes molu8bits/modsecurity-parser:latest

modsecurity-parser's People

Contributors

Stargazers

Watchers

modsecurity-parser's Issues

No modsecurity events found in the specified file

hi,

i am facing issue that when i am running python3 modsecurity-parser.py -f /var/log/modsec_audit.log i am getting below error. please help to resolve the same and attached is the sample logs which i am getting

inputFileName : /var/log/modsec_audit.log
No modsecurity events found in the specified file

modsec_audit.log

ValueError: Single argument to subplot must be a three-digit integer, not AxesSubplot(0.125,0.420962;0.149038x0.148077)

Hi,

I tried to use it on ModSecurity Version 3.

python3 modsecurity-parser.py -f /var/log/modsec_audit.log --version3

inputFileName : /var/log/modsec_audit.log
---------- modsec_audit events processed: 249 ----------
---------- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID : 0 ----------
modsecurity-parser.py:437: MatplotlibDeprecationWarning: Passing non-integers as three-element position specification is deprecated since 3.3 and will be removed two minor releases later.
plt.subplot(ax21)
Traceback (most recent call last):
File "modsecurity-parser.py", line 689, in
outputWithGraphs = modsecViewGraphs(modsec_entries)
File "modsecurity-parser.py", line 437, in modsecViewGraphs
plt.subplot(ax21)
File "/usr/local/lib/python3.8/dist-packages/matplotlib/pyplot.py", line 1272, in subplot
key = SubplotSpec._from_subplot_args(fig, args)
File "/usr/local/lib/python3.8/dist-packages/matplotlib/gridspec.py", line 632, in _from_subplot_args
raise ValueError(
ValueError: Single argument to subplot must be a three-digit integer, not AxesSubplot(0.125,0.420962;0.149038x0.148077)

Graph error

When I execute py via Pycharm terminal, I receive error with graph exporting.

Comman I use is this:

.\modsecurity-parser.py -f modsec_audit.log -g mypicture

Errors are in attached screenshot.
How to resolve that issues?
Thank you!

ValueError: Single argument to subplot must be a three-digit integer, not <Axes: >

First time clone, install pre-requisites and run. It simply throws a stack trace:

[shaund@peregrine modsecurity-parser]$ python3 modsecurity_parser.py -f /home/shaund/tmp/mod_sec/modsec_audit.log
input_filename: /home/shaund/tmp/mod_sec/modsec_audit.log
----- modsec_audit events processed: 990 -----
----- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID: 0 -----
Traceback (most recent call last):
File "/home/shaund/Software/modsecurity-parser/modsecurity_parser.py", line 864, in
output_with_graphs = modsec_view_graphs(modsec_entries)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/shaund/Software/modsecurity-parser/modsecurity_parser.py", line 565, in modsec_view_graphs
plt.subplot(ax21)
File "/usr/lib64/python3.11/site-packages/matplotlib/pyplot.py", line 1323, in subplot
key = SubplotSpec._from_subplot_args(fig, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/site-packages/matplotlib/gridspec.py", line 575, in _from_subplot_args
raise ValueError(
ValueError: Single argument to subplot must be a three-digit integer, not <Axes: >

Json not vaild

Hi,
the .json output is not valid, tested on https://jsonlint.com/?code= on sample file modsec_audit2_2018-06-11_20-16-27.json report the following problem:

Error: Parse error on line 61: ...de": "ENABLED" }} { "transaction": { ---------------------^ Expecting 'EOF', '}', ',', ']', got '{'

Problem with modsecurity 2.9.2

I get the following error when I try to analyse my modsec audit log:

Traceback (most recent call last):
  File "modsecurity-parser.py", line 588, in <module>
    json_modsec_entry = modsecLog2Info(modsec_entry)
  File "modsecurity-parser.py", line 499, in modsecLog2Info
    modsec_f_headers = dict(map(lambda s: [s, '-'] if len(s.split(': ')) == 1 else s.split(': '), modsec_f[1:-1]))
ValueError: dictionary update sequence element #8 has length 3; 2 is required

Time format

Hello =)

I noticed that if difference in time zone is specified with "--" the program gives an error.

python3 modsecurity-parser.py -g LOL.png -f modsec_audit.log inputFileName : modsec_audit.log ---------- modsec_audit events processed: 9577 ---------- ---------- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID : 0 ---------- Traceback (most recent call last): File "modsecurity-parser.py", line 689, in <module> outputWithGraphs = modsecViewGraphs(modsec_entries) File "modsecurity-parser.py", line 360, in modsecViewGraphs event_times = list(map(lambda x: datetime.strptime(x, LOG_TIMESTAMP_FORMAT).replace(tzinfo=None), event_times1)) File "modsecurity-parser.py", line 360, in <lambda> event_times = list(map(lambda x: datetime.strptime(x, LOG_TIMESTAMP_FORMAT).replace(tzinfo=None), event_times1)) File "/usr/lib64/python3.6/_strptime.py", line 565, in _strptime_datetime tt, fraction = _strptime(data_string, format) File "/usr/lib64/python3.6/_strptime.py", line 362, in _strptime (data_string, format)) ValueError: time data '03/Mar/2020:08:52:27 --0500' does not match format '%d/%b/%Y:%H:%M:%S %z'

Not sure if this something wrong with my settings. However quick work-around is to replace the --0500 to +0500 in each line for time and everything looks good.

Also if there are around 10000 events the program exists with the killed error for me.
python3 modsecurity-parser.py -x auditlog_1 -f prefix_aa inputFileName : prefix_aa ---------- modsec_audit events processed: 12430 ---------- ---------- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID : 0 ---------- Killed

I did split the audit log, and this resolves the issue. Suspect this is related to the 1gb of ram on my virtual machine =)

This is just FYI, not sure this is something you will be interested in correcting.

Thanks a lot anyway, this program saved me a lot of time by helping me to analyze the mod_security audit logs.

IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

Hi Guys,
run this script, met issue(as title), can help? thanks

my modsecurity(3.0.9) logs format:

{"transaction":{"client_ip":"10.200.101.16","time_stamp":"Thu May 11 02:13:58 2023","server_id":"6c63a629cf8ef75665cbe6abb55daaf9d4fa7b2b","client_port":33042,"host_ip":"172.22.0.2","host_port":80,"unique_id":"168377123884.428748","request":{"method":"GET","http_version":1.1,"uri":"/pub/","headers":{"Connection":"Keep-Alive","Host":"10.200.101.18","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"}},"response":{"body":"<!--\n\n    Copyright © 2016-2023 The Thingsboard Authors\n\n    Licensed under the Apache License, Version 2.0 (the \"License\");\n    you may not use this file except in compliance with the License.\n    You may obtain a copy of the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS,\n    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n    See the License for the specific language governing permissions and\n    limitations under the License.\n\n-->\n<!doctype html>\n<html lang=\"en\" style=\"width: 100%; height: 100%;\">\n<head>\n  <meta charset=\"utf-8\">\n  <title>ThingsBoard</title>\n  <base href=\"/\">\n\n  <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n  <link rel=\"icon\" type=\"image/x-icon\" href=\"thingsboard.ico\">\n  <link rel=\"preload\" href=\"assets/fonts/MaterialIcons-Regular.ttf\" as=\"font\" type=\"font/ttf\" crossorigin=\"anonymous\"/>\n  <link rel=\"stylesheet\" href=\"assets/fonts/material-icons.css\"/>\n  <style type=\"text/css\">\n\n    body, html {\n      height: 100%;\n      overflow: hidden;\n      background-color: #eee;\n    }\n\n    .tb-loading-spinner {\n      margin: auto;\n      z-index: 1;\n      position: absolute;\n      top: 0;\n      bottom: 0;\n      left: 0;\n      right: 0;\n      width: 136px;\n      height: 30px;\n      text-align: center;\n    }\n\n    .tb-loading-spinner > div {\n      width: 30px;\n      height: 30px;\n      margin-right: 10px;\n      background-color: rgb(43,160,199);\n\n      border-radius: 100%;\n      display: inline-block;\n      -webkit-animation: tb-bouncedelay 1.4s infinite ease-in-out both;\n      -moz-animation: tb-bouncedelay 1.4s infinite ease-in-out both;\n      animation: tb-bouncedelay 1.4s infinite ease-in-out both;\n    }\n\n    .tb-loading-spinner .tb-bounce1 {\n      -webkit-animation-delay: -0.32s;\n      -moz-animation-delay: -0.32s;\n      animation-delay: -0.32s;\n    }\n\n    .tb-loading-spinner .tb-bounce2 {\n      -webkit-animation-delay: -0.16s;\n      -moz-animation-delay: -0.16s;\n      animation-delay: -0.16s;\n    }\n\n    @-webkit-keyframes tb-bouncedelay {\n      0%, 80%, 100% { -webkit-transform: scale(0) }\n      40% { -webkit-transform: scale(1.0) }\n    }\n\n    @-moz-keyframes tb-bouncedelay {\n      0%, 80%, 100% { -moz-transform: scale(0) }\n      40% { -moz-transform: scale(1.0) }\n    }\n\n    @keyframes tb-bouncedelay {\n      0%, 80%, 100% {\n        -webkit-transform: scale(0);\n        -moz-transform: scale(0);\n        transform: scale(0);\n      } 40% {\n          -webkit-transform: scale(1.0);\n          -moz-transform: scale(1.0);\n          transform: scale(1.0);\n        }\n    }\n\n  </style>\n<link rel=\"stylesheet\" href=\"styles.10895964a4a3aa21d65a.css\"></head>\n<body class=\"tb-default\">\n  <tb-root></tb-root>\n  <div id=\"tb-loading-spinner\" class=\"tb-loading-spinner\">\n    <div class=\"tb-bounce1\"></div>\n    <div class=\"tb-bounce2\"></div>\n    <div class=\"tb-bounce3\"></div>\n  </div>\n<script src=\"runtime.286f6982886cb90bbe7a.js\" defer></script><script src=\"polyfills.e2023dc347cde42f7c8d.js\" defer></script><script src=\"scripts.d93c5ee41f6da54bd100.js\" defer></script><script src=\"vendor.3f3611f892c51888617d.js\" defer></script><script src=\"main.ed39576ce9947da26638.js\" defer></script></body>\n</html>\n","http_code":200,"headers":{"Accept-Ranges":"bytes","Vary":"Origin","Vary":"Access-Control-Request-Method","Vary":"Access-Control-Request-Headers","Connection":"keep-alive","Last-Modified":"Tue, 07 Feb 2023 14:18:35 GMT","Last-Modified":"Tue, 07 Feb 2023 14:18:35 GMT","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","Content-Type":"text/html;charset=UTF-8","Content-Length":"3345","Date":"Thu, 11 May 2023 02:13:58 GMT","Server":"nginx/1.22.1","X-Content-Type-Options":"nosniff","X-Content-Type-Options":"nosniff","X-XSS-Protection":"1; mode=block","Pragma":"no-cache","Content-Language":"en","Expires":"0","X-Frame-Options":"SAMEORIGIN"}},"producer":{"modsecurity":"ModSecurity v3.0.9 (Linux)","connector":"ModSecurity-nginx v1.0.3","secrules_engine":"Enabled","components":["OWASP_CRS/4.0.0-rc1\""]},"messages":[{"message":"Host header is a numeric IP address","details":{"match":"Matched \"Operator `Rx' with parameter `(?:^([\\d.]+|\\[[\\da-f:]+\\]|[\\da-f:]+)(:[\\d]+)?$)' against variable `REQUEST_HEADERS:Host' (Value: `10.200.101.18' )","reference":"o0,13o0,13v48,13","ruleId":"920350","file":"/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf","lineNumber":"744","data":"10.200.101.18","severity":"4","ver":"OWASP_CRS/4.0.0-rc1","rev":"","tags":["modsecurity","application-multi","language-multi","platform-multi","attack-protocol","paranoia-level/1","OWASP_CRS","capec/1000/210/272","PCI/6.5.10"],"maturity":"0","accuracy":"0"}}]}}

incorrect HELP=Yes parameters

$ diff -N0u run.sh.orig run.sh
--- run.sh.orig 2024-01-27 15:15:45.961138946 +0100
+++ run.sh 2024-01-27 15:16:03.588852800 +0100
@@ -26,2 +26,2 @@

echo " E={filename} - IP addresses to exclude (space separated, enclosed by parenthesis)"
echo " I={filename} - IP addresses to include (space separated, encolsed by paranthesis)"

echo " EXCLUDE={filename} - IP addresses to exclude (space separated, enclosed by parenthesis)"
echo " INCLUDE={filename} - IP addresses to include (space separated, encolsed by paranthesis)"

--version3 and libmodsecurity 3 parsing exceptions

Hy,
've compile libmodsecurity3 (3.0.9) and apache-connector.
How resolve it?

----- modsec_audit events processed: 68 -----
----- modsec_audit events skipped by INCLUDED/EXCLUDED options or INVALID: 0 -----
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'messages'
Exception in Graph TOP 10 Attacks intercepted 'messages'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses: 'remote_address'
Exception in TOP 20 rule hits: 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc
Exception at modsec_save_xlsx() :'transaction_id', transaction_id :ZF3-elttbGFBrieJDfTmbQAAAAc

ValueError: time data

Hi! I am using ModSecurity 3.0.9 + Nginx and I get this error:
ValueError: time data '19/May/2023:00:21:35 +0500' does not match format '%d/%b/%Y:%H:%M:%S %z'
It seems like the mask is completely suitable, but still something is wrong.
I'll attach part of the log.
1.txt

IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

Team,

Excellent tool. For some reason when I want to generate the image or the excel gives me an error.

Is it a compatibility issue or does the log have something difficult to partial?

What is the level of debugging desired by the tool?

command line : python3 modsecurity-parser.py -x 1.xls -g 1.png --jsonaudit --jsononeperline --version3 -f modsec_audit.log

Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Exception in Graph TOP 10 IP source addresses 'remote_address'
Exception in TOP 20 rule hits 'audit_data'
for transaction_id : -
Exception in Graph TOP 10 Attacks intercepted 'audit_data'
Traceback (most recent call last):
File "modsecurity-parser.py", line 689, in
outputWithGraphs = modsecViewGraphs(modsec_entries)
File "modsecurity-parser.py", line 359, in modsecViewGraphs
event_times1 = np_event_time_action[:, 0]

regards

in Detection only mode on hapee-2.4/libmodsecurity3 no rules are displayed in the report

I've used modsecurity_parser.py to analyse a 27gb modsec_audit.log file with over 4.5million modsec events spread over a total of 171 million rows.

the issue is that it's giving no insight on the rules that are passed and so i get no intel when running modsec on haproxy enterprise in the Dectection only mode. Is there a way this can be enabled?. Thanks for a great tool!

MatplotlibDeprecationWarning using matplotlib 3.3.1

If matplotlib version is 3.3.1 you get the following deprecation warning:

modsecurity-parser.py:454: MatplotlibDeprecationWarning: normalize=None does not normalize if the sum is less than 1 but this behavior is deprecated since 3.3 until two minor releases later. After the deprecation period the default value will be normalize=True. To prevent normalization pass normalize=False 
  patches, texts, autotexts = plt.pie(intercepted_cnt_top10.values(), autopct='%1.1f%%', shadow=True, startangle=90, radius=1.0)

I've read that you're using an older version of that python lib, but in case you're interested in supporting newer versions you can update your code

File config.yaml.example for global settings

This is needed to config more feauters e.g. Choice file outputs.... or max events...or anything..

Doesn't work with ModSecurity 3.0

Haven't seen the log files for previous version but ModSecurity 3.0 Audit logs take the following pattern:
---PAbyO0H9---A--
This is different from the expected patter in the modsecurity-parser.py

# modsec_patterns
a_pattern = re.compile('^--\w{6,10}-A--$')
z_pattern = re.compile('^--\w{6,10}-Z--$')

Doesn't work with python 3.11 on Wondows 10/11 with matplotlib > 3.3

Can't install matplotlib==3.3.2 with python 3.11. PIP thows error.

pip3 install matplotlib works but it thows this error,

  File "C:\Users\Administrator\Downloads\modsecurity-parser-master\modsecurity_parser.py", line 864, in <module>
    output_with_graphs = modsec_view_graphs(modsec_entries)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\Downloads\modsecurity-parser-master\modsecurity_parser.py", line 565, in modsec_view_graphs
    plt.subplot(ax21)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\pyplot.py", line 1323, in subplot
    key = SubplotSpec._from_subplot_args(fig, args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Lib\site-packages\matplotlib\gridspec.py", line 573, in _from_subplot_args
    raise ValueError(
ValueError: Single argument to subplot must be a three-digit integer, not <Axes: >

Edit: Solved uninstalling Python 3.11 and installing Python 3.8.11

Error TypeError: pie() got an unexpected keyword argument 'normalize'

Hi,

I just updated to the most recent version on my server (running Ubuntu 18.04 LTS). Unfortunatelly I get the following error:

Traceback (most recent call last):
File "/opt/modsecurity-parser-master/modsecurity-parser.py", line 689, in
outputWithGraphs = modsecViewGraphs(modsec_entries)
File "/opt/modsecurity-parser-master/modsecurity-parser.py", line 454, in modsecViewGraphs
patches, texts, autotexts = plt.pie(intercepted_cnt_top10.values(), autopct='%1.1f%%', shadow=True, startangle=90, radius=1.0, normalize=True)
TypeError: pie() got an unexpected keyword argument 'normalize'

Any ideas?

The log file format can be json, for example, each log file seperated by blank line.

{ "time": "15/Jan/2024:00:01:32 +0530", "transaction_id": "16573246656640753946", "remote_address": "152.32.153.53",
    "request": {
        "request_line": "GET / HTTP/1.1",
        "headers": {
            "Host": "192.168.5.99",
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0"
        }
   }
    "response": "None",
    "audit_data": {
        "messages": [
            "Message: Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host. [file \"C:\\/Program Files/ModSecurity IIS/owasp_crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"810\"] [id \"920350\"] [rev \"2\"] [msg \"Host header is a numeric IP address\"] [data \"192.168.5.99\"] [severity \"WARNING\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"9\"] [accuracy \"9\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST\"] [tag \"WASCTC/WASC-21\"] [tag \"OWASP_TOP_10/A7\"] [tag \"PCI/6.5.10\"]"]
    }
}

{ "time": "15/Jan/2024:00:01:35 +0530", "transaction_id": "16573246656640753979"}

TypeError: 'set' object does not support indexing

hello, i am trying to see what your script does, but i am now getting: