osmncht / spilp Goto Github PK

# Copyright 2011, Tihomir Kit ([email protected])
# spilp is distributed under the terms of GNU General Public License v3
# A copy of GNU GPL v3 license can be found in LICENSE.txt or at http://www.gnu.org/licenses/gpl-3.0.html


	Simple Python IIS log parser (spilp) is a simple Python parser that takes IIS logs, 
	parses them and creates statistical reports which can be used to discover unusual 
	IP activity more easily. 


 1. Features
 2. Usage
 3. Download links


 
1. Features
--------------------------------------------------------
 - extracts a list of IP addresses with number of hits they made sorted by number of hits
 - extracts a list of "close" IP addresses that made a certain number of hits
 - extracts a list of user agents sorted by number of hits
 - extracts a list of cs-method hits (GET method excluded)
 - extracts a list of file hits sorted by number of hits
   - .pdf, .doc, .xls, .ppt (document files)
   - .js, .htm, .asp (web files)
 - extracts extended information for document and web file hits
   - includes timestamps, client IP addresses, methods, ports, user agent details and http status codes
 - extracts a list of "unusual" http status code hits sorted by number of hits
   - client IP address list
   - a list of files hit by an IP and number of hits for that file
 - filtering results (include or exclude filtering - works in "either-or" way)
   - ability to auto-generate an IP range list as a filter
 - reverse DNS country lookup using MaxMinds GeoIP country downloadable database 
   - additional info in certain reports
   - filtering results by country of origin (as a separate filtering option using spilpconf.py file) 
 - ability to process large amount of IIS log files
 - CONFIG file for performance and output tweaking

 

2. Usage
--------------------------------------------------------
Spilp is written in python3.2 so you will need that installed (if not using Windows binary
version).

Spilp requires pygeoip python module to work properly. It also requires GeoIP country free 
database (find links below). After downloading GeoIP.dat.gz from MaxMind website, extract 
the archive, and put GeoIP.dat database into the same folder as spilp.py. GeoIP country 
database binary should be updated every once in a while to get the latest country IP ranges. 

To tweak the amount of output and/or performance, edit spilpconfig.py.
Note that if you have large amounts of log files (more than 2/3 of your free amount of RAM), 
you must change DB_STORAGE option to ".tempdb".

IIS log files will be automatically parsed if they are in spilp root directory or in any 1st
level subdirectory residing in spilp root directory.

Spilp currently works with default IIS log format meaning that it uses the following IIS fields:
date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status

If you are using a different IIS log format, you will need to add/remove certain variables in
splitLogLine() function in spilp.py (left part of the line that contains "log_line.split").
Be careful when doing so though, because you must have the same number of variables there
as you have fields in your IIS logs and they must be in same order. Keeping variable names
the same is recommended unless you know what you are doing.

To use an IP range list for filtering, use the following syntax:
$$.ip_range(192.168.1.1 - 192.170.127.234)
and put that line into your FILTERS.txt file as you would do with any other filter expression. 

Windows users can use spilp.exe binary. In that case there is no need for Python and pygeoip 
to be installed on the machine. GeoIP country database still needs to be downloaded from
MaxMinds website and it needs to be put into the same folder where spilp.exe is located.



3. Download links
--------------------------------------------------------
pygeoip: http://code.google.com/p/pygeoip
GeoIP country: http://www.maxmind.com/app/geolitecountry
GeoIP database: http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz