dogoncouch / logdissect Goto Github PK

Option (off by default) to include zipped archive log files, if their mtimes are within the time range or if no time range is specified.

Module to output LogData objects as CSV files.

Expected Behavior

Set UTC date forward from GMT-04:00

Actual Behavior

UTC dates set behind GMT-04:00

Steps To Reproduce Behavior

Export log to JSON, check numeric_date_utc attribute.

Specifications

• Command tried:
• logdissect version: 3.0.1
• Python version:
• Operating system:
• Operating system version:
• Log file type:
• Other info that might be useful:

Easier to use and document.

Fix all known bugs so we can start adding functionality.

Add parse option in syslog parser for ISO 8601 timestamps (e.g. 2017-05-02T23:26:40+00:00).

Because syslog timestamps don't have milliseconds, sorting after a merge is only accurate to the nearest second. Entries from each file for each second will be lumped together.

Feature Idea

Looks good.

Can you contribute some parsing tricks to https://github.com/log2timeline/plaso ?

This seems more future-proof. Attributes could be added in new parsers without breaking existing functionality.

Feature Idea

Handle output from pipes.

Geared toward use by other programs.

Add output module for log file format (same as input file, basically).

Parse raw syslog packets.

Finalize all names (variables, objects, methods, etc.).

Get to the point where we can figure out that something is broken.

Entries are sorted by UTC date stamp, which is set using the current system time zone. If log entries span daylight savings time shifts, this could be a problem.

Set with syslog parser.

Add morph_options and output_options objects with argument attributes; add instructions to README-API.md, manual page.

Bug

Overview

logdissect crashes when passed a gzipped file.

Steps To Reproduce Behavior

logdissect .gz

Specifications

• Command tried: logdissect .gz
• logdissect version: 3.0.2
• Python version: any
• Operating system: Ubuntu Linux
• Operating system version:
• Log file type: any gzipped
• Other info that might be useful:

Get to the point where testing can start.

Feature Idea

Add post_parse_action parser method for more customization. Entries will be passed through this method after lines are parsed, to add the ability to customize parsers a bit more.

Fix sorting when log crosses new year.

Add setup script for easy installation.

Add documentation.

Add daemon option to follow log inputs in real time.

Next release will have parsers (and possibly morphers) in an easy-to-use module that is accessible by other programs.

Syslog entries with no source host attribute don't parse right. This is the default configuration on FreeBSD, among others. The --no-host option has been added. It will be included in version 1.3.1 or 1.4, whichever comes next.

Add grep-like morpher.

Get everything off the ground so we can start looking for bugs

Let people pipe output, etc. Add -s option to silence terminal output.

Bug and solution

if want to parse gz log file, the parser has some bugs.
Exactly, the function parser_file() in parser/type.py,

To sove this problem
loglines = reversed(logfile.readlines())
===> loglines = reversed([x.decode('utf-8') for x in logfile.readlines()])

Because, in the original loglines is a list of bytes. python regular expression(re) can parse only strings.
So, the above solution i convert bytes list to strings list .

enjoy this project code and have a nice day :)

Specifications

• Command tried:
• logdissect version:
• Python version:
• Operating system:
• Operating system version:
• Log file type:
• Other info that might be useful:

Remove entries containing a pattern from the final output.

Add a parser for pcap files.

Examples:
--last=5m
--last=1h
--last=3d