An simple log agent aimed for use with Elasticsearch and Kibana. The main use is to take a line of log from disk and tranform it to a json that can be index by ElasticSearch.
Command line:
cmd> nuget install LogFlowPackage Manager console in Visual Studio:
PM> Install-Package LogFlowRead and send a log line
2013-09-11 11:53:43 WARN All log are belong to us
Send it to ElasticSearch formated JSON
{
'@timestamp': '2013-09-11 11:53:43'
'LogLevel': 'WARN'
'Message': 'All log are belong to us'
}Create a project of type Class Library Install nuget package
PM> Install-Package LogFlowTwo config files used by log flow will be added to your project. LogFlow.exe.config, setup for storage and web interface, and NLog.config, log configuration.
A flow has and one input, multiple processors and one output.
public class MyLogFlow : Flow
{
public MyLogFlow()
{
CreateProcess("InsteadOfClassName")
.FromInput(new FileInput("C:\\MyLogPath", Encoding.UTF8, true))
.Then(new MyLogLineParser())
.ToOutput(new ElasticSearchOutput(new ElasticSearchConfiguration()
{
Host = "localhost",
Port = 9200,
IndexNameFormat = @"\m\y\L\o\g\s\-yyyyMM" //new index each month
}));
}
}Included inputs are FileInput
Included outputs are ElasticSearchOutput and ConsoleOutput
No processor are included.
A processor takes a result and returns a result. Result contains the read line, the resulting json structure, event time stamp and the ability to cancel the flow and read next line.
public class MyLogLineParser : LogProcessor
{
public override Result Process(Result result)
{
var logLine = result.Line;
var timestampPosition = logLine.IndexOf(" ", logLine.IndexOf(" ") + 1);
var eventTimeStamp = DateTime.Parse(logLine.Substring(0, timestampPosition).Trim(), CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal);
var logLevelPosition = logLine.IndexOf(" ", timestampPosition + 1);
var logLevel = logLine.Substring(timestampPosition, logLevelPosition - timestampPosition).Trim();
var message = logLine.Substring(logLine.IndexOf(" ", logLevelPosition)).Trim();
result.EventTimeStamp = eventTimeStamp;
result.Json.Add("LogLevel", new JValue(logLevel));
result.Json.Add("Message", new JValue(message));
return result;
}
}Try not to over write the Json property only add to it because erlier steps might have added data to it. There is MetaData to help you to transport data in the flow without it saving to the output.
- Sucessfully build the project in Debug
- Project properties > Debug > Start external program > Select logflow.exe in debug folder.
- Press F5
LogFlow.exe install
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent