Code Monkey home page Code Monkey logo

weevil's Introduction

Weevil

Latest Release Latest Build Security Rating

What is Weevil?

WeevilDemo

Weevil is an open-source .NET project that is used by analysts to extract valuable insights from log files. It's all about "boring log files for tasty bytes".

A complete list of features can be found in the release notes.

Key Features

  1. File and Record Level Notes
    • Capture high-level observations as remarks, or low-level details as record comments.
  2. Persisted State
    • Automatically load filter history, record comments, and file level comments when opening a log file.
    • Share the application's state as an XML sidecar with colleagues.
  3. Non-Destructive Operations
    • The Weevil application ensures that the original log file is never modified.
  4. Simplified Callstacks
    • When a record includes an exception call stack, Weevil simplifies the call stack by only displaying business logic references.
  5. Clear Operations
    • This operation removes records from memory, thus reducing the RAM footprint and speeding up the filtering process.

Filtering

One or more filter criteria can be used to show or hide log file records.

  1. Inclusive and Exclusive Filters
    • Display records matching the inclusive filter while hiding those matching the exclusive filter.
  2. Filter Criteria
    1. Plain Text
    2. Regular Expressions
    3. Aliases
      • Frequently used or complex filters can be assigned a unique key that can be used to speed up the filtering process.
      • For example, the #IpAddress key could be assigned to the following filter criteria ^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$.
    4. Monikers
      • Monikers are built-in keys that can be used to query metadata collected by Weevil.
        • For example, the @Comment can be used to identify records that have a user comment.
  3. Multiple Criteria
    • Multiple filter criteria can be combined together using a logical "OR" operator (||).
  4. Pinned Records
    • Pinned records are guaranteed to be included in the filter results.

Navigation

  1. Find
    • Search for text within filtered results.
  2. Go To
    • Jump to specific line numbers or timestamps.
  3. Pinned Records
    • Effortlessly navigate between important records.
  4. Flagged Records
    • Move between records flagged during prior analysis.
  5. Record Comments
    • Navigate between records containing user comments.

Analysis

Utilize Regular expression named groups to identify key data in log files. Leverage Weevil's analysis tools to then extract data and identify trends.

Each analysis tool updates the Comments fields with the values that match the provided named group(s), and the recor's Flagged field is set.

  1. Detect Data
    • For example: extracting URLs from a log file
  2. Detect Data Transitions
    • For example: when a hardware serial number changes
  3. Detect Rising Edges
    • For example: detecting peek CPU usage
  4. Detect Falling Edges
    • For example: detect when a firmware's uptime has reset
  5. Detect Temporal Anomalies
    • For example: detect when records are logged out of order

Furthermore, Weevil includes the ability to generated graphs based on the extracted data.

Extensible Architecture

Maximize potential by developing domain-specific extensions tailored to your business' needs. Weevil can be enhanced by custom plugins:

  1. Log File Parsers
    • Create tailored parsers to accurately interpret log files from various sources and formats, ensuring seamless integration with Weevil.
  2. Log File Analyzers
    • Design specialized analyzers to process and extract valuable insights from the parsed log data, optimizing the analysis for your specific business domain.
  3. Dashboard Insights
    • Develop custom dashboard visualizations and insights that highlight the most relevant information, enabling efficient decision-making and improved understanding of your log data.

Software Development

WPF Application

NuGet Packages

Latest Release NuGet Package
latest version BlueDotBrigade.Weevil.Common.nupkg
latest version BlueDotBrigade.Weevil.Core.nupkg
latest version BlueDotBrigade.Weevil.Windows.nupkg

A .NET application can use Weevil's feature set by directly referencing the BlueDotBrigade.Weevil.Core NuGet package.

For example, one could determine when equipment was changed using the following sample code:

var engine = Engine
   .UsingPath(@"C:\Temp\hardware.log")
   .Open();

// The `UniqueId` regular expression named group is used to
// capture serial hardware serial numbers.
engine.Filter.Apply(
   FilterType.RegularExpression,
   new FilterCriteria(@"Received hardware message. ID=(?<UniqueId>[a-zA-Z0-9]+)"));

// This type of analysis compares the captured serial numbers,
// and flags the record when a value changes.
engine.Analyzer.Analyze(AnalysisType.DetectDataTransition);

foreach (var record in engine.Filter.Results.Where(r => r.Metadata.IsFlagged == true))
{
   Console.WriteLine(
   $"{record.CreatedAt} {record.Metadata.Comment}");
}

Development

Attribute Description
GitHub Latest Release The list of features & bug fixes for the latest Weevil release.
Latest Stable Source code for the most stable version of Weevil.
Latest Code The most up-to-date source code. This branch includes features that are still under development.
Latest Build A value of passing indicates that the main branch is compiling & that the automated tests have passed.
GitHub Repository Size Total size of Weevil's Git repository.
Lines of code Total number of lines of code in the Git repository.
Last Commit Indicates when the Git repository was last updated.
Security Rating SonarCube: Number of security issues detected.
Vulnerabilities SonarCube: Number of security vulnerabilities detected
Maintainability Rating SonarCube: Represents the project's SQALE rating.
Code Smells SonarCube: Characteristics of the code base that suggest the design may have maintenance issues.

Guidelines

  • When working on the WPF application, please be sure to follow the Style Guide for the user interface.

Compiling

The following steps outline how to build Weevil's WPF application:

  1. Download the latest stable release source code.
  2. If you have implemented a custom Weevil plugin:
    • Prior to starting Visual Studio, create the following Windows [environment variable][EnvironmentVariable]:
      • %WEEVIL_PLUGINS_PATH% which refers to the directory where the Weevil plugin assembly (*.dll) can be found.
  3. Using Visual Studio, compile the WPF project: BlueDotBrigade.Weevil.Gui [EnvironmentVariable]: https://en.wikipedia.org/wiki/Environment_variable#Windows

Verification

Software integrity is verified through a number of automated tests which can be found in the /Weevil/Tst/ directory:

  • UnitTests
  • FunctionalTests

Recognition

  • PostSharp
    • PostSharp`s aspect oriented library helps to simplify a code base by reducing boilerplate. Special thanks to the PostSharp team for donating a license.
  • GitHub
    • Free Git repository hosting platform for this project & many others like it.

Open Source Projects

  • Live Charts
    • Beto Rodriguez et al. have developed an impressive WPF charting library. Am looking forward to future releases.
  • Material Design in XAML
    • An excellent WPF library that helps to standardize themes & improve the overall quality of an application's user interface.

Contributors

A special thanks to all of those who have contributed to this project.

weevil's People

Contributors

bluedotbrigadefounder avatar orbitthree avatar pressacco avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

weevil's Issues

Filter `ComboBox` should only display list of options when down arrow clicked.

Since implementing the new dark theme (#11), I have noticed that the ComboBox behavior has changed.

Expected Behavior

  • Similar to the Windows WPF ComboBox behavior, the drop-down should only expand when you click on the down arrow button:
    • Click on the TextBox area of the ComboBox, and the user can start typing.
    • Click on the down arrow, and a list of choices is displayed.

Actual Behavior

  • Clicking anywhere on the combo box (including the TextBox area) and a list of choices is displayed.
    -- This means: anytime the ComboBox has focus, then the list of choices will visually block the rest of the Weevil UI.

Additional Context

Filter history stops working after log file `Reload()`.

Steps To Reproduce

  1. Open log file.
  2. Apply an inclusive filter (e.g. 001).
  3. Reload the log file using Ctrl+F5.
  4. Apply a new inclusive filter (e.g. 002).

Results

Expected

  • Inclusive filter history should show 002 followed by 001.

Actual

  • Inclusive filter history only shows the original filter: 001

As a developer, I expect the `About` dialog to include CPU & RAM details, to make it easier to troubleshoot problems in the field.

Related articles:

As a developer, the Weevil application should monitor UI responsiveness, to evaluate UX in a production environment.

Having an application monitor it's user interface (UI) responsiveness is not a new concept - there are plenty of articles on the Internet about this subject. For example:

`Value could not be converted` validation error appears when `Pinned` unchecked.

Steps to Reproduce

  1. Open any log file (e.g. sample.log) in Weevil.
  2. Add a comment to record number 5: HelloWorld
  3. Save the metadata by pressing: Ctrl+S
  4. Open the meta data file (e.g. sample.log.xml)
  5. Record number 5 IsPinned will be true
  6. In Weevil, uncheck the Pinned value for record number 5.

Results

Actual

The Pinned checkbox will display a validation error:

  • Value could not be converted.

If you attempt to save the metadata, you will see that the value does not update:

  • IsPinned=true

Expected

Nothing should happen... the Pinned checkbox should remain unchecked.

Exception thrown when switching between different types of log files.

Steps To Reproduce

  1. Open a log file that uses a Weevil extension (implemented by IPlugin)
  • The plugin should return at least one analyzer when ICoreExtension.GetAnalyzers() is called
  • This will create customized menu entries in Weevil.
  1. Open log file that does not use a Weevil extension.

Results

Expected

  • The Custom Analyzers from step 1 should no longer be visible in the menu.

Actual

  • Weevil throws an exception.

`Clear Before & After` edge case results in all records being unexpectedly cleared.

Steps to Reproduce

  1. Open large log file (e.g. Large.log)
  2. Towards the top of the file (e.g. record 1000), add a user comment: START.
  3. Towards the bottom of the file (e.g. record 2,500,000) add user comment: END.
  4. Save metadata & close Weevil.
  5. Start a new instance of Weevil.
  6. Open the previous log file (e.g. Large.log).
  7. Inclusive filter: @Comment
    • When the results appear, START should already be selected.
  8. Holding Shift, select the second record: END
  9. Perform a Clear Before & After

Results

Expected

  • All records before & after the selection should be cleared.

Actual

  • Only the End record remains in memory.
    • TotalRecordCount in the status bar says: 1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.