scorelab / openxdr Goto Github PK

This component contains the log colletor for the following Linux log collectors.

• Application Logs.
• Event Logs.
• Service Logs.
• System Logs.

This repos should follow Gitflow workflow and follow are the items you have to check.

• There is a master branch and a develop branch.
• master branch is locked for direct commits and,
  • Require pull request reviews before merging is active
  • Require status checks to pass before merging is active
    • Require branches to be up to date before merging
    • Appropriate Status checks are required
  • Enforce all configured restrictions for administrators is active
  • Restrict only maintainers group can push to the master branch
• develop branch is locked for direct commits and,
  • Require pull request reviews before merging
    • Dismiss stale pull request approvals when new commits are pushed is active
  • Require pull request reviews before merging is active
  • Require status checks to pass before merging is active
    • Require branches to be up to date before merging
    • Appropriate Status checks are required
  • Enforce all configured restrictions for administrators is active
  • Restrict only maintainers group can push to the master branch

Tasks DoD:

• Literature study on network anomaly detection models that were open-source and deployed in large scale
• Introduce new direction of machine learning models which OpenADS can co-relate with
• Document at project wiki

As explained in the architecture, RDD can used to pass data-points between components, RDD is kind of immutable data-sets that are generated from any type of data objects. But do we have to identify a generic data structure to represent objects, or do we maintain desperate objects per data-source. That should be resilient for future plug-in data sources as well. come up with such

Based on identified libraries, implement syslog data-receiver

Task DoD:

• Implement streaming receiver for any of data-sources listed (e.g. pcap)

Succinct http://succinct.cs.berkeley.edu/wp/wordpress/
[Data-source] -> Spark Streaming -> {RDD} Document store at Spark

• List all possible data-sources with identified integration (e.g. package, third-party tool etc)
• Come up with a strategic plan to embed data sources to OpenADS space

refactor http endpoints to websocket connection

In the long run, we need to have a single dashboard to monitor incoming packets, filtering, visualization and analysis, This task is to come up with UI features