Code Monkey home page Code Monkey logo

azure-eventhub-logscale-ingester's Introduction

CrowdStrike Logscale Ingester
Twitter URL

CrowdStrike LogScale ingester for Event Hubs

The LogScale Azure Event Hub Collector is an open source project and not a CrowdStrike product. As such, it carries no formal support, expressed, or implied.

Event Hubs are data/event ingesters which can be integrated with functions and services (Azure internal and external).
As Event Hubs are often used as temporary storage of data/events, we can utilize Azure Logic Apps to forward the data/events to CrowdStrike LogScale for storage, analytics, or other purposes.

Set Up

1. Set up a LogScale repository, parser, and ingest token

Refer to Logscale Documentation for:

The following script can be copied and pasted for the custom parser:

parseJson(@rawstring)
| split("ContentData.records", strip=true) // Split into individual events
| drop([@rawstring, _index]) // Clean up

2. Create a Logic App within Azure

The Logic App will contain all of the workflows, each of which is used to forward data from a single Event Hub instance. LogicAppSetup

3. Create a Workflow for each Event Hub Instance

Each Event Hub instance to be ingested into LogScale requires a separate Workflow. To create a workflow, a Trigger and Action needs to be specified.
Workflow1

For the Trigger, we are utilizing the "Event Hub" Trigger in the "Azure" tab.
WorkflowTrigger WorkflowTrigger2
The trigger requires a connection with an Event Hub Namespace via the Connection String . The Connection String can be found after creating a Shared Access Policy within the Event Hub after adding an access policy in the "Shared access policies" tab with the "Listen" claim. The "Connection string–primary key" will be generated and can be used for the Workflow trigger.
ConnectEventHub
After the Event Hub Namespace is connected, the trigger can be set up with an Event Hub Instance from the dropdown list, with the Content Type "application/json", the default Consumer group name, Maximum Event Count set to "175" and the interval to check for new events being "1 Second.
TriggerSettings
In the settings tab of the Trigger, turn off "Split On" as shown in the screenshot below.
TriggerSettings2
For the Action, the simple "HTTP" action is needed with the Method being a "POST" request and the URI being the LogScale HEC URL (which should look similar to this: https://your-logscale-url.com/api/v1/ingest/hec/raw). The Header key and value, we follow the LogScale HEC format:

Key: "Authorization"  
Value: "Bearer <Insert your Ingest Token created in Step 1>"

For the Body, "Body" needs to be selected to have the events in the correct format to be parsed into LogScale.
ActionSettings

Finally, the workflow can be saved, which will start the execution of the workflow and forward the data/events to LogScale.

Notes


  • Batching of multiple events coming from Azure as a single log in LogScale as 175 events per second is the maximum allowed by Azure.

Future Additions


  • Dashboard are currently not available, however will be published as a LogScale package on the Marketplace once available


WE STOP BREACHES

azure-eventhub-logscale-ingester's People

Contributors

daningu avatar jacobmovingfwd avatar jshcodes avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.