netevert / sentinel-attack Goto Github PK

Hi everyone,

facing issues getting Sysmon threat hunting workbook to work.
have installed whitelisting CSV storage files to storage container.
from dashboard i am seeing this error " 'project' operator: Failed to resolve table or column expression named 'process_create_whitelist"

not sure if there anything i missed out. appreciate anyone's assistance.

There's no technique_id, technique_name or phase_name attributed in Sysmon EventID 22.

Hi Olaf,

the Sysmon parser includes a time condition (i.e. ago). The best practice is to leave out the time condition from a parser and leave it to the parser user to add when using the parser.

~ Ofer

also FYI, your first link in the workbooks readme links to jupyter notebooks, not azure workbooks. close but not quite :D

Originally posted by @gardnerjr in #9 (comment)

Maybe something has changed in Azure, but the guide in https://github.com/BlueTeamLabs/sentinel-attack/wiki/Sysmon-Threat-Hunting-workbook---post-deployment-configuration can no longer be followed.

Azure does not allow me to use the exact workspace name as the underlying log analytics workspace of my Sentinel instance.
My Sentinel workspace name is over 24 characters, so when I was using my workspace name it gave me an error the name must be between 3 and 24 characters.

Then I tried creating using a shorter name and it gave me an error that the name is already taken, which it actually wasn't.
Only when I used a short name and also a number in the workspace name, the custom deployment worked.

Now inside that storage account container, I'm getting errors when I try to upload the whitelist csv files. And randomly after trying a few days, it worked. I got the files uploaded, but the ATT&CK trigger overview tab still gives me the same error.
Not sure where to go from here.

When running post-deployment:
https://github.com/BlueTeamLabs/sentinel-attack/wiki/Sysmon-Threat-Hunting-workbook---post-deployment-configuration
the ARM template prompts you for your workspace name.
However the template uses your workspace name to create a storage account, which only allows lower case letters.
So the template creation fails with the error:
blobstore is not a valid storage account name. Storage account name must be between 3 and 24 characters in length and use numbers and lower-case letters only.
Suggestion: Add a warning on the above link to enter your workspace name in lowercase letters.

hi

the default alert rules in sentinel for Threat Intelligence uses the security event 4688 as a source, woudl it be possible to rewrite this rule so i can use sysmon eventId 1 instead. from our tests using security events with azure sentinel even with the lowest setting generates a huge amount of logs and thus costs.

below is the original query that relies on eventid 4688

let dt_lookBack = 1h;
let ioc_lookBack = 14d;
ThreatIntelligenceIndicator
| where TimeGenerated >= ago(ioc_lookBack) and ExpirationDateTime > now()
| where Active == true
| where isnotempty(FileHashValue)
|  join (
  SecurityEvent | where TimeGenerated >= ago(dt_lookBack)
      | where EventID in ("8003","8002","8005")
      | where isnotempty(FileHash)
      | extend SecurityEvent_TimeGenerated = TimeGenerated, Event = EventID
)
on $left.FileHashValue == $right.FileHash
| summarize LatestIndicatorTime = arg_max(TimeGenerated, *) by IndicatorId
| project LatestIndicatorTime, Description, ActivityGroupNames, IndicatorId, ThreatType, ExpirationDateTime, ConfidenceScore, SecurityEvent_TimeGenerated, Process, FileHash, Computer, Account, Event
| extend timestamp = SecurityEvent_TimeGenerated, AccountCustomEntity = Account, HostCustomEntity = Computer

in the eventID 1 of the sysmon the hashes are not expilicitly seperated out, can this be "handled" or will I need to amend the sysmon_OSSEM function first / also

below is the result from the azure sentinel log query for eventID 1

TimeGenerated [UTC] | 2019-11-19T12:30:41.82Z
-- | --
  | Source | Microsoft-Windows-Sysmon
  | EventID | 1
  | Computer | XXXXX
  | UserName | NT AUTHORITY\SYSTEM
  | RenderedDescription | Process Create
  | event_creation_time | 2019-11-19T12:30:41.7500000Z
  | process_guid | {817354A9-E071-5DD3-0100-00102DD1DA40}
  | process_id | 11828
  | process_path | CXXXXXXXXXXX
  | file_version | 10.0.14393.0 (rs1_release.160715-1616)
  | file_description | Windows Command Processor
  | file_product | Microsoft® Windows® Operating System
  | file_company | Microsoft Corporation
  | process_commandline | Cmd.Exe
  | file_directory | cmd XXXXXXXXXXXXXXXXXXXX
  | user_name | C:\Program Files\XXXXXXXXXXXXXXXXXXXXXXX
  | user_logon_guid | NT AUTHORITY\SYSTEM
  | user_logon_id | {817354A9-2040-5DB2-0000-0020E7030000}
  | user_session_id | 0x3e7
  | process_integrity_level | 0
  | process_parent_guid | SHA1=99AE9C73E9BEE6F9C76D6F4093A9882DF06832CF,MD5=F4F684066175B77E0C3A000549D2922C,SHA256=935C1861DF1F4018D698E8B65ABFA02D7E9037D8F68CA3C2065B6CA165D44AD2,IMPHASH=3062ED732D4B25D1C64F084DAC97D37A
  | process_parent_id | {817354A9-E06A-5DD3-0100-001012D1D840}
  | process_parent_path | 8608
  | process_parent_command_line | C:\Windows\System32\XXXXXXXXXXXXXXX
  | technique_id | T1059
  | technique_name | Command-Line Interface
  | phase_name | Execution

it seems to me there is something off here with the process parent guid containing the file hashes ?

Convert all detection rules into AZSentinel YAML to enable automatic upload of analytics rules to Sentinel instances

Hello!

So I am deploying this to an instance of Sentinel I already have up and running. I added the parser, created the storage container, and uploaded the whitelists. I can't seem to figure out the custom functions for the whitelist tables. Is there any way I can get a bit of help with this?

Thanks in advance.

hi

what am i doing wrong here, it works for if I used the import-AzsentinelHuntingRule from the detections folder. and everything gets loaded up, ok
if i do the same from the detection folder with import-azsentinelAlertrule i get this error, is there something i missed or what?

the sentinel is in a seperate resource group but the the permissions, and the fact that everything works when i use the import-azsentinelHuntingRule makes it really wired that this is even an issue.

Import-AzSentinelAlertRule -SettingsFile "sentinel_attack_rules.json"

cmdlet Import-AzSentinelAlertRule at command pipeline position 1
Supply values for the following parameters:
WorkspaceName: XXXXXXXXXX
Import-AzSentinelAlertRule : Unable to connect to APi to get Analytic rules with message: The gateway did not receive a response from 'Microsoft.SecurityInsights' within the specified time period.
At line:1 char:1
+ Import-AzSentinelAlertRule -SettingsFile "sentinel_attack_rules.json"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-AzSentinelAlertRule

From any tab in the dashboard, I can see that it can reach the CSV files, but it returns no results.

When I look at the prerequisite section it says:
Whitelisting functions: that allow Sentinel-ATT&CK workbooks to query CSV storage files. The whitelisting functions are automatically provisioned within the Sysmon threat hunting workbook.

What is the whitelisting function? - I'm not sure it has automatically started working for me.
Is there a way to troubleshoot this?

Throughout your template in https://github.com/BlueTeamLabs/sentinel-attack/tree/master/hunting/workbooks, you have this:

        "crossComponentResources": [
          "/subscriptions/{Subscription_Id}/resourceGroups/{Resource_Group}/providers/Microsoft.OperationalInsights/workspaces/{Workspace_Name}"
        ],

in each query step.

if the user is already opening workbooks from inside sentinel, the workspace to query should already be "set" in the workbook's resources (in edit mode, click the gear icon in the toolbar, on the resources tab you should see a workspace already listed there). If that's the case, query steps of the workbook will "inherit" that resource automatically, it doesn't need to be explicitly listed in each step.

also, in the advanced editor, if you scroll all the way to the bottom in this state, there should be a couple properties like

  "defaultResourceIds": [
"something sentinel specific?",    "/subscriptions/subId/resourceGroups/rgName/providers/Microsoft.OperationalInsights/workspaces/workspaceName",
  ],

If that is the case, then the extra crossResourceIds section listing the workspace in every step is only there as an "override" from the default resources. so hypothetically, you can remove all of the crossComponentResources settings from the whole template you have, and just pasting it like it is should keep the window's defaultResourceIds intact and all the query steps would inherit that default workspace. That should just work without having to do a lot of replacements.

(if that doesn't work, the instructions chould probably say to copy the defaultResourceIds section from the bottom of the advanced editor when it opens into your template, and paste that back into the advanced editor).

I'm working with the Sentinel team on trying to clear up some of how they're doing the workbooks, so that this is a little simpler for everyone.

hi is there anywhere i can get input as to caculating what cost a setup like what you have build here for approx 100 servers

i have run sysmon and can see that my servers individually generate an approx abount of data pr day. This is easy to multiply and I can also look at the cost calculator in terms of GB sent to sentinel.

But what about the ksql queries, will i not get additional cost if i set these to run at 5,10,15 min interval, and will this cost not go up as I apply more of the ksql queries ?

In Kusto (the underlying database engine used for Sentinel) : for the cases when the full worked is looked up - it is better (perf-wise) to use 'has' instead of 'contains'.

See Kusto best query practices:
https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/best-practices

"When using string operators:
Prefer has operator over contains when looking for full tokens. has is more performant as it doesn't have to look-up for substrings."

I have followed your guides, however phase_name is not being captured in Sentinel:

However the parser was installed successfully:

And I can clearly see the phase_name in the sysmon event on the server:

So I'm a little lost. Also, when querying the log, the column names do not seem to correspond with the data they contain, which suggests perhaps an issue with the parser (?):

Originally posted by @SRThomson in #15 (comment)

"""If the user is already opening workbooks from inside sentinel, the workspace to query should already be "set" in the workbook's resources (in edit mode, click the gear icon in the toolbar, on the resources tab you should see a workspace already listed there). If that's the case, query steps of the workbook will "inherit" that resource automatically, it doesn't need to be explicitly listed in each step."""

Originally posted by @gardnerjr in #9 (comment)

From the "Configuring whitelisting functions and understanding how to use them" page, the following link is not found: https://github.com/BlueTeamLabs/sentinel-attack/tree/master/hunting/functions

The whitelisting functions are located in the hunting/functions folder and are distributed as .txt files. Each file contains a whitelisting function. To install the whitelisting functions you must:

don't you guys talk to each other at MS security dev/teams/products or am I missing something here?

;-) awesome work!!! really appriciated!

Hello,

The logic searches for several key processes in the process_path attribute and then searches the process_parent_command_line. Within this second filter it is incorrectly using an OR statement instead of another AND since both checks are !contains they should both be true. The OR statement will generate false positives as both can legitimately call the userinit.exe process in this particular example and only one needs to be satisfied as non-existent. This logic issue exists in all of the subprocesses.
or (process_path contains "userinit.exe"
and (process_parent_command_line !contains "dwm.exe" or process_parent_command_line !contains "winlogon.exe"))

Solution:
I propose you update the 'or' to an 'and' between each of the process_parent_command_line !contains statements.

Also, minor tweak the name of this KQL has a spelling issue in 'hollowing.'

I have followed your guides, however phase_name is not being captured in Sentinel:

However the parser was installed successfully:

And I can clearly see the phase_name in the sysmon event on the server:

So I'm a little lost. Also, when querying the log, the column names do not seem to correspond with the data they contain, which suggests perhaps an issue with the parser (?):

Originally posted by @SRThomson in #15 (comment)

when doing a log search with the following data show up
Event
| where Source contains "sysmon"
| where EventID == 3

When using the parser no data shows up

Sysmon | where EventID == 3

when using a different EventID (22) data is being parsed.

Hey,

Thank you for your effort you've put in this.
I deployed the lab with your terraform script but noticed my Workstation could not resolve the custom domain that I've setup on the DC.
It looks like the VNET was still using the default Azure DNS servers, shouldn't it be setup to use the DC as the DNS server?

I had a problem with missing Registry "SetValue" events in most of the Workbook queries, I would only see CreateKey (EventId 12) results.
Noticed that the data was present in the raw xml, and returned data as expected with something like:
Event
| where Source contains "sysmon"
| where EventID in (13,14)
| extend RenderedDescription = tostring(split(RenderedDescription, ":")[0])
| project TimeGenerated, EventID, RenderedDescription, Computer, UserName, Data=parse_xml(EventData).DataItem.EventData.Data

I then noticed that the sentinel-attack/Sysmon saved Search was missing SysmonEvent13_RegistrySetValue at the end in:
(union isfuzzy=true
SysmonEvent1_ProcessCreate,SysmonEvent2_FileCreateTime,SysmonEvent3_NetworkConnect,SysmonEvent4_ServiceStateChange,SysmonEvent5_ProcessTerminate,
SysmonEvent6_DriverLoad,SysmonEvent7_ImageLoad,SysmonEvent8_CreateRemoteThread,SysmonEvent9_RawAccessRead,SysmonEvent10_ProcessAccess,
SysmonEvent11_FileCreate,SysmonEvent12_RegistryObjectAddDel,SysmonEvent14_RegistryObjectRename,
SysmonEvent15_FileCreateStreamHash,SysmonEvent16_ConfigChange,SysmonEvent17_CreateNamedPipe,SysmonEvent18_ConnectNamedPipe,
SysmonEvent19_WMIEventFilter,SysmonEvent20_WMIEventConsumer,SysmonEvent21_WMIEventConsumerToFilter,SysmonEvent22_DNSEvents)

Adding SysmonEvent13_RegistrySetValue in between SysmonEvent12_RegistryObjectAddDel and SysmonEvent14_RegistryObjectRename fixed it.

I'm just seeing that sentinel-attack\deployment\gallery.azuredeploy.json has the corresponding missing item in this section:
"type": "Microsoft.OperationalInsights/workspaces/savedSearches",
"apiVersion": "2020-03-01-preview",
"name": "[concat(parameters('workspace_name'), '/Sysmon')]",
"dependsOn": [
"[resourceId('Microsoft.OperationalInsights/workspaces', parameters('workspace_name'))]"
],
"properties": {
"category": "sentinel-attack",
"displayName": "Sysmon",
...
\r\n(union isfuzzy=true\r\nSysmonEvent1_ProcessCreate,SysmonEvent2_FileCreateTime,SysmonEvent3_NetworkConnect,SysmonEvent4_ServiceStateChange,SysmonEvent5_ProcessTerminate,\r\nSysmonEvent6_DriverLoad,SysmonEvent7_ImageLoad,SysmonEvent8_CreateRemoteThread,SysmonEvent9_RawAccessRead,SysmonEvent10_ProcessAccess,\r\nSysmonEvent11_FileCreate,SysmonEvent12_RegistryObjectAddDel,SysmonEvent14_RegistryObjectRename,\r\nSysmonEvent15_FileCreateStreamHash,SysmonEvent16_ConfigChange,SysmonEvent17_CreateNamedPipe,SysmonEvent18_ConnectNamedPipe,\r\nSysmonEvent19_WMIEventFilter,SysmonEvent20_WMIEventConsumer,SysmonEvent21_WMIEventConsumerToFilter,SysmonEvent22_DNSEvents)\r\n",
"functionAlias": "Sysmon",
"version": 2
}

Hi!
just to let you know that on SysmonEvent13 section seems to have a double __ on the name:

let SysmonEvent13__RegistrySetValue=() {

Congrats for this brilliant project

I'm following your quickstart guide for Hunting Workbooks and I've encountered the following issue - please can you point me in the right direction to resolve...?

Hi!

It's possible that the published parser is not the latest version? I'm seeing some inconsistencies with the fields parsed and the fields used at the queries, or I'm doing something wrong...

Ex:
Parser --> EventID
Detection --> event_id

Sysmon event 1

Parser --> process_parent_path
Detection --> process_parent_name