cammurray / orca Goto Github PK

The Microsoft Defender for Office 365 Recommended Configuration Analyzer (ORCA)

PowerShell 100.00%

exchange-online orca threat-protection powershell-gallery atp mdo

orca's Issues

Module based on ORCA

Hi Cam, I've taken the liberty of modifying your module to make it more modular, also adding checks for Intune ATP and Azure ATP. Please have a look at it at https://github.com/mpiederiet/test-atpbaseline and let me know what you think of it? (sorry to use "issues" for this; unfortunately there is no way to do "private messaging" on Github.)

Best regards,
Maarten

Need for other output types i.e. json

Hi (caveat i know virtually nothing about exchange), im working for an organisation where we want to schedule a run of the ORCA report inspect the results and potentially notify individuals of configuration drift that may make their platform insecure. At present im thinking of leveraging this module for this task. It would be very useful to have a json representation of the report to accomplish this. Is this something you could consider adding? Otherwise i will have to fork and maintain a version of this code that would allow for this and would mean that i somehow have to keep up to date with enhancements that you make to this library which is not ideal.

Whitelisted Domains - Disabled Transport Rule

I just noticted 'Remove whitelisting on domains' portion of ORCA through Transport rules also fails when the transport rule is disabled.

I guess this shouldn't really be the case? As the transport rule isn't enabled, this shouldn't have effect on the score.

Errors after typing "GetORCAReport"

I installed the module fine but every time I type "GetORCAReport" I get the following error:
get-orcareport : The 'get-orcareport' command was found in the module 'ORCA', but the module could not be loaded. For
more information, run 'Import-Module ORCA'.
At line:1 char:1

get-orcareport

  + CategoryInfo          : ObjectNotFound: (get-orcareport:String) [], CommandNotFoundException
  + FullyQualifiedErrorId : CouldNotAutoloadMatchingModule

Unified Audit Log False Positive

Our tenant has unified audit log enabled, but orcareport shows "improvement."

HTML output has several formatting errors

The closing tag is not matching the opening tag on several headers in the HTML output.
Some examples:

<h2 class='card-title'>Office 365 ATP Recommended Configuration Analyzer Report</h5>
-->
<h2 class='card-title'>Office 365 ATP Recommended Configuration Analyzer Report</h2>

<div class='card-header'><h5>Informational</h4></div>
<div class='card-body'>
<h2>1</h5>
-->
<div class='card-header'><<h5>Informational</h5></div>
<div class='card-body'>
<h2>1</h2>

<div class='card-header'><h5>Recommendations</h4></div>
<div class='card-body'>
<h2>13</h3>
-->
<div class='card-header'><h5>Recommendations</h5></div>
<div class='card-body'>
<h2>13</h2>

Will the next release support the new STRICT and STANDARD setting categories?

Will the next release support the new STRICT and STANDARD setting categories?
See https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/recommended-settings-for-eop-and-office365-atp

warning about Intra-org Safe Links points to deprecated page

ORCA 1.9.11

warning about Intra-organization Safe Links (Enable Safe Links between internal users) points to deprecated page:

from top of https://protection.office.com/safelinksconverged:

updated page is:
https://protection.office.com/safelinksv2

orca script does not recognize EOP .us endpoints as EOP

Update orca script to recognize domain.mail.protection.office365.us as EOP.

It appears the orca script only recognizes domain.mail.protection.outlook.com as EOP at this time.

Powershell 7 support

Probably more of a question, ive noted the ExchangeOnlineManagement module has a preview for Powershell 7. Im hoping that at some point (and im wondering when) the ORCA module can be updated to work with this? Atm im being forced to use an automation account in azure to execute the report, i would much prefer to be using an Azure Powershell Function as soon as possible. Ofc ive no idea when ExchangeOnlineManagement will come out of preview..

Get-ORCAReport stalls at - Getting MX Reports for all domains

Hi Cam,

Awesome tool BTW, but currently checking on large Edu tenant and getting stalled at:
Getting MX Reports for all domains - consistently stopping here for 30 - 50 mins plus

12/14/2020 15:23:50 Getting Anti-Spam Settings
12/14/2020 15:23:53 Getting Tenant Settings
12/14/2020 15:23:54 Getting Anti Phish Settings
12/14/2020 15:23:55 Getting Anti-Malware Settings
12/14/2020 15:23:56 Getting Transport Rules
12/14/2020 15:23:58 Getting ATP Policies
12/14/2020 15:23:59 Getting Accepted Domains
12/14/2020 15:24:00 Getting DKIM Configuration
12/14/2020 15:24:01 Getting Connectors
12/14/2020 15:24:01 Getting MX Reports for all domains

Tried running Get-ORCAReport -Collections Get-AcceptedDomain more or less to try "something" and it seemed to get past it - but now reporting that ATP is not installed in Tenant?

12/14/2020 15:50:17 Performing ORCA Version check...
12/14/2020 15:50:26 Analysis - Anti-Spam Policies - Anti-Spam Policy Rules
12/14/2020 15:50:26 Analysis - Anti-Spam Policies - Safety Tips
12/14/2020 15:50:26 Analysis - Anti-Spam Policies - Phish Action
12/14/2020 15:50:26 Analysis - Anti-Spam Policies - Bulk Complaint Level
12/14/2020 15:50:26 Analysis - Anti-Spam Policies - High Confidence Spam Action
12/14/2020 15:50:26 Analysis - Anti-Spam Policies - Allowed Senders
12/14/2020 15:50:26 Analysis - Anti-Spam Policies - Bulk Action
12/14/2020 15:50:27 Analysis - Anti-Spam Policies - IP Allow Lists
12/14/2020 15:50:27 Analysis - Anti-Spam Policies - Domain Whitelisting
12/14/2020 15:50:27 Analysis - Anti-Spam Policies - Quarantine retention period
12/14/2020 15:50:27 Analysis - Anti-Spam Policies - Outbound spam filter policy settings
12/14/2020 15:50:27 Analysis - Anti-Spam Policies - High Confidence Phish Action
12/14/2020 15:50:27 Analysis - Anti-Spam Policies - Mark Bulk as Spam
12/14/2020 15:50:27 Analysis - Anti-Spam Policies - Spam Action
12/14/2020 15:50:27 Analysis - Anti-Spam Policies - Advanced Spam Filter (ASF)
12/14/2020 15:50:27 Analysis - Anti-Spam Policies - End-user Spam notifications
12/14/2020 15:50:27 Analysis - Connectors - Domains
12/14/2020 15:50:27 Analysis - Connectors - Enhanced Filtering Configuration
12/14/2020 15:50:27 Analysis - DKIM - Signing Configuration
12/14/2020 15:50:27 Analysis - DKIM - DNS Records
12/14/2020 15:50:27 Analysis - Malware Filter Policy - Malware Filter Policy Policy Rules
12/14/2020 15:50:27 Analysis - Malware Filter Policy - Internal Sender Notifications
12/14/2020 15:50:27 Analysis - Malware Filter Policy - Common Attachment Type Filter
12/14/2020 15:50:27 Analysis - Malware Filter Policy - External Sender Notifications
12/14/2020 15:50:27 Analysis - Tenant Settings - Unified Audit Log
12/14/2020 15:50:27 Analysis - Transport Rules - Domain Whitelisting
12/14/2020 15:50:27 Analysis - Zero Hour Autopurge - Zero Hour Autopurge Enabled for Malware
12/14/2020 15:50:27 Analysis - Zero Hour Autopurge - Zero Hour Autopurge Enabled for Phish
12/14/2020 15:50:27 Analysis - Zero Hour Autopurge - Zero Hour Autopurge Enabled for Spam
12/14/2020 15:50:27 Analysis - Zero Hour Autopurge - Supported filter policy action
12/14/2020 15:50:27 Generating Output
12/14/2020 15:50:27 Output - HTML
12/14/2020 15:50:28 Complete! Output is in C:\Users......
ORCA--202012141550.zip

Check for insufficient permissions

In case user has not "Exchange service admin" rights first cmdlet which fails is Get-HostedConnectionFilterPolicy.
I recommend to add a simple check like this:

if(!(Get-Command Get-HostedConnectionFilterPolicy -ErrorAction SilentlyContinue))
{
Write-Host "Exchange PowerShell module not loaded" -foregroundcolor red
break
}

Typo in Report Output

The output report has a a typo under the Advanced Spam Filter (ASF) section.

It outputs

"Advanced Spam filter options are runed off"

I believe this should be

"Advanced Spam filter options are turned off"

Output File name and directory

Cannot ouput the file to a custom directory or file name. Tried:

OutputFile
OutputDirectory
-ReportFileName
-Report

All of which save it in app data with the scripted name format. How do achieve custom directory/name?

Thanks.

Add Hours and Minutes to the Report file name

When running more than once a day, it will be overwritten
For example: Run the report, fix issues, run the report again. Now you don't have original to check back on what you changed.

Add Hours and Minutes:
ORCA--MMDDYYHHMM.html

Actually prefer for all Internationals :
ORCA--YYYYMMDDHHMM.html

False positive on DKIM / DNS Records

Getting results of "improvement." Opened Microsoft support ticket and they verified our records are correct.

ORCA fails in Azure Cloud Shell

It appears something to do with $PSScriptRoot based on initial investigations.

Get-ORCAReport returns recommendations in odd order

It would be great if the order the results are returned in was in line with the policy that is being changed. That is, when you have a recommendation to change a policy in the report and the next recommendation is a different policy, and the n recommendation is back to the first policy. Please can you return the results so that all settings for a single policy appear together.

'Invoke-ORCA -PerformVersionCheck $False' Produces Error

Trying to automate this for another reporting script and when I use the '-PerformVersionCheck $False' parameter, I am getting an error:

05/21/2020 22:11:09 Output - HTML
You cannot call a method on a null-valued expression.
At C:\Users\DScoles\Documents\WindowsPowerShell\Modules\ORCAPreview\1.8.1\Outputs\output-html.ps1:149 char:47

... Version $($this.VersionCheck.Version.ToString())</stron ...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CategoryInfo : InvalidOperation: (:) [], RuntimeException

FullyQualifiedErrorId : InvokeMethodOnNull

My guess is that the handling of the no version check is tripping something (I have not delved into it to check). I am using the 1.8.1 version of the Preview module so hopefully this is the right place to report that.

Safe Links settings - Action for unknown or potentially malicious URLs in messages within Microsoft Teams

Hi Cam,
Please add for 1.5

Safe Links settings
Security feature name: Action for unknown or potentially malicious URLs in messages within Microsoft Teams
Standard or Strict: On

Install EXO Remote Session dependency and detect if WinRMRunning

Add ability to detect if dependency (Exchange Online Remote Session) is installed and install it if not. Be sure to prompt user to install prior to doing it.

you can reference https://raw.githubusercontent.com/jingsta/powershell-scripts/master/Microsoft/O365/O365-SecurityAssessment.ps1 and re-use the function I used.

Need to also determine if WinRM service is running and if not, enable it.

$WinRMRunning = (Get-Service 'WinRM').Status
Enable-PSRemoting -Force

Rights in documentation are not sufficient

Hi Community,

the documentation states
`For example you can use the following:

Global Reader - Azure Active Directory role
View-Only Organization Management - built in role group in Exchange Online`

However, upon testing with these permissions we get the error
ORCA was unable to connect to Exchange Online, or you do not have sufficient permissions to check ORCA related configuration.

It seems as if these permissions are not sufficient for all gathering all relevant data. Or am I missing something here?

Silent running?

Just a possible suggestion, is there a way to get an option to silence either the output to the screen and / or the final opening of the report file? Would like to run this headless (automated) and would like to get rid of any extraneous output (without cracking it open and making non-permanent changes). Thanks.

Possible enhancement - Orca Report Per Policy

Hello,

after the evaluation, orca report provides recommendation. In will happen that company has more policies (default policy where probably there will be default recommendation and additional policy where recommendation will be applied). As a result of the orca report, recommendation will still be visible although no user is actually using default policy but second policy is applied to all users. Might be good to have a filter while generating Orca report to include default policy or not to include default policy. That was company could have a clean recommendation that applies only to a certain policy.

Thank you very much for the report tool it made my life easier.

Update Help

In the Get-Help for Get-ORCAReport the '-Report' parameter is listed in Examples 2 and 3. This option does not exist in the cmdlet. Since the help is not updatable nor is there an online version of help ('-online' switch), can the help be updated to show that that '-Report' is not a valid option (removed from Help)? Can we then add Output to examples as well? Or take information from the Readme in this GitHub and place it in the Get-Help for the cmdlet? Thanks.

ORCA doesn't work with the new preview Exchange module

Currently, we require the released Exchange Online PowerShell module (http://aka.ms/exopsmodule) and will not detect the new preview module.

Code will be improved so that we support both the preview module and the exchange online powershell module.

Unsolveable issue in ORCA Report... false positive?

ORCA 1.9.11 - run date 12/12/2020

I got these 2 warnings for ATP:

Domain Impersonation Action → Configure domain impersonation action to Quarantine → Strict Preset Security Policy → TargetedDomainProtectionAction: MoveToJmf (should be: Quarantine)
Anti-spoofing protection → Enable anti-spoofing protection in Anti-phishing policy → Antispoof Enforced: False (should be: True)

Both of the remediation links above take you to the same page: https://protection.office.com/antiphishing ...

Here's the Default policy: (everything already set to enabled/Quarantine)

...and my 1 custom policy: (also all set to enabled/Quarantine)

Trying to remediate via PowerShell: (change is not allowed)

PS C:\> Set-AntiPhishPolicy -Id "Strict Preset Security Policy" -TargetedDomainProtectionAction Quarantine
WARNING: All recommended properties will be controlled by Microsoft.

...and, nothing in here about `Antispoof Enforced` being off or anything else to be enabled, etc... can't find a refrence to this setting anywhere...?

PS C:\> Get-AntiPhishPolicy | fl Id,*able*,*force*

Id                                  : Strict Preset Security Policy
Enabled                             : True
EnableTargetedUserProtection        : True
EnableMailboxIntelligenceProtection : True
EnableTargetedDomainsProtection     : True
EnableOrganizationDomainsProtection : True
EnableMailboxIntelligence           : True
EnableSimilarUsersSafetyTips        : True
EnableSimilarDomainsSafetyTips      : True
EnableUnusualCharactersSafetyTips   : True
EnableSpoofIntelligence             : True
EnableViaTag                        : True
EnableUnauthenticatedSender         : True
EnableSuspiciousSafetyTip           : True

Id                                  : Custom-AntiPhishPolicy
Enabled                             : True
EnableTargetedUserProtection        : True
EnableMailboxIntelligenceProtection : True
EnableTargetedDomainsProtection     : True
EnableOrganizationDomainsProtection : True
EnableMailboxIntelligence           : True
EnableSimilarUsersSafetyTips        : True
EnableSimilarDomainsSafetyTips      : True
EnableUnusualCharactersSafetyTips   : True
EnableSpoofIntelligence             : True
EnableViaTag                        : True
EnableUnauthenticatedSender         : True
EnableSuspiciousSafetyTip           : False

Id                                  : Office365 AntiPhish Default
Enabled                             : True
EnableTargetedUserProtection        : True
EnableMailboxIntelligenceProtection : True
EnableTargetedDomainsProtection     : True
EnableOrganizationDomainsProtection : True
EnableMailboxIntelligence           : True
EnableSimilarUsersSafetyTips        : True
EnableSimilarDomainsSafetyTips      : True
EnableUnusualCharactersSafetyTips   : True
EnableSpoofIntelligence             : True
EnableViaTag                        : True
EnableUnauthenticatedSender         : True
EnableSuspiciousSafetyTip           : False

are these false positives?

possibly related: #114

Basic authentication required

Hi,
Is it possible to use the ORCA report tool without enabling basic auth?
Happy to assist in making this happen if you can let me know where I should start.

Thanks,
Daniel

05/19/2020 12:19:49 Performing ORCA Version check...
05/19/2020 12:21:49 Connecting to Exchange Online..
New-ExoPSSession : Connecting to remote server outlook.office365.com failed with the following error message : The
WinRM client cannot process the request. Basic authentication is currently disabled in the client configuration.
Change the client configuration and try the request again. For more information, see the about_Remote_Troubleshooting
Help topic.
At C:\Users\Username\AppData\Local\Apps\2.0\LR9KCPZ3.8JK\PLATQYJN.38Y\micr..tion_1f16bd4ec4c2bb19_0010.0000_673f37
c317fb5976\CreateExoPSSession.ps1:302 char:30
+ ... PSSession = New-ExoPSSession -UserPrincipalName $UserPrincipalName.Va ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [New-ExoPSSession], PSRemotingTransportException
    + FullyQualifiedErrorId : System.Management.Automation.Remoting.PSRemotingDataStructureException,Microsoft.Exchang
   e.Management.ExoPowershellSnapin.NewExoPSSession```

Output to Desktop

Hi,
I was just looking at the ORCA tool which is excellent, however I'm trying to write a script that will always output to the desktop, using userprofile\Desktop won't work because if people are using OneDrive KFM then the Desktop doesn't exist in that location(I'm guessing it'll be the same for FolderRedirection also), so I've been trying to use:

$DesktopPath = [System.Environment]::GetFolderPath([System.Environment+SpecialFolder]::Desktop)
Invoke-ORCA -Output HTML -OutputOptions @{HTML=@{OutputDirectory=$DesktopPath}}

The desktop path gives the correct result but it doesn't seem like the Output Directory will accept it as a location

Can you please see what you can do to get this working, or advise if I'm doing something wrong.

Thanks!

Invoke-ORCA -CSV Does not produce CSV File

Running Invoke-ORCA -CSV runs successfully, and spits out the last line of "Generating Report" But the log does not indicate that the CSV is actually created. Further, It does not show up in AppData\Local\Microsoft\ORCA or any other location if -OutputDirectory is specified.

[Request] Publish code in repo to allow pull requests

Hi @cammurray,

thanks for the great tool. Have you considered to also publish the actual code in the repo to allow the community to make pull requests.
I understand that there may be licensing issues that would prevent this, but it would make it so much easier to propose changes / features

All the best, Fabian

No Output. Multiple Errors.

Hello,

When attempting to run Get-ORCAReport I receive the following output:

PS C:\Temp> Get-ORCAReport
11/20/2020 09:29:15 Performing ORCA Version check...
11/20/2020 09:29:22 Connecting to Exchange Online (Modern Module)..

The module allows access to all existing remote PowerShell (V1) cmdlets in addition to the 9 new, faster, and more reliable cmdlets.

|--------------------------------------------------------------------------|
| Old Cmdlets | New/Reliable/Faster Cmdlets |
|--------------------------------------------------------------------------|
| Get-CASMailbox | Get-EXOCASMailbox |
| Get-Mailbox | Get-EXOMailbox |
| Get-MailboxFolderPermission | Get-EXOMailboxFolderPermission |
| Get-MailboxFolderStatistics | Get-EXOMailboxFolderStatistics |
| Get-MailboxPermission | Get-EXOMailboxPermission |
| Get-MailboxStatistics | Get-EXOMailboxStatistics |
| Get-MobileDeviceStatistics | Get-EXOMobileDeviceStatistics |
| Get-Recipient | Get-EXORecipient |
| Get-RecipientPermission | Get-EXORecipientPermission |
|--------------------------------------------------------------------------|

To get additional information, run: Get-Help Connect-ExchangeOnline or check https://aka.ms/exops-docs

Send your product improvement suggestions and feedback to [email protected]. For issues related to the module, contact Microsoft support. Don't use the feedback alias for problems or support issues.

New-Object : Cannot find type [html]: verify that the assembly containing this type is loaded.
At line:32 char:16
    return & $origNewObject @psBoundParameters
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo : InvalidType: (:) [New-Object], PSArgumentException

FullyQualifiedErrorId : TypeNotFound,Microsoft.PowerShell.Commands.NewObjectCommand
The property 'DefaultOutputDirectory' cannot be found on this object. Verify that the property exists and can be set.
At C:\Program Files\WindowsPowerShell\Modules\ORCA\1.9.11\ORCA.psm1:410 char:17
            $Output.DefaultOutputDirectory = Get-ORCADirectory
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo : InvalidOperation: (:) [], RuntimeException

FullyQualifiedErrorId : PropertyNotFound
The property 'VersionCheck' cannot be found on this object. Verify that the property exists and can be set.
At C:\Program Files\WindowsPowerShell\Modules\ORCA\1.9.11\ORCA.psm1:413 char:17
            $Output.VersionCheck = $VersionCheck
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo : InvalidOperation: (:) [], RuntimeException

FullyQualifiedErrorId : PropertyNotFound
New-Object : Cannot find type [ORCA100]: verify that the assembly containing this type is loaded.
At line:32 char:16
    return & $origNewObject @psBoundParameters
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo : InvalidType: (:) [New-Object], PSArgumentException

FullyQualifiedErrorId : TypeNotFound,Microsoft.PowerShell.Commands.NewObjectCommand
The property 'ORCAParams' cannot be found on this object. Verify that the property exists and can be set.
At C:\Program Files\WindowsPowerShell\Modules\ORCA\1.9.11\ORCA.psm1:352 char:13
        $Check.ORCAParams = $ORCAParams
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo : InvalidOperation: (:) [], RuntimeException

FullyQualifiedErrorId : PropertyNotFound`
The last two errors repeat several times (for ORCA100 -> ORCA234). Then:
`11/20/2020 09:29:45 Getting Anti-Spam Settings
11/20/2020 09:29:47 Getting Tenant Settings
11/20/2020 09:29:47 Getting Anti Phish Settings
11/20/2020 09:29:48 Getting Anti-Malware Settings
11/20/2020 09:29:48 Getting Transport Rules
11/20/2020 09:29:49 Getting ATP Policies
11/20/2020 09:29:50 Getting Accepted Domains
11/20/2020 09:29:51 Getting DKIM Configuration
11/20/2020 09:29:51 Getting Connectors
11/20/2020 09:29:51 Getting MX Reports for all domains
11/20/2020 09:30:59 Generating Output
11/20/2020 09:30:59 Complete! Output is in
PS C:\Temp>

There doesn't appear to be any output.
I also tried this:

PS C:\reports> Invoke-ORCA -Output HTML -OutputOptions @{HTML=@{OutputDirectory="c:\reports"}}
11/20/2020 09:38:39 Performing ORCA Version check...
New-Object : Cannot find type [html]: verify that the assembly containing this type is loaded.
At line:32 char:16
    return & $origNewObject @psBoundParameters
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo : InvalidType: (:) [New-Object], PSArgumentException

FullyQualifiedErrorId : TypeNotFound,Microsoft.PowerShell.Commands.NewObjectCommand
Get-Member : You must specify an object for the Get-Member cmdlet.
At C:\Program Files\WindowsPowerShell\Modules\ORCA\1.9.11\ORCA.psm1:395 char:54
... $ModProperties = $($Output | Get-Member | Where ...
                                                   ~~~~~~~~~~
CategoryInfo : CloseError: (:) [Get-Member], InvalidOperationException

FullyQualifiedErrorId : NoObjectInGetMember,Microsoft.PowerShell.Commands.GetMemberCommand
There is no option OutputDirectory on output module html
At C:\Program Files\WindowsPowerShell\Modules\ORCA\1.9.11\ORCA.psm1:403 char:29
... Throw("There is no option $($Opt) on output module $($mat ...
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo : OperationStopped: (There is no opt...put module html:String) [], RuntimeException

FullyQualifiedErrorId : There is no option OutputDirectory on output module html

Running on Windows 10 with Global Reader credentials in an elevated PowerShell 5.1 window.
Please let me know if you require any additional information. Thank you so much for the work that you've put into this script.

Application cannot be started..contact the application vendor

Problem when doing the collection.

Running the report gives me the error below.
This error is repeated for ORCA100 to ORCA230.
Can I change something in my powershell setup to fix this?

PS VersionTable:
Name                           Value
----                           -----
PSVersion                      5.1.19041.1
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.1
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Error

Get-ORCAReport
05/29/2020 09:10:31 Performing ORCA Version check...
New-Object : Cannot find type [ORCA100]: verify that the assembly containing this type is loaded.
At line:32 char:16
return & $origNewObject @psBoundParameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo          : InvalidType: (:) [New-Object], PSArgumentException
FullyQualifiedErrorId : TypeNotFound,Microsoft.PowerShell.Commands.NewObjectCommand

The property 'ORCAParams' cannot be found on this object. Verify that the property exists and can be set.
At C:\Program Files\WindowsPowerShell\Modules\ORCA\1.7.5\ORCA.psm1:288 char:13
 $Check.ORCAParams = $ORCAParams
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo          : InvalidOperation: (:) [], RuntimeException
FullyQualifiedErrorId : PropertyNotFound

No output file...

I have updated orca, verified permissions, attempted changing directory before running Get-ORCAReport but not matter what I do I don't get an output file..

final output is always..

08/13/2020 11:17:06 Complete! Output is in

Any suggestions?
(I am on Mac in a Windows VM, could it be a vm/host permission issue maybe?)

EndUserSpamNotificationFrequency < 4 Is OK?

BP: Enable End-user Spam notification and set the frequency to 3 days . If customers have EndUserSpamNotificationFrequency = 1 or 2 this should be OK??

Running without ATP

I realize that this is meant to be run with ATP, however, it would be nice to be able to run it without ATP as a selling tool to show just how many security issues they have.

Control values are inconsistent

The control values are inconsistently formatted. Some are like "ORAC-101", others like "101", and others like 101. We have a downstream process that uses these values, so it would be nice to have them in a consistent format. I attempted to submit a merge request with consistent values, but I got a 403.

Orca Check 180 "Enable anti-spoofing protection" returning false positive

check-ORCA180.ps1 is checking if EnableAntiSpoofEnforcement returns as true, however EnableAntiSpoofEnforcement appears to have been renamed to EnableSpoofIntelligence in the policy.

This is causing ORCA to always fail this check with $Enabled = $False.

Updating the text EnableAntiSpoofEnforcement in the script to EnableSpoofIntelligence resolves this issue.

Return objects don't always have the required properties set

In check-ORCA117.ps1, the return object doesn't return the "Object" property when no ATP policies are present. E.g.:

According to this, the Object property needs to be present:
$this.CheckType=[CheckType]::ObjectPropertyValue

Line 76-79:
$ConfigObject = [ORCACheckConfig]::new()
$ConfigObject.ConfigItem="All"
$ConfigObject.ConfigData="Enabled False"
$ConfigObject.SetResult([ORCAConfigLevel]::Standard,"Fail")
This should be:
$ConfigObject = [ORCACheckConfig]::new()
$ConfigObject.Object ="All"
$ConfigObject.ConfigItem="IsEnabled"
$ConfigObject.ConfigData="False"
$ConfigObject.SetResult([ORCAConfigLevel]::Standard,"Fail")

Had to use Install-Module -Name ExchangeOnlineManagement

I couldn't use the http://aka.ms/exopsmodule I had to use
Install-Module -Name ExchangeOnlineManagement . It kept erroring out that the psmodule.client.exe.manifest did not succeed. Once I used the other Powershell command it worked like a champ. Just sharing in case others hit this.

ORCA will not connect when used to check an Exchange Online Hybrid instance

ORCA fails to connect in Exchange Online Powershell when used in an Exchange Online Hybrid setup,

PS C:\Users\sean.brackeen> Get-OrcaReport
11/18/2019 14:59:08 Performing ORCA Version check...
11/18/2019 14:59:10 Connecting to Exchange Online..
ORCA was unable to connect to Exchange Online.
At C:\Program Files\WindowsPowerShell\Modules\ORCA\1.2.1\ORCA.psm1:1343 char:9
+         Throw "ORCA was unable to connect to Exchange Online."
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (ORCA was unable...xchange Online.:String) [], RuntimeException
    + FullyQualifiedErrorId : ORCA was unable to connect to Exchange Online.

Looking at the other issue in this tracker it appears that it depends on Get-Mailbox to see if a connection is already available, which is not an available command once connected to Exchange Online in Hybrid mode, since Hybrid accounts do not always contain mailboxes.

I'm not sure if this is a WontFix since you may not be interested in fixing this for Hybrid deployments, but using EOP and ATP is valid for Exchange Hybrid, and I would be interested in using this script.

Preset Security Policy shows up even when not set or disabled

Problem: The Preset Security Policies (both Standard and Strict) are shown in the report as "OK" marked with "Standard" or "Strict" tags.

This is the case even when the policies are not configured (i.e. there are no assignments / applies-to yet). Checking with PowerShell "Get-xxxRule" confirms that no rules are in place. PowerShell "Get-ATPProtectionPolicyRule" also returns nothing confirming the same. Yet the ORCA report gladly shows the policies/rules in each section in green and adds to the total "OK" count.

This is also the case when the policies are disabled.

Expected Behaviour: The preset security policies if disabled or not configured should either not be shown in the report, or should show up as informational as in the case of transport rules.

Version: 1.9.11

Steps to reproduce:

Use a tenant where the tile called "Preset Security Policies" under the "Templated Policies" section is never clicked. This means, the preset policies are not created yet.
Run ORCA to generate a report.
The report will not show the preset policies anywhere.
Click on the "Preset Security Policies" tile under the "Templated Policies" section. But do not configure anything and leave the templates/policies in disabled state.
Run ORCA to generate a report.
The report will show all the preset policies (both standard and strict) with respective green tags and adds to the "OK" score.

Note: Toggling enabled/disabled state does not make any difference.

Note 2: Once the "Preset Security Policies" tile is clicked, the present policies are created but without rules unless configured.

ORCA future

I noticed in the Security and Compliance center under Threat Management -> Policy, there is a component called Configuration Analyzer which recommends settings that overlap with recommendations generated in the ORCA reports. Is this configuration analyzer going replace the ORCA reports? I notice there hasn't been a new version of ORCA since October. Is all development now going into the Configuration analyzer? Of does one not have anything to do with the other? Put another way, is there any road map for ORCA?

Possible enhancement - Filter by recommendation state

Possible enhancement would be that the report would filter by Informational, Recommendations or
OK when the state is clicked on top of the html report

Add direct links to settings in order to improve admins changes

On every control made by ORCA, it would be a good idea to add direct links (or at least an information) to easily find the settings that the administrators have to change in order to retrieve the recommanded settings.
It's a little bit hard for some administrators to find where they have to change settings.

DNS Server setting

In order to resolve a potential problem in case of a local split DNS configuration. Could you please add a new parameter in the command line (like -DNSName) to be able to select and force a DNS Server instead of using the default server settings.

Using ORCA report in Azure Automation

Hi, I managed to get this working in Azure Automation, however it left me with several feedback points / suggestions.

I will paste the entire AzA powershell runbook below so that you can provide advice to others & test yourself, comments & suggestions are inline.

#Runbook designed for Azure Automation

#Display verbose output in test pane
$VerbosePreference = "Continue"

#Imported (required) Modules
<#
ExchagneOnlineManagement
ORCA
#>

#Get credentials for use throughout runbook. Note that the user must have "View Only Configuration" role as a minimum.
Write-Output "Getting Credential"
$Credential = Get-AutomationPSCredential -Name 'EOPAdmin' -Verbose

#Set email parameters - Note that Azure servers are in UTC, so convert the time to a local timezone
$EmailFrom = 'valid o365 [email protected]'
$EmailTo = @('[email protected]')
$EmailSubject = 'Exchange Online ORCA Report - ' + [System.TimeZoneInfo]::ConvertTimeBySystemTimeZoneId((Get-Date), 'AUS Eastern Standard Time')
$EmailBody = 'See attached file for the ORCA report
Generated using https://github.com/cammurray/orca
Report ran in Azure Automation'
$ReportPath = "$($env:LOCALAPPDATA)\Microsoft\ORCA\ORCAReport.html"

#Connect to Exchange Online - Don't let the Get-ORCAReport function attempt to connect, it needs to be the newer module & specify creds (No MFA)
Write-Output "Creating session"
Connect-ExchangeOnline -Credential $Credential

#Run the ORCA Report tools
Write-Output "Running ORCA - Expect a warning about no program to open HTML file"
#Note: The synopsis for Get-ORCAReport states to use "-Report" to specify the filename however the parameter is actually named "Output"
#Suggested updates for author ([email protected])

- Update synopsis with correct parameter names

- Update synposis / readme with required role "View Only Configuration"

- Add an option to not automatically attempt to execute the HTML file (running in Azure Automation this does not work) - Generates error "No application is associated with the specified file for this operation"

- Add a "Return" statement which is a string containing the full file path OR just return the HTML as a string so it can be used as an email body

---- Regarding the HTML itself, it does not render in Outlook or Outlook.com well if used as the message body - I believe this is due to the CSS used and the fact that it is not embedded.

---- Consider embedding the CSS and using more cross browser friendly HTML. Some tips here:

------- https://email.uplers.com/blog/email-rendering-issues-in-outlook-and-hacks/

------- https://www.emailonacid.com/blog/article/email-development/how-to-create-excellent-emails-for-outlook-com/

------- https://www.contactmonkey.com/blog/outlook-rendering-issues

------- https://litmus.com/blog/a-guide-to-rendering-differences-in-microsoft-outlook-clients

- Add an option to auto update (no user interaction) so that we don't have to manally import the updated module into our Automation account

---- Note that even with the "NoUpdate" parameter, an error is still generated if the NuGet module is not installed

------- "Find-Module : NuGet provider is required to interact with NuGet-based repositories. Please ensure that '2.8.5.201' or newer version of NuGet provider is installed."

------- This is caused by the "Invoke-ORCAVersionCheck" being run without checking if "NoUpdate" is specified

- Update HTML to include timezone info where it displays the date/time and/or allow user to specify timezone - Timezone of where the script is running can be found with "[System.TimeZoneInfo]::Local.ToString()"

#eg: (attachment)

$ReportFile = Get-ORCAReport -NoConnect -AutoUpdate -DontOpenHTML -OutputType 'FilePath'

Send-MailMessage -Attachments $ReportFile (+other SMTP parameters)

#eg: (string)

$ReportHTML = Get-ORCAReport -NoConnect -AutoUpdate -DontOpenHTML -OutputType 'HTMLString'

Send-MailMessage -Body $ReportHTML (+other SMTP parameters)

#Run the report
Get-ORCAReport -NoConnect -NoUpdate -Output $ReportPath

#If using the HTML as the email body, convert the system.object (array of strings) into a single string first
#Not using this as it doesn't render well as yet
#$FileContent = Get-Content -Path $ReportPath
#$EmailBody = Out-String -InputObject $FileContent

#For some reason, you have to get the credentials again or sending the email fails - Maybe a quirk of AZ Automation?
Write-Output "Getting Credential"
$Credential = Get-AutomationPSCredential -Name 'EOPAdmin' -Verbose

#Send an email with the report attached
Write-Output "Sending Email"
Send-MailMessage -Attachments $ReportPath
-Body $EmailBody -BodyAsHtml:$True
-DeliveryNotificationOption @('OnFailure') -From $EmailFrom
-SmtpServer 'smtp.office365.com' -Priority 'Normal'
-Subject $EmailSubject -To $EmailTo
-Credential $Credential -UseSsl:$True
-Port 587 -Verbose:$True
-Debug:$False -ErrorAction Continue
-WarningAction Continue

Get-ORCAReport does not work

Lots of failures, but most importantly does not show output.

Get-ORCAReport
02/22/2021 16:41:42 Performing ORCA Version check...
New-Object : Cannot find type [html]: verify that the assembly containing this type is loaded.
At line:32 char:18

    return @(& $origNewObject @psBoundParameters)

```
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
- CategoryInfo : InvalidType: (:) [New-Object], PSArgumentException
- FullyQualifiedErrorId : TypeNotFound,Microsoft.PowerShell.Commands.NewObjectCommand

The property 'DefaultOutputDirectory' cannot be found on this object. Verify that the property exists and can be set.
At C:\Program Files\WindowsPowerShell\Modules\ORCA\1.9.11\ORCA.psm1:410 char:17

            $Output.DefaultOutputDirectory = Get-ORCADirectory

```
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
- CategoryInfo : InvalidOperation: (:) [], RuntimeException
- FullyQualifiedErrorId : PropertyNotFound

The property 'VersionCheck' cannot be found on this object. Verify that the property exists and can be set.
At C:\Program Files\WindowsPowerShell\Modules\ORCA\1.9.11\ORCA.psm1:413 char:17

            $Output.VersionCheck = $VersionCheck

```
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
- CategoryInfo : InvalidOperation: (:) [], RuntimeException
- FullyQualifiedErrorId : PropertyNotFound

New-Object : Cannot find type [ORCA100]: verify that the assembly containing this type is loaded.
At line:32 char:18

    return @(& $origNewObject @psBoundParameters)

```
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
- CategoryInfo : InvalidType: (:) [New-Object], PSArgumentException
- FullyQualifiedErrorId : TypeNotFound,Microsoft.PowerShell.Commands.NewObjectCommand