As organizations go through the Autonomic Security modernization journey, this repository serves as a community-driven list of sample security analytics for auditing cloud usage and for detecting threats to your data & workloads in Google Cloud. These may assist detection engineers, threat hunters and data governance analysts.
CSA is a set of foundational security analytics designed to provide organizations with a rich baseline of pre-built queries and rules that they can readily use to start analyzing their Google Cloud logs including Cloud Audit logs, VPC Flow logs, DNS logs, and more using cloud-native or third-party analytics tools. The source code is provided as is, without warranty. See Copyright & License below.
To learn more about the variety of Google Cloud logs, how to enable and natively export these logs to destinations like BigQuery or Google Security Operations for in-depth analytics, refer to Google Cloud Security and access analytics solution guide.
The dataform folder contains the Dataform repo to automate deployment of CSA queries in BigQuery for optimized performance and cost. Use this Dataform repo to operationalize CSA use cases as reports and alerts powered by BigQuery. This Dataform project deploys and orchestrates pre-built ELT pipelines to filter, normalize and model log data leveraging incremental summary tables, lookup tables and views for fast, cost-effective and simpler querying. See underlying README for more details.
CI/CD for CSA on Google Security Operations
The cicd folder contains a set of scripts to help you with storing CSA YARA-L detection rules as code and testing/deploying updates you and your team make in an automated fashion. Whether you use GitHub Actions, Google Cloud Build or Azure DevOps, you can use the corresponding scripts to automatically test and deploy new or modified rules into your Google Security Operations instance. See underlying README for more details.
Support
This is not an officially supported Google product. Queries, rules and other assets in Community Security Analytics (CSA) are community-supported. Please don't hesitate to open a GitHub issue if you have any question or a feature request.
Queries, rules and other assets under Community Security Analytics (CSA) are licensed under the Apache license, v2.0. Details can be found in LICENSE file.
When running the github actions workflow, it errors out at the pre-upload dry run phase.
When run locally the process works fine, our created rules are ingested into our chronicle rules editor, confirming we are using the correct Google API Key.
However, when we format our Google API key using the cat ~/malachite-abc-7ba40dd4f123.json | tr '\n' ' ' | sed -r 's/\"/\\"/g' (as instructed in https://github.com/GoogleCloudPlatform/security-analytics/blob/main/cicd/README.md) once we insert the resulting formatted key into the repository SA_CREDENTIAL the github action fails.
Screenshot(s)
HELP
Please let us know where we have gone wrong or if there is a bug in the process that's blocking us from creating the CICD.
We will continue working on our end to identify any mistakes we've made too.
The queries for the BigQuery backend do not have a standardized format for dataset tables with multiple days of data. For example, BigQuery stores multiple days of Cloud Audit Logs in a dataset like the following:
The queries provided in a majority of the samples only query on a single table, such as 3_01_logging_settings_modified.sql. I believe all the sample queries should accommodate both single day tables and multiple day formats to improve their ease of use. Some queries, like 4_01_unusually_high_api_usage_by_user_identity.sql, accommodate multiple tables already. However the _* suffix formatting used in 4_01 would not accommodate a single day table.
I propose that all queries should pull from a wildcard table to alleviate this issue like the following:
To prevent the queries from running over massive amounts of data for tables spanning months of logs, we could also include the following delimiter to the queries themselves to allow users to granularly select the time slot for the queries to run.
WHERE
_TABLE_SUFFIX = FORMAT("_%s", FORMAT_DATE("%Y%m%d", CURRENT_DATE()))
AND โฆ
Suggest that the file should be placed in a .github/workflow, however correct folder should be .github/workflows for Github action pipeline to work properly.