perfectthymetech / terraform-azurerm-data-landing-zone Goto Github PK

Feature or Idea - What?

Expose Integration Runtimes and sharing of those.

Feature or Idea - Why?

Necessary to host runtimes for on-premises pull use-cases.

Steps to reproduce

Deployed the module and terraform apply almost goes through. There is a data block referencing a group that was never created. I have defined databricks_admin_groupname as an existing aad group name in tfvars. Not sure how this should work.

./modules/databricksconfiguration/data.tf

data "databricks_group" "group" {
  count        = var.databricks_admin_groupname != "" ? 1 : 0
  display_name = var.databricks_admin_groupname
  provider = databricks.account
}

Error Message

No response

Screenshots

No response

Steps to reproduce

I was using the project defaults to not create private link/a-record for storage account queue. This caused the below error on the terraform plan. I then turned on the private links/a-record for storage account queue and the /28 network ran out of ip. addresses for the storage subnet. I changed it to /27 to fix that.

Error Message

Error: retrieving queue properties for Storage Account (Subscription: "xxxxxx"
│ Resource Group Name: "xxxx"
│ Storage Account Name: "xxxxx"): queues.Client#GetServiceProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
│ 
│   with module.data_landing_zone.module.datalake_workspace.azurerm_storage_account.datalake,
│   on ../../modules/analytics/datalake/datalake.tf line 1, in resource "azurerm_storage_account" "datalake":
│    1: resource "azurerm_storage_account" "datalake" {

Screenshots

No response

Documentation Issue

Document Data Landing Zone including:

• Architecture

Feature or Idea - What?

I know this is a new project. But I wanted to get an understanding of where you plan on keeping the state?

Feature or Idea - Why?

Above.

Feature or Idea - What?

Allow creating more than a single subnet per Data Product

Feature or Idea - Why?

Required as Data Products may require additional tools such as a Logic App or an Azure Function.

Feature or Idea - What?

Add the possibility to automatically create clusters for Data Products.

Feature or Idea - Why?

This is required for the full e2e configuration for new data products being onboarded to the data platform.

Feature or Idea - What?

Pass data_platform_subscription_ids to storage module

Feature or Idea - Why?

data_platform_subscription_ids must be configure

Feature or Idea - What?

Allow to create multiple containers per data products.

Feature or Idea - Why?

Each data product may want to have a separate container per source system.

Feature or Idea - What?

Deploy Data Product Subnets differently as current approach will lead to concurrency issues.

Feature or Idea - Why?

Required as all subnets need to be created one by one. Otherwise, the deployment will fail because of concurrent transactions on Vnet.

Feature or Idea - What?

Add option to deploy ADB auth PE

Feature or Idea - Why?

Might be required for workspaces in new region

Update reference for data product library path to actual input variable. Needs to be changed from locals to var here: https://github.com/PerfectThymeTech/CloudScaleAnalytics-v2-Terraform/blob/07f8806b8c0ed8d4bbf8a129d402e982648a676a/code/data-landing-zone/locals.tf#L11

Feature or Idea - What?

Expose Databricks cluster policies as variable instead of pre-defined configurations

Feature or Idea - Why?

Required as customers have different requirements here.

Feature or Idea - What?

This should be supported; https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/catalog_workspace_binding

Feature or Idea - Why?

Further isolation of catalogs.

Feature or Idea - What?

Change role assignments for data products and don't add users to it.

Feature or Idea - Why?

Adding UAI or others to groups may not be possible. This is a wrong assumption.

Feature or Idea - What?

Add Key Vault Reference for Data Product Team to have Key Vault backed secret scope available.

Feature or Idea - Why?

This configures the last mile for this team to automatically have a Key Vault backed secret scope available.