Add to README.md the services that must be enabled in GCP for terraform to run successfully.

The documentation links on the README are currently directing to holding pages. Once the documentation pages are published we need to update the README to point to the final URLs.

We've published a number of patches to AWS Terraform Modules. These should be bumped and a patch for 20.08 released.

The Snowflake Loader is now improved with v3 of the RDB Loader and ready to be added to the Quick Start Examples.

Iglu Central now has schema lists, therefore the documentation in the .tfvars files about seeding Iglu Servers with the Iglu Central schemas is no longer required as the PG Loader can now work with Iglu Central.

As Iglu Central now has schema lists published, we no longer need to seed the Iglu Server with Iglu Central schemas for the Postgres Loader to work.

We should remove the comments in the .tfvars files which suggest seeding is required.

We are now not leveraging the more complex setup flow for the loader so can remove this entirely

I created a new azure account with pay as you go subscription and gave myself all administrator roles as well as the contributor, user access admin, storage blob data contributor for the resource groups created. But I still get this error.

Error: Could not create application

with module.lake_loader[0].azuread_application.app_registration,
on .terraform/modules/lake_loader/main.tf line 115, in resource "azuread_application" "app_registration":
115: resource "azuread_application" "app_registration" {

ApplicationsClient.BaseClient.Post(): unexpected status 403 with OData error: Authorization_RequestDenied: Insufficient
privileges to complete the operation.

This Project is really awesome - Right now I'm trying to figure out how I can load the Available Enrichments https://docs.snowplowanalytics.com/docs/enriching-your-data/available-enrichments/ using the Terraform configuration

I have been unable to find anything on main.tf that references which enrichments are currently being loaded.
It seems this open source version is using the AWS Kinesis Stream to enrich data

Can anyone help with this?

As I also wrote here, there is currently no way to disable HTTP when using the Terraform modules. I suggested to fix this on the load balancer side, but perhaps there is a better way. In the end we want to be able to disable it for the Terraform module setup as a whole.

snowplow-devops/terraform-aws-alb#7

• Provider blob added to main.tf

provider "google" {
  project = var.project_id
  region  = var.region
}

• Project_id added to terraform.tfvars and variable added to variables.tf

This enables users to deploy the Lake Loader to build a data-lake in Azure within a Storage Container that can then be connected to Azure Synapse, Databricks and Azure Fabric.

This is such a great repo and I've followed the Quickstart guide on the snowplow website. The issue I'm experiencing is that the ELB is not accepting a HTTPS curl and the DNS name is not working either.

What I've done

1. Created a certificate using the Amazon Certificate Manager
2. Added the ARN to the pipeline/secure and igluserver/secure configuration
3. Run Terraform Apply

Any chance anyone can help with this?

ssl_information = {
  certificate_arn = "arn:aws:acm:us-east-1:631219223101:certificate/874481b6-510b-48c1-a4e1-4c719637016d"
  enabled         = true
}

RDB Loader v5.7.0 has critical vulnerabilities which this patch will update and mitigate.

All Compute Engine modules have just received a bug fix. The examples needs bumping to use them.

AWS is planning to stop supporting launch configurations and replace them with launch templates.

Is there a plan already to migrate these? All ec2 are running on top of the launch configs.

Add Terraform example for deploying Iglu Server and other Pipeline components to AWS.

With bigquery default pipeline, when prefix is more than 6 characters long, terraform plan errors with:

│ Error: "name_prefix" cannot be longer than 37 characters, name is limited to 63
│ 
│   with module.bigquery_loader[0].google_compute_instance_template.tpl["streamloader"],
│   on .terraform/modules/bigquery_loader/main.tf line 309, in resource "google_compute_instance_template" "tpl":
│  309:   name_prefix = "${var.name}-${each.key}-"

Info to reproduce:

• On Terraform 1.3.0
• On gcp/pipeline/default
• In bigquery.terraform.tfvars supply a prefix with a value more than 6 chars long,e.g. testersp
• terraform init and terraform plan -var-file=bigquery.terraform.tfvars

When setting up the default pipeline on GCP and running terraform plan with bigquery specific var-file, i.e.

terraform plan -var-file=bigquery.terraform.tfvars

there is error about missing postgres variables (Error: No value for required variable), namely for postgres_db_name, postgres_db_username, postgres_db_password. As those don't have defaults in variables.tf, a workaround was to provide dummy values for them in the bigquery tfvars file.

From the README, it seems that the configuration in this repository only supports the 3 big cloud providers.

I would like to try to setup things locally first before attempting to do it in the cloud.

Is it possible to create a Terraform config using development using Docker?

Update gce-proxy from 1.19 to 1.31.2 (see cloud-sql-proxy changelog https://github.com/GoogleCloudPlatform/cloud-sql-proxy/blob/main/CHANGELOG.md) in the GCP Quick Start examples.

We encountered some issues with a fresh install of Iglu server and SQL over the last few days with the proxy failing to connect to SQL. The error message was 2023/01/13 08:03:19 couldn't connect to \"xx-snowplow:rrrrrrr:terraform-xxxxx\": x509: certificate is valid for 12-a123456b-1ab2-1a1b-a123-1a123456b12c.rrrrrrr.sql.goog, not xx-snowplow:terraform-xxxxx. The Iglu pods were unable to start and kept recycling. Updating to the latest gce-proxy resolved the issue and allowed the Iglu pods to restart.

This solves an issue with deploying outside of the eu-west-1 region in respect to default enrichment's.

Source: snowplow-devops/terraform-aws-enrich-kinesis-ec2#3

Once the documentation is published for GCP, we should split the Deployment Summary so it links to both AWS and GCP as they have different deployments.

I'd recommend moving this into the top table, and placing the links next to the Pipeline sections.

A common issue reported on Discourse is around Iglu Resolvers not working due to a missing prefix. By adding a validation into the variables we can catch this issue earlier.

When you have a prefix value with underscores in Azure, e.g. snowplow_test, everything will run as expected in the base module. However, in the iglu_server module you then run into the following error after trying to terraform apply:

│ Error: domain_name_label must contain only lowercase alphanumeric characters, numbers and hyphens. It must start with a letter, end only with a number or letter and not exceed 63 characters in length
│ 
│   with module.iglu_lb.azurerm_public_ip.ip,
│   on .terraform/modules/iglu_lb/main.tf line 33, in resource "azurerm_public_ip" "ip":
│   33:   domain_name_label = "${var.name}-${random_uuid.ip_domain_name_label.result}"
│ 

This error also persists when then trying to run terraform destroy to restart everything, which can be solved by using the terraform state rm module.iglu_lb.random_uuid.ip_domain_name_label command to clean it up.

We've published a number of patches to GCP Terraform Modules. These should be bumped and a patch for 20.08 released.

Bump Google Provider to latest version