Additional questions about hashicorp-vault-with-cloud-run HOT 8 CLOSED

Hi!

Certs

You need certs to send SSL traffic. It's not just about accepting it. And because I'm using a scratch image for my final image base, it has no CA cert of its own and hence one from another lightweight distro, Alpine was used

Internal deployment

If you read the instructions, I deploy it as internal first, initialize it, then redeploy as public. I also go on to say that you don't have to do that and deploy directly as public and initialize from there. This is what I do. The internal-first option is just (in theory) a safer option so you don't have someone hijack your vault and initialize it before you get the chance to, but if you think about it, the chances of that happening are very slim (unless you leave it lying around for a week or something). If you deploy it as public, then you don't need an HTTPS LB or Cloud Armor. An LB is ~$20/month and it gives gives the option to add Cloud Armor and restrict soure IP, but I don't really need it because I don't have any real secrets in my Vault other than random ones I use in my demos or for testing purposes. The purpose of this Vault instances is more for testing/learning rather than real (production) secrets.

TLS

Yes you can disable TLS in the Vault config because the Cloud Run provided URL already has one and that's what I use. You would set your VAULT_ADDR to the HTTPS Cloud Run address and that's it.

HA Vault

In order to have an HA Vault setup, these nodes (and clusters) need to communicate and that is done on a different port (default: 8201). Cloud Run can only have one serving port (8200), so there's no way for a this setup to be setup in HA. This is why ha_enabled is set to false for the GCS storage backend config and why this setup shouldn't be used as prod. But it also depends on how big of an operation it is you run. I run one of these Vault instances for my own demos and stuff and I've had no failures or issues in the 2.5 years that it's been up, but I also don't ask a lot of it. Could you use it for prod? Probably, if it's not a big environment, you take backups and you can tolerate some downtime if something does happen. But generally, I would say no. This shouldn't be used for prod.

from hashicorp-vault-with-cloud-run.

Thank you a lot for the explanation !

Just something that you may add in your tutorial, when I deployed the setup on cloud run, I encountered the following error
Error initializing: context deadline exceeded when trying to unseal it from kms.

The issue has been reported here hashicorp/vault#16257 and the proposed solution works for me, by setting VAULT_CLIENT_TIMEOUT to 300 as cloud run env var.

from hashicorp-vault-with-cloud-run.

@ntang86 are you in the Asia region by any chance? The person who reported the issue was having issues in Asia. Based on your last name, there's at least a chance that you could be. I've never had this issue deploying in the Canadian or US East and West regions.

from hashicorp-vault-with-cloud-run.

I'm running everything on northamerica-northeast1, Montreal, Canada. And using VAULT_CLIENT_TIMEOUT env var as workaround did not solve the issue at the end, I'm seeing the error poping up again.

from hashicorp-vault-with-cloud-run.

What version of Vault are you deploying? Thought maybe I can try to replicate it.

I've recently deployed a 1.15.3 (in Toronto) without issue. I deployed one in us-west back in October for a talk demo. And I've had mine since 1.7.2 and through upgrades throughout the years currently is running at 1.14.7 (in us-east, also no issues)

from hashicorp-vault-with-cloud-run.

I deployed Vault 1.15.3 with the following config

vault-server.hcl

default_max_request_duration = "90s"
disable_clustering           = true
disable_mlock                = true
ui                           = true

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "true"
}

seal "gcpckms" {
  crypto_key = "auto_unseal"
  project = "-"
  region     = "northamerica-northeast1"
  key_ring    = "vault-server-stg"
}

storage "gcs" {
  ha_enabled = "false"
}

init.json

{
  "recovery_shares": 1,
  "recovery_threshold": 1,
  "stored_share": 1
}

Cloud Run settings

Env VAR

GOOGLE_PROJECT=-
GOOGLE_STORAGE_BUCKET=vault-backend-stg
VAULT_GCPCKMS_SEAL_KEY_RING=vault-server-stg
VAULT_CLIENT_TIMEOUT=500
VAULT_ADDR=https://vault-server-pfuzsvjs6q-nn.a.run.app

When you first deploy, it's always successful

Then the unseal happens pretty quickly

This is new, timeout on storage.googleapis.com...

I'm connecting this on my serverless VPC because I want to auto rotate the database (private IP only) secret, disabling the VPC did not help either, I notice here that it appears every 5min (300s) but my timeout is set to 500s :/

Added, VAULT_LOG_LEVEL to debug, can't find more information :/

from hashicorp-vault-with-cloud-run.

It looks like you're just taking bits and pieces from my repo rather than the entire setup as a whole. I use global as my KMS key ring region and a lot of things I pass via env vars, so I don't know what other settings you used for the other items. I also don't know whether your service account has the roles necessary as well. I don't have any of the issues you're experiencing.

from hashicorp-vault-with-cloud-run.

Yes I do, It's rare to find a perfect blueprint to match each use case.

The error persist but Vault works !
I use a regional KMS as storing everything in Canada is important to me, but this does not explain the error and how the first time it was able to unlock itself. I'll try to change to Global KMS to test next week.

I agree, it is too difficult to troubleshoot a hole setup when you just have pieces and parts to works with. Your guide was really helpful, but unfortunately Vault on Cloudrun is still too unstable at the moment.

from hashicorp-vault-with-cloud-run.