aaronsgithub / homelab Goto Github PK

Describe the bug

ArgoCD reports a difference in the live manifest and the manifest generated by https://github.com/aaronsgithub/homelab/tree/965f4fe88f99115e742c0485ab0b966c142db75b/appsets/system/vault, giving an OutOfSync status.

Expected behavior
We either want to have the status showing as syncrhonized or have ArgoCD ignore expected mutations of the manifest after deployment.

Current behavior

• A PersistentVolumeClaim named data-vault-0 is produced in the Live manifest but is absent from the desired manifest
• A caBundle is added to the MutatingWebhookConfiguration vault-agent-injector-cfg in the live manifest, which does not appear in the desired manifest.

To reproduce

See https://github.com/aaronsgithub/homelab/tree/965f4fe88f99115e742c0485ab0b966c142db75b/appsets/system/vault

Possible Solutions

• Get ArgoCD to ignore the diff of the caBundle
• Understand why the PersistentVolumeClaim is absent from the desired manifest, find out if this is expected behaviour, and get ArgoCD to ignore this difference too.

See https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/

Describe the bug

We have deployed a Certificate resource which is a cert-manager CRD via:
https://github.com/aaronsgithub/homelab/blob/1a0f7eef5275e8cb5a313e71f6c7524203777e65/appsets/system/certificates/resources/wildcard.bert.local-tls.certificate.cert-manager.yaml

with the following patch which gets updated when we want to add a new namespace to the certificate:
https://github.com/aaronsgithub/homelab/blob/1a0f7eef5275e8cb5a313e71f6c7524203777e65/appsets/system/certificates/patches/wildcard.bert.local-tls.certificate.cert-manager.reflector.namespaces.yaml

This has been deployed by ArgoCD as an Application via an ApplicationSet defined here:
https://github.com/aaronsgithub/homelab/blob/1a0f7eef5275e8cb5a313e71f6c7524203777e65/appsets/system/system.applicationset.yaml

There is an OutOfSync status with a diff showing between the "live" and the "desired" manifest in ArgoCD

Expected Behaviour

There should be no OutOfSync error.

Current Behaviour

live| desired diff
107 | 107 | - bert.local  
108 |     | duration: 8760h0m0s 
    | 108 | duration: 8760h 
    | 109 | isCA: false 
109 | 110 | issuerRef: 
110 | 111 | group: cert-manager.io 
116 | 117 | rotationPolicy: Always 
117 | 118 | size: 384 
118 |     | renewBefore: 720h0m0s 
    | 119 | renewBefore: 720h 
119 | 120 | secretName: wildcard.bert.local-tls 
120 | 121 | secretTemplate: 

It is not clear why isCA disappears from the live manifest.

Possible Solutions

• Get ArgoCD to ignore the difference:
  https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/
• Change manifest deployment to add minutes and seconds to renewBefore.
• Investigate why isCA does not appear in live manifest.

Please describe the problem to be solved

Once Vault is installed, the vault servers are started in a "sealed" state. The data stored in Vault is encrypted.

The Vault servers therefore need to be manually initialised and unsealed in order to read the decryption key and access the data stored in the storage backend. See:

• https://www.github.com/hashicorp/vault-helm/issues/17
• https://developer.hashicorp.com/vault/docs/concepts/seal#why
• https://www.kloia.com/blog/comparison-of-unseal-options-in-hashicorp-vaul

The init process reveals the master keys and initial root token which can then be used to unseal Vault.

Can you propose a solution

A client side script could be made to automate this process but this would not allow for unattended install.

Another option might be to have an InitContainer automate this process.

PGP could be used to encrypt the keys and export them from Kubernetes.

Additional context

Many discussions have taken place on the Vault issues tracker and pull requests submitted:

• https://www.github.com/hashicorp/vault-helm/issues/251
• https://www.github.com/hashicorp/vault-helm/pull/252
• https://www.github.com/hashicorp/vault-helm/pull/315
• https://www.github.com/hashicorp/vault-helm/issues/720
• https://github.com/jasonodonnell/vault-agent-demo/blob/master/configs/bootstrap.sh

Is your feature request related to a problem? Please describe.

At present, the cluster uses a self signed root certificate intended for boostrapping / testing:
https://cert-manager.io/docs/configuration/selfsigned/

⚠️ SelfSigned issuers are generally useful for bootstrapping a PKI locally, which is a complex topic for advanced users. To be used safely in production, running a PKI introduces complex planning requirements around rotation, trust store distribution and disaster recovery.

A working cluster needs good secret management. Vault offers a centralised secret management service which handles secret rotation allowing secrets to be ephemeral, as well as providing other cryptographic APIs.

Describe the solution you'd like
See:

• https://cert-manager.io/docs/configuration/vault/
• https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-cert-manager

Describe alternatives you've considered

Cert manager can be configured with a variety of Issuers:
https://cert-manager.io/docs/configuration/

We may also consider having a vault service run outside the cluster, or even a second instance for HA / disaster recovery.

Vault seems to be the most secure way of issuing / managing certificates on a local network not exposed to the internet.

Additional context

These third party guides may be useful / provide additional context:

• https://www.ibm.com/docs/en/cloud-private/3.1.2?topic=manager-adding-certificates-by-using-vault-issuer
• https://www.ibm.com/docs/en/cloud-private/3.1.2?topic=manager-adding-certificates-by-using-vault-issuer