Code Monkey home page Code Monkey logo

gocd-vault-secret-plugin's Introduction

Vault secret manager plugin

This is a GoCD Secrets plugin which allows users to use Vault as a secret manger for the GoCD server.

The plugin supports Version 2 of KV Secrets Engine.

Table of Contents

Setup Vault using docker

  1. Run following command to start docker container for vault
docker run --cap-add=IPC_LOCK -e VAULT_DEV_ROOT_TOKEN_ID=some-token -p8200:8200  -d --name=dev-vault vault:latest
  1. Once container is up and running exec to container in order to create secrets
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='some-token'

The above environment variables are used by vault client to connect to the vault server.

  1. Create secret
export BASE_PATH='secret/gocd'
vault kv put $BASE_PATH AWS_ACCESS_KEY=ABDASDKDLKM \
    AWS_SECRET_KEY=asdsfksfhkdfgfhfghhg; \
    GITHUB_TOKEN=97dasdd9789sd79sdf7sd9f7s9f9sd7f9sdvfd9gd9

Configure the plugin

The plugin needs to be configured with a secret config in order to connect to Vault. The configuration can be added from the Secrets Management page under Admin > Secret Management.

Alternatively, the configuration can be added directly to the config.xml using the configuration.

<secretConfigs>
    <secretConfig id="vault" pluginId="com.thoughtworks.gocd.secretmanager.vault">
      <description>All secrets for env1</description>
      <configuration>
        <property>
          <key>VaultUrl</key>
          <value>http://127.0.0.1:8200</value>
        </property>
        <property>
          <key>VaultPath</key>
          <value>secret/gocd</value>
        </property>
        <property>
          <key>AuthMethod</key>
          <value>token</value>
        </property>
        <property>
          <key>Token</key>
          <value>some-auth-token</value>
        </property>
      </configuration>
      <rules>
          <allow action="refer" type="environment">env_*</allow>
          <deny action="refer" type="pipeline_group">my_group</deny>
          <allow action="refer" type="pipeline_group">other_group</allow>
      </rules>
    </secretConfig>
</secretConfigs>

<rules> tag defines where this secretConfig is allowed/denied to be referred. For more details about rules and examples refer the GoCD Secret Management documentation

Field Required Description
VaultUrl Yes The url of the Vault server instance. If no address is explicitly set, the plugin will look to the VAULT_ADDR environment variable.
VaultPath Yes The root vault path which holds the secrets, and should be prepended to all secret values (e.g. secret/gocd)
ConnectionTimeout No The number of seconds to wait before giving up on establishing an HTTP(s) connection to the Vault server. If no openTimeout is explicitly set, then the object will look to the VAULT_OPEN_TIMEOUT environment variable. Defaults to 5 seconds.
ReadTimeout No Once connection has already been established, this is the number of seconds to wait for all data to finish downloading. If no readTimeout is explicitly set, then the object will look to the VAULT_READ_TIMEOUT environment variable. Defaults to 30 seconds.
ServerPem No An X.509 certificate, in unencrypted PEM format with UTF-8 encoding to use when communicating with Vault over HTTPS
AuthMethod Yes The auth method to use to authenticate with the Vault server, can be one of token, approle or cert
Token No Required if using token auth method. This is the token used to read secrets from Vault. Ensure this token has a longer ttl, the plugin will not be renewing the token.
RoleId No Required if using approle auth method. The plugins will use the configured RoleId and SecretId to authenticate with Vault.
SecretId No Required if using approle auth method.
ClientKeyPem No Required if using cert auth method. An RSA private key, in unencrypted PEM format with UTF-8 encoding.
ClientPem No Required if using cert auth method. An X.509 client certificate, in unencrypted PEM format with UTF-8 encoding.
Max Retries No Number of times to attempt to gather secrets from Vault. Defaults to 0.
Retry Interval Milliseconds No Duration between retry attempts (set by Max Retries). Defaults to 100 milliseconds.

Using the secret plugin in GoCD

Since version 1.3.0, the complete path used to look up the secret in Vault can be varied on individual secret retrieval, allowing a given Secret Configuration to be shared for all paths within a given root.

Sub-paths can be optionally configured by prefixing the key with subpath:, i.e the retrieved path is the concatenation of the plugin config VaultPath and the optional extra path specified in the secret key before :.

Assuming the the vault config's root VaultPath is secret/gocd, then:

  • {{SECRET:[vault][my_key]}} looks up the key my_key in the secret at secret/gocd
  • {{SECRET:[vault][my_server:my_key]}} looks up the key my_key in the secret at secret/gocd/my_server
  • {{SECRET:[vault][/a/b/c/d:my_key]}} looks up the key my_key in the secret at secret/gocd/a/b/c/d
  • {{SECRET:[vault][a:b:my_key]}} looks up the key a:b:my_key in the secret at secret/gocd (multiple :s are ignored)

Building the code base

To build the jar, run ./gradlew clean test assemble

Troubleshooting

Enable Debug Logs

If you are on GoCD version 19.6 and above:

Edit the file wrapper-properties.conf on your GoCD server and add the following options. The location of the wrapper-properties.conf can be found in the installation documentation of the GoCD server.

# We recommend that you begin with the index `100` and increment the index for each system property
wrapper.java.additional.100=-Dplugin.com.thoughtworks.gocd.secretmanager.vault.log.level=debug

If you're running with GoCD server 19.6 and above on docker using one of the supported GoCD server images, set the environment variable GOCD_SERVER_JVM_OPTIONS:

docker run -e "GOCD_SERVER_JVM_OPTIONS=-Dplugin.com.thoughtworks.gocd.secretmanager.vault.log.level=debug" ...

The plugin logs are written to LOG_DIR/plugin-com.thoughtworks.gocd.secretmanager.vault.log. The log dir

  • on Linux is /var/log/go-server
  • on Windows are written to C:\Program Files\Go Server\logs
  • on docker images are written to /godata/logs

gocd-vault-secret-plugin's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gocd-vault-secret-plugin's Issues

[Feature Request] Support for auth method using kubernetes service account

A lot of users running their GoCD servers within kubernetes.
Setting up a secret in GoCD involves choosing an auth method, which is currently either Token, AppRole or TLS.
For simplicity of setup, it's a little tempting for admins to just put in the root token, which basically grants GoCD access to do anything in vault.
For those users running GoCD on kubernetes, it would be simpler to select kubernetes auth.
The plugin could then fetch the kubernetes service account token from the usual location (/var/run/secrets/kubernetes.io/serviceaccount/token).
The user would simply need to configure the auth path (defaulting to auth/kubernetes) and role, and their access is done!
This would be a lot simpler to set up, and doesn't require the user to do any further setup or AppRole or to go and fetch any sensitive credentials to paste in.

-Dplugin.com.thoughtworks.gocd.secretmanager.vault.log.level=debug does not increase log level

Troubleshooting error responses from a remote vault server is almost impossible with the provided logs.

Having a go-server running with parameter -Dplugin.com.thoughtworks.gocd.secretmanager.vault.log.level=debug doesn't seem to increase the log level.

What would be of very interest would be the body of the request made against the vault server, headers inclusive, as well as the full response with headers. This would help to understand how queries are made and if there was a wrong path provided etc.

sample stacktrace:

ERROR [110@MessageListener for WorkFinder] Executor:127 - Failed to lookup secret from vault.
com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 403
Response body: {"errors":["1 error occurred:\n\t* permission denied\n\n"]}

    at com.bettercloud.vault.api.Logical.read(Logical.java:98)
    at com.bettercloud.vault.api.Logical.read(Logical.java:74)
    at com.thoughtworks.gocd.secretmanager.vault.SecretConfigLookupExecutor.execute(SecretConfigLookupExecutor.java:51)
    at com.thoughtworks.gocd.secretmanager.vault.SecretConfigLookupExecutor.execute(SecretConfigLookupExecutor.java:33)
    at cd.go.plugin.base.executors.secrets.LookupExecutor.execute(LookupExecutor.java:28)
    at cd.go.plugin.base.dispatcher.RequestDispatcher.lambda$dispatch$0(RequestDispatcher.java:41)
    at java.base/java.util.Optional.map(Unknown Source)
    at cd.go.plugin.base.dispatcher.RequestDispatcher.dispatch(RequestDispatcher.java:41)
    at com.thoughtworks.gocd.secretmanager.vault.VaultPlugin.handle(VaultPlugin.java:59)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:208)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:164)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131)
    at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
    at com.thoughtworks.go.plugin.access.secrets.v1.SecretsExtensionV1.lookupSecrets(SecretsExtensionV1.java:101)
    at com.thoughtworks.go.plugin.access.secrets.SecretsExtension.lookupSecrets(SecretsExtension.java:79)
    at com.thoughtworks.go.server.service.SecretParamResolver.lambda$lookupAndUpdateSecretParamsValue$0(SecretParamResolver.java:89)
    at java.base/java.util.HashMap.forEach(Unknown Source)
    at com.thoughtworks.go.server.service.SecretParamResolver.resolve(SecretParamResolver.java:80)
    at com.thoughtworks.go.server.service.SecretParamResolver.resolve(SecretParamResolver.java:66)
    at com.thoughtworks.go.server.service.BuildAssignmentService.lambda$createWork$2(BuildAssignmentService.java:298)
    at com.thoughtworks.go.server.transaction.TransactionTemplate.lambda$execute$1(TransactionTemplate.java:31)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
    at com.thoughtworks.go.server.transaction.TransactionTemplate.execute(TransactionTemplate.java:28)
    at com.thoughtworks.go.server.service.BuildAssignmentService.lambda$createWork$3(BuildAssignmentService.java:288)
    at com.thoughtworks.go.server.transaction.TransactionTemplate.transactionSurrounding(TransactionTemplate.java:60)
    at com.thoughtworks.go.server.service.BuildAssignmentService.createWork(BuildAssignmentService.java:272)
    at com.thoughtworks.go.server.service.BuildAssignmentService.assignWorkToAgent(BuildAssignmentService.java:181)
    at com.thoughtworks.go.server.service.BuildAssignmentService.assignWorkToAgent(BuildAssignmentService.java:160)
    at com.thoughtworks.go.server.messaging.scheduling.WorkFinder.onMessage(WorkFinder.java:60)
    at com.thoughtworks.go.server.messaging.scheduling.WorkFinder.onMessage(WorkFinder.java:32)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:84)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:64)
    at java.base/java.lang.Thread.run(Unknown Source)

NameSpaces

There does not seem to be any current support for namespaces. Is there a way to make this work with vault namespaces or any plans to implement it in the future?

How to access secrets in multipath under vault kv?

Hello, I got this tree structure in my vault:
kv/app1/dev/bdd_secret/
kv/app1/prod/app_secret/

kv/app2/dev/bdd_secret/
kv/app2/prod/app_secret/

How can I get secret from app1 or app2 tree according the environnement ?
Like this:
VaultPath=kv
var1={{SECRET:[testvault][app1/dev/bdd_secret/myuser1]}}
var2={{SECRET:[testvault][app1/prod/app_secret/myuser2]}}

I cannot a way to do this.
Thanks

Not resolving Vault Secrets

I am running GoCD installed via Helm in my kubernetes cluster. My configuration looks like:

id: vault
Vault URL: https://myvaultserver.com
Vault Path: /my/path/to/secrets
Auth Method: Token
Token: MY_AUTH_TOKEN
Rules:
Allow All *

However when I assign a secret {{SECRET:[vault][password]}} to an environment variable my jobs fail after agent registration with this error:

com.thoughtworks.go.plugin.access.exceptions.SecretResolutionFailureException: Expected plugin to resolve secret param(s) `password` using secret config `vault` but plugin failed to resolve secret param(s) `password`. Please make sure that secret(s) with the same name exists in your secret management tool.

I can retrieve this just fine with vault CLI:

=========== Data ===========
Key                    Value
---                    -----
password               My_Password

What am I missing?

X-509 format ServerPem

What is the string form that I need to put in ServerPem?

I'm trying a lot ways, but I always receive error:

Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input

Thank you

ServerPem

I'm trying to setup server pem but I'm receiving this error:

Could not parse certificate: java.io.IOException: Incomplete data

I'm pasting the certificate, in this format:

-----BEGIN CERTIFICATE-----
MIIFRTCCBC2gAwIBAgIUD3duYYpQE9qOYwMUSlnqFygVNMUwDQYJKoZIhvcNAQEL
BQAwgdIxCzAJBgNVBAYTAkJSMRIwEAYDVQQIEwlTYW8gUGF1bG8xEjAQBgNVBAcT
CVNhbyBQYXVsbzFQME4GA1UECQxHUnVhIEdvbWVzIGRlIENhcnZhbGhvLCAxNjI5
IC0gMsK6IGFuZGFyIC0gVmlsYSBPbMOtbXBpYSwgU8OjbyBQYXVsbywgU1AxEjAQ
BgNVBAoTCUNvcmEgQmFuazEPMA0GA1UECxMGRGV2T3BzMSQwIgYDVQQDExtDb3Jh
IEludGVybWVkaWF0ZSBBdXRob3JpdHkwHhcNMjAwOTAzMTUyODI5WhcNNDcxMDEy
MTUyODU4WjCBszELMAkGA1UEBhMCQlIxEjAQBgNVBAgTCVNhbyBQYXVsbzESMBAG
A1UEBxMJU2FvIFBhdWxvMUAwPgYDVQQJDDdSdWEgR29tZXMgZGUgQ2FydmFsaG8s
IDE2MjkgLSAywrogYW5kYXIgLSBWaWxhIE9sw61tcGlhMRIwEAYDVQQKlDb3Jh
IEJhbmsxDzANBgNVBAsTBkRldk9wczEVMBMGA1UEAwwMKi5jb3JhLmxvY2FsMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApvnA8fbMBzOgS6w3Qxzaxvb9
FYIa88CaFIH+7V9gbCFjYgvj7qvyIXM1pJ4SmBIJxU+DJgocE4MMada56VyMC1JG
W5V+Rsk8WJQ9zxX/I/b15jaA9gWN+slCndjp+MUbGgOgGAMKNlIHd/jC1H1ofcZO
u9qpvVjumjvdIjzFp33FshfYigy11Tb+lHUzpJ0Aj2GfcShX1fVT1xhlRkCPZ
aauBd6Wo4IQQ8fp8as4fgZF5GRK8X0cowhi7LT+XgQxaR40dQFXMh/8Gt7zKWfW+
7mwNt2HcK2w7BaLobgl/wqKyF2ZQpYkmVj73KVycxoKjjIyfUwuqMoGGLPJrOwID
AQABo4IBLjCCASowDgYDVR0PAQH/BAQDAgOoMB0GA1UdJQQWMBQGCCsGAQUFBwMB
BggrBgEFBQcDAjAdBgNVHQ4EFgQU1kxunkAp5QZxk62Gi0zd/ksK/3cwHwYDVR0j
BBgwFoAULAlbb+rIouaJ/pTurMrIhae0k+IwRAYIKwYBBQUHAQEEODA2MDQGCCsG
AQUFBzAChihodHRwczovL3Z0LmNvcmEudG9vbHMvdjEvY29yYV9wa2lfaW50L2Nh
MDcGA1UdEQQwMC6CDCouY29yYS5sb2NhbIISKi5zdGFnZS5jb3JhLmxvY2Fsggpj
b3JhLmxvY2FsMDoGA1UdHwQzMDEwL6AtoCuGKWh0dHBzOi8vdnQuY29yYS50b29s
cy92MS9jb3JhX3BraV9pbnQvY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQBVUXyzq+Eb
1909/J166dyRcRoanka5k1MuuvZBRMpWnW2FpGBRtSuZwAnT/EpfylyJNfB0lgdr
WoqcO6ZNDraAuRafvi/HMjX0r4WWxxSK20M2kpoF23U+lTsa0Z/SO34mW+LoYon3
au0qkHcV7w/pIV7YVpczrpDM3SO1XK77ZWYlhnA8/8wi1uvbKaI/lo6K0/+hT63A
zE7Ti6BkPo35C615EsfpvE6eN8pe0mn5T+tO2EOX1G1/E9Pp7CK4288lJpVrl3gU
u5iNANbIZ7uri+pKIA5LmCXZNy8A2QKMirdxDD/qd8P8aYdlV88OjwZxdDqt3MdC
r4tv4zMY0hE9
-----END CERTIFICATE-----

The certificate here was modified, but the format is this.

The certificate that I have to put here is the vault address correct?

Thank you

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.