Code Monkey home page Code Monkey logo

medusa's Introduction

medusa logo medusa logo

Medusa

GoDoc Go Report Card Build status codecov CodeQL

Table of Contents

About

Medusa is a cli tool currently for importing and exporting a json or yaml file into HashiCorp Vault.
Medusa currently supports kv1 and kv2 Vault secret engines.

Supported HashiCorp Vault versions

The minimum required HashiCorp Vault version that is supported by Medusa is Vault version 0.10.0. Medusa has not been tested with earlier verisons than Vault version 0.10.0.

How to use

In this section you can read about how to configure and use Medusa.
You can also watch the Medusa 101 introduction video to get a quick introduction on how to use Medusa for importing and exporting secrets in HashiCorp Vault.

Setting up Medusa

Config file

It's possible to create a config file for Medusa to read in your homefolder ~/.medusa/config.yaml that looks like this

VAULT_ADDR: https://192.168.86.41:8201
VAULT_SKIP_VERIFY: true
VAULT_TOKEN: 00000000-0000-0000-0000-000000000000

If you haven't set any environment variables, or given any parameters, this file will tell Medusa where to connect, the token to use and to VAULT_SKIP_VERIFY should be enabled or not.

Environment variables

It's also possible configure Medusa via environment variables by setting them like this:

export VAULT_ADDR=https://192.168.86.41:8201
export VAULT_SKIP_VERIFY=true
export VAULT_TOKEN=00000000-0000-0000-0000-000000000000

Parameters

Get help with ./medusa -h You can configure Medusa in the commands you run like this :

  -a, --address string   Address of the Vault server
  -k, --insecure         Allow insecure server connections when using SSL
  -t, --token string     Vault authentication token

Use them like this:

./medusa import secret ./test/data/import-example-1.yaml --address="https://0.0.0.0:8201" --token="00000000-0000-0000-0000-000000000000" --insecure
./medusa export secret/A --address="https://0.0.0.0:8201" --token="00000000-0000-0000-0000-000000000000" --format="json" --insecure

Importing secrets

Get help with ./medusa import -h

Import a yaml file into a Vault instance

Usage: medusa import [vault path] ['file' to import | '-' read from stdin] [flags]

  Flags:
  -d, --decrypt              Decrypt the Vault data before importing
  -m, --engine-type string   Specify the secret engine type [kv1|kv2] (default "kv2")
  -h, --help                 help for import
  -p, --private-key string   Location of the RSA private key

  Global Flags:
  -a, --address string       Address of the Vault server
  -k, --insecure             Allow insecure server connections when using SSL
  -n, --namespace string     Namespace within the Vault server (Enterprise only)
  -t, --token string         Vault authentication token

Example:

# Read from file
./medusa import secret ./test/data/import-example-1.yaml -a="https://0.0.0.0:8201" -t="00000000-0000-0000-0000-000000000000" --insecure
Secret successfully written to Vault instance on path [/A/B/E]
Secret successfully written to Vault instance on path [/A/Xa/Z]
Secret successfully written to Vault instance on path [/A/F/G]
Secret successfully written to Vault instance on path [/A/B/C/D]
Secret successfully written to Vault instance on path [/A/B/C/D/Db]

# Read from file
./medusa import secret/folder ./test/data/import-example-1.yaml -a="https://0.0.0.0:8201" -t="00000000-0000-0000-0000-000000000000" --insecure
Secret successfully written to Vault instance on path [folder/A/F/G]
Secret successfully written to Vault instance on path [folder/A/B/C/D]
Secret successfully written to Vault instance on path [folder/A/B/C/D/Db]
Secret successfully written to Vault instance on path [folder/A/B/E]
Secret successfully written to Vault instance on path [folder/A/Xa/Z]

# Read from stdin
cat ./test/data/import-example-1.yaml | ./medusa import secret -
Secret successfully written to Vault [https://localhost:8201] using path [/A/F/G]
Secret successfully written to Vault [https://localhost:8201] using path [/A/B/C/D]
Secret successfully written to Vault [https://localhost:8201] using path [/A/B/C/D/Db]
Secret successfully written to Vault [https://localhost:8201] using path [/A/B/E]
Secret successfully written to Vault [https://localhost:8201] using path [/A/Xa/Z]

Exporting secrets

Get help with ./medusa export -h and yaml is the default output format Medusa import will take a [vault path] with [flags]

  Flags:
  -e, --encrypt              Encrypt the exported Vault data
  -m, --engine-type string   Specify the secret engine type [kv1|kv2] (default "kv2")
  -f, --format string        Specify the export format [yaml|json] (default "yaml")
  -h, --help                 help for export
  -o, --output string        Write to file instead of stdout
  -p, --public-key string    Location of the RSA public key

  Global Flags:
  -a, --address string       Address of the Vault server
  -k, --insecure             Allow insecure server connections when using SSL
  -n, --namespace string     Namespace within the Vault server (Enterprise only)
  -t, --token string         Vault authentication token

Example:

./medusa export secret --address="https://0.0.0.0:8201" --token="00000000-0000-0000-0000-000000000000" --format="yaml" --insecure
A:
  B:
    C:
      D:
        Db:
          DBa: value 1
          DBb: value 2
    E:
      Ea: value 1
      Eb: value 2
  F:
    G:
      Ga: value1
  Xa:
    Z:
      Za: value 1
      Zb: value 2

Deleting secrets

Get help with ./medusa delete -h Medusa delete will take a [vault path] with [flags]

  Flags:
  -y, --auto-approve         Skip interactive approval of plan before deletion
  -m, --engine-type string   Specify the secret engine type [kv1|kv2] (default "kv2")
  -h, --help                 help for import

  Global Flags:
  -a, --address string       Address of the Vault server
  -k, --insecure             Allow insecure server connections when using SSL
  -n, --namespace string     Namespace within the Vault server (Enterprise only)
  -t, --token string         Vault authentication token

Example:

./medusa delete secret/production --address="https://0.0.0.0:8201" --token="00000000-0000-0000-0000-000000000000" --insecure
Deleting secret [secret/production/users/cart/database]
Deleting secret [secret/production/users/cart/database/users/readuser]
Deleting secret [secret/production/users/cart/database/users/writeuser]
Deleting secret [secret/production/users/user/database]
Deleting secret [secret/production/users/user/database/users/readuser]
? Do you want to delete the 25 secrets listed above? Only 'y' will be accepted to approve.? [y/N] y
The secrets has now been deleted


./medusa delete secret/staging --address="https://0.0.0.0:8201" --token="00000000-0000-0000-0000-000000000000" --insecure --auto-approve
Deleting secret [secret/staging/users/cart/database]
Deleting secret [secret/staging/users/cart/database/users/readuser]
Deleting secret [secret/staging/users/cart/database/users/writeuser]
Deleting secret [secret/staging/users/user/database]
Deleting secret [secret/staging/users/user/database/users/readuser]
The secrets has now been deleted

Decrypt secrets

Get help with ./medusa decrypt -h Medusa decrypt will take a [FILE path] with [flags]

  Flags:
  -p, --private-key string   Location of the RSA private key

Example:

# Write to stdout
./medusa decrypt encrypted-export.txt --private-key private-key.pem
env:
  dev:
    nomad:
      token: secret-token
  production:
    nomad:
      token: secret-other-token


# Write to file
./medusa decrypt encrypted-export.txt --private-key private-key.pem > plaintext-export.yaml

Encrypt secrets

Get help with ./medusa encrypt -h Medusa encrypt will take a [FILE path] with [flags]

  Flags:
  -o, --output string       Write to file instead of stdout
  -p, --public-key string   Location of the RSA public key

Example:

# Write to stdout
./medusa encrypt plaintext-export.txt --public-key public-key.pem
<Encrypted data>

# Write to file
./medusa encrypt plaintext-export.txt --public-key public-key.pem --output encrypted-export.txt.b64

Secure secret management outside Vault

Medusa will help you securely manage your secrets outside Vault. This could for instance be as a backup of your Vault data or while your secrets are being transported between Vault instances.
Medusa uses a hybrid encryption solution in order to keep your secrets safe.

Key generation

When exporting your Vault secrets using Medusa, the secrets are encrypted using the AES symmetric encryption algorithm. The 256-bit AES encryption key is randomly generated by Medusa every time the export command is being called.
Then the AES key is encrypted by the provided RSA public key and then stored together with the encrypted secrets.
This ensures that both the exported secrets and AES enctyption key can be transfered safely between Vault instances.
The exported secrets and AES enctyption key can only be decrypted by a person who is in possession of the RSA private key.

The RSA key-pair can be generated by the following two commands:

# Generate private key
openssl genrsa -out private-key.pem 4096

# Generate public key
openssl rsa -in private-key.pem -pubout -out public-key.pem

Exporting and encrypting Vault secrets

Encrypting your Vault export is easy using Medusa. Simply add the following two flags to your command:

-e, --encrypt bool       Encrypt the exported Vault data [true/false]
-p, --public-key string  Location of the RSA public key

Use them like this:

./medusa export kv --address="https://my-vault-server.com" --token="00000000-0000-0000-0000-000000000000" --insecure --encrypt="true" --public-key="public-key.pem" --output="encrypted-vault-secrets.txt"

Importing and decrypting Vault secrets

Decrypting and importing your encrypted Vault export can be done by adding the following two flags to your command:

-d, --decrypt bool        Decrypt the Vault data before importing [true/false]
-p, --private-key string  Location of the RSA private key

Use them like this:

./medusa import kv encrypted-vault-secrets.txt --address="https://my-vault-server.com" --token="00000000-0000-0000-0000-000000000000" --insecure --decrypt="true" --private-key="private-key.pem"

Help

To test out medusa on your laptop

Medusa is a cli tool currently for importing a json or yaml file into HashiCorp Vault.
Created by Jonas Vinther & Henrik Høegh.

Usage:
  medusa [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  decrypt     Decrypt an encrypted Vault output file into plaintext in stdout
  delete      Recursively delete all secrets below the given path
  encrypt     Encrypt a Vault export file onto stdout or to an output file
  export      Export Vault secrets as yaml
  help        Help about any command
  import      Import a yaml file into a Vault instance
  version     Print the version number of Medusa

Flags:
  -a, --address string                Address of the Vault server
  -h, --help                          help for medusa
  -k, --insecure                      Allow insecure server connections when using SSL
      --kubernetes                    Authenticate using the Kubernetes JWT token
      --kubernetes-auth-path string   Authentication mount point within Vault for Kubernetes
  -n, --namespace string              Namespace within the Vault server (Enterprise only)
  -r, --role string                   Vault role for Kubernetes JWT authentication
  -t, --token string                  Vault authentication token

Use "medusa [command] --help" for more information about a command.

medusa's People

Contributors

buster1253 avatar cwien avatar dependabot[bot] avatar elopsod avatar gramosg avatar hoeghh avatar jonasvinther avatar kpaulisse avatar landrew57 avatar linant avatar madsbaggesen avatar maroskukan avatar motilevy avatar rdrgmnzs avatar tim-goto avatar vanveele avatar vvaltia avatar zoeimogen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

medusa's Issues

Document use of Medusa

We need to create documentation for how to use Medusa, and fix some old documentation

Describe how to contribute

We need to document how people can contribute to the project.
The documentation should be located under /docs and be pointed to from the root README.md

Import from stdin

Thanks for the great tool. I already have SOPS+age set up to manage encryption and decryption of files in my projects, so I only need Medusa for the import/export functionality. It does not seem obvious to me from the documentation and command line help if it is possible to import from stdin. The scenario is using SOPS to decrypt to stdout and then using Medusa to import the decrypted backup from stdin without having to write the decrypted backup to an intermediate file. Is this possible with the current version of Medusa?

Request for installation instructions

Good day,

I'm trying to test Medusa but not exactly sure how to install it. When I run make I see Go getting some stuff but then I get the below:

?       medusa  [no test files]
?       medusa/cmd      [no test files]
?       medusa/pkg/encrypt      [no test files]
=== RUN   TestImport
--- PASS: TestImport (0.00s)
PASS
ok      medusa/pkg/importer     0.005s
?       medusa/pkg/vaultengine  [no test files]

Would be nice to have clear installation instructions, but am investigating how to proceed.

Export Error '<' looking for beginning of value OR missing vaultaddr parameter

URL: http://207.254.25.36:8000/secrets/kv/secret/
it has subfolders in it, but for testing i used this http://207.254.25.36:8000/secrets/kv/secret/android/dev/keys

Cmd Executed:
./medusa export secret/android/dev/ -k --address="http://207.254.25.36:8000/secrets/kv/secret/android/dev/"

./medusa export secret/android/dev/ -k --address="http://207.254.25.36:8000/secrets/kv/"

let me know if i am passing the urlFlag or VaultPath wrong
Error:
invalid character '<' looking for beginning of value
Error: invalid character '<' looking for beginning of value

apart from the above error, what exactly this mountpathMean when importing?

Doesn't handle nested secrets with the same name.

Firstly, great product, helped me out with a Vault migration in record time. While I was doing the migration, I found an issue:

  • We have a secret path of foo/bar
  • Under foo/bar, there is a secret called 'moo'
  • Also under foo/bar, this is a directory, also called 'moo' that contains further secrets.

eg:

foor/bar
    moo
    moo/wibble

Medusa skipped exporting the secret 'moo' and instead only exported the directory called 'moo'. No errors in logs. I suspect it just passed over the secret due to it having the same name as the directory in the same path.

This is exporting from Vault 1.7.1. The secret engine is kv2

Import is wrapping the keys within extra "data"

Hi,
May be a silly question but I couldn't find anything related this in readme so posting it here.

here are the commands I'm using for export/import.

docker run
--rm
-v $(shell pwd):/tmp/
-e VAULT_ADDR=$(VAULT_ADDR_SOURCE)
-e VAULT_TOKEN=$(VAULT_TOKEN_SOURCE)
-e VAULT_SKIP_VERIFY=true
jonasvinther/medusa:latest
export -m=kv1 myfolder --insecure --encrypt=true --public-key=/tmp/public-key.pem -o /tmp/backup.json

docker run
-v $(shell pwd):/tmp/
--rm
-e VAULT_ADDR=$(VAULT_ADDR_DESTINATION)
-e VAULT_TOKEN=$(VAULT_TOKEN_DESTINATION)
-e VAULT_SKIP_VERIFY=true
jonasvinther/medusa:latest
import -m=kv1 myfolder --insecure --decrypt=true --private-key=/tmp/private-key.pem /tmp/backup.json

For example this the export via medusa

project:
myapp:
data:
foo2:
data:
foo: bar
pizza: cheese
options:
cas: 0
dummy:
foo: bar
kv:
mysecret:
foo: bar
mysecret:
data:
mykey1: myjmv
name: nitin

But this what being imported via medusa with extra "data" tag.

project:
myapp:
data:
foo2:
data:
data:
foo: bar
pizza: cheese
options:
data:
cas: 0
dummy:
data:
foo: bar
kv:
mysecret:
data:
foo: bar
mysecret:
data:
data:
mykey1: myjmv
name: nitin

Does not handle JSON secrets

We have a secret in a kv2 engine that has a big JSON object as data.
Exporting this secret with medusa works fine, both in yaml and json format, but when importing back, it tries to create separate secrets from every key/value pair instead of creating just one secret with the whole JSON object.
Plus during this try, it's unable to handle some keys with this error: Error while writing secret. json: unsupported type: importer.RawYaml.

unexpected behavior when importing "complexly structured secrets"

I have a handful of "complex, structured JSON objects" currently stored in Vault. specifically, they are Hyperledger Fabric connection profiles. it's just a big chunk of JSON with mostly lists of URLs and public keys.

here's a heavily redacted chunk of one for a specific example.

image

however, when dumped and loaded with medusa, it produces many lines of log output like the following:

Secret successfully written to Vault [https://target-vault:8200] using path [/fabric-profile/channels/mainnet/peers/hlf-peer--...]
Secret successfully written to Vault [https://target-vault:8200] using path [/fabric-profile/channels/mainnet]
Secret successfully written to Vault [https://target-vault:8200] using path [/fabric-profile/certificateAuthorities/hlf-ca--redacted]
Secret successfully written to Vault [https://target-vault:8200] using path [/fabric-profile/organizations/Redacted]
Secret successfully written to Vault [https://target-vault:8200] using path [/fabric-profile/client/connection/timeout]

and creates a directory structure instead of a single object ??

image

I'm willing to accept that "maybe Vault itself doesn't like this Blob of JSON" and I'm storing my data incorrectly. I also have NOT tested if my applications reading from this directory structure is equivalent from reading the Blob of JSON. (even if it is, the Hyperledger ecosystem of tools and daemons often support this One Config JSON and it would be nice to keep it as a unified format instead of making the Vault representation some "Vault Normalized" variant.)

I don't think I know enough about Vault internals and/or Medusa to determine if "I'm doing it wrong" or if this is a bug in Medusa. certainly the behavior surprised me: the "social contract" around a dump&&load tool is that the data is not transformed (or transformed as minimally as possible to support the load, ie: modifying timestamps)

edit: I realize now this is basically a duplicate of #47

Medusa import with encrypt -- "panic: runtime error: index out of range [0] with length 0"

Hey there, so I am importing several sets of encrypted secrets into a private vault instance. About half of them import just fine, the other half fail with the following message:
`panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/jonasvinther/medusa/pkg/encrypt.Decrypt(0x7ffd3766a689, 0x19, 0x7ffd3766a5fd, 0x30, 0xc, 0xc0001296d7, 0x1, 0x0)
/vault/medusa/pkg/encrypt/encrypt.go:46 +0x41d
github.com/jonasvinther/medusa/cmd.glob..func4(0xc62a20, 0xc000184300, 0x2, 0x8, 0x0, 0x0)
/vault/medusa/cmd/import.go:51 +0x4a5
github.com/spf13/cobra.(*Command).execute(0xc62a20, 0xc000184280, 0x8, 0x8, 0xc62a20, 0xc000184280)
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:850 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc62240, 0xc000000180, 0xc00018df78, 0x407d05)
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:895
github.com/jonasvinther/medusa/cmd.Execute(...)
/vault/medusa/cmd/cmd.go:59
main.main()
/vault/medusa/main.go:10 +0x32`

All sets were successfully exported from vault using "medusa export" with --encrypt. The difference between the sets that successfully decrypt and import and the ones that do not is that the failing sets are larger, with more sets of secrets.

Thanks for any help you can provide.

required policy ?

Hi,

I create a policy dedicated for my needs with :

path "secret/*" {  capabilities = ["read","list"] }

and works as expected with CLI

But when using a token bind to this policy I hit the error bellow

$ ./medusa export secret -m kv1 -a https://127.0.0.1:8200 -k -t xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Error making API request.

URL: GET https://127.0.0.1:8200/v1/sys/internal/ui/mounts/secret
Code: 403. Errors:

* permission denied

I don't get why medusa needs to access v1/sys/internal/ui/mounts/secret for looping over what is in secret.

Regards,

build error after initial clone

New to medusa. cloned the repo, and performed a go build and received the following output.

go build . go: github.com/hashicorp/vault/[email protected] requires github.com/hashicorp/vault/[email protected] requires golang.org/x/[email protected]: invalid pseudo-version: git fetch --unshallow -f origin: fatal: git fetch-pack: expected shallow list

any suggestions would be greatly appreciated.

Unable to export a KV path due to a `segmentation violation`

When attempting export a specific path on vault, it will throw the following error:

[sudo] password for robloxadmin:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x98 pc=0x8ad3d2]

goroutine 1 [running]:
github.com/jonasvinther/medusa/pkg/vaultengine.(*Client).SecretRead(0xc0000d1d08, 0xc0000831c0, 0x3f, 0xc00024f800)
        /home/runner/work/medusa/medusa/pkg/vaultengine/secret_read.go:54 +0x2b2
github.com/jonasvinther/medusa/pkg/vaultengine.(*Client).PathReader(0xc0000d1d08, 0xc0000d1b28, 0xc0000e2780, 0x2d, 0x0, 0x0)
        /home/runner/work/medusa/medusa/pkg/vaultengine/folder_export.go:79 +0x1c5
github.com/jonasvinther/medusa/pkg/vaultengine.(*Client).FolderExport(0xc0000d1d08, 0xc0000e2780, 0x2d, 0x10, 0xc0000e2780, 0x2d)
        /home/runner/work/medusa/medusa/pkg/vaultengine/folder_export.go:16 +0x69
github.com/jonasvinther/medusa/cmd.glob..func3(0xd375a0, 0xc0000dfa40, 0x1, 0x7, 0x0, 0x0)
        /home/runner/work/medusa/medusa/cmd/export.go:42 +0x3fe
github.com/spf13/cobra.(*Command).execute(0xd375a0, 0xc0000df9d0, 0x7, 0x7, 0xd375a0, 0xc0000df9d0)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:850 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xd37060, 0xc00003a778, 0xc0000d1f78, 0x4062c5)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:895
github.com/jonasvinther/medusa/cmd.Execute(...)
        /home/runner/work/medusa/medusa/cmd/cmd.go:59
main.main()
        /home/runner/work/medusa/medusa/main.go:10 +0x32

There are 2 paths that have data in them
/secret/teams/neteng/grid-service-v2/release -- throws
/secret/teams/neteng/grid-service-v2/debug -- can be exported, despite it having the same KV layout

The export command is as follows:
./medusa export secret/teams/neteng/grid-service-v2/release --address="https://chi1-vault.simulprod.com" --token="s.token" --format="json" --insecure

It can export /debug but no /release

If you need any more information, please ask

Clean up cli

Clean up the interface of the cli. We try to follow the same naming conventions as Vault.

./medusa export /secret/A --format="yaml" --address="http://..." --token="000-000-000" --output="secret.yaml"

global:
--tls-skip-verify

VAULT_ADDR
VAULT_TOKEN

Make use of ~/.medusa/.env

Medusa should be able to read configuration from a home folder, so that end users can setup where and how Medusa should connect.

Right now Medusa looks for a .env file in the scripts folder, which is not doable for end users. The script start-vault.sh under /scripts should copy the outputted .env file to ~/.medusa/.env for Medusa to use.

Not able to export data of transit and totp

I am not able to export the data of transit and totp, here is the command I used when I started my vault :

vault secrets enable totp transit
vault secrets disable secret
vault secrets enable kv -path=secret -version=1

And when I run command to export the data it showing me error:

./medusa export transit/ -m kv1 --address="http://0.0.0.0:8200" --token="s.token" -o testing.yaml

Response :

no keys found using path [transit//] on Vault instance [http://0.0.0.0:8200]
Error: no keys found using path [transit//] on Vault instance [http://0.0.0.0:8200]
Usage:
  medusa export [vault path] [flags]

Flags:
  -e, --encrypt              Encrypt the exported Vault data
  -m, --engine-type string   Specify the secret engine type [kv1|kv2] (default "kv2")
  -f, --format string        Specify the export format [yaml|json] (default "yaml")
  -h, --help                 help for export
  -o, --output string        Write to file instead of stdout
  -p, --public-key string    Location of the RSA public key

Global Flags:
  -a, --address string     Address of the Vault server
  -k, --insecure           Allow insecure server connections when using SSL
  -n, --namespace string   Namespace within the Vault server (Enterprise only)
  -t, --token string       Vault authentication token

no keys found using path [transit//] on Vault instance [http://0.0.0.0:8200]

I did try these too but same error :

./medusa export secrets/transit -m kv1 --address="http://0.0.0.0:8200" --token="s.token"  -o testing.yaml
./medusa export secrets/transit/secret_file -m kv1 --address="http://0.0.0.0:8200" --token="s.token"  -o testing.yaml
 / # vault version
Vault v1.3.0

Please let me know if there is anything I am doing wrong.

Delete secrets

Medusa should be able to clean up a sub path of a secret engine so that the the sub path is empty before performing the import.
The command could be like: ./medusa delete kv/my/sub/path

PathSplitPrefix helper fails to properly handle engine mounts with '/' characters

The method used to predict the secret engine mountpath in PathSplitPrefix fails to handle cases where the mount path includes one or more '/' characters.
eg.

$ vault secrets list 
Path                         Type         Accessor              Description
----                         ----         --------              -----------
cubbyhole/                   cubbyhole    cubbyhole_9bb160ea    per-token private secret storage
identity/                    identity     identity_0e3746c2     identity store
non-prod/staging/secrets/    kv           kv_a914dea7           n/a
prod/secrets/                kv           kv_55c5c07c           n/a
sys/                         system       system_11cdf9da       system endpoints used for control, policy and debugging

For example, the k/v v2 mount path 'prod/secrets' or 'non-prod/staging/secrets', which are valid and permitted by Vault, may contain the following secrets:

prod/secrets
├── appA
│   └── creds
└── appB
    ├── creds
    └── tls
non-prod/staging/secrets
├── appA
│   └── creds
└── appB
    ├── creds
    └── tls

when importing or exporting using medusa the path prod/secrets/appB/tls will be broken into:
{ "engine": "prod", "prefix": "secrets/appB/tls" }

which causes the client api request (for k/v v2) to target https://vault.example.com:8200/v1/prod/data/secrets/appB/tls which fails.

The correct split would result in:
{ "engine": "prod/secrets", "prefix": "appB/tls" }

and the api url: https://vault.example.com:8200/v1/prod/secrets/data/appB/tls.

I have confirmed a workaround that borrows heavily from Vault clitool's own method of splitting the mountpath from the secret path which I can include here unless you prefer a cleanroom implementation.

Add an option to print version

Hello,

could you please add a flag or a command to print the version of medusa?
E.g. "medusa version" or "medusa --version".

(Our current workaround for that is "strings medusa | fgrep 'main.version'")

Thanks!

BUG: Medusa fails to backup KV password version number, resulting in wrong version number on import

I labeled it as a bug instead of a feature request, because it causes an unintended behavior. In our case, it caused our infrastructure to error out when we tried to restore a backup, because it could not map to the passwords.

Why KV version numbers are important

KV (Key-Value) passwords are version controlled. Each password is assigned a new version number every time it's changed. In gitOps, that version number is very important, because each password placeholder is declared in git as <path:kv/data/foo#3>, #3 being the password version number. This number is incremented in git on password change, which in return signals the gitOps operator (i.e ArgoCD) to update the resource password.

The issue with Medusa

Medusa does not export previous version of the password. It also does not export the version number of that password. When the file is imported back, the version numbers on all KV passwords are reset to 1, causing the platform to not being able to retrieve the passwords.

Solution

  • export/import the version number of each KV secret
  • export/import all previous versions of KV secret

feature request - decrypt export

Sometimes I only want to restore some of the keys. So I think the easiest way would be for medusa to provide a command to decrypt a export into a plaintext version, which I can then view to work out which keys I want, or I can edit it before importing

Only timestamp and <nil> output sometimes on export

os: mac os 11.2.3
binary downloaded from here: https://github.com/jonasvinther/medusa/releases/download/v0.2.0/medusa_0.2.0_darwin_amd64.tar.gz
vault: 1.5.3

Steps:
Set VAULT_ADDR, and run export:
./medusa export di-secrets/ -t s.tokenstuff --format="yaml" --insecure
output:
2021/04/05 21:04:47 <nil> <nil>

Expected:
All the secret data in the engine at di-secrets (v2) should be exported to stdout.

Notes:

  • di-secrets is a v2 kv store
  • some exports work, for example: ./medusa export di-secrets/application -t s.foo

config.yaml Issue

Medusa does not take my configuration listed on config.yaml. So I have to put in variable to make it work

panic: runtime error: invalid memory address or nil pointer dereference

Running: medusa export Redacted -o Redacted.yaml

Getting:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x98 pc=0xc9693d]

goroutine 1 [running]:
github.com/jonasvinther/medusa/pkg/vaultengine.(*Client).SecretRead(0xc0000e3d08, 0xc00001efc0, 0x3b, 0xc0006abd00)
        /home/runner/work/medusa/medusa/pkg/vaultengine/secret_read.go:54 +0x2bd
github.com/jonasvinther/medusa/pkg/vaultengine.(*Client).PathReader(0xc0000e3d08, 0xc000758018, 0xc0003bc0c0, 0x22, 0x0, 0x0)
        /home/runner/work/medusa/medusa/pkg/vaultengine/folder_export.go:79 +0x1cf
github.com/jonasvinther/medusa/pkg/vaultengine.(*Client).PathReader(0xc0000e3d08, 0xc00075a038, 0xc00046ee40, 0x17, 0x0, 0x0)
        /home/runner/work/medusa/medusa/pkg/vaultengine/folder_export.go:67 +0x345
github.com/jonasvinther/medusa/pkg/vaultengine.(*Client).PathReader(0xc0000e3d08, 0xc000222000, 0xc00041c0c0, 0x10, 0x0, 0x0)
        /home/runner/work/medusa/medusa/pkg/vaultengine/folder_export.go:67 +0x345
github.com/jonasvinther/medusa/pkg/vaultengine.(*Client).PathReader(0xc0000e3d08, 0xc0000e3b40, 0xc0001c86f5, 0x1, 0xc000173860, 0x0)
        /home/runner/work/medusa/medusa/pkg/vaultengine/folder_export.go:67 +0x345
github.com/jonasvinther/medusa/pkg/vaultengine.(*Client).FolderExport(0xc000123d08, 0xc0001c86f5, 0x1, 0xc0001c86d0, 0xa, 0xc0001c86f5)
        /home/runner/work/medusa/medusa/pkg/vaultengine/folder_export.go:16 +0x70
github.com/jonasvinther/medusa/cmd.glob..func3(0x111b2e0, 0xc00017ccc0, 0x1, 0x3, 0x0, 0x0)
        /home/runner/work/medusa/medusa/cmd/export.go:47 +0x49e
github.com/spf13/cobra.(*Command).execute(0x111b2e0, 0xc00017cc90, 0x3, 0x3, 0x111b2e0, 0xc00017cc90)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:850 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x111ada0, 0xc000059f78, 0xc000123f78, 0x7f662b)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:895
github.com/jonasvinther/medusa/cmd.Execute(...)
        /home/runner/work/medusa/medusa/cmd/cmd.go:59
main.main()
        /home/runner/work/medusa/medusa/main.go:10 +0x39

unable to export with --insecure flag.

image
image

Tried both Kv/kv1/kv2 and secrets flag. Nothing worked out. When tried without --insecure flag got ": x509: certificate signed by unknown authority" error.

medusa import: Unable to import empty secret files

When transferring secrets from one instance to another, I run medusa export which gives me structure similar to this:

A:
  B:
    key1: value1
  C: {}

Here A/C is empty secret.

When I run medusa import, only A/B is imported.

A:
  B:
    key1: value1

Secret A/C is not imported.

Feature: Implement Export & Import of secret versions

Thx for this great tool.

kv-2 supports multiple versions of a secret. Medusa currently exports and imports the latest version of a secret, only.
As an improvement medusa can be extended to also ex- and import all versions of a secret.
This allows the user to create a full backup of all his secrets versions for Vaults kv-2 secrets engines.

Feature Request: Import and Export of Policies.

Thank you for this tool - it was very handy migrating secrets to new Vault server (fortunately, we don't care about the version history yet).

The next step of my migration was to move my Policies over in a manner very much like import/export. I was able to do that with some simple shell scripting, but would be cool to have as part of Medusa.

Putting shell script fragments here in case someone else trying to migrate might find them useful.

dump-policies.sh:

dump_dir="./dumped-policies"

policies=$(vault policy list | grep -v root)

mkdir -p "${dump_dir}"
for p in ${policies} ; do
    echo "reading policy: ${p}"
    vault policy read "${p}" > "${dump_dir}/${p}-policy-dumped.hcl"
done

load-dumped-policies.sh:

dump_dir="./dumped-policies"

policies=$(cd ${dump_dir}/ ; ls *-policy-dumped.hcl | sed -e 's/-policy-dumped.hcl//g')

for p in ${policies} ; do
    echo "writing policy: ${p}"
    vault policy write "${p}" "${dump_dir}/${p}-policy-dumped.hcl"
done

os exit codes

Thank you for this tool!
I am running it as a Kubernetes job and noticed that medusa export ... always returns an exit code of 0.
I wanted to know if you're open for a pull request to make sure all errors are returned back to main.go and change main.go to:

	err := cmd.Execute()
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

Or use log.Fatal/log.Fatalf where appropriate ?
let me know which approach makes sense and I'll open a PR.

Cannot export data from KV v1 secret engine

Running an export fails on KV v1 secret engine, because the v1 engine doesn't use the metadata path:

docker run medusa -a https://vault:8200 -k -t $(cat .vault-token) export secret/secretthings

Error: no keys found using path [secret/metadata/secretthings/] on Vault instance [https://vault:8200]
no keys found using path [secret/metadata/secretthings/] on Vault instance [https://vault:8200]`

Any chance of getting KV v1 support?

KV2 "Permanently Destroyed" secrets causes fatal crash

If a version of a secret is "Permanently Destroyed" this causes issues with medusa export

ex:

vault kv put test/test/test2 secret=my-secret
vault kv put test/test/test2 secret=my-secret-update
vault kv destroy -versions=2 test/test/test2

medusa export test
2022/03/09 14:26:44 Error while reading secret
Path:   test/data//test/test2
Data:   <nil>

0.3.2 breaks kv1 export (Darwin release)

Not sure if this happens on other architectures, but something about 0.3.2 broke export with --engine-type="kv1".
Version 0.3.1 works for me though. Not sure if import is also affected as I'm only using medusa for export at the moment.

JSON switch disabled in Vault UI after secrets imported( in yaml or json file)

Hi,
Medusa is a great tool and I enjoy the great convenience it brought to me when transfer secrets between different Vault instances time to time.

I just noticed that JSON switch disabled after I import the secrets from another Vault server.

Here is the procedures to re-produce my scenario:

1.Export a specific group of secrets to yaml or json file
medusa export secret2/cicd/env_vars/ --format="yaml" --insecure -o env_vars.yaml

2.Update env_vars.yaml with bundle of changes and import back
medusa import secret2 env_vars.yaml

  1. Go to Vault UI and JSON switch is disabled and can't switch on

图片

Vault Version:1.11.2
Medusa Version:0.3.6

Thank you.

Create k8s cronjob manifest example

We want an example manifest for a Kubernetes cronjob that uses medusa to backup a vault instance.

Something like :

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: medusa
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: jonasvinther/medusa:latest
            imagePullPolicy: IfNotPresent
            command:
            - ./medusa
            - -h
          restartPolicy: OnFailure

Then from there we will add env's to control what and where.

IP doesnt exist on mac

We use ip in /scripts/start-vault.sh and that command doesn't exists on mac's. So we need to catch missing dependencies, and tell people instead of simply continuing.

medusa import: Access Entities support

I am trying to import the Access Entity using medusa. I wonder if this is possible? I have tried importing the ACL Policies and it works fine. Consider I have the userpass Auth Method enabled with the Accessor "auth_userpass_966f2d29"

Let's say there is a "test" user created. Exporting those user works fine

./medusa export -n sys identity/entity/name -m kv1 -f json -o output.json
It produces the file output.json.zip

My next step is to import that user back

./medusa import identity -m kv1 output.json
Error while writing secret. json: unsupported type: importer.RawYaml

For us, it is important to back up not only the secrets but also the ACL Policies and Access Entities info as well.

Feature Request

Oh Lord, please add a functionality to import java property files (eg. for spring boot cloud config)

Suggestions

Thank you for a nice tool. Here are some improvements that will be welcome:

  • Autodetect the engine version if the token has enough permissions
  • Re-use the vault cli token ~/.vault-token if present and valid
  • Implement more user-friendly encryption that does not require openssl but instead relies on a key derivation function
  • in addittion to -insecure option allows to supply your own CA for validation
  • If we gone as far as creating the delete command adding a move/copy now sounds logical
  • Support cross-engine workflows better: have an option to include the engine name in the export/import format instead of requiring it to be present on command line only
  • Add an option to display the program's version

Describe how to sync a fork when contributing

To help beginners, we should add a section on how to sync a fork of this repo, so people can keep contributing.

Something like

# Add the remote, call it "upstream":

git remote add upstream https://github.com/jonasvinther/medusa

# Fetch all the branches of that remote into remote-tracking branches

git fetch upstream

# Make sure that you're on your master branch:

git checkout main

# Rewrite your main branch so that any commits of yours that
# aren't already in upstream/main are replayed on top of that
# other branch:

git rebase upstream/main

Medusa export : ability to export all the vault keys for key value engine

Hi Team,

I'm really impressed with the tool. It beautifully exports the given vault path and has very minimal and simple setup. But my main use case was to export all the keys for all the mounts for key value engine. I had few keys of generic/kv1/kv2.

Since this module doesnt support this right now out of the box, i have forked it and coded the basic minimal version at sumanthkumarc@47296bb This is my first golang code, so code might not be very standard but works for me.

medusa export /

will export all keys of all mounts belonging to generic/kv1/kv2 secret engines and skips any other engines. I have also added go routine support for parallel fetch, and timing.

I'm all good for giving up my code or even contributing, if i get some mentoring.

Support for namespaces (Vault Enterprise)

Vault Enterprise has the concept of "Namespaces" to isolate different environments.

The Vault API provides passing the namespace as an additional parameter, but medusa does not accept such a value on the command line yet. As we are using the Enterprise version of Vault we cannot use medusa until this option can be set.

In the vaultclient.go there is already the stub to support namespaces, but the commands are not yet aware of this parameter.

I added those parameters and will create a PR that will add a "namespace flag" to the commands.

HashiCorp Vault Version Compatibility

Remote vault server is running version 1.3.2.

When I tried to run - medusa export, it prompts an error:

Error: no keys found using path [secret/metadata//] on Vault instance [https://xxx]

Is there a minimum vault server version requirement? If so, better to mention in README

Unable to import an encrypted backup

Hello, I just follow the README.md, and I can't re-import the data.

pomme@shark medusa % openssl version
OpenSSL 1.1.1k  25 Mar 2021
pomme@shark medusa % openssl genrsa -out private-key.pem 4096
Generating RSA private key, 4096 bit long modulus (2 primes)
...........................++++
..........................................................................++++
e is 65537 (0x010001)
pomme@shark medusa % openssl rsa -in private-key.pem -pubout -out public-key.pem
writing RSA key
pomme@shark medusa % ./bin/medusa export kv-certificates_backup --token="$(vault print token)" --encrypt true --public-key="public-key.pem" --output="encrypted-vault-secrets.txt"
pomme@shark medusa % file encrypted-vault-secrets.txt 
encrypted-vault-secrets.txt: ASCII text, with very long lines (65536), with no line terminators
pomme@shark medusa % ./bin/medusa import kv-temporaire encrypted-vault-secrets.txt --token="$(vault print token)" --decrypt="true" --private-key="private-key.pem"
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
medusa/pkg/encrypt.Decrypt(0x7ffeeea13e06, 0xf, 0x7ffeeea13daa, 0x1b, 0xc000126c3b, 0x1, 0x0, 0xc000187bd0)
        /home/runner/work/medusa/medusa/pkg/encrypt/encrypt.go:46 +0x41d
medusa/cmd.glob..func3(0xcdce00, 0xc000100a50, 0x2, 0x5, 0x0, 0x0)
        /home/runner/work/medusa/medusa/cmd/import.go:45 +0x38f
github.com/spf13/cobra.(*Command).execute(0xcdce00, 0xc000100a00, 0x5, 0x5, 0xcdce00, 0xc000100a00)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:850 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xcdc8c0, 0xc000052778, 0xc000187f78, 0x4062c5)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:895
medusa/cmd.Execute(...)
        /home/runner/work/medusa/medusa/cmd/cmd.go:50
main.main()
        /home/runner/work/medusa/medusa/main.go:10 +0x32

I use medusa v0.2.2 .

I can figure what I m doing wrong, do you have an idea ?

Seb 🍎

[Feature Request] Add search option

Hi, thanks for that great tool it's super handy 👏

Since the tool actually is able to export, I believe it would be easy to have a search option so I can find where exactly a key or a value is used given a certain path.

What I do at the moment is export everything in the path and search that using the text editor which is suboptimal.

Example:

medusa search secret/folder foo

It could also have the option to search in keys only or values only, but that's nice to have.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.