getsops / sops Goto Github PK

I'm trying to use sops in some automation where the PGP private key I'm using for decryption has a passphrase. The passphrase can't be entered interactively. Is there a way to supply a passphrase through sops? Glancing at the source code it looks like it just shells out to gpg -d, so this may not be possible right now. It looks like it'd need to explicitly accept a passphrase argument to pass on to GPG. If this isn't possible through sops right now, any ideas for workarounds? Thanks much.

When a file claims to be yaml or json but does not load into a tree as this type, sops fails to detect it and continues anyway, resulting into a tree failure later on, because the tree is None.

sops must detect failure to load a yaml or json file and panic accordingly.

When encrypting, we should check each branch to see if it ends with the unencrypted suffix, and if so, leave it unencrypted.

Hi. Great stuff here. Putting a PoC together for the team using git as a storage and distribution mechanism. We are currently using https://www.passwordstore.org/ and I'm attracted to sops for its support for AWS KMS.

What are your recommendations for settling merges since they (almost?) always result in a conflict on the mac signature with neither signature going to be the correct one.

I'm comfortable with sops --ignore-mac and :wq to re-generate a mac, but this could be a roadblock for adoption since it's additional training to micro-manage the MAC every time there is a merge.

Any recommendations for reducing the scope of this problem? How do you folks handle this issue?

The timestamp generated when encrypting a data key currently uses unixtime with nanosecond precision. This is not optimal, we should use RFC3339 timestamps instead.

When I assume another users privileges I'm unable to run sops -d cat.json and inspect encrypted data, it looks like sops is unable to get write permissions on /dev/stdout.

I can, however, run sops -d cat.json | cat to inspect the contents.

Steps to reproduce:

bhourigan@system$ sudo su
root@system# su machineaccount
machineaccount@system$ sops -d cat.json
Traceback (most recent call last):
  File "/usr/local/bin/sops", line 9, in <module>
    load_entry_point('sops==1.3', 'console_scripts', 'sops')()
  File "/usr/local/lib/python2.7/dist-packages/sops/__init__.py", line 226, in main
    write_file(tree, path=dest, filetype=otype)
  File "/usr/local/lib/python2.7/dist-packages/sops/__init__.py", line 1034, in write_file
    fd = open(path, "wb")
IOError: [Errno 13] Permission denied: u'/dev/stdout'

Path permission/ownership chain:

$ ls -ald /dev/stdout
lrwxrwxrwx 1 root root 15 Jan 19 20:10 /dev/stdout -> /proc/self/fd/1
$ ls -ald /proc/self/fd/1
lrwx------ 1 machineaccount machineaccount 64 Jan 22 14:29 /proc/self/fd/1 -> /dev/pts/0
$ ls -ald /dev/pts/0
crw--w---- 1 bhourigan tty 136, 0 Jan 22 14:29 /dev/pts/0

Is this a feature, or a bug?

I recently upgraded sops and wanted to verify that pip worked properly, as my local python installation has become convoluted.

I found the version is available at the end of the output with --help, but I would like to request the addition of -v or --version to print the version explicitly.

We have some tar.gz files that we encrypt and then decrypt with sops. The file was decrypting properly til recent upgrade. Now, however, (after decryption) when de-tarring the file, the contents are correctly unzipped, however tar throws an error complaining the file is terminated incorrectly. We have since switched to using tar.bz2 instead and don't see this issue with that file format.

would be great to have ini support - much simpler format

So in our initial toe-dipping investigations we had discovered when running sops on hardware inside our corporate network, it would take > 30 seconds to execute, but from elsewhere it would take < 1 second. It turns out the root cause is due to us not declaring the --no-latest-check argument when running it, as the method it calls is not aware of common proxy environment variables (https_proxy and the likes) and the wait is due to proxy timeout, in which case it gives up and just completes the task anyhow.

We could just send a pull request adding proxy support to the check, however I think now is a good time to have a discussion about wether or not the idea of a security product like sops 'phoning home' on every execution by default is a good idea and that it outweighs the risks of out-of-date and potentially vulnerable software being exploited. A very large number of our machines will be automatically executing this code frequently as well as engineers and not necessarily be in an immediate position to upgrade the moment a new release is made available. At the same time we could look at other options - disabling the check everywhere or forking code being the first to mind.

Is this the right default to have?

for yaml file foo.yaml

foo:
    baz: 'blah'

sops -d foo.yaml --extract '["foo"]["baz"]'

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/bin/sops", line 9, in <module>
    load_entry_point('sops==1.7', 'console_scripts', 'sops')()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sops/__init__.py", line 241, in main
    write_file(tree, path=dest, filetype=otype)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sops/__init__.py", line 1106, in write_file
    fd.write(tree.encode('utf-8'))
UnboundLocalError: local variable 'fd' referenced before assignment

When neither -e, -d or -r are passed, that means we should open an editor and let the user edit the decrypted file

Do not crash when encrypted yaml files where the content is entirely commented out.

• Encryption
• Decryption

My $EDITOR variable is currently set to bbedit --wait --resume. When I try to edit a file I get the following error:

Traceback (most recent call last):
  File "/Users/jbuckley/Library/Python/2.7/bin/sops", line 11, in <module>
    sys.exit(main())
  File "/Users/jbuckley/Library/Python/2.7/lib/python/site-packages/sops/__init__.py", line 298, in main
    run_editor(tmppath)
  File "/Users/jbuckley/Library/Python/2.7/lib/python/site-packages/sops/__init__.py", line 1282, in run_editor
    subprocess.call([editor, path])
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Looking at the documentation for subprocess.call this is occurring because shell=False by default. I can work around this by creating a shell script that runs the bbedit command with arguments but I wonder if this should be set to shell=True. It's recommended to not do this with untrusted input so maybe it should actually remain False. Thoughts?

When a syntax error is detected in a document in edit mode, sops alerts and invites the user to reenter edit mode to fix it. This is broken, and sops fails to reenter edit mode with error:

$ sops /tmp/testval.yaml
/tmp/testval.yaml doesn't exist, creating it.
please wait while a data encryption key is being generated and stored securely
temp file created at /tmp/tmpRpB1Fw.yaml
Syntax error: while scanning a simple key
  in "/tmp/tmpRpB1Fw.yaml", line 12, column 1
could not find expected ':'
  in "/tmp/tmpRpB1Fw.yaml", line 13, column 1
Press a key to return into the editor, or ctrl+c to exit without saving.

Traceback (most recent call last):
  File "/usr/local/bin/sops", line 9, in <module>
    load_entry_point('sops==0.9', 'console_scripts', 'sops')()
  File "/home/ulfr/git/svc/sops/sops/__init__.py", line 252, in main
    restore_sops=stash['sops'])
  File "/home/ulfr/git/svc/sops/sops/__init__.py", line 341, in load_file_into_tree
    tree = ruamel.yaml.load(fd, ruamel.yaml.RoundTripLoader)
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/main.py", line 74, in load
    return loader.get_single_data()
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/constructor.py", line 57, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/composer.py", line 46, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/composer.py", line 66, in compose_document
    node = self.compose_node(None, None)
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/composer.py", line 97, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/composer.py", line 149, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/parser_.py", line 119, in check_event
    self.current_event = self.state()
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/parser_.py", line 491, in parse_block_mapping_key
    if self.check_token(KeyToken):
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/scanner.py", line 1529, in check_token
    self.fetch_more_tokens()
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/scanner.py", line 177, in fetch_more_tokens
    self.stale_possible_simple_keys()
  File "/usr/local/lib/python2.7/dist-packages/ruamel/yaml/scanner.py", line 308, in stale_possible_simple_keys
    "could not find expected ':'", self.get_mark())
ruamel.yaml.scanner.ScannerError: while scanning a simple key
  in "/tmp/tmpRpB1Fw.yaml", line 12, column 1
could not find expected ':'
  in "/tmp/tmpRpB1Fw.yaml", line 13, column 1

pynacl provides libsodium implementation in python. We should add support for it and propose it as an alternate encryption method to AES_GCM.

• Encryption
• Decryption
• Example file

I have users of our SOPS implementation that would prefer to use subdirectories to determine what encryption keys to use. Our project structure is as follows:

├── engineering
├── ops
├── resources
└── ssl

Right now, we have a .sops.yaml file in each directory and require users to cd to the directory before creating a new file. I get a lot of forgetful users who will create secrets from the top level.

So instead of

cd engineering
sops foo

we get

sops engineering/foo

If I could extend the configuration featureset to include a directory_regex I can run a regex on the first filename that is checked and see if it matches. If no match, pass through to the filename_regex as it works today.

creation_rules:
  - directory_regex: ^engineering/
    kms: '...'
    pgp: '...'
  - directory_regex: ^ops/
    kms: '...'
    pgp: '...'

happy to submit a patch if you are interested in accepting

This should remove a number of tests for unicode(), str() and bytes().
https://pypi.python.org/pypi/six

sops could inform a user that a newer version is available and should be installed via pip. Not sure if there's a clean API to do that...

It would be useful to support extracting only a subsection of a sops encrypted file. For example, given the file:

branchA:
    keyA: valueA
branchB:
    keyB: value B

The following command should extract branchA.keyA only:

$ sops -d -q 'branchA'->'keyA' file.yaml 

or something similar.

Since we have to allow output formats different from the input ones (i.e., input yaml, output json), it would be easier to assume sops can only encrypt trees, as the Python version does. The Go version currently does not make such assumption and lets Stores do whatever they want, but by doing this, we'd need each store to create a function to port their data to and from every other store. It's easier to only support trees.

This way, the stores will only have to know how to load the tree, and dump the tree including the sops branch or not. They don't need to take care of encrypting themselves or decrypting.

I've tried decrypting a file using both sops version 1.10 (latest) and earlier version 1.4, but getting the following traceback. Perhaps recent dependency upgrade broke something?

$ sops -d /tmp/my_config.xml.ENC > config.xml
Traceback (most recent call last):
File "/usr/local/bin//sops", line 9, in
load_entry_point('sops==1.4', 'console_scripts', 'sops')()
File "/usr/local/lib/python2.7/dist-packages/sops/init.py", line 218, in main
tree = walk_and_decrypt(tree, key, ignoreMac=args.ignore_mac)
File "/usr/local/lib/python2.7/dist-packages/sops/init.py", line 641, in walk_and_decrypt
branch[k] = decrypt(v, key, aad=caad, stash=nstash, digest=digest)
File "/usr/local/lib/python2.7/dist-packages/sops/init.py", line 698, in decrypt
default_backend()
File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/init.py", line 35, in default_backend
_default_backend = MultiBackend(_available_backends())
File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/init.py", line 22, in _available_backends
"cryptography.backends"
AttributeError: 'EntryPoint' object has no attribute 'resolve'

When following the project README.rst step by step, I got the following output when running sops example.yaml. Maybe this is expected, but wasn't explicitly stated in the README. I can confirm it successfully opened the file in an editor with the secrets decrypted.

Thanks!

(sops)pmoore@Petes-iMac:~/git/sops master $ sops example.yaml
[warning] skipping kms arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext references a key that either does not exist or you do not have access to. 
[warning] skipping kms arn:aws:kms:ap-southeast-1:656532927350:key/9006a8aa-0fa6-4c14-930e-a2dfb916de1d: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext references a key that either does not exist or you do not have access to. 
gpg: encrypted with 1024-bit RSA key, ID 7CD79CC0, created 2015-10-08
      "SOPS Functional Tests (https://github.com/mozilla/sops/) <[email protected]>"
temp file created at /var/folders/v9/mll6p_rj5h94dt_m5m8j0f9c0000gn/T/tmpF6qNM1.yaml
file written to example.yaml

no details yet, needs investigation
should add testing in travis

@floatingatoll suggested that being able to decrypt a diff would be useful. Something like git diff HEAD~1 | sops -d.

This isn't trivial, as sops must be able to parse the diff format and obtain a data key to decrypt the entries. Moreover, if a data key rotation has happened between the diffs, sops must retrieve the old data key from a previous version of the file.
We could probably recompose each file based on the + and - characters, and use that to perform the decryption.

by moving the encrypted/decrypted tempfile to the original path

I browsed __init__.py but I was unable to find a way to programmatically set the KMS ARN or PGP FP without using an environment variable.

This was the only code snippet I could find that specified the decryption resources, and it appears to only be invoked in main(). I think making direct API calls bypasses this code section entirely.

    kms_arns = ""
    if 'SOPS_KMS_ARN' in os.environ:
        kms_arns = os.environ['SOPS_KMS_ARN']
    if args.kmsarn:
        kms_arns = args.kmsarn

    pgp_fps = ""
    if 'SOPS_PGP_FP' in os.environ:
        pgp_fps = os.environ['SOPS_PGP_FP']
    if args.pgpfp:
        pgp_fps = args.pgpfp

Via the command line flags --(add|remove)-(kms|pgp).

After a user edits a file, and prior to sops encrypting it, we should perform a syntax checking step and invite the user to fix any issue there might be with yaml or json syntax.

Currently, sops will just fail if there are syntax errors, and changes will be lost.

As a newcomer to secret management (used to either an ignored file or committing into memory but in a private repo), I had trouble figuring out how to apply SOPS to my current setup (e.g.would I load into memory at boot? do I store in plaintext on disk?). After enough reasoning, I resolved that:

• Loading into memory at boot isn't great for cross-language
  • Requires per-language implementation which is bad for security since higher chance of human error
  • If someone uses an autoloader like nodemon which restarts server upon changes during development, then that can get costly with AWS (could use caching but then we are storing in disk)
• Plaintext on disk is fine because if attacker gains server access then they can run sops --decrypt and look at secrets

However, to prevent all future newcomers from running into the same hurdles, I suggest we add an examples section.

I have thought of 2 configurations that might be nice to showcase:

Catch-all secrets file:

• File inside of configuration folder (e.g. config/secret.enc.yml) which contains all secrets
• Upon plaintext decryption, store without .enc suffix (e.g. config/secret.yml)
• Ignore config/secret.yml via .gitignore or similar
• In other configuration files, load in config/secret.yml and use in respective configurations

Here's an example setup:

https://github.com/twolfson/twolfson.com/blob/3.79.0/config/secret.enc.json

https://github.com/twolfson/twolfson.com/blob/3.79.0/config/static.js#L14

https://github.com/twolfson/twolfson.com/blob/3.79.0/bin/decrypt-config.sh

Secrets in each file:

This setup will likely be more sane once #24 is resolved but each secret would be managed inline in each file. This would be practical if developers want to visually cross-check test vs production values easily.

• Store all secrets inlined in encrypted configuration files (e.g. inside config.enc)
• Decode all files from config.enc to config
  • Bonus points for scripting editing + syncing automatically
  • https://github.com/twolfson/twolfson.com/blob/12a84f0c66bf10f80ae18e26c8d306779ce88b47/bin/edit-config-file.sh
• Ignore decrypted files via .gitignore for config

I can reliably reproduce this by creating a JSON kv with an empty value, such as in the example below:

{
    "bar": "",
    "example_key": "example_value",
    "example_array": [
        "example_value1",
        "example_value2"
    ],
    "example_number": 1234.5678
}

sops -d test.json prints the following:

{
    "bar": "ENC[AES256_GCM,data:,iv:3CNjsdQS/SaFNDup0pbcsfgg41712CPj/Q4b58YT5g0=,tag:Aqg5vJjahkFOjVfjOgWadg==,type:str]",
    "example_key": "example_value",
    "example_array": [
        "example_value1",
        "example_value2"
    ],
    "example_number": 1234.5678
}

Add command line flags to add/remove master keys, such as:

• --add-kms <arn>
• --remove-kms <arn>
• --add-pgp <fingerprint>
• --remove-pgp <fingerprint>

This could be a known limitation and improper usage on my part, but I ran into an issue while encrypting a json file and redirecting the encrypted version to test.json.enc.

Creating a new file via sops test.json works as expected
Encrypting a file in place with sops -i -e test.json works as expected
Encrypting a file with sops -e test.json > test.json.enc renders the file unreadable, until test.json.enc is renamed to <something>.json

Steps to reproduce:

bash-3.2$ cat test.json
{
   "bar": "baz",
   "foo": ""
}
bash-3.2$ sops -e test.json > test.json.enc
please wait while a data encryption key is being generated and stored securely
bash-3.2$ sops -d test.json.enc
bash-3.2$ cat test.json.enc
{
    "bar": "ENC[AES256_GCM,data:XBgd,iv:RuJg8xWgkyvsDQb+u8zMzuVsB0OpWyAbsUi9xz8T0Ms=,tag:XTmtCVJ/nzsUr7c7zzbKtA==,type:str]",
    "foo": "ENC[AES256_GCM,data:,iv:5HPcYQ9i+ntnEklmLyY7fYAiPDWlGPOyagu238gE7hY=,tag:ZAlB6zZrgz9iXASjrcP5bg==,type:str]",
    "sops": {
        "lastmodified": "2016-02-03T16:40:59Z",
        "attention": "This section contains key material that should only be modified with extra care. See `sops -h`.",
        "mac": "ENC[AES256_GCM,data:sPBMhe9F3Bm1kGwO/kQvF295nVHKSaUzq/usH1V3yF3e1G1FTyt2FUx5HuX/nYuu5OqWpWzWoWNDqMYB1Rfyz3ZdWxhph0NT5X7HH7TSSadMtSib5HevgI6BJnUCmlcBuNPX6GWMQHPA3kN/yMGcjJAYD2r6Qd+s/vip9xT0Hxs=,iv:W6dUFubWGsaWLKdfgIWsQKe7heHmT7+Iuzyduu/AUy8=,tag:T9ugdVlJR/22kvAibwR2Qw==,type:str]",
        "version": 1.4,
        "kms": [
REDACTED
        ],
        "pgp": [
REDACTED
        ]
    }
}
bash-3.2$ sops --help | grep Version
Version 1.4 - See the Readme at github.com/mozilla/sops

Deploying sops to instances can be difficult because of python's complex dependencies management. Instances only need to decrypt files and don't need all the fancy editing features sops provides, so we should write a small Go client that takes an encrypted file and decrypts it.

Can be limited to yaml and kms for a first shot.

When the --in-place or -i command line flag is passed, sops should encrypt and decrypt in-place, not output the file to stdout.

We should be able to get KMS ARNs and PGP fingerprints based on the matches on the GPG config file. See the python version for more details

• Key source selection based on config file (c06f5bd)
• Config file path override with --config CLI flag (0550fcc)

I am very intrigued by this project, and I would like to work it into some of the processes that I oversee.

However, I have some use cases where it would be very convenient to only have some values encrypted, while others remain plaintext, in the same YAML file. For example, I may need to store an encrypted username and password or key for an FTP site, however there may be data in the YAML file that would benefit from remaining unencrypted, such as a custom report date range, or the path to a working directory, etc. If there were a top level "no_encryption" key or something of the sort, I assume this could be accomplished rather trivially.

Is this type of support compatible with the purpose of this project?

Looking forward to your reply.

Thanks,
Legomaniac

Following @ametaireau 's comment, we should create a Tree class, which handles its state internally, so we don't have to expose the tree variable to the user of the object.

tree = Tree() # This loads the tree. In the constructor, do everything you need.

# Then, directly use functions or arguments of the tree. e.g.

tree.key # or tree.get_key() if it takes some time to get the key.
tree.encrypt()
tree.decrypt()
tree.write()

When opening a binary file with flag -s and modifying the sops json data, saving the file fails with the following error:

Traceback (most recent call last):
  File "/usr/local/bin/sops", line 9, in <module>
    load_entry_point('sops==0.9', 'console_scripts', 'sops')()
  File "/home/ulfr/git/svc/sops/sops/__init__.py", line 253, in main
    if check_master_keys(tree):
  File "/home/ulfr/git/svc/sops/sops/__init__.py", line 476, in check_master_keys
    if 'kms' in tree['sops']:
KeyError: u'sops'

The current implementation of SOPS is functionally based. This is good for programs that rely on transforming data. However, it's beginning to get unwieldy with all of the state passing it is doing (e.g. digest, iv, key, and in #47 unencrypted_suffix).

I suggest considering a refactor to swap out the core with something more class based so state is always located on self. I will create a gist to provide examples of what I mean.

While the benefit of encrypting booleans is clearly close to null, they should be encrypted regardless to follow the same logic as the rest of the file.

You should be able to override the input and output types via the CLI --input-type and --output-type flags.

Using two different types might be difficult, as you'd have to transfer the data from YAMLStore to JSONStore, for example.

Add an option to reencrypt a file with a new data key, which would allow for key rotation.

Is it possible to add feature where we can only encrypt part of file using _encrypted and rest of the files unencrypted.

same feature _unencrypted

Text files are currently encrypted as blobs, so there's no benefit to put a requirement on them being actual texts. Instead, we should treat any file that isn't JSON or YAML as binary, which would allow for encryption of any binary file with sops.

test.json - array of hashes (test payload)

[{
    "example_key": "exaaasasasasple_value",
    "example_array": [
        "example_value1",
        "example_value2"
    ],
    "example_number": 1234.56789
}]

When using sops for encrypting test.json get the following error.

$ sops test.json 
temp file created at /var/folders/k6/660mymls4wbfxxh3j933wp480000gn/T/tmp4bkTf5.json
Traceback (most recent call last):
  File "/usr/local/bin/sops", line 11, in <module>
    sys.exit(main())
  File "/Library/Python/2.7/site-packages/sops/__init__.py", line 292, in main
    restore_sops=stash['sops'])
  File "/Library/Python/2.7/site-packages/sops/__init__.py", line 411, in load_file_into_tree
    tree['sops'] = restore_sops.copy()
TypeError: list indices must be integers, not unicode

sops should prevent malicious modifications of the content of the file:

1. Prevent entry modification by adding the keys on the key/value pairs in the AAD of the AES_GCM of a given entry.
2. Prevent addition/removal of entries by signing a global hash on all entries.