gmr / consulate Goto Github PK

I confirmed that its not a consul server configuration issue - I am able to use HTTP API update (upsert) for ACL and it works ok.

However when i do the same with Consulate, it does not error out but does not create the ACLs in Consul.

Thanks for looking into this.

Currently, the dependency to simplejson is optional in consulate. When running under Python3 without simplejson installed, consulate fails to decode the response properly and returns an indigested bytestring.

Example without simplejson installed:

>>> c = consulate.Consul()
>>> c.catalog.datacenters()
b'["services","test"]'

Example with simplejson installed:

>>> c = consulate.Consul()
>>> c.catalog.datacenters()
['services', 'test']

Return values from other operations differ accordingly.

I see basically two options to fix it: either decode the HTTP response body explicitely or make simplejson an explicit dependency.

install consul and vault at same instance and clustered (totally 3 instances)

When run consulate kv backup -b -f consult.json I got an empty file:

$ cat consult.json
[]

Then I did below test:

run consul members I can see them all.

vault works fine.

login http://<consule_IP>:8500, I can see the services of both consul and vault and all nodes.

But when click key/value, there is no any key/value displayed.

What should I check?

Hello,

It seems we can't use catalog.register in 0.5.x. It's working in 0.4.0 but with

consul.catalog.register(hostname,address)

I get the following stack trace :

Traceback (most recent call last):
  File "./consul-register", line 84, in <module>
    register(consul, args)
  File "./consul-register", line 66, in register
    consul.catalog.register(hostname,address)
  File "/home/rvrignaud/.virtualenvs/consul-register/lib/python2.7/site-packages/consulate/api/catalog.py", line 95, in register
    return self._adapter.put(['register'], None, payload)
  File "/home/rvrignaud/.virtualenvs/consul-register/lib/python2.7/site-packages/consulate/adapters.py", line 39, in inner
    return fun(*args, **kwargs)
TypeError: put() takes at most 3 arguments (4 given)

Am I doing something wrong?

from the readme:

usage: consulate register [-h] [-s SERVICE_ID] [-t TAGS]
                          {check,no-check,ttl} ... name port

from the shell:

usage: consulate register [-h] [-a ADDRESS] [-p PORT] [-s SERVICE_ID]
                          [-t TAGS]
                          name {check,no-check,ttl} ...

Hi there,

Without PR #1 the version of consulate on PyPi is unusable without tornado.
Can this fix be propagated so we can pip install this library cleanly?

Thanks!

Hello,

consulate kv backup saves an array of key-values, but consulate kv restore complains if you give it a file that does not contain an object.

Seems like the high level functions should also take an optional query_params dict argument so you can capture most of the variations of requests to the API, including blocking requests. Want a PR?

I would like to use the consulate CLI against a server that requires TLS authentication. Ideally the user could provide a TLS certificate and key with command line options, and provide defaults by setting environment variables (e.g. CONSULATE_TLS_CERT).

Do you have any plans to implement anything like this? If not, would you be willing to accept a pull request?

Thanks.

I'm getting the following when attempting to run a consulate backup on my consul cluster:

(con)[Tue Nov 03 17:25:50] ~/r/temp/consulate$ consulate --api-host 10.1.15.125 --api-port 8500 kv backup --base64 -f consul_backup.json
Traceback (most recent call last):
  File "/Users/neil1/virtualenvs/con/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.6.0', 'console_scripts', 'consulate')()
  File "build/bdist.macosx-10.11-intel/egg/consulate/cli.py", line 469, in main
  File "build/bdist.macosx-10.11-intel/egg/consulate/cli.py", line 214, in kv_backup
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/base64.py", line 53, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2014' in position 562: ordinal not in range(128)    

Looks like #41

Python 2.7.10
Consulate 0.6.0 (Both pip and master)

We've tracked the issue to Hashicorp Vault writing binary data.

Using python3.5 we instead get the error:

(test)[Tue Nov 03 17:41:37] ~/r/temp$ consulate --api-host 10.1.15.125 --api-port 8500 kv backup --base64 -f consul_backup.json
Traceback (most recent call last):
  File "/usr/local/bin/consulate", line 11, in <module>
    sys.exit(main())
  File "/Library/Python/2.7/site-packages/consulate/cli.py", line 469, in main
    KV_ACTIONS[args.action](consul, args)
  File "/Library/Python/2.7/site-packages/consulate/cli.py", line 214, in kv_backup
    records = [(k, f, base64.b64encode(v)) for k,f,v in records]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/base64.py", line 53, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
TypeError: must be string or buffer, not None

If there's any other information we can provide please let us know!

Error from cli:

consulate register -p 3306 mysql-master no-check
Traceback (most recent call last):
  File "/usr/local/bin/consulate", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 283, in main
    register(consul, args)
  File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 255, in register
    args.port, tags, check, interval, ttl)
  File "/usr/local/lib/python2.7/dist-packages/consulate/api/agent.py", line 181, in register
    raise ValueError('port must be an integer')
ValueError: port must be an integer

Can't see any mention of it.

I'm able to connect, set and read kv pairs from Consul with consulate without too much trouble. set_record() throws a fit whenever I try to use replace=False. As you can see by the func_code line, set_record only shows the self, item, flags and value params. I'm running 0.4.0 on Python 2.7.6 and have never had an older version of consulate on this box.

>>> key='moo/too'
>>> value='foobar'
>>> s.kv.set_record(item=key, value=value, replace=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: set_record() got an unexpected keyword argument 'replace'
>>> print s.kv.set_record
<bound method KV.set_record of <consulate.api.kv.KV object at 0x10527e7d0>>
>>> print s.kv.set_record.func_code.co_varnames
('self', 'item', 'flags', 'value')
>>> print s.kv.set(key,value)
None
>>> print s.kv.get(key)
foobar
>>> print s.kv.delete(key)
<consulate.api.base.Response object at 0x105288d50>
>>> print s.kv.get(key)
None
>>> print s.kv.set_record(item=key,value=value)
None
>>> print s.kv.get(key)
foobar

$ pip show consulate

---
Metadata-Version: 2.0
Name: consulate
Version: 0.4.0
Summary: A Client library for the Consul
Home-page: https://consulate.readthedocs.org
Author: Gavin M. Roy
Author-email: [email protected]
License: BSD
Location: /Library/Python/2.7/site-packages
Requires: requests

$ python --version
Python 2.7.6

$ uname -a
Darwin martins.local 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64

$ sw_vers -productVersion
10.10.3

Wonderful tool. Thanks for supporting this.

Similar to the kv backup and restore option, I would like to place a feature request to backup and restore ACLs.

I get $title error with both the 0.2.0 from pip install consulate and also with current Git of consulate + PR #18 when I try to backup the KV store

I do have international non-ASCII chars in the KV store and consulate should definitely be able to backup + restore these.

Cheers,
Gavin.

According to the Docs of the HTTP API it's possible to register a custom address for a service - the default only uses the Agent's address. This is insufficent for many use cases.

I'm trying to prepare a pull request in a minute.

When using the -f,--file option with kv restore, the kv_restore function expects that a string, but a file object is being provided:

root@vagrant-ubuntu-trusty-64:~# consulate kv restore --file dump.json
Traceback (most recent call last):
  File "/usr/local/bin/consulate", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 274, in main
    kv_restore(consul, args)
  File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 196, in kv_restore
    handle = open(args.file, 'r') if args.file else sys.stdin
TypeError: coercing to Unicode: need string or buffer, file found

Looks like when --file is specified, the file is opened automatically:

    restore = kvsparsers.add_parser('restore',
                                    help='Restore from stdin or a JSON file')
    restore.add_argument('-f', '--file',
                         help='JSON file to read instead of stdin',
                         nargs="?", type=open)

So, simply swapping

    handle = open(args.file, 'r') if args.file else sys.stdin

with:

    handle = args.file

in cli.py on line 196 should fix the issue. Piping data in kv restore does work correctly however:

cat SOME_FILE.json | consulate kv restore

Hi,

What's the api equivalent of "consul members list"? I'm trying to write a utility using consulate which starts with a snapshot of all known nodes.

TIA,
-T

Hello,

It's possible in Consul API to put a lock with a value associated (https://www.consul.io/docs/agent/http/kv.html).
It seems that it's not currently possible with consulate.
Would you mind adding that support ?

Are there any plans for a release soon? Specifically, I'm hoping to get one that includes #70 (fix for #68) so I can stop running off of my own fork...

Hi,

With value = u'some string'
If I do session.kv[full_key] = value, in consul I get "some string".
But if I do session.kv[full_key] = str(value), I do get some string without the quotes.

CentOS 65 with Python 3.3.5 (sandbox) or 2.7.6 (virtualenv)

(consulate) [root@cns-ui-prg001 consul-api]# python --version
Python 3.3.5
(consulate) [root@cns-ui-prg001 consul-api]# pip install consulate
Downloading/unpacking consulate
Downloading consulate-0.2.0.tar.gz
Running setup.py (path:/tmp/pip_build_root/consulate/setup.py) egg_info for package consulate

Requirement already satisfied (use --upgrade to upgrade): requests in /usr/local/lib/python3.3/site-packages (from consulate)
Installing collected packages: consulate
Running setup.py install for consulate

Installing consulate script to /usr/local/bin

Successfully installed consulate
Cleaning up...
(consulate) [root@cns-ui-prg001 consul-api]# consulate -v

Traceback (most recent call last):
File "/usr/local/bin/consulate", line 9, in
load_entry_point('consulate==0.2.0', 'console_scripts', 'consulate')()
File "/usr/local/lib/python3.3/site-packages/setuptools-7.0-py3.3.egg/pkg_resources.py", line 339, in load_entry_point
File "/usr/local/lib/python3.3/site-packages/setuptools-7.0-py3.3.egg/pkg_resources.py", line 2470, in load_entry_point
File "/usr/local/lib/python3.3/site-packages/setuptools-7.0-py3.3.egg/pkg_resources.py", line 2184, in load
File "/usr/local/lib/python3.3/site-packages/consulate/init.py", line 7, in
from consulate.api import Consulate
File "/usr/local/lib/python3.3/site-packages/consulate/api.py", line 10, in
from consulate import adapters
File "/usr/local/lib/python3.3/site-packages/consulate/adapters.py", line 59, in
class TornadoRequest(Request):
File "/usr/local/lib/python3.3/site-packages/consulate/adapters.py", line 66, in TornadoRequest
@gen.coroutine
AttributeError: 'NoneType' object has no attribute 'coroutine'

Then using consulate kv restore from previous consulate kv backup, the values which had double quotes are inserted with quotes removed, so that if i had a "somevalue" or would become somevalue after restore.

On release 0.5.0 the automatic decoding of JSON when reading KV values was removed, please add a flag to will enable it when needed.

What do you think about adding a -r option to consulate kv get to request all keys in a given path? Like this:

$ consulate kv get test/recursive/one
1

$ consulate kv get test/recursive/two
2

$ consulate kv get test/recursive/three
3

$ consulate kv get test/recursive
None

$ consulate kv get -r test/recursive
test/recursive/one   1
test/recursive/two   2
test/recursive/three 3

I'd be happy to add it to the CLI and submit a pull request, just wanted to make sure that it's a welcome contribution.

Consulate currenlty supports only two types of checks: ttl and Script + Interval. But there is third one: HTTP + Interval that lets you configure an URL endpoint to check for a 200 response.

In the documentation for the methods it states Set a value in the Consul Key/Value store, using the CAS mechanism to ensure that the set is atomic. However it appears that only ?index= is used for a blocking call to the consul server. This doesn't prevent concurrent writes I believe as another consul server could update the value before the original call is set. I could be mistaken, however, as since it using a blocking query the call to consul should return until the raft quorum has accepted the value.

What I would suggest though is that consul does include a cas method with ?cas=<index> instead of just ?index=<index> as specified in the consul http api doc. I could be wrong though.

When issuing an API call that returns nothing (as expected), consulate prints a spurious error message to stdout:

>>> c.agent.service.register('myservice', address='myaddr')
('Error:', ValueError('No JSON object could be decoded',))
True

The HTTP call just exits with status 200 and returns no body. This is ok by Consul's specs. So I'd suggest just to remove the print() statement in api/base.py.

I'm using vaultproject.io from Hashicorp that stores encrypted data in the Key/Value store of consul.

On backup I'm getting

Traceback (most recent call last):
  File "/usr/local/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.5.1', 'console_scripts', 'consulate')()
  File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 285, in main
  File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 134, in kv_backup
  File "build/bdist.linux-x86_64/egg/consulate/api/kv.py", line 236, in records
  File "build/bdist.linux-x86_64/egg/consulate/api/kv.py", line 303, in _get_all_items
  File "build/bdist.linux-x86_64/egg/consulate/api/base.py", line 77, in _get_list
  File "build/bdist.linux-x86_64/egg/consulate/api/base.py", line 61, in _get
  File "build/bdist.linux-x86_64/egg/consulate/adapters.py", line 77, in get
  File "build/bdist.linux-x86_64/egg/consulate/adapters.py", line 108, in _process_response
  File "build/bdist.linux-x86_64/egg/consulate/api/base.py", line 119, in __init__
  File "build/bdist.linux-x86_64/egg/consulate/api/base.py", line 147, in _demarshal
  File "/usr/local/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 6: invalid start byte

I'm running consulate from master (as of 3c6ce3c) with current Consul (0.5.2). I originally tried using the version of consulate on PyPi and ran into the same issue as #61. Installing from master, that issue (when backing up) is fixed, but I get the corresponding issue on the "other side" when trying to restore the backup I just made:

(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ consulate --api-host myhost.mydomain.com --api-port 8500 kv backup --base64 -f consul_backup.json
(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ echo $?
0
(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ ls -l consul_backup.json 
-rw-r--r-- 1 jantman jantman 851677 Oct 14 14:35 consul_backup.json
(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ consulate --api-host 127.0.0.1 --api-port 8500 kv restore -b -n -f consul_backup.json
Traceback (most recent call last):
  File "/tmp/jantman/consulate/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.6.0', 'console_scripts', 'consulate')()
  File "/tmp/jantman/consulate/src/consulate/consulate/cli.py", line 469, in main
    KV_ACTIONS[args.action](consul, args)
  File "/tmp/jantman/consulate/src/consulate/consulate/cli.py", line 310, in kv_restore
    row[2] = base64.b64decode(row[2])
  File "/usr/lib64/python2.7/base64.py", line 73, in b64decode
    return binascii.a2b_base64(s)
TypeError: must be string or buffer, not None

If I apply the following patch:

diff --git a/consulate/cli.py b/consulate/cli.py
index 09a838f..d08153b 100644
--- a/consulate/cli.py
+++ b/consulate/cli.py
@@ -307,7 +307,11 @@ def kv_restore(consul, args):
             row = [row['Key'], row['Flags'], row['Value']]

         if args.base64:
-            row[2] = base64.b64decode(row[2])
+            try:
+                row[2] = base64.b64decode(row[2])
+            except Exception as ex:
+                print(row)
+                raise ex

         # Here's an awesome thing to make things work
         if not utils.PYTHON3 and isinstance(row[2], unicode):

Then it becomes a bit more clear:

(consulate)jantman@phoenix:pts/14:~/tmp/consulate$ consulate --api-host 127.0.0.1 --api-port 8500 kv restore -b -n -f consul_backup.json
[u'configs/test/', 0, None]
Traceback (most recent call last):
  File "/tmp/jantman/consulate/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.6.0', 'console_scripts', 'consulate')()
  File "/tmp/jantman/consulate/src/consulate/consulate/cli.py", line 473, in main
    KV_ACTIONS[args.action](consul, args)
  File "/tmp/jantman/consulate/src/consulate/consulate/cli.py", line 314, in kv_restore
    raise ex
TypeError: must be string or buffer, not None

Hi @gmr

Using consulate 0.6.0, it appears that the run_once command is not working.

With a local Consul agent:

consulate --datacenter chef run_once -i 60 echo_hello 'echo hello'

Nothing is happening. I also checked Consul KV, but not echo_hello locks are created.

I tried to edit the run_once method to print anything (before the try: https://github.com/gmr/consulate/blob/master/consulate/cli.py#L392-L399), but it doesn't look like it gets executed.

Any help appreciated 😃

The pypi page links to https://consulate.readthedocs.org/ which does not exist. :(

Hi,

I followed the below steps for backup / restore of Consul KV. After the restore Vault was throwing error as "* failed to decode seal configuration: invalid character 'e' looking for beginning of value". Can you please suggest?

$ consulate --api-host <hostname> --api-port 8500 kv backup -b --file dump.json

$consulate --api-host  <hostname> --api-port 8500 kv restore --file dump.json

$ vault status
Error checking seal status: Error making API request.

URL: GET http://<hostname>:8200/v1/sys/seal-status
Code: 500. Errors:

* failed to decode seal configuration: invalid character 'e' looking for beginning of value

Thanks,
Govind

On Python 3.5.2 / consulate 0.6 I was seeing an intermittent error:

    ...
      File "/app/__init__.py", line 65, in consul_lock
        sid = consul.session.create(session_name, ttl=ttl)
      File "/opt/conda/lib/python3.5/site-packages/consulate/api/session.py", line 67, in create
        return self._put_response_body(['create'], None, payload).get('ID')
    AttributeError: 'bytes' object has no attribute 'get'

After splicing in a debug print(self._put_response_body(['create'], None, payload)), it would tell me this:
*** DEBUG <class 'bytes'> : b'No cluster leader'.

This indicated problems with my consul setup, and I was able to simply restart all my instances as this is the only application they are currently used for. I'm not sure what the proper handling should be in this case, but something to handle the case where the response body isn't a dict seems needed.

The ability to add a timeout option to the adapter.Request() object, in PR #31, was broken by the change made in 7dec136 to add Unix Socket support the passed in adapter.

It used to be possible to pass in an instantiated adapter (allowing one to set the timeout in their own code), but now the adapter is always instantiated with no options inside Consul.__init__() - regardless of whether it's already an instance of the adapter class itself or not: relevant code.

Happy to supply a PR for fixing this but not sure what the best fix is - either reverting the change and making it necessary to instantiate any custom adapter passed in, or checking the passed in adapter to work out if it's instantiated or not and then instantiating if necessary.

Thoughts?

Hi, I installed 0.3.0 with pip install https://github.com/gmr/consulate/archive/master.zip on CentOS 7 / Python 2.7.5 - the CLI tool launches OK after install simplejson / requests, but I am getting the error in $title

[root@ip-172-18-22-87 stub]# consul members
Node               Address             Status  Type    Build  Protocol
ip-172-18-22-87    172.18.22.87:8301   alive   client  0.4.1  2
bastion-i535bffb5  172.18.20.199:8301  alive   server  0.4.1  2
jenkins-ifa37e51d  172.18.22.165:8301  alive   client  0.4.1  2
bastion-i5d9840b9  172.18.21.36:8301   alive   server  0.4.1  2
bastion-idc50853b  172.18.19.30:8301   alive   server  0.4.1  2

[root@ip-172-18-22-87 stub]# consulate -h
usage: consulate [-h] [--api-host API_HOST] [--api-port API_PORT]
                 [--datacenter DATACENTER] [--token TOKEN]
                 {register,kv} ...

CLI utilities for Consul

optional arguments:
  -h, --help            show this help message and exit
  --api-host API_HOST   The consul host to connect on
  --api-port API_PORT   The consul API port to connect to
  --datacenter DATACENTER
                        The datacenter to specify for the connection
  --token TOKEN         ACL token

Commands:
  {register,kv}
    register            Register a service for this node
    kv                  Key/Value Database Utilities

[root@ip-172-18-22-87 stub]# consulate kv set mykey 123
Traceback (most recent call last):
  File "/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.3.0', 'console_scripts', 'consulate')()
  File "/usr/lib/python2.7/site-packages/consulate/cli.py", line 75, in main
    session = api.Consulate(args.api_host, args.api_port, args.datacenter,
AttributeError: 'module' object has no attribute 'Consulate'

Any help would be appreciated :)

Cheers,
Gavin.

Thanks for the acquire_lock(key,session) feature. Is it also possible to add a function that returns the session id, if the key is already locked.

Sample code:

def check_lock_on_key(myKey):
    url="http://localhost:8500/v1/kv/"
    response = requests.get(url+myKey)
    jsondata=response.json() if (response.status_code == requests.codes.ok) else [{}]
    return jsondata[0].get('Session')

It could be great to have a 'ls' subcommand to the KV store.

Is there any plan to implement a behavior like https://github.com/coreos/etcdctl ?

Hello,

With consulate 0.6.0 I still can't use catalog.register. See following stack trace:

>>> import consulate
>>> c = consulate.Consul('192.168.59.103')
>>> c.catalog.datacenters()
u'dc1'
>>> c.catalog.register('test','192.168.0.1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/rvrignaud/.virtualenvs/cr/lib/python2.7/site-packages/consulate/api/catalog.py", line 95, in register
    return self._adapter.put(['register'], payload)
  File "/Users/rvrignaud/.virtualenvs/cr/lib/python2.7/site-packages/consulate/adapters.py", line 42, in inner
    return fun(*args, **kwargs)
  File "/Users/rvrignaud/.virtualenvs/cr/lib/python2.7/site-packages/consulate/adapters.py", line 99, in put
    timeout=self.timeout))
  File "/Users/rvrignaud/.virtualenvs/cr/lib/python2.7/site-packages/requests/sessions.py", line 518, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/Users/rvrignaud/.virtualenvs/cr/lib/python2.7/site-packages/requests/sessions.py", line 451, in request
    prep = self.prepare_request(req)
  File "/Users/rvrignaud/.virtualenvs/cr/lib/python2.7/site-packages/requests/sessions.py", line 382, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/Users/rvrignaud/.virtualenvs/cr/lib/python2.7/site-packages/requests/models.py", line 304, in prepare
    self.prepare_url(url, params)
  File "/Users/rvrignaud/.virtualenvs/cr/lib/python2.7/site-packages/requests/models.py", line 362, in prepare_url
    to_native_string(url, 'utf8')))
requests.exceptions.MissingSchema: Invalid URL "['register']": No schema supplied. Perhaps you meant http://['register']?

Thoughts ?

On Ubuntu 14.04.2 LTS, consulate fails as its dependency on urllib3.connection no longer exists.

sbapp@ppop-repo:~$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import consulate
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/consulate/init.py", line 24, in
from consulate import adapters
File "/usr/local/lib/python2.7/dist-packages/consulate/adapters.py", line 9, in
import requests_unixsocket
File "/usr/local/lib/python2.7/dist-packages/requests_unixsocket/init.py", line 4, in
from .adapters import UnixAdapter
File "/usr/local/lib/python2.7/dist-packages/requests_unixsocket/adapters.py", line 9, in
from urllib3.connection import HTTPConnection
ImportError: No module named connection

sbapp@ppop-repo:~$ cat /etc/issue
Ubuntu 14.04.2 LTS \n \l

sbapp@ppop-repo:~$ pip freeze
argparse==1.2.1
chardet==2.0.1
colorama==0.2.5
consulate==0.5.1
html5lib==0.999
meld3==0.6.10
requests==2.2.1
requests-unixsocket==0.1.4
six==1.5.2
ssh-import-id==3.21
supervisor==3.0b2
urllib3==1.7.1
wheel==0.24.0
wsgiref==0.1.2

After reviewing the code, I'm a little concerned about the performance implications of all directory queries being recursive. I would like to propose an items() method that accepts 2 parameters:

1. the key to retrieve the set of items for (to restrict the search to a single directory)
2. Whether or not the search should be recursive

I need this feature anyways, so I'm happy to submit a patch for it.

Inconsistance restore, additional quoting for values:
to reproduce, create a variable:

$# consulate kv set foo bar
$# curl loclahost:8500/v1/kv/foo?pretty=1
[
    {
        "CreateIndex": 980,
        "ModifyIndex": 980,
        "LockIndex": 0,
        "Key": "foo",
        "Flags": 0,
        "Value": "YmFy"
    }
]

Now make a backup:

$# consulate kv backup /tmp/foo    
$# cat /tmp/foo
[
  [
    "foo", 
    0, 
    "bar"
  ]
]

Remove the value and restore:

$# consulate kv del foo
$# consulate kv restore /tmp/foo    

Check the value of key 'foo':

$# curl localhost:8500/v1/kv/foo?pretty=1
[
    {
        "CreateIndex": 992,
        "ModifyIndex": 992,
        "LockIndex": 0,
        "Key": "foo",
        "Flags": 0,
        "Value": "ImJhciI="
    }
]

Difference:

$# perl -MMIME::Base64 -e 'print decode_base64("YmFy")'
bar

$# perl -MMIME::Base64 -e 'print decode_base64("ImJhciI=")'
"bar"

Help text says:

$ consulate kv backup --h
usage: consulate kv backup [-h] [-b] [-f [FILE]]

[...]

  -f [FILE], --file [FILE]
                        JSON file to read instead of stdin

But -f works for both backup and restore. For backup the (expected, actual) result is that the argument passed to -f is the file output is written to.

It's missing, and in order to add timeout to all possible gets within adapters, One actually have to overwrite whole adapter.

This is very like issue #41. If we run without the -b, we get:

wwalker@hatter:~/git/consulate (master *$%=)$ consulate --api-host tst-log3 --datacenter qa kv backup
Traceback (most recent call last):
File "/home/wwalker/.virtualenvs/consulate/bin/consulate", line 9, in
load_entry_point('consulate==0.6.0', 'console_scripts', 'consulate')()
File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 469, in main
File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 216, in kv_backup
File "/usr/lib64/python2.7/json/init.py", line 243, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x94 in position 6: invalid start byte

If we run with -b we fail with:

wwalker@hatter:~/git/consulate (master $%=)$ consulate --api-host tst-log3 --datacenter qa kv backup -b
Traceback (most recent call last):
File "/home/wwalker/.virtualenvs/consulate/bin/consulate", line 9, in
load_entry_point('consulate==0.6.0', 'console_scripts', 'consulate')()
File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 469, in main
File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 214, in kv_backup
File "/usr/lib64/python2.7/base64.py", line 53, in b64encode
encoded = binascii.b2a_base64(s)[:-1]
TypeError: must be string or buffer, not None

We have a key with a None value.

When backing up my Consul kv store, I get lots (but not all) folders dumped as null values like this:

[
    "my-folder/",
    0,
    null
],
[
    "top-level-folder/nested-folder/",
    0,
    "null"
]

This creates non-folder entries on restore with the folder name as the key and null or "null" as the value.

If these were simply not present in the backup data, the restore would do what I expect. If I go through and delete them from the resulting JSON, everything restores just fine.

I haven't yet been able to discern a common pattern for the folders that this happens to vs. the ones it does not.

I have seen this with both 0.6.0 and master HEAD as of a few minutes ago (reporting itself as 0.7.0.rc0).

If I have a key in consul with a value of "" (empty string), a consulate kv backup will dump that with null instead.

Installing it & trying to import results in:

[10:56:04] vmhost1-cluster1:~# pip install consulate
Downloading/unpacking consulate
  Downloading consulate-0.2.0.tar.gz
  Running setup.py (path:/tmp/pip-build-UdHBo_/consulate/setup.py) egg_info for package consulate

Downloading/unpacking requests (from consulate)
  Downloading requests-2.5.3-py2.py3-none-any.whl (468kB): 468kB downloaded
Installing collected packages: consulate, requests
  Running setup.py install for consulate

    Installing consulate script to /usr/bin
Successfully installed consulate requests
Cleaning up...
[10:56:40] vmhost1-cluster1:~# python
Python 2.7.5 (default, Jun 17 2014, 18:11:42) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import consulate
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/consulate/__init__.py", line 7, in <module>
    from consulate.api import Consulate
  File "/usr/lib/python2.7/site-packages/consulate/api.py", line 10, in <module>
    from consulate import adapters
  File "/usr/lib/python2.7/site-packages/consulate/adapters.py", line 59, in <module>
    class TornadoRequest(Request):
  File "/usr/lib/python2.7/site-packages/consulate/adapters.py", line 66, in TornadoRequest
    @gen.coroutine
AttributeError: 'NoneType' object has no attribute 'coroutine'
>>> 
[10:57:21] vmhost1-cluster1:~# cat /etc/redhat-release 
CentOS Linux release 7.0.1406 (Core) 
[10:57:31] vmhost1-cluster1:~# python --version
Python 2.7.5
[10:57:43] vmhost1-cluster1:~# 

Hi,

I'm currently running some code against a test installation of consul in a single-node vagrant environment. I stumbled over the issue that peers() will return a single string instead of a list in this case:

(Pdb) p connection.status.peers()
2015-05-19 11:03:38,533 INFO [requests.packages.urllib3.connectionpool][Dummy-4] Starting new HTTP connection (1): consul-ext.service.services.vgr.consul.local
u'10.0.2.15:8300'