Code Monkey home page Code Monkey logo

pymemcache's Introduction

pymemcache

image

Master Documentation Status

A comprehensive, fast, pure-Python memcached client.

pymemcache supports the following features:

  • Complete implementation of the memcached text protocol.
  • Connections using UNIX sockets, or TCP over IPv4 or IPv6.
  • Configurable timeouts for socket connect and send/recv calls.
  • Access to the "noreply" flag, which can significantly increase the speed of writes.
  • Flexible, modular and simple approach to serialization and deserialization.
  • The (optional) ability to treat network and memcached errors as cache misses.

Installing pymemcache

Install from pip:

pip install pymemcache

For development, clone from github and run the tests:

git clone https://github.com/pinterest/pymemcache.git
cd pymemcache

Run the tests (make sure you have a local memcached server running):

tox

Usage

See the documentation here: https://pymemcache.readthedocs.io/en/latest/

Django

Since version 3.2, Django has included a pymemcache-based cache backend. See its documentation.

On older Django versions, you can use django-pymemcache.

Comparison with Other Libraries

pylibmc

The pylibmc library is a wrapper around libmemcached, implemented in C. It is fast, implements consistent hashing, the full memcached protocol and timeouts. It does not provide access to the "noreply" flag. It also isn't pure Python, so using it with libraries like gevent is out of the question, and its dependency on libmemcached poses challenges (e.g., it must be built against the same version of libmemcached that it will use at runtime).

python-memcached

The python-memcached library implements the entire memcached text protocol, has a single timeout for all socket calls and has a flexible approach to serialization and deserialization. It is also written entirely in Python, so it works well with libraries like gevent. However, it is tied to using thread locals, doesn't implement "noreply", can't treat errors as cache misses and is slower than both pylibmc and pymemcache. It is also tied to a specific method for handling clusters of memcached servers.

memcache_client

The team at mixpanel put together a pure Python memcached client as well. It has more fine grained support for socket timeouts, only connects to a single host. However, it doesn't support most of the memcached API (just get, set, delete and stats), doesn't support "noreply", has no serialization or deserialization support and can't treat errors as cache misses.

External Links

The memcached text protocol reference page:

https://github.com/memcached/memcached/blob/master/doc/protocol.txt

The python-memcached library (another pure-Python library):

https://github.com/linsomniac/python-memcached

Mixpanel's Blog post about their memcached client for Python:

https://engineering.mixpanel.com/we-went-down-so-we-wrote-a-better-pure-python-memcache-client-b409a9fe07a9

Mixpanel's pure Python memcached client:

https://github.com/mixpanel/memcache_client

Bye-bye python-memcached, hello pymemcache (migration guide)

https://jugmac00.github.io/blog/bye-bye-python-memcached-hello-pymemcache/

Credits

We're Hiring!

Are you really excited about open-source? Or great software engineering? Pinterest is hiring!

pymemcache's People

Contributors

4383 avatar adamchainz avatar akatrevorjay avatar bwalks avatar cclauss avatar cgordon avatar davedash avatar dcrosta avatar dependabot[bot] avatar iurisilvio avatar jianguopinterest avatar jogo avatar jparise avatar julian avatar krallin avatar mainanick avatar martinnj avatar methane avatar morrme avatar msabramo avatar ngnpope avatar nichochar avatar opapy avatar pankrat avatar shargan avatar sirosen avatar slingamn avatar sontek avatar suhailpatel avatar tomkins avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pymemcache's Issues

Security issue in input handling

Key is not sanitized and something like \r\nflush_all\r\n in it could destroy all values and maybe more, if it comes from any user input.

Pinterest sends email that are not compliant with CAN-SPAM

I am received spam emails from Pinterest that I did not sign up for.

screen shot 2014-04-28 at 5 18 28 pm
screen shot 2014-04-28 at 5 18 23 pm

In the United States, the CAN-SPAM law requires that commercial emails, which this is, include the full mailing address of the sender and a one-click unsubscribe, which this doesn't.

WHY AM I REPORTING THIS HERE?

Because a link on that email goes to a webpage, shown, which does not have a CONTACT LINK. It does have a GitHub link, which establishes that you are affiliated win Pinterest.

screen shot 2014-04-28 at 5 20 25 pm

ALSO

Because I have reported this illegal behavior to you and because you are related to Pinterest you have a responsibility to triage this issue as appropriate. Also, with my documentation of send this complaint, you will be liable for subsequent offenses which I may collect $500 per email from you.

MacOS X 10.11 Issue

When installed on MacOS X 10..11

macbookproloreto:pymemcache admin$ python setup.py nosetests
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'nosetests

Issues when using pymemcache from a CI

Hello there,

In OpenStack CI we are gating with pymemcache and we have seen a failure lately with it, the issue is that pip is trying to download the macosx binary of pymemcache, see from:

http://logs.openstack.org/10/131410/2/gate/gate-pecan-tox-ceilometer-tip/b119a86/console.html

 2014-11-08 22:35:26.982 | Downloading/unpacking zake>=0.1.6 (from tooz>=0.3->ceilometer)
 2014-11-08 22:35:26.982 |   http://pypi.DFW.openstack.org/simple/zake/ uses an insecure transport scheme (http). Consider using https if pypi.DFW.openstack.org has it available
 2014-11-08 22:35:26.983 |   Downloading zake-0.1.6.tar.gz
 2014-11-08 22:35:26.983 |   Running setup.py (path:/home/jenkins/workspace/gate-pecan-tox-ceilometer-tip/.tox/ceilometer-tip/build/zake/setup.py) egg_info for package zake
 2014-11-08 22:35:26.983 |
 2014-11-08 22:35:26.983 | Downloading/unpacking pymemcache>=1.2 (from tooz>=0.3->ceilometer)
 2014-11-08 22:35:26.983 |   http://pypi.DFW.openstack.org/simple/pymemcache/ uses an insecure transport scheme (http). Consider using https if pypi.DFW.openstack.org has it available
 2014-11-08 22:35:26.983 |   Downloading pymemcache-1.2.6.macosx-10.9-intel.tar.gz
 2014-11-08 22:35:26.983 |   Running setup.py (path:/home/jenkins/workspace/gate-pecan-tox-ceilometer-tip/.tox/ceilometer-tip/build/pymemcache/setup.py) egg_info for package pymemcache
 2014-11-08 22:35:26.983 |     Traceback (most recent call last):
 2014-11-08 22:35:26.983 |       File "<string>", line 17, in <module>
 2014-11-08 22:35:26.983 |     IOError: [Errno 2] No such file or directory: '/home/jenkins/workspace/gate-pecan-tox-ceilometer-tip/.tox/ceilometer-tip/build/pymemcache/setup.py'
 2014-11-08 22:35:26.983 |     Complete output from command python setup.py egg_info:
 2014-11-08 22:35:26.984 |     Traceback (most recent call last):
 2014-11-08 22:35:26.984 |
 2014-11-08 22:35:26.984 |   File "<string>", line 17, in <module>

Chatting with the pip maintainer (dstuffit) has this recommendation (sorryy for the irc screenshot) :

2014-11-12__13-54-04

which tl;dr to :

dstufft: pymemcache should delete the bdist_dumb

any ideas?

No tags, no changelog...

How can I find out the differences between versions? On pypi are different versions, but in git are no tags. Not even a Changelog file...

Unpack error with Ubuntu's (14.04) release of memcached

on line 702 of pymemcache/client/base.py

701                 elif name == b'stats' and line.startswith(b'STAT'):
702                     _, key, value = line.split()
703                     result[key] = value

line 702 throws an exception with the default memcached from Ubuntu.

The stats command expects all output to be exactly three values separated by spaces, however, this is the result for the stats commands on Ubuntu 14.04

b'STAT version 1.4.14 (Ubuntu)'

Connecting to memcached through telnet and issuing the stats command gives you:

STAT pid 993
STAT uptime 11057
STAT time 1458238357
STAT version 1.4.14 (Ubuntu)
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 1.037698
STAT rusage_system 0.325248
.
.   <- cut for brevity ->
.
STAT reclaimed 2

As you can see, all output does follow this pattern, except for the version line.
This could be submitted to Ubuntu upstream as a bug since they appear to have broken the pattern, but I'm not sure it's documents (haven't researched it) as being required to fix this pattern.

Should get pymemcache added as a supported backend to dogpile.cache

dogpile.cache is a very popular way to interact with cache in python and some people might choose their memcached backend specifically based on what it supports so we should do our best to have good support inside it as well.

I've done the initial implementation here, its not perfect but I've run out of time for the weekend =)

https://bitbucket.org/zzzeek/dogpile.cache/pull-requests/34/this-gives-initial-support-for-pymemcache/diff

Drop support for python 2.6

We currently support it, but it's in end of life, and a lot of major libraries are dropping support in the near future:

It also forces us to use an old version of gevent in our tests, because of the following error:

```Collecting gevent (from -r test-requirements.txt (line 4))
  Downloading gevent-1.2.1.tar.gz (2.8MB)
    100% |████████████████████████████████| 2.8MB 510kB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-rPyEfW/gevent/setup.py", line 39, in <module>
        raise Exception("Please install gevent 1.1 for Python 2.6")
    Exception: Please install gevent 1.1 for Python 2.6```

@jparise @cgordon any thoughts?

Remove docs from repo, and add them to gitignore

Checking in the docs means they will get added to each PR, adding extra noise for review and potential bugs, with little upside.
The docs are built automatically in readthedocs from the docs/conf.py file.

We could add the apidoc and the _build doc to the gitignore, and take them out of the repo

Support for AWS ElastiCache "config" command (needed to support cluster configuration)

The AWS implementation of memcached adds a new command "config" (for now only "config get cluster") that returns the list of hosts in the cluster, for auto-discovery. I would like to be able to use pymemcache on AWS and get the cluster configuration:

http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/AutoDiscovery.AddingToYourClientLibrary.html

A couple of ways:

  • a low level command (like "_misc_cmd") that can return multiple lines for the result (_misc_cmd currently reads just one line). The "config" command currently returns 2 lines terminated with "END")
  • update _fetch_cmd to add support for CONFIG (but if AWS adds more subcommands, the format may change).

I would be fine with the first way (and then expose either a "config" or "aws_config" command or add a generic high-level command that would allow to send more "non-standard" requests. I have hacked something together that implement this (with a new "aws_config" method on the base client class) and I can submit a pull request.

Python version issue

Hi,
It seems only Py2.7 or higher could be used. There're some syntaxs of string format are only valid in Py2.7 or higher python version. Any plan to support some lower python version such as 2.5/2.6?

noreply=False by default or safe close

PyMemcache has noreply option and it is default to True.
It may cause data lost when close after write operation.
Please see https://travis-ci.org/KLab/Flask-PyMemcache/builds/23411560

I think this is dangerous for normal users.
I think default value of noreply option should be False.
But it may not good for pinterest's use.

Another option is ringering close.

Current:

                self.sock.close()

Possibly:

            self.sock.shutdown(socket.SHUT_WR)
            data = self.sock.recv(4096)
            assert not data
            self.sock.close()

how to get keys list which are older than 60 seconds

How my requirement is get all the keys that are older than 60 seconds. Is there any way I can get all the keys that are older than 60 seconds. One way i can think of is set the timeout and then check for each key if is expired. But can we set the timeout in pymemcache and check if item is expired? but in this case also i have to go through each key and check which is very costly. Can you suggest any better way?
Thanks,
Shashi

Missing six from dependencies

Immediately after running pip install pymemcache:

Python 2.7.5 (default, Aug 25 2013, 00:04:04) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import pymemcache.client
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-4281dde3c57b> in <module>()
----> 1 import pymemcache.client

/Users/fletcher/Work/Current/domfwd/venv/lib/python2.7/site-packages/pymemcache/client.py in <module>()
     72 
     73 import socket
---> 74 import six
     75 
     76 

ImportError: No module named six

It looks like the six package is missing from your requirements in setup.py.

Support default parameter for 'noreply'

When integrating this with Django I found it annoying that noreply was set, but I want to disable it. The Django interface is for generic k/v caches and thus doesn't support adding arguments - thus I had to subclass client and for each method pass noreply=False to the super() call. Would be nice if the Client classes by default did noreply=None and then if noreply is None: noreply = self.default_noreply so at instantiation time you could decide whether you wanted to noreply or not.

Wheel for 1.3.3 includes spurious client.py

I was looking at the installed version of 1.3.3 (installed via pip 1.5.6) in a virtualenv. It picked the wheel file ( pymemcache-1.3.3-py2.py3-none-any.whl ). It appears that this file has a pymemcache/client.py as well as a pymemcache/client/ dir:

riva% unzip pymemcache-1.3.3-py2.py3-none-any.whl 
Archive:  pymemcache-1.3.3-py2.py3-none-any.whl
  inflating: pymemcache/__init__.py  
  inflating: pymemcache/client.py    
  inflating: pymemcache/exceptions.py  
  inflating: pymemcache/fallback.py  
  inflating: pymemcache/pool.py      
  inflating: pymemcache/serde.py     
  inflating: pymemcache/client/__init__.py  
  inflating: pymemcache/client/base.py  
  inflating: pymemcache/client/hash.py  
  inflating: pymemcache/client/murmur3.py  
  inflating: pymemcache/client/rendezvous.py  
  inflating: pymemcache/test/__init__.py  
  inflating: pymemcache/test/benchmark.py  
  inflating: pymemcache/test/conftest.py  
  inflating: pymemcache/test/integration.py  
  inflating: pymemcache/test/test_benchmark.py  
  inflating: pymemcache/test/test_client.py  
  inflating: pymemcache/test/test_client_hash.py  
  inflating: pymemcache/test/test_integration.py  
  inflating: pymemcache/test/test_rendezvous.py  
  inflating: pymemcache/test/test_utils.py  
  inflating: pymemcache/test/utils.py  
  inflating: pymemcache-1.3.3.dist-info/DESCRIPTION.rst  
  inflating: pymemcache-1.3.3.dist-info/metadata.json  
  inflating: pymemcache-1.3.3.dist-info/pbr.json  
  inflating: pymemcache-1.3.3.dist-info/top_level.txt  
  inflating: pymemcache-1.3.3.dist-info/WHEEL  
  inflating: pymemcache-1.3.3.dist-info/METADATA  
  inflating: pymemcache-1.3.3.dist-info/RECORD

In testing, it appears that at least PyPy 2.6.0 prefers the module instead of the file, but it would be good to double-check the packaging steps to ensure it's not accidentally included any more.

Is lock needed for pool with gevent?

This is not an issue. I am just curious that when this lib is used with gevent, is it still necessary to guard the members of ObjectPool with lock? What's the possible race conditions without greenlet locks?

Support unicode characters in keys

It could be a nice feature to be able to support unicode characters in the keys used for the cache. Since the memcached protocol only supports ascii, this would require some kind of different approach of preparing keys.

One solution is to use byte arrays, another one would be to use some kind of hashing (murmur hash is probably a good fit here) of the keys, which would entail that we obfuscate them to the user in the middleware.

Useful references:

Key question that remains unclear:

What are the encoding constrains on memcached keys?

Keys cannot have spaces, new lines, carriage returns, or null characters.

Probably the best answer

Cut a Release

Hi.

Seems like it's been awhile since a pymemcache release -- any chance you could publish one?

In particular it would be great to have #169 in a released version.

Thanks!

Change the serializer API to be a single class

I don't think taking serializer and deserializer as arguments is a clean API because they really should be treated as a protocol. I can't imagine a time you would provide a serializer and not have to provide a deserializer for it to successfully work.

I suggest the following:

class JsonSerializer:
     def serialize(self, key, value):
         if type(value) == str:
             return value, 1
         return json.dumps(value), 2

    def deserialize(self, key, value, flags):
        if flags == 1:
            return value
        if flags == 2:
            return json.loads(value)
        raise Exception("Unknown serialization format")

client = Client(('localhost', 11211), serializer=JsonSerializer())

HashClient does not appear to be thread-safe

Reading over HashClient, it does not appear to be thread-safe. I'm not clear why it offers the use_pooling option which while that providers a thread-safe underlying client, doesn't fix the fact that multiple threads using the same HashClient instance will probably have issues. e.g., looking at _safely_run_func, two threads can simultaneously notice the same server in self._failed_clients at which point only one thread can win this: self._failed_clients.pop(client.server) with the other getting a KeyError. (That's just one example... there are several instance variables that are not thread safe).

In fact, the design of HashClient w/r/t to PooledClient seems inside-out. Wouldn't it make more sense for the PooledClient class to create instances of HashClient? That seems to be what cgordon had in mind. Otherwise, it seems that HashClient either needs to have locking added, or one must use an instance per-thread.

Thoughts?

Release on PyPI

Problems with github-only pip packages:

  • pip install -r requirements.txt is slower - the git package must always be fetched even in the case of no change. I use the tarball syntax (https://github.com/pinterest/pymemcache/archive/642c094aa561e99125dd9955670b65974ea6b16b.tar.gz) which is a bit faster since it bypasses git but it's still slower since it's not a version compare
  • No wheels, no caching - especially relevant with the pip 7 changes to wheel caching - all the PyPI-based packages are cached in my build pipeline and take milliseconds to install
  • No versions - often an incremental upgrade won't be installed because pip sees the version in setup.py as being the same. Also as I user I have to just "trust" that a commit is safe, whereas a released version is an indicator of a bit more thought.

Please put releases on PyPI with universal wheels :)

Unicode character in key not detected if python2 str

User tried to use key 'rpgv:https://www.pacificsales.com/brands/viking?ref=166_725&loc=soc_\xb1901520_%c3%abuy!' and no MemcacheIllegalInputError was raised, which causes obsure bug with stack trace below

Traceback (most recent call last):
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/services/utils/task_flow.py", line 165, in _run
return self.run()
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/services/utils/task_flow.py", line 160, in run
return self.f(*self.args, **self.kwargs)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/common/utils/decorators.py", line 556, in __get__
value = self.func(obj)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/api/serializers/contexts.py", line 1040, in rich_summary_map
return core.RichPinGridData.manager.get_many(ids)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/core/managers/rich_pin/rich_pin_view_manager.py", line 143, in get_many
return _get_many(model_ids)
File "/mnt/virtualenv/local/lib/python2.7/site-packages/pinstatsd/statsd.py", line 77, in decorated_function
return f(*args, **kwargs)
File "/mnt/virtualenv/local/lib/python2.7/site-packages/pinstatsd/statsd.py", line 98, in decorated_function
return_value = func(*args, **kwargs)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/core/managers/rich_pin/rich_pin_view_manager.py", line 138, in _get_many
models = super(FilteredRichPinViewManager, self).get_many(model_ids)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/core/managers/hbase_managers.py", line 572, in get_many
many_as_dicts = self.get_many_as_dicts(model_ids)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/core/managers/hbase_managers.py", line 552, in get_many_as_dicts
fetched_dicts = _get_many_as_dicts(self, model_ids)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/data_clients/mc_objects.py", line 154, in fn
lzero_missed_keys) or {}
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/data_clients/memcache.py", line 222, in wrap_f
return f(self, *args, **kwargs)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/data_clients/memcache.py", line 451, in get_many
result = mc.get_many(keys)
File "/mnt/virtualenv/local/lib/python2.7/site-packages/pymemcache/client/base.py", line 407, in get_many
return self._fetch_cmd(b'get', keys, False)
File "/mnt/virtualenv/local/lib/python2.7/site-packages/pymemcache/client/base.py", line 689, in _fetch_cmd
key = checked_keys[key]
KeyError: 'rpgv:https://www.pacificsales.com/brands/viking?ref=166_725&loc=soc_\xb1901520_%c3%abuy!'

Name 'basestring' is not defined (django error)

Using python 3.4.2 and django 1.7

Here is the traceback:

File "C:\Python34\lib\site-packages\django\core\handlers\base.py" in get_response
  111.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\User\Project\miraFrontEnd\manageDb\views.py" in mira_cacl
  329.     cache_chains.set(1 + cache_chains.get('count'), a, None)
File "C:\Python34\lib\site-packages\django\core\cache\backends\memcached.py" in set
  89.         self._cache.set(key, value, self.get_backend_timeout(timeout))
File "C:\Python34\lib\site-packages\pymemcache\client.py" in set
  335.         return self._store_cmd(b'set', key, expire, noreply, value)
File "C:\Python34\lib\site-packages\pymemcache\client.py" in _store_cmd
  749.             data, flags = self.serializer(key, data)
File "C:\Python34\lib\site-packages\djpymemcache\backend.py" in serialize_pickle
  27.     if isinstance(value, basestring):
Exception Type: NameError at /manageDb/mira/miracalc/
Exception Value: name 'basestring' is not defined

Possible fix:

I have added following lines in the beginning of \site-packages\djpymemcache\backend.py and it worked:

try:
    unicode = unicode
except NameError:
    # 'unicode' is undefined, must be Python 3
    str = str
    unicode = str
    bytes = bytes
    basestring = (str,bytes)
else:
    # 'unicode' exists, must be Python 2
    str = str
    unicode = unicode
    bytes = str
    basestring = basestring

Note that I didn't write those lines. I took it from here

Feature Suggestion: Timeout between connection attempts

Rather than jumping to a pull request, I wanted to open a discussion here to get feedback on a proposed feature.

The python-memcache client (and others) allow the user to specify a delay before reconnecting in the event that the connection to the memcache server is lost. pymemcache's behavior is to try to reconnect immediately when performing the next operation.

I'd like to add a new keyword argument to Client.__init__ to support this feature. When disconnecting, an attribute would be set which indicates the time after which a new connection should be made. _connect() would return either the socket it just created or None (indicating that the timeout is still in effect); the methods which call _connect would check its return value, and raise an appropriate exception if the timeout is still in effect.

An alternate idea would be to add this behavior to a subclass of Client, so that users could opt in, though that seems like a lot of extra infrastructure for a single, generally-useful feature.

Note that python-memcache silently ignores disconnections -- get-like commands return empty results, and set-like commands silently do nothing (well, technically they return a value to signal that they silently did nothing). On the one hand, it's nice not to have to wrap everything in try/except; on the other, this seems decidedly un-Pythonic. I'm open to suggestions in this regard.

Fix Documentation

The documentation written inline in the code isn't viewable in readthedocs. The whole sphinx documentation building needs a fix.

KeyError when no connections are available using HashClient

When the HashClient runs out connections it will start raising KeyError (even with ignore_exc=True). Reproduce with this:

>>> from pymemcache.client.hash import HashClient
>>> client = HashClient([('localhost', 11211)], use_pooling=True, ignore_exc=True,
...                     timeout=2, connect_timeout=2)
>>> client.set('foo', 'bar')
True
>>> client.get('foo')
'bar'
>>> # Stop Memcached server
...
>>> client.get('foo')
>>> client.get('foo')
>>> client.get('foo')
>>> client.get('foo')
>>> client.get('foo')
>>> client.get('foo')
>>> client.get('foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Peter/.vim/venv/lib/python2.7/site-packages/pymemcache/client/hash.py", line 226, in get
    return self._run_cmd('get', key, None, *args, **kwargs)
  File "/Users/Peter/.vim/venv/lib/python2.7/site-packages/pymemcache/client/hash.py", line 214, in _run_cmd
    client = self._get_client(key)
  File "/Users/Peter/.vim/venv/lib/python2.7/site-packages/pymemcache/client/hash.py", line 134, in _get_client
    client = self.clients[server]
KeyError: None

Add support for pluggable compression

By default memcached has a 1mb limit and when you are working with larger objects using compression can get you a lot of additional caching for little overhead.

Primary cache algorithms that should be supported:

  1. lz4
  2. speedy
  3. zlib

The compression shouldn't kick in automatically, we should have a sane threshold for when compression starts

stats value for evictions is boolean when it should be numeric?

per https://github.com/memcached/memcached/blob/b1debc4c96134e9014a929c32d6990cb5d66a22c/doc/protocol.txt#L562-L563 the value for evictions from a stats command is numeric.

but pymemcache is treating it as on/off:

b'evictions': lambda value: value == b'on',

this seems odd, because if memcached returns evictions as 55, pymemcache is returning this as False.

i'd send a pull request, but it seems like this might have done for a reason that's escaping me at the moment ;) let me know.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.