Add support for pluggable compression about pymemcache HOT 8 CLOSED

What python library provides all of those?

from pymemcache.

You don't need a single python library for them. i.e if you allow this:

class ICompressable:
    def compress(obj):
        pass
    def decompress(obj):
        pass

and on the client you accept a compressor argument:

from pymemcache import Client
import lz4

class LZ4Compressor:
    def compress(obj):
        return lz4.dumps(obj)

    def decompress(obj):
        return lz4.loads(obj)

client = Client(compressor=lzr_compressor)

from pymemcache.

Fair enough that works to.

from pymemcache.

Only thing would be if we wanted to flag each type of compressor in the data that way if you had two clients:

client = Client(compressor=lz4_compressor)
client2 = Client(compressor=speedy_compressor)

If client1 tried to open a key from client2 it would fail. Would we want to detect its using a different compressor and raise a nice error like "Hey, that key was compressed with lz4!" or do we just blow up with a weird error and let them figure it out?

If we wanted to provide the nice flags then we'd have to maintain them internally like:

FLAG_LZ4 = 1 << 3
FLAG_SPEEDY = 1 <<  4
FLAG_ZLIB = 1 << 5

from pymemcache.

I put up a refactor branch to help with this in #160 and #161 along with a json codec. I wrote something similar for my own projects in the past, so it was mostly copypasta from that. I need to dig deeper but I also have a pipeline that would be helpful in this case, ie compression. We can just pass the bitflags in modified, that's at least what I did in the past.

from pymemcache.

You can now implement this in 3.0 with a custom serde object which makes these decisions.
(You can do it in 2.x with serializer/deserializer too.)

@jogo, @jparise, should this be kept open? If so, the issue would seem to be adding built-in support for compression -- e.g. a built-in PickleZlibSerde.

from pymemcache.

I think we could emphasize the fact that the serde interface is the place to implement (de)compression (or other data transformations). I don't think pymemcache needs to provide compression implementations itself.

from pymemcache.

Closing this because this functionality is now available through the serde object interface.

from pymemcache.