Code Monkey home page Code Monkey logo

Comments (7)

michelp avatar michelp commented on July 24, 2024

Here is an example PickleEncoder that mimics the exact output of the way pq does it now:

import pickle
from json import dumps, JSONEncoder


class PickleEncoder(JSONEncoder):

    def encode(self, obj):
        return dumps(pickle.dumps(obj, 0).decode('latin-1'))

from pq.

malthe avatar malthe commented on July 24, 2024

The idea with the current design is that you can subclass Queue and implement your own loads and dumps methods or even just pass callables directly to the PQ constructor using the loads and dumps keyword arguments.

The built-in json module has an object hook mechanism that can be used to encode and decode a DateTime object, but you might want to use a more optimized JSON module as well.

The "double encoding" option is really a feature. If you have complex objects that you don't want to implement custom JSON object hooks for, then you can just use the pickle format and not have to do more work. In terms of performance then it is two-fold: we're using a slower pickle type (version 0) and we're JSON-escaping the output string. If you need better performance then you must override the loads and dumps callables and provide custom serialization for your objects.

from pq.

michelp avatar michelp commented on July 24, 2024

The idea with the current design is that you can subclass Queue and implement your own loads and dumps methods or even just pass callables directly to the PQ constructor using the loads and dumps keyword arguments.

There is no way to override the outer dumps function in the expression dumps(self.dumps(data)) on line 226 of init.py. Yes, you can override the inner dumps, but not the outer, so the only option is to have the inner dumps return a string which the outer dumps just reserializes as one big json string. This makes the payload pretty much completely useless on the SQL side of things, being a json string that contains escaped json. This is the double encoding. Every payload must be serialized and deserialized twice and the payload is effectively inaccessible from SQL.

The PR I submitted removes this double encoding, it's more efficient, the payload is only encoded once, and because it's actual json, not json escaped into a big json string, the payload is usable from sql using postgres default json operators.

from pq.

malthe avatar malthe commented on July 24, 2024

I have taken the liberty to take a different stab at the issue in the referenced pull request. This also circumvents the database driver from doing the JSON decoding by itself.

from pq.

michelp avatar michelp commented on July 24, 2024

Ok I can work with that, it does sidestep the double encoding issue and allows the data to be sql accessible, so it works for me. Thanks!

from pq.

malthe avatar malthe commented on July 24, 2024

Great. I'll merge the request. It's not that your proposed fix wouldn't work, I just think it introduced more code than necessary.

from pq.

malthe avatar malthe commented on July 24, 2024

This feature is now available in release 1.8.1.

from pq.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.