Comments (7)
Here is an example PickleEncoder that mimics the exact output of the way pq does it now:
import pickle
from json import dumps, JSONEncoder
class PickleEncoder(JSONEncoder):
def encode(self, obj):
return dumps(pickle.dumps(obj, 0).decode('latin-1'))
from pq.
The idea with the current design is that you can subclass Queue
and implement your own loads
and dumps
methods or even just pass callables directly to the PQ
constructor using the loads
and dumps
keyword arguments.
The built-in json
module has an object hook mechanism that can be used to encode and decode a DateTime
object, but you might want to use a more optimized JSON module as well.
The "double encoding" option is really a feature. If you have complex objects that you don't want to implement custom JSON object hooks for, then you can just use the pickle format and not have to do more work. In terms of performance then it is two-fold: we're using a slower pickle type (version 0) and we're JSON-escaping the output string. If you need better performance then you must override the loads
and dumps
callables and provide custom serialization for your objects.
from pq.
The idea with the current design is that you can subclass Queue and implement your own loads and dumps methods or even just pass callables directly to the PQ constructor using the loads and dumps keyword arguments.
There is no way to override the outer dumps function in the expression dumps(self.dumps(data))
on line 226 of init.py. Yes, you can override the inner dumps, but not the outer, so the only option is to have the inner dumps return a string which the outer dumps just reserializes as one big json string. This makes the payload pretty much completely useless on the SQL side of things, being a json string that contains escaped json. This is the double encoding. Every payload must be serialized and deserialized twice and the payload is effectively inaccessible from SQL.
The PR I submitted removes this double encoding, it's more efficient, the payload is only encoded once, and because it's actual json, not json escaped into a big json string, the payload is usable from sql using postgres default json operators.
from pq.
I have taken the liberty to take a different stab at the issue in the referenced pull request. This also circumvents the database driver from doing the JSON decoding by itself.
from pq.
Ok I can work with that, it does sidestep the double encoding issue and allows the data to be sql accessible, so it works for me. Thanks!
from pq.
Great. I'll merge the request. It's not that your proposed fix wouldn't work, I just think it introduced more code than necessary.
from pq.
This feature is now available in release 1.8.1.
from pq.
Related Issues (20)
- 1.6? HOT 2
- pq as a "jobs queue" - items can not be read concurrently when using transactional get HOT 19
- Support for schema qualified tables HOT 3
- connection pool exhausted HOT 3
- DuplicatePreparedStatement error HOT 10
- Prepared statement name not properly escaped HOT 1
- Question: why am I seeing "NOTICE: function pq_notify does not exist" in the logs? HOT 3
- Delete executed tasks?! HOT 1
- Wait on a job? HOT 3
- get() timeout not honoured HOT 1
- Any interest in porting to cockroach HOT 3
- Pipenv using old create.sql HOT 5
- Get id of current task HOT 2
- Performance with a very large queue HOT 9
- lost trigger when setting up multiple queue-tables within the same schema
- Adding recurring tasks? HOT 4
- queue.put() inside a transaction sets enqueued_at to the transaction start time, not the current time HOT 4
- Adding name of the job producer HOT 2
- Dashboard for PQ HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pq.