twisted / txmongo Goto Github PK
View Code? Open in Web Editor NEWasynchronous python driver for mongo
Home Page: https://txmongo.readthedocs.io
License: Apache License 2.0
asynchronous python driver for mongo
Home Page: https://txmongo.readthedocs.io
License: Apache License 2.0
The new authentication approach crashes with an exception in defer.py:
exceptions.TypeError: object of type 'generator' has no len()
The cause looks like this line of code in connect.py: method authenticate, around line 292:
yield defer.gatherResults(
(connection.authenticate(database, username, password) for connection in self.__pool),
consumeErrors=True
)
Looks like what is being passed into gatherResults is a generator, not a proper list, and defer.py doesn't like this. The function gatherResults is expecting a list of Deferred, not a generator.
Here's the full traceback:
Traceback (most recent call last):
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1213, in unwindGener
ator
return _inlineCallbacks(None, gen, Deferred())
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1070, in _inlineCall
backs
result = g.send(result)
File "/usr/local/pakedge/proxy/python_pakedge_proxy/local/lib/python2.7/site-packages/txmongo-0.6-py2.7.egg/txmongo/database.py", lin
e 100, in authenticate
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1213, in unwindGener
ator
return _inlineCallbacks(None, gen, Deferred())
--- ---
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1070, in _inlineCall
backs
result = g.send(result)
File "/usr/local/pakedge/proxy/python_pakedge_proxy/local/lib/python2.7/site-packages/txmongo-0.6-py2.7.egg/txmongo/connection.py", l
ine 292, in authenticate
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 867, in gatherResult
s
consumeErrors=consumeErrors)
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 793, in init
self.resultList = [None] * len(deferredList)
exceptions.TypeError: object of type 'generator' has no len()
In current code when factory lose connection, it queues all requests and finish them on successful reconnection. I think it is better to have possibility to specify the behaviour: to queue or raise exception for requests, that were called while retrying to connect. How do you think?
I guess it was removed from previous version. Why?
Hi
I notice uri type assertion in class Connection Pool.init has been changed from
assert isinstance(uri, basestring) to assert isinstantce(uri, str) like following:
ConnectionPool(object):
def init(self, uri="mongodb://127.0.0.1:27017", pool_size=1, ssl_context_factory=None, **kwargs):
assert isinstance(uri, str)
when I use an unicode string, The Connection pool init will fail.
Is it possible to use basestring back?
Regards
gelin yan
getLastError supports multiple options (see http://www.mongodb.org/display/DOCS/getLastError+Command#getLastErrorCommand-Options), but providing these options are currently not possible in txmongo.
This is supported by pymongo by passing extra keyword arguments to the insert/update/remove commands. Default options are possible to set on both the database and collection instances. It would be great having this feature here.
The problem is that when the fist attempt to connect to db on application start failed (e.g. mongo was down) - reconnector fails to connect when db is up in some some (but tries to do reconnection). Possibly the problem is in ClientReconnectingFactory itself. Can you give some directions or a workaround for that?
Hi All
The title is my question. I have read some source codes and notice there is no a bit of info about replication support.
Thanks.
There's a recursion in dataReceived method of protocol, which leads to exceptions.RuntimeError: maximum recursion deptch exceeded in __instancecheck__
.
self.dataReceived
method is called about more that thousand times.
pymongo supports this it seems.
http://api.mongodb.org/python/1.0/pymongo.master_slave_connection-pysrc.html
There is no minimum (or recommended minimum) mongo version. While this probably supports the wire protocol down to 0 like the pymongo
package, the tests and examples reference features that aren't in legacy versions.
I realized this because I accidentally spun up a old 2.1 version, and the example/tests all failed.
Im getting the following ran twistd -ny cyclone_server.tac and test_rest.sh
2010-09-14 15:44:20+0800 [MongoProtocol,client] Unhandled Error
Traceback (most recent call last):
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 345, in errback
self._startRunCallbacks(fail)
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 424, in _startRunCallbacks
self._runCallbacks()
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, _args, *_kw)
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 949, in gotResult
_inlineCallbacks(r, g, deferred)
--- ---
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 891, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/python/failure.py", line 338, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
exceptions.TypeError: exceptions must be classes, or instances, not type
When using txmongo, we use code like this:
mongoConnection=yield txmongo.MongoConnection(MONGODB_HOST, MONGODB_PORT, pool_size=CONNECTION_POOL_SIZE)
db=getattr(self.mongoConnection, DEFAULT_MONGO_DB_NAME)
yield db.authenticate(MONGODB_USER, MONGODB_PASS)
It would be nice to know if authentication is successful. However, the database authenticate method does not return anything to let you know this. It only calls connection authenticate_with_nonce and that sets a variable db._authenticated to True when authenticated. But it is bad form for any external python script to check on the value of a member variable that starts with an underscore.
I suggest that you add a method like is_authenticated() to db that returns True or False. This way we know when the connection is ready for successful writing or other database interactions.
Any particular reason that TxMongo's args order differ from PyMongo (e.g. find()
)? I'm migrating some code (and have other code that I'd prefer to have working on both platforms). Obviously, I'll update the overlapping code to use kwargs, but it was an unexpected discovery (and might catch others).
Seems like there is no possibility to connect to a replica set, I mean like this:
http://api.mongodb.org/python/1.9%2B/examples/replica_set.html#id1
Am I right?
It seems like GridFS.get and .put don't work at all...
from twisted.internet import reactor
import txmongo
from txmongo.gridfs import GridFS
d = txmongo.MongoConnectionPool()
def gotDB(db):
files = GridFS(db.fs, collection='foo')
print files
return files
def gotFiles(files):
obj_id = files.put('some contents', filename='foo', content_type='image/png')
print 'saved', repr(obj_id)
obj = files.get(str(obj_id))
print 'got', repr(obj)
print dir(obj)
print obj.read()
print obj.name
print obj.content_type
d.addCallback(gotDB)
d.addCallback(gotFiles)
reactor.run()
gives me this:
<txmongo._gridfs.GridFS object at 0x12885d0>
saved ObjectId('4fe84a9bf648c2b4df000001')
got 4fe84a9bf648c2b4df000001
['_GridOut__buffer', '_GridOut__chunks', '_GridOut__current_chunk', '_GridOut__position', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattr__', '__getattribute__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_file', '_id', 'aliases', 'chunk_size', 'close', 'content_type', 'length', 'md5', 'metadata', 'name', 'read', 'seek', 'tell', 'upload_date']
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 464, in _startRunCallbacks
self._runCallbacks()
File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 551, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "badmongo.py", line 19, in gotFiles
print obj.read()
File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 1187, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 1045, in _inlineCallbacks
result = g.send(result)
File "build/bdist.macosx-10.3-fat/egg/txmongo/_gridfs/grid_file.py", line 345, in read
File "build/bdist.macosx-10.3-fat/egg/txmongo/_gridfs/grid_file.py", line 331, in __getattr__
exceptions.AttributeError: GridOut object has no attribute 'length'
<Deferred at 0x128aee0 current result: <twisted.python.failure.Failure <type 'exceptions.AttributeError'>>>
I think it's because GridFS.put
returns grid_file._id
, and GridFS.get
takes a file_id
, but uses that id as if it were a document in this line:
return GridOut(self.__collection, file_id)
Hi, using txmongo for a while I hadn't updated for about a year and am discovering now the new versions.
I tried installing on a blank ubuntu machine both mongo3 (following new method given here) then pip install txmongo==0.6
and resulted in a non-functional install, as shown by the following error when running python examples/inlinecallbacks/query.py
:
File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 424, in errback self._startRunCallbacks(fail) File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 491, in _startRunCallbacks self._runCallbacks() File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 578, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1163, in gotResult _inlineCallbacks(r, g, deferred) --- --- File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1105, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/tmp/test", line 7, in example docs = yield mongo.foo.test.find(limit=10) File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1107, in _inlineCallbacks result = g.send(result) File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/txmongo/collection.py", line 150, in find defer.returnValue([d.decode(as_class=as_class) for d in documents]) exceptions.TypeError: decode() got an unexpected keyword argument 'as_class'
For what I understand, it seems txmongo0.6 uses methods from bson2.7 while bson3 (which dropped some kwargs used in txmongo including 'as_class') is shipped with pymongo3 that is coming along naturally with txmongo.
My solution for now for my projects consists in installing first separately pymongo pinned to version 2.7 and then txmongo pinned to 0.6 in my requirements.txt but that can only be temporary.
I tried installing from git sources instead of pip but had the same issue, which makes sense since decode(as_class=as_class) is still being used line 167 of collection.py in the actual repo.
Hi. I'm testing the following code using txmongo 16.1.0. Basically, I'm sending 1001 objects to an insert_many. It's raising pymongo.errors.OperationFailure: TxMongo: command SON ... ('writeConcern', {})]) on namespace foo.$cmd failed with 'exceeded maximum write batch size of 1000'
import time
import txmongo
from twisted.internet import defer, reactor
@defer.inlineCallbacks
def example():
mongo = yield txmongo.connection.ConnectionPool('mongodb://127.0.0.1:27018')
foo = mongo.foo # `foo` database
test = foo.test # `test` collection
# insert some data
insert_data = []
for x in range(1001):
insert_data.append({"something": x*time.time()})
result = yield test.insert_many(insert_data)
if __name__ == '__main__':
example().addCallback(lambda ign: reactor.stop())
reactor.run()
Mongo has a limit of 1000 for Bulk operations, but drivers usually handle this by dividing batch operations. Pymongo does not fail with many more objects.
Hi,
I'm using txmongo in an environment with a lot of running parallel tasks (twitter and rss collection mostly) and was wondering which best practice would be recommended to use txmongo :
When I try to connect to GridFS on 2.6.3, I get an error while creating index on the "fs" collection.
The error is something like this:
pymongo.errors.OperationFailure: not authorized to create index on dbname.fs.chunks
Then I looked up the init.py file for the GridFS object and I found the offending line:
self.__chunks.create_index(filter.sort(ASCENDING("files_id") + ASCENDING("n")),
unique=True)
However it's not clear to me why this happens because finally it's an insert operation that takes place when we create_index. So does this mean that the authentication protocol I pushed earlier is now failing for 2.6.3? This driver works perfectly on 2.4.10
Any responses would be appreciated for those who might have observed similar behavior.
Hi
I noticed the count() function of collection returned floating value but pymongo return integer value which is correct.
Is it a bug?
Regards
gelin yan
Perhaps this is a deliberate design choice, but shouldn't Collection.find_one() return None rather than {} when the document doesn't exist?
So it will be possible to install it with pip install txmongo.
EDIT: clarify issue
Only the top one of these works:
import os
import sys
import txmongo
import pymongo.uri_parser
import pymongo
from twisted.internet import defer, reactor
mongo_uri = os.environ.get('MONGOLAB_URI')
mongo_creds = pymongo.uri_parser.parse_uri(mongo_uri)
if sys.argv[1] == 'normal':
@defer.inlineCallbacks
def example():
conn = yield txmongo.MongoConnection(*mongo_creds['nodelist'][0])
db = getattr(conn, mongo_creds['database'])
yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
test = db.test
yield test.insert({'1':'1'})
docs = yield test.find(limit=10)
for doc in docs:
print doc
if sys.argv[1] == 'pool':
@defer.inlineCallbacks
def example():
conn = yield txmongo.MongoConnectionPool(*mongo_creds['nodelist'][0])
db = getattr(conn, mongo_creds['database'])
yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
test = db.test
yield test.insert({'1':'1'})
# fetch some documents
docs = yield test.find(limit=10)
for doc in docs:
print doc
if sys.argv[1] == 'lazy':
@defer.inlineCallbacks
def example():
conn = txmongo.lazyMongoConnection(*mongo_creds['nodelist'][0])
db = getattr(conn, mongo_creds['database']) # `foo` database
yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
test = db.test # `test` collection
yield test.insert({'1':'1'})
docs = yield test.find(limit=10)
for doc in docs:
print doc
if sys.argv[1] == 'lazypool':
@defer.inlineCallbacks
def example():
conn = txmongo.lazyMongoConnectionPool(*mongo_creds['nodelist'][0])
db = getattr(conn, mongo_creds['database']) # `foo` database
yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
test = db.test # `test` collection
yield test.insert({'1':'1'})
docs = yield test.find(limit=10)
for doc in docs:
print doc
if __name__ == '__main__':
example().addCallback(lambda ign: reactor.stop())
reactor.run()```
The lazy methods fail with 'not connected' and the pooled method fail with 'auth fails'.
I want to use authentication inside a cyclone request, but I don't want to authenticate if it already has happened.
I don't think txmongo supports this naturally at the moment.
is ok?
Pleas take a look at the code below. This was my best guess. I am sure I am doing something wrong here but switching from pymongo to txmongo has caused a 95% decrease in performance. I am wondering if there is way to use the txmongo.MonoConnection object global so I only need to connect to mongo once. Sorry I am still a bit of a noob to python, twisted and mongo. but basic idea is store udp packet for later processing.
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor,defer
import txmongo
import struct
import string
x = 0
class Collector(DatagramProtocol):
def init(self):
print "Initializing Collector";
@defer.inlineCallbacks
def datagramReceived(self, data, (host, port)):
#print "Recieved Packet"
ch = yield txmongo.MongoConnection()
dbh = ch.flowpackets.packets
#print "received %r from %s:%d type:%s" % (data, host, port,type(data))
#self.transport.write(data, (host, port))
global x
packet = dict()
packet = { 'packet': data.encode('hex')}
dbh.insert(packet)
x += 1
reactor.listenUDP(2055, Collector())
print "Starting Reactor"
reactor.run()
print "Packet Count: ", x
Hi,
I would really like to benefit from the latest version including PR #49 but it seems like it was not updated on pypi yet https://pypi.python.org/pypi/txmongo
Could you please push it?
Thanks a lot in advance!
I believe that the _initializeProto is used incorrectly
It returns a deferred-generator, but we do not yield it (moreover I think yield would not work inside buildProtocol).
https://github.com/twisted/txmongo/blob/master/txmongo/connection.py#L51
I'm really having a heck of a time getting the current version of the library to run, I've tried a build on debian linux and osx and in both locations I can't get my script to do anything but have a key error inside the library, so I've reverted to trying to verify that I have the library installed correctly by trying to get the test scripts to run.
If run on a machine with mongo on its normal port, should "make test" in the project directory succeed?
I'm getting different counts between debian (FAILED (failures=1, errors=82)) and osx (FAILED (failures=1, errors=75, successes=1)), but on both I'm getting almost complete failure.
So after a fresh clone, is a "make test" after a "sudo python setup.py install" working for you?
The package requires twisted, but does not specify a minimum version or the known target/working version.
Is there any way to issue a custom command (like findAndModify) through public txmongo API? The only solution I found so far is via private methods:
An example would be lovely.
Hi.
When I try to execute the example "insert.py", I get the next exception (not sure if only in insert):
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
File "build/bdist.linux-i686/egg/txmongo/protocol.py", line 240, in handle
File "build/bdist.linux-i686/egg/txmongo/protocol.py", line 335, in handle_REPLY
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback
self._startRunCallbacks(result)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
self._runCallbacks()
--- ---
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
current.result = callback(current.result, _args, *_kw)
File "build/bdist.linux-i686/egg/txmongo/connection.py", line 91, in _configureCallback
File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 624, in decode
self, as_class, tz_aware, uuid_subtype, compile_re)
exceptions.TypeError: function takes exactly 4 arguments (5 given)
I got this error in my application, so I decided to try with the examples.
I'm on de Debian Wheezy machine, with python 2.7 and Mongo 2.4.10.
Thanks for any help.
Docs:
http://www.mongodb.org/display/DOCS/Tailable+Cursors
Upstream:
mongodb/mongo-python-driver@66292c6
This would be very nice.
The PyMongo driver supports finding fields as either list or dict. They internally convert list to dict if they receive a list.
This would be helpful as I would very much like to specify both exclusions and inclusions on the field parameter.
Currently txmongo raises Exception because the field isn't a list. But it looks like you are converting to dict internally too, so I think you can just add
[code]
if not isinstance(fields, dict):
[/code]
before converting.
I'm in the process of learning txmongo so I went to the examples (I haven't found other docs, am I correct in thinking there aren't any for the current release?).
The examples tend to call one of the following:
MongoConnection
MongoConnectionPool
lazyMongoConnection
lazyMongoConnectionPool
So I started out trying to do that, but error-ing out because they all want 2 args (plus self)
After tracing I worked out that these are defined in a block at the end of connection.py
, and are wrapped by comments listing them as "Legacy Wrapper". They all equate back to MongoConnection
, which has an __init__
that requires a host and port argument (no defaults), turning them into a uri. That init in-turn calls ConnectionPool
's __init__
with has no required agreements (because it defaults to a uri of "mongodb://127.0.0.1:27017")
The ConnectionPool
class isn't exposed via the projects __init__.py
file though it seems like that is what is being implied to be the proper way to open the library these days.
So for new projects what is the intended setup function?
Hello. A quick support request on how to transform BSON object to json.
At the moment I'm doing following (result is return result from collection.find_one({})
d = result[0]
del d['_id']
json.dumps(d)
I have to del '_id' , because otherwise json cannot be serialized from objectid. I've found BSON class in _pymongo but couldn't figure out how to use. Can you help?
When running in windows the following error is received:
Traceback (most recent call last):
File "webmain.tac", line 4, in
import cyclone.web
File "C:\Python27\lib\site-packages\cyclone\web.py", line 21, in
from cyclone import __version__, escape, template, httpserver
File "C:\Python27\lib\site-packages\cyclone\httpserver.py", line 24, in
import fcntl
ImportError: No module named fcntl
it seems that fcntl is not required in httpserver since there are no refernces to it.
removing it solved the problem
This will be the release thread for the 15.0 milestone.
If there are comments about the current release, please ask/post them here. :)
@fiorix We good to go?
@IlyaSkriblovsky Any additional things we should do before release, you've done a lot already for this project.
In my logs, I get the following:
2016-03-01 16:53:43.398197 +0100 ERROR [] 'Connection lost.'
2016-03-01 16:53:43.399087 +0100 ERROR [] 'Connection lost.'
which come about when there aren't any mongodbs left alive:
https://github.com/twisted/txmongo/blob/master/txmongo/protocol.py#L325
Normally this would be fine, but for people reading logs (for the fun of it?), they won't have any context as to what is going on so we should provide better error messages such as:
'TxMongo lost connection to MongoDB.
It would probably not be so bad an idea to have something in connectionReady that say that TxMongo is ready as some TxMongo users (like me) put all all the twisted protocol log.msg to DEBUG to help cut down on log spam (3 messages per attempt to re-connect).
Hey hey
It would be awesome if ObjectId's are nativeley (re)serializable. E.g. to embed them in json return values of json-rpc handlers. That way, one can avoid iterating over docs to replace certain objects with the string value ( or to write a specific handlers for the parser, if supported )
I did it by inheriting from str, making it serializable by anything which knows how to handle str. Just a couple of changes here and there to make it immutable
if you're interested, here comes a diff
diff --git a/txmongo/_pymongo/bson.py b/txmongo/_pymongo/bson.py
index aef08f8..7587ff1 100644
--- a/txmongo/_pymongo/bson.py
+++ b/txmongo/_pymongo/bson.py
@@ -416,6 +416,8 @@ def _element_to_bson(key, value, check_keys):
full_length = struct.pack("<i", 8 + len(cstring) + len(scope))
length = struct.pack("<i", len(cstring))
return "\x0F" + name + full_length + length + cstring + scope
+ if isinstance(value, ObjectId):
+ return "\x07" + name + value.binary
if isinstance(value, str):
cstring = _make_c_string(value)
length = struct.pack("<i", len(cstring))
@@ -429,8 +431,6 @@ def _element_to_bson(key, value, check_keys):
if isinstance(value, (list, tuple)):
as_dict = SON(zip([str(i) for i in range(len(value))], value))
return "\x04" + name + _dict_to_bson(as_dict, check_keys)
- if isinstance(value, ObjectId):
- return "\x07" + name + value.binary
if value is True:
return "\x08" + name + "\x01"
if value is False:
diff --git a/txmongo/_pymongo/objectid.py b/txmongo/_pymongo/objectid.py
index 1c8de98..4c43cea 100644
--- a/txmongo/_pymongo/objectid.py
+++ b/txmongo/_pymongo/objectid.py
@@ -41,7 +41,7 @@ def _machine_bytes():
return machine_hash.digest()[0:3]
-class ObjectId(object):
+class ObjectId( str ):
"""A Mongo ObjectId.
"""
@@ -50,7 +50,7 @@ class ObjectId(object):
_machine_bytes = _machine_bytes()
- def __init__(self, oid=None):
+ def __new__(klass, oid=None):
"""Initialize a new ObjectId_.
If `oid` is ``None``, create a new (unique)
@@ -70,11 +70,16 @@ class ObjectId(object):
.. _ObjectId: http://www.mongodb.org/display/DOCS/Object+IDs
"""
if oid is None:
- self.__generate()
+ oid = klass._generate()
else:
- self.__validate(oid)
+ oid = klass._validate(oid)
- def __generate(self):
+ self = str.__new__( klass, oid.encode("hex") )
+ self.__id = oid
+ return self
+
+ @classmethod
+ def _generate(klass):
"""Generate a new value for this ObjectId.
"""
oid = ""
@@ -94,9 +99,10 @@ class ObjectId(object):
ObjectId._inc = (ObjectId._inc + 1) % 0xFFFFFF
ObjectId._inc_lock.release()
- self.__id = oid
+ return oid
- def __validate(self, oid):
+ @classmethod
+ def _validate(self, oid):
"""Validate and use the given id for this ObjectId.
Raises TypeError if id is not an instance of (str, ObjectId) and
@@ -106,13 +112,13 @@ class ObjectId(object):
- `oid`: a valid ObjectId
"""
if isinstance(oid, ObjectId):
- self.__id = oid.__id
+ return oid.__id
elif isinstance(oid, basestring):
if len(oid) == 12:
- self.__id = oid
+ return oid
elif len(oid) == 24:
try:
- self.__id = oid.decode("hex")
+ return oid.decode("hex")
except TypeError:
raise InvalidId("%s is not a valid ObjectId" % oid)
else:
@@ -154,9 +160,6 @@ class ObjectId(object):
return datetime.datetime.utcfromtimestamp(t)
generation_time = property(generation_time)
- def __str__(self):
- return self.__id.encode("hex")
-
def __repr__(self):
return "ObjectId('%s')" % self.__id.encode("hex")
I have been thinking for quite a while about being able to run queries without loading the entire result set into memory. Here's what I came up with (view the Changes):
collection.find_with_cursor returns a deferred which results in a tuple: (docs, deferred). If there are no more documents remaining in the current query, a deferred will be returned which results in this tuple: ([], None)
For example:
docs, d = yield collection.find_with_cursor()
# collection.find(cursor=True) also works
while docs:
# Do something with this page of docs here
for i in docs:
pass
# Get the next set of docs from the query.
docs, d = yield d
I have already implemented this (here), but let me know what you think, or if there are any improvements that could be made on this, then I will send over a pull request
Otherwise a save of a document that does have an _id, but is not yet in the collection doesn't work:
diff -r d4e6d68c7815 txmongo/collection.py
--- a/txmongo/collection.py Wed Dec 22 15:26:53 2010 +0000
+++ b/txmongo/collection.py Wed Dec 22 15:41:34 2010 +0000
@@ -215,7 +215,7 @@
objid = doc.get("_id")
if objid:
- return self.update({"_id": objid}, doc, safe=safe)
+ return self.update({"_id": objid}, doc, safe=safe, upsert=True)
else:
return self.insert(doc, safe=safe)
When the fields
argument in Collection.find()
is an empty list ([]
), the returned documents only contain the _id
field when they should contain all of the fields. When the fields
argument is None
all fields in the documents are returned which is the expected behavior of setting fields
to an empty list.
NOTE: This issue is mostly trivial but a slight annoyance.
Here's a diff:
@@ -83,11 +83,6 @@ if not isinstance(limit, types.IntType): raise TypeError("limit must be an instance of int") - if fields is not None: - if not fields: - fields = ["_id"] - fields = self._fields_list_to_dict(fields) - if isinstance(filter, (qf.sort, qf.hint, qf.explain, qf.snapshot)): spec = SON(dict(query=spec)) for k, v in filter.items():
"fakeroot debian/rules binary" complained about python2.5 missing. Fixed, but it's unclear if my solution is too fragile.
$ debian/rules binary
debian/rules:6: /usr/share/cdbs/1/rules/debhelper.mk: No such file or directory
debian/rules:7: /usr/share/cdbs/1/class/python-distutils.mk: No such file or directory
make: *** No rule to make target `/usr/share/cdbs/1/class/python-distutils.mk'. Stop.
resolved with "apt-get install cdbs"
Docs:
http://www.mongodb.org/display/DOCS/Tailable+Cursors
Upstream:
mongodb/mongo-python-driver@66292c6
This would be very nice.
I was wondering about GEO2D indexes and ensure_index support. As far as I can see, this is not supported in the current txmongo, but I am not sure how hard it would be to support. When I try to create_index a GEO2D index, I get the following exception:
sort_fields must be an instance of filter.sort
Trying to use ensure_index failed as well - this would be a nice to have.
Let me know if I can help in any way. I have a good knowledge of MongoDB but I am not that well-versed in Twisted so I didn't want to just fork start hacking, but I'll be happy to look at it if you point me in the right direction.
On my development server which is running Ubuntu 12.04, with bundled twisted 11.1 and mongodb 2.04.
I keep getting the following error.
2012-06-07 02:18:29+0000 [MongoProtocol,client] <twisted.internet.tcp.Connector instance at 0xbcc882c> will retry in 10 seconds
2012-06-07 02:18:29+0000 [MongoProtocol,client] Stopping factory <txmongo._MongoFactory instance at 0xbcc880c>
Upon further investigation,
using lsof | grep mongod | grep TCP | wc -l
I got 448 established connection after restarting mongodb and my twisted app for 1/2 day.
Take note that this is a development server, so the only person using it is me. And the twisted app keeps getting restarted to update the latest changes. So to have 448 established connection seems crazy, it seems that the connection is not closed after each query.
Anyone facing the same issue?
Is there any plans to support authentication in txmongo?
When use insert_may with ordered=False I expected have in BulkWriteError whith all duplicated keys but only return the last, how can i get all error (knowing data that induce error)
Hi. I just cloned the txmongo repository and I'm trying to run the tests. In test_queries.py, in TestInsertMany, I'm getting two FAILs for test_UnorderedUnack and test_OrderedUnack.
I'm using python3.5 with mongo3.2.3. Any clues?. Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.