I was wondering about GEO2D indexes and ensure_index support. As far as I can see, this is not supported in the current txmongo, but I am not sure how hard it would be to support. When I try to create_index a GEO2D index, I get the following exception:

sort_fields must be an instance of filter.sort

Trying to use ensure_index failed as well - this would be a nice to have.

Let me know if I can help in any way. I have a good knowledge of MongoDB but I am not that well-versed in Twisted so I didn't want to just fork start hacking, but I'll be happy to look at it if you point me in the right direction.

getLastError supports multiple options (see http://www.mongodb.org/display/DOCS/getLastError+Command#getLastErrorCommand-Options), but providing these options are currently not possible in txmongo.

This is supported by pymongo by passing extra keyword arguments to the insert/update/remove commands. Default options are possible to set on both the database and collection instances. It would be great having this feature here.

is ok?

When using txmongo, we use code like this:

mongoConnection=yield txmongo.MongoConnection(MONGODB_HOST, MONGODB_PORT, pool_size=CONNECTION_POOL_SIZE)
db=getattr(self.mongoConnection, DEFAULT_MONGO_DB_NAME)
yield db.authenticate(MONGODB_USER, MONGODB_PASS)

It would be nice to know if authentication is successful. However, the database authenticate method does not return anything to let you know this. It only calls connection authenticate_with_nonce and that sets a variable db._authenticated to True when authenticated. But it is bad form for any external python script to check on the value of a member variable that starts with an underscore.

I suggest that you add a method like is_authenticated() to db that returns True or False. This way we know when the connection is ready for successful writing or other database interactions.

Pleas take a look at the code below. This was my best guess. I am sure I am doing something wrong here but switching from pymongo to txmongo has caused a 95% decrease in performance. I am wondering if there is way to use the txmongo.MonoConnection object global so I only need to connect to mongo once. Sorry I am still a bit of a noob to python, twisted and mongo. but basic idea is store udp packet for later processing.

from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor,defer
import txmongo
import struct
import string

x = 0

class Collector(DatagramProtocol):
def init(self):
print "Initializing Collector";

@defer.inlineCallbacks
def datagramReceived(self, data, (host, port)):
    #print "Recieved Packet"
    ch = yield txmongo.MongoConnection()
    dbh = ch.flowpackets.packets
    #print "received %r from %s:%d  type:%s" % (data, host, port,type(data))
    #self.transport.write(data, (host, port))
    global x
    packet = dict()
    packet = { 'packet': data.encode('hex')}
    dbh.insert(packet)
    x += 1

reactor.listenUDP(2055, Collector())
print "Starting Reactor"
reactor.run()
print "Packet Count: ", x

Hi,
I would really like to benefit from the latest version including PR #49 but it seems like it was not updated on pypi yet https://pypi.python.org/pypi/txmongo
Could you please push it?
Thanks a lot in advance!

In current code when factory lose connection, it queues all requests and finish them on successful reconnection. I think it is better to have possibility to specify the behaviour: to queue or raise exception for requests, that were called while retrying to connect. How do you think?
I guess it was removed from previous version. Why?

EDIT: clarify issue

Only the top one of these works:

import os
import sys
import txmongo
import pymongo.uri_parser
import pymongo
from twisted.internet import defer, reactor

mongo_uri = os.environ.get('MONGOLAB_URI')
mongo_creds = pymongo.uri_parser.parse_uri(mongo_uri)

if sys.argv[1] == 'normal':

    @defer.inlineCallbacks
    def example():
        conn = yield txmongo.MongoConnection(*mongo_creds['nodelist'][0])

        db = getattr(conn, mongo_creds['database'])  
        yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
        test = db.test  

        yield test.insert({'1':'1'})        
        docs = yield test.find(limit=10)
        for doc in docs:
            print doc


if sys.argv[1] == 'pool':

    @defer.inlineCallbacks
    def example():
        conn = yield txmongo.MongoConnectionPool(*mongo_creds['nodelist'][0])

        db = getattr(conn, mongo_creds['database']) 
        yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
        test = db.test  
        yield test.insert({'1':'1'})

        # fetch some documents
        docs = yield test.find(limit=10)
        for doc in docs:
            print doc

if sys.argv[1] == 'lazy':

    @defer.inlineCallbacks
    def example():
        conn = txmongo.lazyMongoConnection(*mongo_creds['nodelist'][0])

        db = getattr(conn, mongo_creds['database'])  # `foo` database
        yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
        test = db.test  # `test` collection

        yield test.insert({'1':'1'})        
        docs = yield test.find(limit=10)
        for doc in docs:
            print doc

if sys.argv[1] == 'lazypool':

    @defer.inlineCallbacks
    def example():
        conn = txmongo.lazyMongoConnectionPool(*mongo_creds['nodelist'][0])

        db = getattr(conn, mongo_creds['database'])  # `foo` database
        yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
        test = db.test  # `test` collection

        yield test.insert({'1':'1'})        
        docs = yield test.find(limit=10)
        for doc in docs:
            print doc


if __name__ == '__main__':
    example().addCallback(lambda ign: reactor.stop())
    reactor.run()```

The lazy methods fail with 'not connected' and the pooled method fail with 'auth fails'.

I want to use authentication inside a cyclone request, but I don't want to authenticate if it already has happened.  
I don't think txmongo supports this naturally at the moment.

Seems like there is no possibility to connect to a replica set, I mean like this:
http://api.mongodb.org/python/1.9%2B/examples/replica_set.html#id1
Am I right?

Is there any way to issue a custom command (like findAndModify) through public txmongo API? The only solution I found so far is via private methods:

http://pastie.org/pastes/809589

There is no minimum (or recommended minimum) mongo version. While this probably supports the wire protocol down to 0 like the pymongo package, the tests and examples reference features that aren't in legacy versions.

I realized this because I accidentally spun up a old 2.1 version, and the example/tests all failed.

Hello. A quick support request on how to transform BSON object to json.

At the moment I'm doing following (result is return result from collection.find_one({})
d = result[0]
del d['_id']
json.dumps(d)

I have to del '_id' , because otherwise json cannot be serialized from objectid. I've found BSON class in _pymongo but couldn't figure out how to use. Can you help?

It seems like GridFS.get and .put don't work at all...

from twisted.internet import reactor
import txmongo
from txmongo.gridfs import GridFS

d = txmongo.MongoConnectionPool()

def gotDB(db):
    files = GridFS(db.fs, collection='foo')
    print files
    return files


def gotFiles(files):
    obj_id = files.put('some contents', filename='foo', content_type='image/png')
    print 'saved', repr(obj_id)
    obj = files.get(str(obj_id))
    print 'got', repr(obj)
    print dir(obj)
    print obj.read()
    print obj.name
    print obj.content_type

d.addCallback(gotDB)
d.addCallback(gotFiles)

reactor.run()

gives me this:

<txmongo._gridfs.GridFS object at 0x12885d0>
saved ObjectId('4fe84a9bf648c2b4df000001')
got 4fe84a9bf648c2b4df000001
['_GridOut__buffer', '_GridOut__chunks', '_GridOut__current_chunk', '_GridOut__position', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattr__', '__getattribute__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_file', '_id', 'aliases', 'chunk_size', 'close', 'content_type', 'length', 'md5', 'metadata', 'name', 'read', 'seek', 'tell', 'upload_date']
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 464, in _startRunCallbacks
    self._runCallbacks()
  File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "badmongo.py", line 19, in gotFiles
    print obj.read()
  File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 1187, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 1045, in _inlineCallbacks
    result = g.send(result)
  File "build/bdist.macosx-10.3-fat/egg/txmongo/_gridfs/grid_file.py", line 345, in read

  File "build/bdist.macosx-10.3-fat/egg/txmongo/_gridfs/grid_file.py", line 331, in __getattr__

exceptions.AttributeError: GridOut object has no attribute 'length'
<Deferred at 0x128aee0 current result: <twisted.python.failure.Failure <type 'exceptions.AttributeError'>>>

I think it's because GridFS.put returns grid_file._id, and GridFS.get takes a file_id, but uses that id as if it were a document in this line:

return GridOut(self.__collection, file_id)

When I try to connect to GridFS on 2.6.3, I get an error while creating index on the "fs" collection.

The error is something like this:

pymongo.errors.OperationFailure: not authorized to create index on dbname.fs.chunks

Then I looked up the init.py file for the GridFS object and I found the offending line:
self.__chunks.create_index(filter.sort(ASCENDING("files_id") + ASCENDING("n")),
unique=True)
However it's not clear to me why this happens because finally it's an insert operation that takes place when we create_index. So does this mean that the authentication protocol I pushed earlier is now failing for 2.6.3? This driver works perfectly on 2.4.10

Any responses would be appreciated for those who might have observed similar behavior.

$ debian/rules binary
debian/rules:6: /usr/share/cdbs/1/rules/debhelper.mk: No such file or directory
debian/rules:7: /usr/share/cdbs/1/class/python-distutils.mk: No such file or directory
make: *** No rule to make target `/usr/share/cdbs/1/class/python-distutils.mk'. Stop.

resolved with "apt-get install cdbs"

The package requires twisted, but does not specify a minimum version or the known target/working version.

Hi. I just cloned the txmongo repository and I'm trying to run the tests. In test_queries.py, in TestInsertMany, I'm getting two FAILs for test_UnorderedUnack and test_OrderedUnack.

I'm using python3.5 with mongo3.2.3. Any clues?. Thanks.

Im getting the following ran twistd -ny cyclone_server.tac and test_rest.sh

2010-09-14 15:44:20+0800 [MongoProtocol,client] Unhandled Error
Traceback (most recent call last):
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 345, in errback
self._startRunCallbacks(fail)
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 424, in _startRunCallbacks
self._runCallbacks()
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, _args, *_kw)
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 949, in gotResult
_inlineCallbacks(r, g, deferred)
--- ---
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 891, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/python/failure.py", line 338, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
exceptions.TypeError: exceptions must be classes, or instances, not type

pymongo supports this it seems.

http://api.mongodb.org/python/1.0/pymongo.master_slave_connection-pysrc.html

Hi,

I'm using txmongo in an environment with a lot of running parallel tasks (twitter and rss collection mostly) and was wondering which best practice would be recommended to use txmongo :

• sometimes my connexion just seem to hang, how do I check whether it's still up?
• if I want to briefly open a new connexion before closing it again, should I close my conenxion afterwards? How? (conn[db]._Database__factory.doStop() seems to do the trick but does it really?)
• which one of the lazyMongoConnexionPool or MongoConnectionPool should I use? Looking at the code I'm not sure to get the difference here

So it will be possible to install it with pip install txmongo.

The problem:

When running in windows the following error is received:

Traceback (most recent call last): File "webmain.tac", line 4, in import cyclone.web File "C:\Python27\lib\site-packages\cyclone\web.py", line 21, in from cyclone import __version__, escape, template, httpserver File "C:\Python27\lib\site-packages\cyclone\httpserver.py", line 24, in import fcntl ImportError: No module named fcntl

Proposed solution

it seems that fcntl is not required in httpserver since there are no refernces to it.
removing it solved the problem

The PyMongo driver supports finding fields as either list or dict. They internally convert list to dict if they receive a list.
This would be helpful as I would very much like to specify both exclusions and inclusions on the field parameter.

Currently txmongo raises Exception because the field isn't a list. But it looks like you are converting to dict internally too, so I think you can just add

[code]
if not isinstance(fields, dict):
[/code]

before converting.

I'm in the process of learning txmongo so I went to the examples (I haven't found other docs, am I correct in thinking there aren't any for the current release?).

The examples tend to call one of the following:

• MongoConnection
• MongoConnectionPool
• lazyMongoConnection
• lazyMongoConnectionPool
  and all with no arguments.

So I started out trying to do that, but error-ing out because they all want 2 args (plus self)

After tracing I worked out that these are defined in a block at the end of connection.py, and are wrapped by comments listing them as "Legacy Wrapper". They all equate back to MongoConnection, which has an __init__ that requires a host and port argument (no defaults), turning them into a uri. That init in-turn calls ConnectionPool's __init__ with has no required agreements (because it defaults to a uri of "mongodb://127.0.0.1:27017")

The ConnectionPool class isn't exposed via the projects __init__.py file though it seems like that is what is being implied to be the proper way to open the library these days.

So for new projects what is the intended setup function?

Hi All

The title is my question. I have read some source codes and notice there is no a bit of info about replication support. 

Thanks.

Any particular reason that TxMongo's args order differ from PyMongo (e.g. find())? I'm migrating some code (and have other code that I'd prefer to have working on both platforms). Obviously, I'll update the overlapping code to use kwargs, but it was an unexpected discovery (and might catch others).

"fakeroot debian/rules binary" complained about python2.5 missing. Fixed, but it's unclear if my solution is too fragile.

An example would be lovely.

This will be the release thread for the 15.0 milestone.

If there are comments about the current release, please ask/post them here. :)

@fiorix We good to go?
@IlyaSkriblovsky Any additional things we should do before release, you've done a lot already for this project.

Docs:
http://www.mongodb.org/display/DOCS/Tailable+Cursors

Upstream:
mongodb/mongo-python-driver@66292c6

This would be very nice.

Perhaps this is a deliberate design choice, but shouldn't Collection.find_one() return None rather than {} when the document doesn't exist?

Hi, using txmongo for a while I hadn't updated for about a year and am discovering now the new versions.

I tried installing on a blank ubuntu machine both mongo3 (following new method given here) then pip install txmongo==0.6 and resulted in a non-functional install, as shown by the following error when running python examples/inlinecallbacks/query.py:

  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 424, in errback
    self._startRunCallbacks(fail)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 491, in _startRunCallbacks
    self._runCallbacks()
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 578, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1163, in gotResult
    _inlineCallbacks(r, g, deferred)
---  ---
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1105, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/tmp/test", line 7, in example
    docs = yield mongo.foo.test.find(limit=10)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1107, in _inlineCallbacks
    result = g.send(result)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/txmongo/collection.py", line 150, in find
    defer.returnValue([d.decode(as_class=as_class) for d in documents])
exceptions.TypeError: decode() got an unexpected keyword argument 'as_class'

For what I understand, it seems txmongo0.6 uses methods from bson2.7 while bson3 (which dropped some kwargs used in txmongo including 'as_class') is shipped with pymongo3 that is coming along naturally with txmongo.

My solution for now for my projects consists in installing first separately pymongo pinned to version 2.7 and then txmongo pinned to 0.6 in my requirements.txt but that can only be temporary.

I tried installing from git sources instead of pip but had the same issue, which makes sense since decode(as_class=as_class) is still being used line 167 of collection.py in the actual repo.

Otherwise a save of a document that does have an _id, but is not yet in the collection doesn't work:

diff -r d4e6d68c7815 txmongo/collection.py
--- a/txmongo/collection.py Wed Dec 22 15:26:53 2010 +0000
+++ b/txmongo/collection.py Wed Dec 22 15:41:34 2010 +0000
@@ -215,7 +215,7 @@

         objid = doc.get("_id")
         if objid:
-            return self.update({"_id": objid}, doc, safe=safe)
+            return self.update({"_id": objid}, doc, safe=safe, upsert=True)
         else:
             return self.insert(doc, safe=safe)

Docs:
http://www.mongodb.org/display/DOCS/Tailable+Cursors

Upstream:
mongodb/mongo-python-driver@66292c6

This would be very nice.

Hi

I noticed the count() function of collection returned floating value but pymongo return integer value which is correct.

Is it a bug?

Regards

gelin yan

When the fields argument in Collection.find() is an empty list ([]), the returned documents only contain the _id field when they should contain all of the fields. When the fields argument is None all fields in the documents are returned which is the expected behavior of setting fields to an empty list.

NOTE: This issue is mostly trivial but a slight annoyance.

Here's a diff:

@@ -83,11 +83,6 @@
         if not isinstance(limit, types.IntType):
             raise TypeError("limit must be an instance of int")
 
-        if fields is not None:
-            if not fields:
-                fields = ["_id"]
-            fields = self._fields_list_to_dict(fields)
-
         if isinstance(filter, (qf.sort, qf.hint, qf.explain, qf.snapshot)):
             spec = SON(dict(query=spec))
             for k, v in filter.items():

I believe that the _initializeProto is used incorrectly
It returns a deferred-generator, but we do not yield it (moreover I think yield would not work inside buildProtocol).

https://github.com/twisted/txmongo/blob/master/txmongo/connection.py#L51

There's a recursion in dataReceived method of protocol, which leads to exceptions.RuntimeError: maximum recursion deptch exceeded in __instancecheck__.

self.dataReceived method is called about more that thousand times.

The new authentication approach crashes with an exception in defer.py:
exceptions.TypeError: object of type 'generator' has no len()

The cause looks like this line of code in connect.py: method authenticate, around line 292:

yield defer.gatherResults(
(connection.authenticate(database, username, password) for connection in self.__pool),
consumeErrors=True
)
Looks like what is being passed into gatherResults is a generator, not a proper list, and defer.py doesn't like this. The function gatherResults is expecting a list of Deferred, not a generator.

Here's the full traceback:

Traceback (most recent call last):
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1213, in unwindGener
ator
return _inlineCallbacks(None, gen, Deferred())
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1070, in _inlineCall
backs
result = g.send(result)
File "/usr/local/pakedge/proxy/python_pakedge_proxy/local/lib/python2.7/site-packages/txmongo-0.6-py2.7.egg/txmongo/database.py", lin
e 100, in authenticate

File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1213, in unwindGener
ator
return _inlineCallbacks(None, gen, Deferred())
--- ---
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1070, in _inlineCall
backs
result = g.send(result)
File "/usr/local/pakedge/proxy/python_pakedge_proxy/local/lib/python2.7/site-packages/txmongo-0.6-py2.7.egg/txmongo/connection.py", l
ine 292, in authenticate

File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 867, in gatherResult
s
consumeErrors=consumeErrors)
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 793, in init
self.resultList = [None] * len(deferredList)
exceptions.TypeError: object of type 'generator' has no len()

On my development server which is running Ubuntu 12.04, with bundled twisted 11.1 and mongodb 2.04.

I keep getting the following error.

2012-06-07 02:18:29+0000 [MongoProtocol,client] <twisted.internet.tcp.Connector instance at 0xbcc882c> will retry in 10 seconds
2012-06-07 02:18:29+0000 [MongoProtocol,client] Stopping factory <txmongo._MongoFactory instance at 0xbcc880c>

Upon further investigation,

using lsof | grep mongod | grep TCP | wc -l

I got 448 established connection after restarting mongodb and my twisted app for 1/2 day.
Take note that this is a development server, so the only person using it is me. And the twisted app keeps getting restarted to update the latest changes. So to have 448 established connection seems crazy, it seems that the connection is not closed after each query.

Anyone facing the same issue?

Is there any plans to support authentication in txmongo?

Hi.
When I try to execute the example "insert.py", I get the next exception (not sure if only in insert):

Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
File "build/bdist.linux-i686/egg/txmongo/protocol.py", line 240, in handle

File "build/bdist.linux-i686/egg/txmongo/protocol.py", line 335, in handle_REPLY

File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback
self._startRunCallbacks(result)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
self._runCallbacks()
--- ---
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
current.result = callback(current.result, _args, *_kw)
File "build/bdist.linux-i686/egg/txmongo/connection.py", line 91, in _configureCallback

File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 624, in decode
self, as_class, tz_aware, uuid_subtype, compile_re)
exceptions.TypeError: function takes exactly 4 arguments (5 given)

I got this error in my application, so I decided to try with the examples.

I'm on de Debian Wheezy machine, with python 2.7 and Mongo 2.4.10.

Thanks for any help.

When use insert_may with ordered=False I expected have in BulkWriteError whith all duplicated keys but only return the last, how can i get all error (knowing data that induce error)

I'm really having a heck of a time getting the current version of the library to run, I've tried a build on debian linux and osx and in both locations I can't get my script to do anything but have a key error inside the library, so I've reverted to trying to verify that I have the library installed correctly by trying to get the test scripts to run.

If run on a machine with mongo on its normal port, should "make test" in the project directory succeed?

I'm getting different counts between debian (FAILED (failures=1, errors=82)) and osx (FAILED (failures=1, errors=75, successes=1)), but on both I'm getting almost complete failure.

So after a fresh clone, is a "make test" after a "sudo python setup.py install" working for you?

I have been thinking for quite a while about being able to run queries without loading the entire result set into memory. Here's what I came up with (view the Changes):

collection.find_with_cursor returns a deferred which results in a tuple: (docs, deferred). If there are no more documents remaining in the current query, a deferred will be returned which results in this tuple: ([], None)

For example:

docs, d = yield collection.find_with_cursor()
# collection.find(cursor=True) also works

while docs:
    # Do something with this page of docs here
    for i in docs:
        pass
    # Get the next set of docs from the query.
    docs, d = yield d

I have already implemented this (here), but let me know what you think, or if there are any improvements that could be made on this, then I will send over a pull request

Hey hey

It would be awesome if ObjectId's are nativeley (re)serializable. E.g. to embed them in json return values of json-rpc handlers. That way, one can avoid iterating over docs to replace certain objects with the string value ( or to write a specific handlers for the parser, if supported )

I did it by inheriting from str, making it serializable by anything which knows how to handle str. Just a couple of changes here and there to make it immutable

if you're interested, here comes a diff

diff --git a/txmongo/_pymongo/bson.py b/txmongo/_pymongo/bson.py
index aef08f8..7587ff1 100644
--- a/txmongo/_pymongo/bson.py
+++ b/txmongo/_pymongo/bson.py
@@ -416,6 +416,8 @@ def _element_to_bson(key, value, check_keys):
         full_length = struct.pack("<i", 8 + len(cstring) + len(scope))
         length = struct.pack("<i", len(cstring))
         return "\x0F" + name + full_length + length + cstring + scope
+    if isinstance(value, ObjectId):
+        return "\x07" + name + value.binary
     if isinstance(value, str):
         cstring = _make_c_string(value)
         length = struct.pack("<i", len(cstring))
@@ -429,8 +431,6 @@ def _element_to_bson(key, value, check_keys):
     if isinstance(value, (list, tuple)):
         as_dict = SON(zip([str(i) for i in range(len(value))], value))
         return "\x04" + name + _dict_to_bson(as_dict, check_keys)
-    if isinstance(value, ObjectId):
-        return "\x07" + name + value.binary
     if value is True:
         return "\x08" + name + "\x01"
     if value is False:
diff --git a/txmongo/_pymongo/objectid.py b/txmongo/_pymongo/objectid.py
index 1c8de98..4c43cea 100644
--- a/txmongo/_pymongo/objectid.py
+++ b/txmongo/_pymongo/objectid.py
@@ -41,7 +41,7 @@ def _machine_bytes():
     return machine_hash.digest()[0:3]


-class ObjectId(object):
+class ObjectId( str ):
     """A Mongo ObjectId.
     """

@@ -50,7 +50,7 @@ class ObjectId(object):

     _machine_bytes = _machine_bytes()

-    def __init__(self, oid=None):
+    def __new__(klass, oid=None):
         """Initialize a new ObjectId_.

         If `oid` is ``None``, create a new (unique)
@@ -70,11 +70,16 @@ class ObjectId(object):
         .. _ObjectId: http://www.mongodb.org/display/DOCS/Object+IDs
         """
         if oid is None:
-            self.__generate()
+            oid = klass._generate()
         else:
-            self.__validate(oid)
+            oid = klass._validate(oid)

-    def __generate(self):
+        self = str.__new__( klass, oid.encode("hex") )
+        self.__id = oid
+        return self
+
+    @classmethod
+    def _generate(klass):
         """Generate a new value for this ObjectId.
         """
         oid = ""
@@ -94,9 +99,10 @@ class ObjectId(object):
         ObjectId._inc = (ObjectId._inc + 1) % 0xFFFFFF
         ObjectId._inc_lock.release()

-        self.__id = oid
+        return oid

-    def __validate(self, oid):
+    @classmethod
+    def _validate(self, oid):
         """Validate and use the given id for this ObjectId.

         Raises TypeError if id is not an instance of (str, ObjectId) and
@@ -106,13 +112,13 @@ class ObjectId(object):
           - `oid`: a valid ObjectId
         """
         if isinstance(oid, ObjectId):
-            self.__id = oid.__id
+            return oid.__id
         elif isinstance(oid, basestring):
             if len(oid) == 12:
-                self.__id = oid
+                return oid
             elif len(oid) == 24:
                 try:
-                    self.__id = oid.decode("hex")
+                    return oid.decode("hex")
                 except TypeError:
                     raise InvalidId("%s is not a valid ObjectId" % oid)
             else:
@@ -154,9 +160,6 @@ class ObjectId(object):
         return datetime.datetime.utcfromtimestamp(t)
     generation_time = property(generation_time)

-    def __str__(self):
-        return self.__id.encode("hex")
-
     def __repr__(self):
         return "ObjectId('%s')" % self.__id.encode("hex")

The problem is that when the fist attempt to connect to db on application start failed (e.g. mongo was down) - reconnector fails to connect when db is up in some some (but tries to do reconnection). Possibly the problem is in ClientReconnectingFactory itself. Can you give some directions or a workaround for that?

Hi

I notice uri type assertion in class Connection Pool.init has been changed from
assert isinstance(uri, basestring) to assert isinstantce(uri, str) like following:

ConnectionPool(object):
def init(self, uri="mongodb://127.0.0.1:27017", pool_size=1, ssl_context_factory=None, **kwargs):
assert isinstance(uri, str)

when I use an unicode string, The Connection pool init will fail.

Is it possible to use basestring back?

Regards

gelin yan

In my logs, I get the following:

2016-03-01 16:53:43.398197 +0100 ERROR [] 'Connection lost.'
2016-03-01 16:53:43.399087 +0100 ERROR [] 'Connection lost.'

which come about when there aren't any mongodbs left alive:
https://github.com/twisted/txmongo/blob/master/txmongo/protocol.py#L325

Normally this would be fine, but for people reading logs (for the fun of it?), they won't have any context as to what is going on so we should provide better error messages such as:
'TxMongo lost connection to MongoDB.

It would probably not be so bad an idea to have something in connectionReady that say that TxMongo is ready as some TxMongo users (like me) put all all the twisted protocol log.msg to DEBUG to help cut down on log spam (3 messages per attempt to re-connect).

Hi. I'm testing the following code using txmongo 16.1.0. Basically, I'm sending 1001 objects to an insert_many. It's raising pymongo.errors.OperationFailure: TxMongo: command SON ... ('writeConcern', {})]) on namespace foo.$cmd failed with 'exceeded maximum write batch size of 1000'

import time
import txmongo
from twisted.internet import defer, reactor


@defer.inlineCallbacks
def example():
    mongo = yield txmongo.connection.ConnectionPool('mongodb://127.0.0.1:27018')

    foo = mongo.foo  # `foo` database
    test = foo.test  # `test` collection

    # insert some data
    insert_data = []
    for x in range(1001):
        insert_data.append({"something": x*time.time()})

    result = yield test.insert_many(insert_data)

if __name__ == '__main__':
    example().addCallback(lambda ign: reactor.stop())
    reactor.run()

Mongo has a limit of 1000 for Bulk operations, but drivers usually handle this by dividing batch operations. Pymongo does not fail with many more objects.