Code Monkey home page Code Monkey logo

txmongo's Introduction

TxMongo

https://travis-ci.org/twisted/txmongo.png?branch=master https://coveralls.io/repos/twisted/txmongo/badge.svg?branch=master https://badge.fury.io/py/txmongo.png Documentation Status

TxMongo is an asynchronous Python/Twisted driver for MongoDB that implements the wire protocol on non-blocking sockets. The API derives from the original PyMongo.

Compatibility

Python 2.7, 3.3+ and PyPy MongoDB 2.6+

Installing

You can use setuptools to install:

`sh sudo python setup.py install `

Docs and examples

Generate them with make docs. You will need sphinx installed. There are some examples in the examples/ directory.

Hacking

Run make env to create clean hacking environment with virtualenv. Run make to torture your code with tests and code style tools.

Take a look in Makefile for commonly used commands and tools we use to develop.

Packages

Debian

Packing for debian exists in debian/, you can build yourself a package (remember to update debian/changelog) if you make changes.

`sh dpkg-buildpackage -b `

Then look for the package in your home directory.

Fedora

`sh rpmbuild -bb python-txmongo.spec `

You might need to download Source0 from the .spec and place it in rpmbuild/SOURCES first.

txmongo's People

Contributors

aleksi avatar amferraz avatar aschobel avatar chergert avatar dynamikdev avatar evert2410 avatar fiorix avatar flanked avatar gleicon avatar glyph avatar gowee avatar hawkowl avatar ilyaskriblovsky avatar jdavisp3 avatar lud4ik avatar oddjobz avatar psi29a avatar sevikkk avatar shylent avatar silas avatar stevenkaras avatar stiletto avatar timgates42 avatar toby avatar tonal avatar touilleman avatar trenton42 avatar trupheenix avatar virajkanwade avatar xadhoom avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

txmongo's Issues

GEO2D and ensure_index Support

I was wondering about GEO2D indexes and ensure_index support. As far as I can see, this is not supported in the current txmongo, but I am not sure how hard it would be to support. When I try to create_index a GEO2D index, I get the following exception:

sort_fields must be an instance of filter.sort

Trying to use ensure_index failed as well - this would be a nice to have.

Let me know if I can help in any way. I have a good knowledge of MongoDB but I am not that well-versed in Twisted so I didn't want to just fork start hacking, but I'll be happy to look at it if you point me in the right direction.

No proper way of knowing whether authentication is successful

When using txmongo, we use code like this:

mongoConnection=yield txmongo.MongoConnection(MONGODB_HOST, MONGODB_PORT, pool_size=CONNECTION_POOL_SIZE)
db=getattr(self.mongoConnection, DEFAULT_MONGO_DB_NAME)
yield db.authenticate(MONGODB_USER, MONGODB_PASS)

It would be nice to know if authentication is successful. However, the database authenticate method does not return anything to let you know this. It only calls connection authenticate_with_nonce and that sets a variable db._authenticated to True when authenticated. But it is bad form for any external python script to check on the value of a member variable that starts with an underscore.

I suggest that you add a method like is_authenticated() to db that returns True or False. This way we know when the connection is ready for successful writing or other database interactions.

performance issues

Pleas take a look at the code below. This was my best guess. I am sure I am doing something wrong here but switching from pymongo to txmongo has caused a 95% decrease in performance. I am wondering if there is way to use the txmongo.MonoConnection object global so I only need to connect to mongo once. Sorry I am still a bit of a noob to python, twisted and mongo. but basic idea is store udp packet for later processing.


from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor,defer
import txmongo
import struct
import string

x = 0

class Collector(DatagramProtocol):
def init(self):
print "Initializing Collector";

@defer.inlineCallbacks
def datagramReceived(self, data, (host, port)):
    #print "Recieved Packet"
    ch = yield txmongo.MongoConnection()
    dbh = ch.flowpackets.packets
    #print "received %r from %s:%d  type:%s" % (data, host, port,type(data))
    #self.transport.write(data, (host, port))
    global x
    packet = dict()
    packet = { 'packet': data.encode('hex')}
    dbh.insert(packet)
    x += 1

reactor.listenUDP(2055, Collector())
print "Starting Reactor"
reactor.run()
print "Packet Count: ", x

Optional queueing of requests while disconnecting

In current code when factory lose connection, it queues all requests and finish them on successful reconnection. I think it is better to have possibility to specify the behaviour: to queue or raise exception for requests, that were called while retrying to connect. How do you think?
I guess it was removed from previous version. Why?

Authentication problems with lazy and pooled connections

EDIT: clarify issue

Only the top one of these works:

import os
import sys
import txmongo
import pymongo.uri_parser
import pymongo
from twisted.internet import defer, reactor

mongo_uri = os.environ.get('MONGOLAB_URI')
mongo_creds = pymongo.uri_parser.parse_uri(mongo_uri)

if sys.argv[1] == 'normal':

    @defer.inlineCallbacks
    def example():
        conn = yield txmongo.MongoConnection(*mongo_creds['nodelist'][0])

        db = getattr(conn, mongo_creds['database'])  
        yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
        test = db.test  

        yield test.insert({'1':'1'})        
        docs = yield test.find(limit=10)
        for doc in docs:
            print doc


if sys.argv[1] == 'pool':

    @defer.inlineCallbacks
    def example():
        conn = yield txmongo.MongoConnectionPool(*mongo_creds['nodelist'][0])

        db = getattr(conn, mongo_creds['database']) 
        yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
        test = db.test  
        yield test.insert({'1':'1'})

        # fetch some documents
        docs = yield test.find(limit=10)
        for doc in docs:
            print doc

if sys.argv[1] == 'lazy':

    @defer.inlineCallbacks
    def example():
        conn = txmongo.lazyMongoConnection(*mongo_creds['nodelist'][0])

        db = getattr(conn, mongo_creds['database'])  # `foo` database
        yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
        test = db.test  # `test` collection

        yield test.insert({'1':'1'})        
        docs = yield test.find(limit=10)
        for doc in docs:
            print doc

if sys.argv[1] == 'lazypool':

    @defer.inlineCallbacks
    def example():
        conn = txmongo.lazyMongoConnectionPool(*mongo_creds['nodelist'][0])

        db = getattr(conn, mongo_creds['database'])  # `foo` database
        yield db.authenticate(mongo_creds['username'], mongo_creds['password'])
        test = db.test  # `test` collection

        yield test.insert({'1':'1'})        
        docs = yield test.find(limit=10)
        for doc in docs:
            print doc


if __name__ == '__main__':
    example().addCallback(lambda ign: reactor.stop())
    reactor.run()```

The lazy methods fail with 'not connected' and the pooled method fail with 'auth fails'.

I want to use authentication inside a cyclone request, but I don't want to authenticate if it already has happened.  
I don't think txmongo supports this naturally at the moment.

No minimum mongo version

There is no minimum (or recommended minimum) mongo version. While this probably supports the wire protocol down to 0 like the pymongo package, the tests and examples reference features that aren't in legacy versions.

I realized this because I accidentally spun up a old 2.1 version, and the example/tests all failed.

Mongo BSON to json

Hello. A quick support request on how to transform BSON object to json.

At the moment I'm doing following (result is return result from collection.find_one({})
d = result[0]
del d['_id']
json.dumps(d)

I have to del '_id' , because otherwise json cannot be serialized from objectid. I've found BSON class in _pymongo but couldn't figure out how to use. Can you help?

GridOut object has no attribute X

It seems like GridFS.get and .put don't work at all...

from twisted.internet import reactor
import txmongo
from txmongo.gridfs import GridFS

d = txmongo.MongoConnectionPool()

def gotDB(db):
    files = GridFS(db.fs, collection='foo')
    print files
    return files


def gotFiles(files):
    obj_id = files.put('some contents', filename='foo', content_type='image/png')
    print 'saved', repr(obj_id)
    obj = files.get(str(obj_id))
    print 'got', repr(obj)
    print dir(obj)
    print obj.read()
    print obj.name
    print obj.content_type

d.addCallback(gotDB)
d.addCallback(gotFiles)

reactor.run()

gives me this:

<txmongo._gridfs.GridFS object at 0x12885d0>
saved ObjectId('4fe84a9bf648c2b4df000001')
got 4fe84a9bf648c2b4df000001
['_GridOut__buffer', '_GridOut__chunks', '_GridOut__current_chunk', '_GridOut__position', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattr__', '__getattribute__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_file', '_id', 'aliases', 'chunk_size', 'close', 'content_type', 'length', 'md5', 'metadata', 'name', 'read', 'seek', 'tell', 'upload_date']
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 464, in _startRunCallbacks
    self._runCallbacks()
  File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "badmongo.py", line 19, in gotFiles
    print obj.read()
  File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 1187, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/Volumes/FatMan/home/matt/venvs/things/lib/python2.6/site-packages/Twisted-12.1.0-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 1045, in _inlineCallbacks
    result = g.send(result)
  File "build/bdist.macosx-10.3-fat/egg/txmongo/_gridfs/grid_file.py", line 345, in read

  File "build/bdist.macosx-10.3-fat/egg/txmongo/_gridfs/grid_file.py", line 331, in __getattr__

exceptions.AttributeError: GridOut object has no attribute 'length'
<Deferred at 0x128aee0 current result: <twisted.python.failure.Failure <type 'exceptions.AttributeError'>>>

I think it's because GridFS.put returns grid_file._id, and GridFS.get takes a file_id, but uses that id as if it were a document in this line:

return GridOut(self.__collection, file_id)

GridFS issue on Mongo DB 2.6.3

When I try to connect to GridFS on 2.6.3, I get an error while creating index on the "fs" collection.

The error is something like this:

pymongo.errors.OperationFailure: not authorized to create index on dbname.fs.chunks

Then I looked up the init.py file for the GridFS object and I found the offending line:
self.__chunks.create_index(filter.sort(ASCENDING("files_id") + ASCENDING("n")),
unique=True)
However it's not clear to me why this happens because finally it's an insert operation that takes place when we create_index. So does this mean that the authentication protocol I pushed earlier is now failing for 2.6.3? This driver works perfectly on 2.4.10

Any responses would be appreciated for those who might have observed similar behavior.

allow creation of Debian packages

$ debian/rules binary
debian/rules:6: /usr/share/cdbs/1/rules/debhelper.mk: No such file or directory
debian/rules:7: /usr/share/cdbs/1/class/python-distutils.mk: No such file or directory
make: *** No rule to make target `/usr/share/cdbs/1/class/python-distutils.mk'. Stop.

resolved with "apt-get install cdbs"

test_UnorderedUnack and test_OrderedUnack tests fail

Hi. I just cloned the txmongo repository and I'm trying to run the tests. In test_queries.py, in TestInsertMany, I'm getting two FAILs for test_UnorderedUnack and test_OrderedUnack.

I'm using python3.5 with mongo3.2.3. Any clues?. Thanks.

Getting an Unhandled Error ( exceptions must be classes, or instances, not type)

Im getting the following ran twistd -ny cyclone_server.tac and test_rest.sh

2010-09-14 15:44:20+0800 [MongoProtocol,client] Unhandled Error
Traceback (most recent call last):
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 345, in errback
self._startRunCallbacks(fail)
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 424, in _startRunCallbacks
self._runCallbacks()
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, _args, *_kw)
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 949, in gotResult
_inlineCallbacks(r, g, deferred)
--- ---
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 891, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/raikage/envs/cyclone/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/python/failure.py", line 338, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
exceptions.TypeError: exceptions must be classes, or instances, not type

How to properly test and close a connexion ?

Hi,

I'm using txmongo in an environment with a lot of running parallel tasks (twitter and rss collection mostly) and was wondering which best practice would be recommended to use txmongo :

  • sometimes my connexion just seem to hang, how do I check whether it's still up?
  • if I want to briefly open a new connexion before closing it again, should I close my conenxion afterwards? How? (conn[db]._Database__factory.doStop() seems to do the trick but does it really?)
  • which one of the lazyMongoConnexionPool or MongoConnectionPool should I use? Looking at the code I'm not sure to get the difference here

No module named fcntl in httpserver.py when running in windows

The problem:

When running in windows the following error is received:

Traceback (most recent call last): File "webmain.tac", line 4, in import cyclone.web File "C:\Python27\lib\site-packages\cyclone\web.py", line 21, in from cyclone import __version__, escape, template, httpserver File "C:\Python27\lib\site-packages\cyclone\httpserver.py", line 24, in import fcntl ImportError: No module named fcntl

Proposed solution

it seems that fcntl is not required in httpserver since there are no refernces to it.
removing it solved the problem

Support for find fields as dict

The PyMongo driver supports finding fields as either list or dict. They internally convert list to dict if they receive a list.
This would be helpful as I would very much like to specify both exclusions and inclusions on the field parameter.

Currently txmongo raises Exception because the field isn't a list. But it looks like you are converting to dict internally too, so I think you can just add

[code]
if not isinstance(fields, dict):
[/code]

before converting.

Question: if [lazy]MongoConnection[Pool] are legacy connection methods what should be used now?

I'm in the process of learning txmongo so I went to the examples (I haven't found other docs, am I correct in thinking there aren't any for the current release?).

The examples tend to call one of the following:

  • MongoConnection
  • MongoConnectionPool
  • lazyMongoConnection
  • lazyMongoConnectionPool
    and all with no arguments.

So I started out trying to do that, but error-ing out because they all want 2 args (plus self)

After tracing I worked out that these are defined in a block at the end of connection.py, and are wrapped by comments listing them as "Legacy Wrapper". They all equate back to MongoConnection, which has an __init__ that requires a host and port argument (no defaults), turning them into a uri. That init in-turn calls ConnectionPool's __init__ with has no required agreements (because it defaults to a uri of "mongodb://127.0.0.1:27017")

The ConnectionPool class isn't exposed via the projects __init__.py file though it seems like that is what is being implied to be the proper way to open the library these days.

So for new projects what is the intended setup function?

Does txmongo support replication?

Hi All

The title is my question. I have read some source codes and notice there is no a bit of info about replication support. 

Thanks.

Args order

Any particular reason that TxMongo's args order differ from PyMongo (e.g. find())? I'm migrating some code (and have other code that I'd prefer to have working on both platforms). Obviously, I'll update the overlapping code to use kwargs, but it was an unexpected discovery (and might catch others).

Get a new release (15.0) out the door!

This will be the release thread for the 15.0 milestone.

If there are comments about the current release, please ask/post them here. :)

@fiorix We good to go?
@IlyaSkriblovsky Any additional things we should do before release, you've done a lot already for this project.

Compatibility issues with pymongo3 & bson3

Hi, using txmongo for a while I hadn't updated for about a year and am discovering now the new versions.

I tried installing on a blank ubuntu machine both mongo3 (following new method given here) then pip install txmongo==0.6 and resulted in a non-functional install, as shown by the following error when running python examples/inlinecallbacks/query.py:

  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 424, in errback
    self._startRunCallbacks(fail)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 491, in _startRunCallbacks
    self._runCallbacks()
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 578, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1163, in gotResult
    _inlineCallbacks(r, g, deferred)
---  ---
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1105, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/tmp/test", line 7, in example
    docs = yield mongo.foo.test.find(limit=10)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1107, in _inlineCallbacks
    result = g.send(result)
  File "/home/boo/.virtualenvs/hyphe/local/lib/python2.7/site-packages/txmongo/collection.py", line 150, in find
    defer.returnValue([d.decode(as_class=as_class) for d in documents])
exceptions.TypeError: decode() got an unexpected keyword argument 'as_class'

For what I understand, it seems txmongo0.6 uses methods from bson2.7 while bson3 (which dropped some kwargs used in txmongo including 'as_class') is shipped with pymongo3 that is coming along naturally with txmongo.

My solution for now for my projects consists in installing first separately pymongo pinned to version 2.7 and then txmongo pinned to 0.6 in my requirements.txt but that can only be temporary.

I tried installing from git sources instead of pip but had the same issue, which makes sense since decode(as_class=as_class) is still being used line 167 of collection.py in the actual repo.

Save should use upsert?

Otherwise a save of a document that does have an _id, but is not yet in the collection doesn't work:

diff -r d4e6d68c7815 txmongo/collection.py
--- a/txmongo/collection.py Wed Dec 22 15:26:53 2010 +0000
+++ b/txmongo/collection.py Wed Dec 22 15:41:34 2010 +0000
@@ -215,7 +215,7 @@

         objid = doc.get("_id")
         if objid:
-            return self.update({"_id": objid}, doc, safe=safe)
+            return self.update({"_id": objid}, doc, safe=safe, upsert=True)
         else:
             return self.insert(doc, safe=safe)

Collection.find() field selection: empty list only returns `_id` field in documents.

When the fields argument in Collection.find() is an empty list ([]), the returned documents only contain the _id field when they should contain all of the fields. When the fields argument is None all fields in the documents are returned which is the expected behavior of setting fields to an empty list.

NOTE: This issue is mostly trivial but a slight annoyance.

Here's a diff:

@@ -83,11 +83,6 @@
         if not isinstance(limit, types.IntType):
             raise TypeError("limit must be an instance of int")
 
-        if fields is not None:
-            if not fields:
-                fields = ["_id"]
-            fields = self._fields_list_to_dict(fields)
-
         if isinstance(filter, (qf.sort, qf.hint, qf.explain, qf.snapshot)):
             spec = SON(dict(query=spec))
             for k, v in filter.items():

Exception in new authentication code

The new authentication approach crashes with an exception in defer.py:
exceptions.TypeError: object of type 'generator' has no len()

The cause looks like this line of code in connect.py: method authenticate, around line 292:

yield defer.gatherResults(
(connection.authenticate(database, username, password) for connection in self.__pool),
consumeErrors=True
)
Looks like what is being passed into gatherResults is a generator, not a proper list, and defer.py doesn't like this. The function gatherResults is expecting a list of Deferred, not a generator.

Here's the full traceback:

Traceback (most recent call last):
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1213, in unwindGener
ator
return _inlineCallbacks(None, gen, Deferred())
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1070, in _inlineCall
backs
result = g.send(result)
File "/usr/local/pakedge/proxy/python_pakedge_proxy/local/lib/python2.7/site-packages/txmongo-0.6-py2.7.egg/txmongo/database.py", lin
e 100, in authenticate

File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1213, in unwindGener
ator
return _inlineCallbacks(None, gen, Deferred())
--- ---
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 1070, in _inlineCall
backs
result = g.send(result)
File "/usr/local/pakedge/proxy/python_pakedge_proxy/local/lib/python2.7/site-packages/txmongo-0.6-py2.7.egg/txmongo/connection.py", l
ine 292, in authenticate

File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 867, in gatherResult
s
consumeErrors=consumeErrors)
File "/usr/local/pakedge/proxy/python_pakedge_proxy/lib/python2.7/site-packages/twisted/internet/defer.py", line 793, in init
self.resultList = [None] * len(deferredList)
exceptions.TypeError: object of type 'generator' has no len()

Too Many Open Files on ubuntu 12.04

On my development server which is running Ubuntu 12.04, with bundled twisted 11.1 and mongodb 2.04.

I keep getting the following error.

2012-06-07 02:18:29+0000 [MongoProtocol,client] <twisted.internet.tcp.Connector instance at 0xbcc882c> will retry in 10 seconds
2012-06-07 02:18:29+0000 [MongoProtocol,client] Stopping factory <txmongo._MongoFactory instance at 0xbcc880c>

Upon further investigation,

using lsof | grep mongod | grep TCP | wc -l

I got 448 established connection after restarting mongodb and my twisted app for 1/2 day.
Take note that this is a development server, so the only person using it is me. And the twisted app keeps getting restarted to update the latest changes. So to have 448 established connection seems crazy, it seems that the connection is not closed after each query.

Anyone facing the same issue?

Error in insert.py

Hi.
When I try to execute the example "insert.py", I get the next exception (not sure if only in insert):

Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
File "build/bdist.linux-i686/egg/txmongo/protocol.py", line 240, in handle

File "build/bdist.linux-i686/egg/txmongo/protocol.py", line 335, in handle_REPLY

File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback
self._startRunCallbacks(result)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
self._runCallbacks()
--- ---
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
current.result = callback(current.result, _args, *_kw)
File "build/bdist.linux-i686/egg/txmongo/connection.py", line 91, in _configureCallback

File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 624, in decode
self, as_class, tz_aware, uuid_subtype, compile_re)
exceptions.TypeError: function takes exactly 4 arguments (5 given)

I got this error in my application, so I decided to try with the examples.

I'm on de Debian Wheezy machine, with python 2.7 and Mongo 2.4.10.

Thanks for any help.

insert_many with BulkWriteError

When use insert_may with ordered=False I expected have in BulkWriteError whith all duplicated keys but only return the last, how can i get all error (knowing data that induce error)

Should "make test" on the library succeed?

I'm really having a heck of a time getting the current version of the library to run, I've tried a build on debian linux and osx and in both locations I can't get my script to do anything but have a key error inside the library, so I've reverted to trying to verify that I have the library installed correctly by trying to get the test scripts to run.

If run on a machine with mongo on its normal port, should "make test" in the project directory succeed?

I'm getting different counts between debian (FAILED (failures=1, errors=82)) and osx (FAILED (failures=1, errors=75, successes=1)), but on both I'm getting almost complete failure.

So after a fresh clone, is a "make test" after a "sudo python setup.py install" working for you?

Experimental: expose the cursor in find() queries

I have been thinking for quite a while about being able to run queries without loading the entire result set into memory. Here's what I came up with (view the Changes):

collection.find_with_cursor returns a deferred which results in a tuple: (docs, deferred). If there are no more documents remaining in the current query, a deferred will be returned which results in this tuple: ([], None)

For example:

docs, d = yield collection.find_with_cursor()
# collection.find(cursor=True) also works

while docs:
    # Do something with this page of docs here
    for i in docs:
        pass
    # Get the next set of docs from the query.
    docs, d = yield d

I have already implemented this (here), but let me know what you think, or if there are any improvements that could be made on this, then I will send over a pull request

serializable ObjectId

Hey hey

It would be awesome if ObjectId's are nativeley (re)serializable. E.g. to embed them in json return values of json-rpc handlers. That way, one can avoid iterating over docs to replace certain objects with the string value ( or to write a specific handlers for the parser, if supported )

I did it by inheriting from str, making it serializable by anything which knows how to handle str. Just a couple of changes here and there to make it immutable

if you're interested, here comes a diff

diff --git a/txmongo/_pymongo/bson.py b/txmongo/_pymongo/bson.py
index aef08f8..7587ff1 100644
--- a/txmongo/_pymongo/bson.py
+++ b/txmongo/_pymongo/bson.py
@@ -416,6 +416,8 @@ def _element_to_bson(key, value, check_keys):
         full_length = struct.pack("<i", 8 + len(cstring) + len(scope))
         length = struct.pack("<i", len(cstring))
         return "\x0F" + name + full_length + length + cstring + scope
+    if isinstance(value, ObjectId):
+        return "\x07" + name + value.binary
     if isinstance(value, str):
         cstring = _make_c_string(value)
         length = struct.pack("<i", len(cstring))
@@ -429,8 +431,6 @@ def _element_to_bson(key, value, check_keys):
     if isinstance(value, (list, tuple)):
         as_dict = SON(zip([str(i) for i in range(len(value))], value))
         return "\x04" + name + _dict_to_bson(as_dict, check_keys)
-    if isinstance(value, ObjectId):
-        return "\x07" + name + value.binary
     if value is True:
         return "\x08" + name + "\x01"
     if value is False:
diff --git a/txmongo/_pymongo/objectid.py b/txmongo/_pymongo/objectid.py
index 1c8de98..4c43cea 100644
--- a/txmongo/_pymongo/objectid.py
+++ b/txmongo/_pymongo/objectid.py
@@ -41,7 +41,7 @@ def _machine_bytes():
     return machine_hash.digest()[0:3]


-class ObjectId(object):
+class ObjectId( str ):
     """A Mongo ObjectId.
     """

@@ -50,7 +50,7 @@ class ObjectId(object):

     _machine_bytes = _machine_bytes()

-    def __init__(self, oid=None):
+    def __new__(klass, oid=None):
         """Initialize a new ObjectId_.

         If `oid` is ``None``, create a new (unique)
@@ -70,11 +70,16 @@ class ObjectId(object):
         .. _ObjectId: http://www.mongodb.org/display/DOCS/Object+IDs
         """
         if oid is None:
-            self.__generate()
+            oid = klass._generate()
         else:
-            self.__validate(oid)
+            oid = klass._validate(oid)

-    def __generate(self):
+        self = str.__new__( klass, oid.encode("hex") )
+        self.__id = oid
+        return self
+
+    @classmethod
+    def _generate(klass):
         """Generate a new value for this ObjectId.
         """
         oid = ""
@@ -94,9 +99,10 @@ class ObjectId(object):
         ObjectId._inc = (ObjectId._inc + 1) % 0xFFFFFF
         ObjectId._inc_lock.release()

-        self.__id = oid
+        return oid

-    def __validate(self, oid):
+    @classmethod
+    def _validate(self, oid):
         """Validate and use the given id for this ObjectId.

         Raises TypeError if id is not an instance of (str, ObjectId) and
@@ -106,13 +112,13 @@ class ObjectId(object):
           - `oid`: a valid ObjectId
         """
         if isinstance(oid, ObjectId):
-            self.__id = oid.__id
+            return oid.__id
         elif isinstance(oid, basestring):
             if len(oid) == 12:
-                self.__id = oid
+                return oid
             elif len(oid) == 24:
                 try:
-                    self.__id = oid.decode("hex")
+                    return oid.decode("hex")
                 except TypeError:
                     raise InvalidId("%s is not a valid ObjectId" % oid)
             else:
@@ -154,9 +160,6 @@ class ObjectId(object):
         return datetime.datetime.utcfromtimestamp(t)
     generation_time = property(generation_time)

-    def __str__(self):
-        return self.__id.encode("hex")
-
     def __repr__(self):
         return "ObjectId('%s')" % self.__id.encode("hex")

First connect fail causes txmongo to never reconnect again

The problem is that when the fist attempt to connect to db on application start failed (e.g. mongo was down) - reconnector fails to connect when db is up in some some (but tries to do reconnection). Possibly the problem is in ClientReconnectingFactory itself. Can you give some directions or a workaround for that?

connection uri doesn't support unicode in python 2.x

Hi

I notice uri type assertion in class Connection Pool.init has been changed from
assert isinstance(uri, basestring) to assert isinstantce(uri, str) like following:

ConnectionPool(object):
def init(self, uri="mongodb://127.0.0.1:27017", pool_size=1, ssl_context_factory=None, **kwargs):
assert isinstance(uri, str)

when I use an unicode string, The Connection pool init will fail.

Is it possible to use basestring back?

Regards

gelin yan

Logging, we can do better.

In my logs, I get the following:

2016-03-01 16:53:43.398197 +0100 ERROR [] 'Connection lost.'
2016-03-01 16:53:43.399087 +0100 ERROR [] 'Connection lost.'

which come about when there aren't any mongodbs left alive:
https://github.com/twisted/txmongo/blob/master/txmongo/protocol.py#L325

Normally this would be fine, but for people reading logs (for the fun of it?), they won't have any context as to what is going on so we should provide better error messages such as:
'TxMongo lost connection to MongoDB.

It would probably not be so bad an idea to have something in connectionReady that say that TxMongo is ready as some TxMongo users (like me) put all all the twisted protocol log.msg to DEBUG to help cut down on log spam (3 messages per attempt to re-connect).

Txmongo is unable to handle more than 1000 objects in bulk operations

Hi. I'm testing the following code using txmongo 16.1.0. Basically, I'm sending 1001 objects to an insert_many. It's raising pymongo.errors.OperationFailure: TxMongo: command SON ... ('writeConcern', {})]) on namespace foo.$cmd failed with 'exceeded maximum write batch size of 1000'

import time
import txmongo
from twisted.internet import defer, reactor


@defer.inlineCallbacks
def example():
    mongo = yield txmongo.connection.ConnectionPool('mongodb://127.0.0.1:27018')

    foo = mongo.foo  # `foo` database
    test = foo.test  # `test` collection

    # insert some data
    insert_data = []
    for x in range(1001):
        insert_data.append({"something": x*time.time()})

    result = yield test.insert_many(insert_data)

if __name__ == '__main__':
    example().addCallback(lambda ign: reactor.stop())
    reactor.run()

Mongo has a limit of 1000 for Bulk operations, but drivers usually handle this by dividing batch operations. Pymongo does not fail with many more objects.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.