Code Monkey home page Code Monkey logo

namlook / mongokit Goto Github PK

View Code? Open in Web Editor NEW
676.0 26.0 134.0 2.58 MB

MongoKit framework try to keep its simplicity when you manage mongodb in python. MongoKit was developed to be fast and light with KISS and DRY in mind. MongoKit brings structured schema and validation layer on top of the great pymongo driver. Discuss with us on Google group : http://groups.google.com/group/mongokit or follow the news on Twitter: http://twitter.com/namlook

Home Page: http://namlook.github.com/mongokit/

License: Other

Makefile 0.22% Python 99.78%

mongokit's Introduction

maintainers wanted ! See #182 for more infos.

MongoKit Build Status

MongoDB is a great schema-less document oriented database. It has a lot of drivers for many languages (python, ruby, perl, java, php...).

MongoKit is a python module that brings a structured schema and validation layer on top of the great pymongo driver. It has been written to be as simple and light as possible with the KISS and DRY principles in mind.

Philosophy

MongoKit is designed to be:

  • simple: MongoKit uses plain python types to describe document structure
  • fast: MongoKit is fast but if you really need to be fast you have access to the raw pymongo layer without changing the API
  • powerful: MongoKit brings many features like document auto-reference, custom types or i18n support.

Your data is clean:

"Tools change, not data". In order to follow this "credo", MongoKit won't add any information into your data saved into the database. So if you need to use other mongo tools or ODMs in other languages, your data won't be polluted by MongoKit's stuff.

Features

  • schema validation (which uses simple python types for the declaration)
  • schema-less feature
  • dot notation
  • nested and complex schema declaration
  • untyped field support
  • required fields validation
  • default values
  • custom validators
  • cross database document reference
  • random query support (which returns a random document from the database)
  • inheritance and polymorphism support
  • versionized document support (in beta stage)
  • partial auth support (it brings a simple User model)
  • operator for validation (currently : OR, NOT and IS)
  • simple web framework integration
  • import/export to json
  • i18n support
  • GridFS support
  • document migration support

Go to the full documentation

A quick example

Documents are enhanced python dictionaries with a validate() method. A Document declaration look as follows:

>>> from mongokit import *
>>> import datetime

>>> connection = Connection()

>>> @connection.register
... class BlogPost(Document):
...     structure = {
...             'title':unicode,
...             'body':unicode,
...             'author':unicode,
...             'date_creation':datetime.datetime,
...             'rank':int
...     }
...     required_fields = ['title','author', 'date_creation']
...     default_values = {'rank':0, 'date_creation':datetime.datetime.utcnow}
...

We establish a connection and register our objects.

>>> blogpost = con.test.example.BlogPost() # this uses the database "test" and the collection "example"
>>> blogpost['title'] = u'my title'
>>> blogpost['body'] = u'a body'
>>> blogpost['author'] = u'me'
>>> blogpost
{'body': u'a body', 'title': u'my title', 'date_creation': datetime.datetime(...), 'rank': 0, 'author': u'me'}
>>> blogpost.save()

Saving the object will call the validate() method.

And you can use a more complex structure as follows:

>>>  @connection.register
...  class ComplexDoc(Document):
...     __database__ = 'test'
...     __collection__ = 'example'
...     structure = {
...         "foo" : {"content":int},
...         "bar" : {
...             'bla':{'spam':int}
...         }
...     }
...     required_fields = ['foo.content', 'bar.bla.spam']

Please see the tutorial for more examples.

Suggestions and patches are really welcome. If you find mistakes in the documentation (English is not my primary language) feel free to contact me. You can find me (namlook) on the freenode #mongodb irc channel or on twitter

Recent Change Log

v0.9.1

  • fixed #131 - Use PEP8 recommendation for import
  • fixed tests (thanks @JohnBrodie and @bneron)
  • Added a Makefile for running tests in venv (thanks to @gulbinas)
  • fixed pep8 error (thanks to @gulbinas)
  • added support for MongoReplicaSetClient (thanks to @inabhi9)
  • Added __getstate__ and __setstate__ to DotedDict and i18nDotedDict. Problems appeared here when pickling mongokit documents due to apparent lack of these functions. (thanks to @petersng)
  • Fixed english mistake and typos into the documentation (thanks to @biow0lf, @SeyZ, @gianpaj and @1123)
  • Fixed inherited queries when accessing cursor by index (thanks to @asivokon)
  • changed the namespace on schema document errors (thanks to @rtjoseph11)

v0.9.0

  • now MongoKit requires PyMongo >= 2.5
  • find_and_modify returns None if the query fails (thanks to @a1gucis)
  • Fix off-by-one error on SchemaDocument (thanks to @John Brodie)
  • Fix inherited queries (issue #106) (thanks to @effem-git)
  • Fix for serialization of nested structures with type validation (thanks to @LK4D4)
  • Remove unnecessary path arguments in to_json._convert_to_python (thanks to @Alexandr Morozov)
  • big refactorization by using multiple inheritance for DRYness (thanks to @liyanchang)
  • Add find_fulltext method for convenience (thanks to @astronouth7303) (not official and not documented yet)
  • Allow text indexes in document definitions (thanks to @astronouth7303)
  • Adding replica set support (thanks to @liyanchang)
  • Fix typos on README (thanks to @girasquid)
  • add pagination helper (not yet documented)(thanks to @jarrodb) https://github.com/namlook/mongokit/blob/master/mongokit/paginator.py

v0.8.3

  • allow keyword arguments (like read_preferences, slave_okay, etc) to be set in Connection (thanks to @petersng)
  • Add find_and_modify again. It was removed by an unexpected rollback.
  • use MongoClient with MasterSlaveConnection

v0.8.2

  • fix #101 - validators condition fix
  • fix #110 - support PyMongo >= 2.4 (import MongoClient) -- thanks to @mattbodman and @zavatskiy
  • Fixed some spelling/grammar (thanks to @gekitsuu)

v0.8.1

  • support python 2.3
  • small updates to validation messages (Merge pull request #94 from unpluggd/master)
  • Fixes formatting error when throwing MaxDocumentSizeError in Document.validate() (Merge pull request #99 from apavlo/master)
  • Fixed typo when throwing MaxDocumentSizeError in validate() (thanks to Andy Pavlo)
  • added fix for unconditional access to `__wrap on cursors (thanks to David T. Lehmann)
  • added test for __getitem__ on cursor with undefined __wrap (thanks to David T. Lehmann)
  • __getitem__ on unwrapped cursor checks if __wrap is None (Merge pull request #97 from dtl/fix-getitem-on-unwrapped-cursor)
  • Add .travis.yml for Travis CI (http://travis-ci.org/) (Merge pull request #96 from msabramo/travis)
  • Fixed a very minor rendering issue in the docs (Merge pull request #95 from d0ugal/master)
  • Fixed rendering issue in the docs. (thanks to Dougal Matthews)
  • tweaked the error messages in validation for missing and unknown fields to aid in debugging projects (thanks to Phillip B Oldham)

v0.8.0

  • Add spec file for rpm-based distributions (Merge pull request #63 from linuxnow/master)
  • change document size limitation for mongodb 1.8 or later. Thanks to Aleksey Sivokon (Merge pull request #74 from key/master)
  • validation of "" for an int (Merge pull request #79 from barnybug/master)
  • Fix exception when loading documents with a custom type field missing (Merge pull request #80 from barnybug/master)
  • Big documentation restructuring made by Sean Lynch (Merge pull request #82 from sean-lynch/master)
  • Using rename no longer causes migrations throw an exception (Merge pull request #86 from matthewh/master)
  • Some test is modified and added tox (Merge pull request #91 from aircastle/modifiytest)
  • Replace pymongo.objectid with bson.objectid (Merge pull request #88 from behackett/master)
  • Added Support for additional keyword-arguments for index-creation (Merge pull request #85 from mfelsche/master)
  • Remove anyjson dependency and use builtin json instead

Thank you all for all your patches !

v0.7.2

v0.7.1

  • change MongokitMasterSlaveConnection to MasterSlaveConnection for consistency
  • fix #57 -- support pymongo > 1.9 in grid.py
  • fix #45 -- remove automatic index creation
  • fix #43 -- slicing a cursor should return a mongokit document, not dict
  • Dont try to convert None struct to json (patch from @mLewisLogic thanks !)
  • fix schemaless issue (thanks to Mihai Pocorschi for reporting it)

v0.7

  • add use_schemaless feature ! please see the documentation for more information
  • Add equality test for mongokit operators (thanks to @allancaffee) This allows developers to write unit tests on the structure of their document classes when operators are used
  • roll back find_and_modify for master branch (need pymongo 1.10 for that)
  • many documentation fixes
  • fix #55 -- Bug in VersionedDocument remove() method
  • fix #53 -- Fixed a few spelling errors in README
  • fix #52 -- Advanced bulk migration docs example is broken
  • fix #51 -- pymongo.dbref is deprecated, use bson.dbref instead
  • fix #49 -- Can't specify default values for lists of embedded objects
  • fix #48 -- uuid.UUID support
  • fix #41 -- add basestring to authorized types
  • fix #40 -- Made some enhancements
  • fix #39 -- KeyError when saving partially loaded documents
  • fix #34 -- add find_and_modify method to Document
  • fix #32 -- allow the structure to be empty (was: document.to_json())
  • fix #24 -- Don't handle __database__ and __collection__ attributes for virtual documents

mongokit's People

Contributors

1123 avatar a1gucis avatar ajessup avatar apavlo avatar behackett avatar biow0lf avatar bitmole avatar bneron avatar d0ugal avatar davidjb avatar dtl avatar eneepo avatar gianpaj avatar girasquid avatar gulbinas avatar idiot1347 avatar jarrodb avatar johnpaulett avatar kedar2a avatar key avatar last-g avatar liyanchang avatar matthewh avatar msabramo avatar osyvokon avatar pawciobiel avatar petersng avatar seyz avatar wasabi0522 avatar winnetou avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mongokit's Issues

typo in the doc

So if you need to use another mongo tools or another ODM in another languages, your data wonโ€™t be pollued by MongoKitโ€™s staff.

should be

So if you need to use other mongo tools or ODMs in other languages, your data wonโ€™t be polluted by MongoKitโ€™s stuff.

tag version

hello you should tag released version, it will automatically make an archive in the download section.

Don't prepopulate non-required fields

MongoKit tip, testcase in attachment:

gauss@pmain:~/Projects/mongokit_test$ ./test1.py 
Traceback (most recent call last):
  File "./test1.py", line 20, in 
    a.save()
  File "/home/gauss/python-site-packages/mongokit/document.py", line 362, in save
    self.validate(auto_migrate=False)
  File "/home/gauss/python-site-packages/mongokit/document.py", line 174, in validate
    super(Document, self).validate()
  File "/home/gauss/python-site-packages/mongokit/schema_document.py", line 290, in validate
    self._validate_doc(self, self.structure)
  File "/home/gauss/python-site-packages/mongokit/schema_document.py", line 538, in _validate_doc
    self._validate_doc(doc[key], struct[key],  new_path)
  File "/home/gauss/python-site-packages/mongokit/schema_document.py", line 549, in _validate_doc
    self._validate_doc(obj, struct, path)
  File "/home/gauss/python-site-packages/mongokit/schema_document.py", line 517, in _validate_doc
    "missed fields : %s" % struct_doc_diff )
  File "/home/gauss/python-site-packages/mongokit/schema_document.py", line 472, in _raise_exception
    raise exception(message)
mongokit.schema_document.StructureError: missed fields : ['a2']

But a2 isn't marked as required.

see http://groups.google.com/group/mongokit/browse_thread/thread/242aaaa73cb4b457

Can't specify default values for lists of embedded objects

I'm pretty new to MongoKit so please bear with me if I'm confused, but it doesn't seem to be possible to specify default values for embedded objects. Using this Document:

class UserInfo(Document):
    use_dot_notation = True
    structure = {
        'email_addresses': [{
            'email': unicode,
            'verified': bool,
        }],
    }
    required_fields = []
    default_values = {
        'email_addresses': [{
            'verified': True,
        }],
    }

raises this exception at object-instantiation time:

Traceback (most recent call last):
  File "./mongofun.py", line 3, in <module>
    from core.model import con
  File "...core/model.py", line 12, in <module>
    class UserInfo(Document):
  File ".../lib/python2.7/site-packages/mongokit/document.py", line 76, in __new__
    return SchemaProperties.__new__(cls, name, bases, attrs)        
  File ".../lib/python2.7/site-packages/mongokit/schema_document.py", line 154, in __new__
    cls._validate_descriptors(attrs)
  File ".../lib/python2.7/site-packages/mongokit/document.py", line 80, in _validate_descriptors
    SchemaProperties._validate_descriptors(attrs)
  File ".../lib/python2.7/site-packages/mongokit/schema_document.py", line 174, in _validate_descriptors
    raise ValueError("Error in default_values: can't find %s in structure" % dv )
ValueError: Error in default_values: can't find email_addresses in structure

Changing the required_fields parameter to this instead:

default_values = { 'email_addresses.verified': False, }

raises this exception at validation time, because MongoKit hasn't used the defaults to fill out the fields:

Traceback (most recent call last):
  File "./mongofun.py", line 23, in <module>
    user.validate()
  File ".../lib/python2.7/site-packages/mongokit/document.py", line 247, in validate
    super(Document, self).validate()
  File ".../lib/python2.7/site-packages/mongokit/schema_document.py", line 348, in validate
    self._validate_doc(self, self.structure)
  File ".../lib/python2.7/site-packages/mongokit/schema_document.py", line 585, in _validate_doc
    self._validate_doc(doc[key], struct[key],  new_path)
  File ".../lib/python2.7/site-packages/mongokit/schema_document.py", line 596, in _validate_doc
    self._validate_doc(obj, struct, path)
  File ".../lib/python2.7/site-packages/mongokit/schema_document.py", line 564, in _validate_doc
    "missed fields : %s" % struct_doc_diff )
  File ".../lib/python2.7/site-packages/mongokit/schema_document.py", line 519, in _raise_exception
    raise exception(message)
mongokit.schema_document.StructureError: missed fields : ['verified']

This issue has been brought up before, and the solution suggested there indicates that it's supposed to work:

https://groups.google.com/forum/#!topic/mongokit/ZjPjE9CgO0E

GEO2D index

MongoKit only allows INDEX_ASCENDING and INDEX_DESCENDING. However, since 1.5.1 mongodb also has a GEO2D index specifier. Please allow this.

A '_type' field that can be used to automatically case documents

Currently mongokit uses the structure dictionary to decide which class of document to cast to when autoreferencing. That can cause problems when you want to have a document autoreference another document, if that referenced document's class is not known ahead of time.
Below is a some code where verification fails because the Run document casts run.result into its base class when you save run.
Rather that using the structure property to cast an autoreferenced document to its proper class, why not include a _type field that is used to select which class to cast a document to. Along with that, a generic, mongokit 'document reference type' would need to be created to allow one document to reference another, without specifying what kind of document it is. When the document is loaded from the database, it would be automatically converted into the correct class based on the _type field.

http://gist.github.com/636106

Advanced bulk migration docs example is broken

class BlogPostMigration(DocumentMigration):
   def allmigration01__add_update_field_and_fill_it(self):
       self.target = {'blog_post.update_date':{'$exists':False}, 'blog_post':{'$exists':True}}
       if not self.status:
           for doc in self.collection.find(self.target):
               self.update = {'$set':{'blog_post.update_date': self.doc['blog_post']['creation_date']}}
               self.collection.update(self.target, self.update, multi=True, safe=True)

It's doc['blog_post']['creation_date'] not self.doc

unable to install (debian lenny, py 2.5)

cron:/# easy_install mongokit
Searching for mongokit
Reading http://pypi.python.org/simple/mongokit/
Reading http://bitbucket.org/namlook/mongokit/
Best match: mongokit 0.5.13
Downloading http://pypi.python.org/packages/source/m/mongokit/mongokit-0.5.13.tar.gz#md5=e8d65011e112571968c32d1a92a812ee
Processing mongokit-0.5.13.tar.gz
Running mongokit-0.5.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-xjR6j4/mongokit-0.5.13/egg-dist-tmp-BkZU97
Traceback (most recent call last):
File "/usr/bin/easy_install", line 8, in
load_entry_point('setuptools==0.6c12dev-r85381', 'console_scripts', 'easy_install')()
File "/usr/lib/python2.5/site-packages/easy_install.py", line 1712, in main

  File "/usr/lib/python2.5/site-packages/easy_install.py", line 1700, in with_ei_usage

  File "/usr/lib/python2.5/site-packages/easy_install.py", line 1716, in <lambda>

  File "/usr/lib/python2.5/distutils/core.py", line 151, in setup
dist.run_commands()
  File "/usr/lib/python2.5/distutils/dist.py", line 974, in run_commands
self.run_command(cmd)
  File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command
cmd_obj.run()
  File "/usr/lib/python2.5/site-packages/easy_install.py", line 211, in run

  File "/usr/lib/python2.5/site-packages/easy_install.py", line 446, in easy_install

  File "/usr/lib/python2.5/site-packages/easy_install.py", line 476, in install_item

  File "/usr/lib/python2.5/site-packages/easy_install.py", line 655, in install_eggs

  File "/usr/lib/python2.5/site-packages/easy_install.py", line 930, in build_and_install

  File "/usr/lib/python2.5/site-packages/easy_install.py", line 919, in run_setup

  File "/usr/lib/python2.5/site-packages/setuptools-0.6c12dev_r85381-py2.5.egg/setuptools/sandbox.py", line 62, in run_setup
  File "/usr/lib/python2.5/site-packages/setuptools-0.6c12dev_r85381-py2.5.egg/setuptools/sandbox.py", line 105, in run
  File "/usr/lib/python2.5/site-packages/setuptools-0.6c12dev_r85381-py2.5.egg/setuptools/sandbox.py", line 64, in <lambda>
  File "setup.py", line 66, in <module>
  File "/usr/lib/python2.5/distutils/core.py", line 151, in setup
dist.run_commands()
  File "/usr/lib/python2.5/distutils/dist.py", line 974, in run_commands
self.run_command(cmd)
  File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command
cmd_obj.run()
  File "/usr/lib/python2.5/site-packages/setuptools-0.6c12dev_r85381-py2.5.egg/setuptools/command/bdist_egg.py", line 167, in run
  File "/usr/lib/python2.5/distutils/cmd.py", line 333, in run_command
self.distribution.run_command(command)
  File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command
cmd_obj.run()
  File "/usr/lib/python2.5/site-packages/setuptools-0.6c12dev_r85381-py2.5.egg/setuptools/command/egg_info.py", line 170, in run
  File "/usr/lib/python2.5/site-packages/setuptools-0.6c12dev_r85381-py2.5.egg/setuptools/command/egg_info.py", line 379, in write_pkg_info
  File "/usr/lib/python2.5/distutils/dist.py", line 1076, in write_pkg_info
self.write_pkg_file(pkg_info)
  File "/usr/lib/python2.5/distutils/dist.py", line 1101, in write_pkg_file
file.write('Description: %s\n' % long_desc)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 1292: ordinal not in range(128)

Made some enhancements

I am bringing data into Mongo from an external process using YAML. Because of that, I needed to loosen up some of the validation that MongoKit was doing for me and have MongoKit fill in some gaps in my data. So I added the following config variables:

autofill_incomplete_records = False
allow_nonrequired_to_be_missing = False
allow_unspecified_fields = False
auto_promote_str_and_int = False

The first one is the major enhancement. It tells MongoKit to accept a sparse record from Mongo and just fill in the missing data for me. This turned out to be really easy to do since namlook already had most of the work done to make this work. This is great for creating initial data, and for automatically adding to your structure without having to do an actual migrate.

In my case, if my collection is empty, I drop a bunch of documents in with only the 'url' filled out. When I read through the records, MongoKit will fix them up and add everything I left out.

One set of data I am storing is a dict of attributes on a corresponding file. Some files have 5 attributes while others have 30. Some are numeric, some dates, etc. So I want to specify the types for those I care about, but I don't even want to list many of the attributes in my structure. This is where allow_nonrequired_to_be_missing and allow_unspecified_fields come in. If allow_nonrequired_to_be_missing is True, validate will not complain about a missing field unless it is on the required fields list. If allow_unspecified_fields in True, validate won't complain about extra fields it doesn't know about.

Finally, since the YAML output often contains strings rather than unicode, I added auto_promote_str_and_int. I don't really care if a field should be unicode but it actually contains a str. I also don't care if it is supposed to be long but actually has an int. So set this to True and MongoKit will accept str for unicode and int for long.

I only had to change about a dozen lines of code in schema_document.py and a few lines in document.py. I'm happy to share but I don't know what the standard procedure is for submitting possible enhancements. All of the new code still passes all of the unit tests that 0.6 passed so nothing should change for current users unless they turn on some of my switches. (I'm not married to any of my config names, BTW.)

Is there an easy way for me to submit my changes for review?

Cheers,

Scott Maxwell
Code Cobblers, Inc.

KeyError when using a builtin -> custom type mapping.

There's a test case for this issue here: https://gist.github.com/795871

It appears that using a custom type in a structure dict with a type as the key causes a key error to be raised. For example:

structure = {
    'dates' : {unicode: CustomDate()}
}

would cause an error on validation, but not:

structure = {
    'dates' : {unicode: datetime.datetime}
}

The exception raised is:

Traceback (most recent call last):
  File "testcase.py", line 32, in <module>
    doc.validate()
  File "/Users/allanc/externals/mongokit/mongokit/document.py", line 247, in validate
    super(Document, self).validate()
  File "/Users/allanc/externals/mongokit/mongokit/schema_document.py", line 347, in validate
    self._process_custom_type('bson', self, self.structure)
  File "/Users/allanc/externals/mongokit/mongokit/schema_document.py", line 656, in _process_custom_type
    self._process_custom_type(target, doc[key], struct[key], new_path, root_path)
  File "/Users/allanc/externals/mongokit/mongokit/schema_document.py", line 642, in _process_custom_type
    if not isinstance(doc[key], struct[key].python_type) and doc[key] is not None:
KeyError: <type 'unicode'>

Document.simplify() method

It would be useful if there was a .simplify() method on Document objects, which would recursively convert the object to the base data-types as defined in the document's structure, eg:

class Foo(Document):
    structure = {
      'title': unicode,
      'properties': {unicode:None}
     }

x = Foo()
x['title'] = u'my document'
x['properties'] = {u'active': True}
x.save()

print type(x) # mongokit.Document
print type(x['_id']) # pymongo.objectid.ObjectId
print type(x['properties']) # mongokit.helpers.DotedDict

y = x.simplify()
print type(x) # dict
print type(x['_id']) # unicode
print type(x['properties']) # dict

allow circular references

As auto referencing is currently implement, there can be no circular references.
This is not allowed currently: Doc1 referencing Doc2 and Doc2 referencing Doc1
Circular references are common in graph data structures. Currently, to implement circular references the user needs to implement their own dbref functions, and can not use the mongokit's auto referencing feature.
Graphs are a common data structure as can be seen in the numerous non-sql graph databases. Adding the ability to handle circular references in mongokit would greatly enhance its usefulness.

Structure aliases to save disk space

from Christian Joudrey
We could add a dict called "structure_aliases" to Document which would serve as an alias when performing writes and reads.
For instance:

class User(Document):
    structure = {
      'u': unicode,
      'p': unicode
    }
    required_fields = [
      'u', 'p'
    ]
    structure_aliases = {
      'u': 'username',
      'p': 'password'
    }

The size of a collection can dramatically increase in size when there are many columns with long names.
Structure aliases would allow one to perform read and writes using "username" and "password" as column names, but they would really be stored as "u" and "p" in MongoDb thus saving (in theory) 14 bytes per row.

VersionedDocument.get_last_revision_id raises StopIteration

The type of strings used in a pymongo collection.find() need to match. The original code doesn't find just inserted IDs, because it's not looking for a unicode string.
The one liner patch (excuse the cut/paste) below corrects the issue for me (it now finds the relevant record).
StopIteration occurs due to the generator iterating over an empty query result set when its .next() method is called.
The test did not fail before; I'm presently working on modifying the test so that it fails before the patch, below, is applied. I'll send a pull request at that time if you'd prefer.

$ hg diff diff -r 43dd8afd9305 mongokit/versioned_document.py --- a/mongokit/versioned_document.py  Mon Aug 09 16:12:14 2010 +0200 +++ b/mongokit/versioned_document.py Thu Aug 26 23:53:36 2010 +1000 @@ -97,7 +97,7 @@
def get_last_revision_id(self): last_doc = self.versioning_collection.find(
- {'id':self['_id']}).sort('revision', -1).next()
+ {'id': unicode(self['_id'])}).sort('revision', -1).next() if last_doc: return last_doc['revision']

typo in the doc

hello,
i think this is a bug:
default_values = {'blog_post.created_at':datetime.utcnow()}

should be datetime.utcnow (the callable)

otherwise all document will have the same date

Remove old doc (was: ImportError: No module named ext.pylons_env)

When I try to setup pylons with mongokit I get the error

from mongokit.ext.pylons_env import MongoPylonsEnv
ImportError: No module named ext.pylons_env

from the line:

from mongokit.ext.pylons_env import MongoPylonsEnv

from this document: http://bytebucket.org/namlook/mongokit/wiki/html/pylons_support.html
Have mongokit and mongokit-pylons not been fully merged yet? Though I tried using the origin mongokit-pylons and i still get the same error. Any help would be appreciated.

put string version into the setup.py

This is necessary to prevent some buildout issues:

via Andrew Degtiariov:

I have pymongo 1.9.0, but there is buildout.
It overrides python path:

#!/opt/python/bin/python -S

import sys
sys.path[0:0] = [
    '/opt/project/parts/buildout',
    ]


import os
path = sys.path[0]
if os.environ.get('PYTHONPATH'):
    os.environ['BUILDOUT_ORIGINAL_PYTHONPATH'] = os.environ['PYTHONPATH']
    path = os.pathsep.join([path, os.environ['PYTHONPATH']])
os.environ['PYTHONPATH'] = path
import site # imports custom buildout-generated site.py

import zc.buildout.buildout

if __name__ == '__main__':
    zc.buildout.buildout.main()

pymongo installed into /opt/project/eggs directory, so ./bin/buildout known nothing about pymongo during processing setup.py.
Could you make version independent from external imports? E.g. could you make it as string as you was doing in previous versions of MongoKit?

Allow custom cursor class

At the moment all Document classes must use MongoDocumentCursor.

It should be possible to instead have a cursor_class class property (with MongoDocumentCursor as default) to allow a custom cursor class to be used instead - for example like the "objects" property in the Django ORM.

allow the structure to be empty (was: document.to_json() failing)

Traceback (most recent call last):
  ...
  File "eggs/mongokit-0.5.13.1-py2.6.egg/mongokit/document.py", line 508, in to_json
    _convert_to_python(obj, self.structure)
  File "eggs/mongokit-0.5.13.1-py2.6.egg/mongokit/document.py", line 485, in _convert_to_python
    _convert_to_python(doc[key], struct[key], new_path, root_path)
  File "eggs/mongokit-0.5.13.1-py2.6.egg/mongokit/document.py", line 481, in _convert_to_python
    new_path = ".".join([path, new_key]).strip('.')
TypeError: sequence item 1: expected string, type found

>>> import mongokit
>>> print mongokit.__version__
0.5.13.1

Enable `with` for object assignment

via Phillip Oldham:

Would it be possible to enable mongokit to use the python with statement?

For instance:

>>> from mongokit import *
>>> class MyObj(MongoDocument):
...   structure = {
...     "settings": { unicode: OR(unicode, bool, int) }
...   }
... 
>>> document = MyObj()
>>> with document.settings:
...   foo = 'bar'
... 
>>> print document.settings.foo
bar

Not completely sure that's accurate, but even using:

>>> with document.settings as s: 
...   s.foo = 'bar'

would help keep things a little neater when working with larger objects/structures.

Populating a Document using the constructor causes problems with migrations

Sample code:

from mongokit import Connection, Document, DocumentMigration

class TestDocMigration(DocumentMigration):
  pass

class TestDoc(Document):
  structure = {'name': unicode}
  migration_handler = TestDocMigration

con = Connection()
con.register([TestDoc])

collection = con['test'].foos

a = collection.TestDoc()
a['name'] = a

b = collection.TestDoc({'name': 'b'})

The latter syntax (with b) doesn't seem to be documented, but it works until you add migrations to your document. Once you add a migration handler, a KeyError is raised because MongoKit thinks the document has already been saved to the db:

...
File "python2.6/site-packages/mongokit/document.py", line 132, in _migrate
  old_doc = self.collection.get_from_id(self['_id'])
KeyError: '_id'

I suppose the solution is to either (a) discourage the use of the constructing documents with an initial structure โ€” I'm not sure how I started doing that since it doesn't seem to be documented anyway โ€” or (b) prevent migration of documents which haven't yet been saved.

MongoKit is great. Keep up the good work!

got an unexpected keyword argument 'from_son'

i got this exception

Exception Value:
init() got an unexpected keyword argument 'from_son'
Exception Location: /usr/local/lib/python2.6/dist-packages/mongokit-0.5.12-py2.6.egg/mongokit/database.py in _fix_outgoing, line 73

it seems Document constructor doesn't have this argument:
def init(self, doc=None, gen_skel=True, collection=None, lang='en', fallback_lang='en', generate_index=True):

creating index on fields not in structure

i have a structure like this:
structure = { 'meta': dict }

when i set indexes like this:

indexes = [{'field': 'meta.title', 'checked': False}]

i get the following exception:
ValueError: Error in indexes: can't find meta.title in structure

i thought 'checked': False would prevent this?

i can't put 'title' in the structure for 2 reason:

  1. it's optionnal
  2. it would conflict with the fact that meta contains a dict of arbitrary key/value

required_fields weird behavior with autorefs

via Peter Bengtsson

I've upgraded to Pymongo 1.9 and MongoKit 0.5.13-dev and this code
used to work just fine before. Here's my structure (simplified)::

class BaseDocument(Document):
   structure = {
     'add_date': datetime.datetime,
     'modify_date': datetime.datetime,
   }

   default_values = {
     'add_date': datetime.datetime.now,
     'modify_date': datetime.datetime.now
   }
   use_autorefs = True
   use_dot_notation = True

class User(BaseDocument):
   structure = {
     'guid': unicode,
     'username': unicode,
     'email': unicode,
     'password': unicode,
     'first_name': unicode,
     'last_name': unicode,
   }

   required_fields = ['guid']
   default_values = {
     'guid': lambda:unicode(uuid.uuid4()),
   }

   indexes = [
     {'fields': 'guid',
      'unique': True},
   ]

class Event(BaseDocument):
   structure = {
     'user': User,
     'title': unicode,
   }
   required_fields = ['user', 'title']

   indexes = [
     {'fields': ['user']},
   ]

And here is my unit test that now breaks:

   def test_create_event(self):
       user = self.db.users.User()
       user.save()
       event = self.db.events.Event()
       event.user = user
       event.title = u"Test"
       print event.user
       event.validate()
       event.save()

Traceback (most recent call last):
 File "/home/peterbe/dev/TORNADO/worklog/tests/test_models.py", line
52, in test_create_event
   event.validate()
 File "/home/peterbe/virtualenvs/worklog/lib/python2.6/site-packages/
mongokit-0.5.13_dev-py2.6.egg/mongokit/document.py", line 249, in
validate
   super(Document, self).validate()
 File "/home/peterbe/virtualenvs/worklog/lib/python2.6/site-packages/
mongokit-0.5.13_dev-py2.6.egg/mongokit/schema_document.py", line 351,
in validate
   self._validate_required(self, self.structure)
 File "/home/peterbe/virtualenvs/worklog/lib/python2.6/site-packages/
mongokit-0.5.13_dev-py2.6.egg/mongokit/schema_document.py", line 772,
in _validate_required
   self._raise_exception(RequireFieldError, req, "%s is required" %
req)
 File "/home/peterbe/virtualenvs/worklog/lib/python2.6/site-packages/
mongokit-0.5.13_dev-py2.6.egg/mongokit/schema_document.py", line 519,
in _raise_exception
   raise exception(message)
RequireFieldError: user is required

It succeeds if I make 'user' NOT a required field.

Allow Set to accept list type

We should be able to do this:

class MyDoc(Document):
    structure = {
        "tags": Set(unicode),
    }
>>> doc = db.col.MyDoc()
>>> doc['tags'] = [u'foo', u'bar', u'foo']
>>> doc.save()

>>> fetched_doc = db.col.get_from_id(doc['_id'])
>>> fetched_doc['tags']
set([u'foo', u'bar'])

fixture generation

Via structure introspection, we can guess the type of field and then generate some fixtures on the fly.
The api could be :

>>> connection.mydb.mycol.Doc.fixtures(how_many=10)

this will load to mycol 10 Doc objects with random value.
Thanks to Vladimir Dronnikov for the original idea...

KeyError when saving partially loaded documents

class User(Document):
    structure = {
        "username": unicode,
        "password": unicode,
    }
    required_fields = ["username", "password"]
    validators = {
        "username": lambda v: re.match(r"[A-Za-z0-9@_-]{3,}", v),
        "password": lambda v: len(v) >= 6,
    }

>>> from app import *
>>> u = connection.User.find_one({}, {"password": 0})
>>> u
{u'_id': ObjectId('4d44fdefebd2536140000000'), u'username': u'iggy'}
>>> u.save()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "models.py", line 47, in save
    super(User, self).save(*args, **kwargs)
  File "/home/bobry/.virtualenvs/mb-env/lib/python2.7/site-packages/mongokit/document.py", line 398, in save
    self.validate(auto_migrate=False)
  File "/home/bobry/.virtualenvs/mb-env/lib/python2.7/site-packages/mongokit/document.py", line 247, in validate
    super(Document, self).validate()
  File "/home/bobry/.virtualenvs/mb-env/lib/python2.7/site-packages/mongokit/schema_document.py", line 346, in validate
    self._process_validators(self, self.structure)
  File "/home/bobry/.virtualenvs/mb-env/lib/python2.7/site-packages/mongokit/schema_document.py", line 613, in _process_validators
    if doted_doc[key] is not None:
KeyError: 'password'

Mongokit + Pylons

I tried setting up a default pylons project to use mongokit as directed here:
http://namlook.github.com/mongokit/pylons.html

However it gives me the error:

Traceback (most recent call last):
File "/usr/bin/paster", line 18, in
command.run()
File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 84, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 123, in invoke
exit_code = runner.run(args)
File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 218, in run
result = self.command()
File "/usr/lib/pymodules/python2.6/paste/script/serve.py", line 276, in command
relative_to=base, global_conf=vars)
File "/usr/lib/pymodules/python2.6/paste/script/serve.py", line 313, in loadapp
*_kw)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 204, in loadapp
return loadobj(APP, uri, name=name, *_kw)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 224, in loadobj
global_conf=global_conf)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 248, in loadcontext
global_conf=global_conf)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 278, in _loadconfig
return loader.get_context(object_type, name, global_conf)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 409, in get_context
section)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 431, in _context_from_use
object_type, name=use, global_conf=global_conf)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 361, in get_context
global_conf=global_conf)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 248, in loadcontext
global_conf=global_conf)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 285, in _loadegg
return loader.get_context(object_type, name, global_conf)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 561, in get_context
object_type, name=name)
File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 587, in find_egg_entry_point
possible.append((entry.load(), protocol, entry.name))
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 1954, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/home/ciferkey/Projects/runnr-site/runnrsite/config/middleware.py", line 11, in
from runnrsite.config.environment import load_environment
File "/home/ciferkey/Projects/runnr-site/runnrsite/config/environment.py", line 8, in
import runnrsite.lib.app_globals as app_globals
File "/home/ciferkey/Projects/runnr-site/runnrsite/lib/app_globals.py", line 2, in
from ekeet.models import register_models
ImportError: No module named ekeet.models

I haven't been able to find any help anywhere else. Do you have any ideas as to why this is happening?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.