Could not create. python manage.py dumps create about alchemydumps HOT 14 CLOSED

The error happens when AlchemyDumps is trying to get the models from your application.

Can you share your models so we can debug this, please?

If there's nothing new (or wrong) with your models, maybe we need a filter in the add_subclasses() helper to be sure that everything passed to get_data() helper is an object valid for pickling. But without your models I can't reproduce this error here to investigate it.

from alchemydumps.

This my project. https://github.com/vlikin/car_budget/tree/master/flask
if you have a time we could have a Hangout meeting.

Yes. I have Table object. These define data structure. There are models these are inherited from tables. These contains the entity logic. So Alchemy could count them twice.

from alchemydumps.

Ok, let me play around with your code a bit, then we talk. Can you point me to the exact file where you define your mapped classes (models)? And what version of Python should I use?

Regarding the Hangouts, sounds a good idea, maybe some pair programming to sort this out. Where are you in terms of timezone? I'm in BST (British Summer Time).

UPDATE

@VLikin, it looks like you have some issues with your application. I just cloned it and change the settings to use SQLite. Take a look at these little steps.

As I couldn't see any migration script, I started a fresh migration:

cuducos@localhost ~/car_budget/flask (car_budget) $ rm -rf migrations/
cuducos@localhost ~/car_budget/flask (car_budget) $ python manage.py db init
  Creating directory /Users/cuducos/car_budget/flask/migrations ... done
  Creating directory /Users/cuducos/car_budget/flask/migrations/versions ... done
  Generating /Users/cuducos/car_budget/flask/migrations/alembic.ini ... done
  Generating /Users/cuducos/car_budget/flask/migrations/env.py ... done
  Generating /Users/cuducos/car_budget/flask/migrations/env.pyc ... done
  Generating /Users/cuducos/car_budget/flask/migrations/README ... done
  Generating /Users/cuducos/car_budget/flask/migrations/script.py.mako ... done
  Please edit configuration/connection/logging settings in '/Users/cuducos/car_budget/flask/migrations/alembic.ini'
  before proceeding.

And then ran the migration script:

cuducos@localhost ~/car_budget/flask (car_budget) $ python manage.py db upgrade
INFO  [alembic.migration] Context impl SQLiteImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.

However, there is nothing in your database (only the Alembic control table):

cuducos@localhost ~/car_budget/flask (car_budget) $ sqlite3 db.sqlite
SQLite version 3.8.5 2014-08-15 22:37:57
Enter ".help" for usage hints.
sqlite> .tables
alembic_version
sqlite>

So it looks like the problem is in your application I guess, since not even Alembic is finding your models (in order to create the proper tables).

What am I getting wrong?

from alchemydumps.

Hello. I am from Ukraine, Chrnigov. GMT+3. My email is [email protected]. To create the database structure run "python manage.py db_rebuild" Than "python manage.py dumps create". I receive errors described above.

from alchemydumps.

Do not use the migration script.

from alchemydumps.

Sorry, but I have to try it for the third time:

• Where is the file where you define your models?

That's the crucial point here.

from alchemydumps.

ext.user.table, ext.user.model, ext.budget.table, ext.budget.model. The models inherit the table Objects.

from alchemydumps.

https://github.com/vlikin/car_budget/tree/master/flask/ext/budget/model
https://github.com/vlikin/car_budget/tree/master/flask/ext/budget/table
https://github.com/vlikin/car_budget/tree/master/flask/ext/user/table
https://github.com/vlikin/car_budget/tree/master/flask/ext/user/model

from alchemydumps.

Ok, now I see! Thanks for the links, they saved a lot of time.

It looks like the issue is with InstanceModel: as it requires an argument in its __init__ method, it messes up with __reduce__ and __reduce_ex__ class methods used for pickling (dumping).

I still studying how to fix it, and this is a completely new field of study for me. So I'm just dropping it here in case anybody can jump in and help.

Some useful references:

• Can't pickle int object error when object comes from SQLAlchemy?
• Pickling and unpickling extension types

from alchemydumps.

Thank you. I try also to resolve it.

from alchemydumps.

Ok, wrong guess, sorry. I just implemented tests for the situation I descried earlier and they pass. If you wanna play around, check the fix-issue-10 branch (take a look on the latest commit to see the changes).

My new guess comes from this StackOverflow answer:

Nesting classes makes pickle fail, since it relies on the path of the object inside your application to reconstruct it later.

AlchemyDumps uses pickle to dump data, to do the backup. If nested classes mess with pickle, and your models are declared that way, probably that's not gonna work. Does it seem to fit you case?

And, finally, any ideas on how to overcome this pickle limitation?

If you're trying to fix it too, share your findings here. It's easier to do it sharing and debating :)

from alchemydumps.

from sqlalchemy.ext.serializer import loads, dumps
from ext.user.table.user import UserTable as User
users = User.query.all()
s = dumps(users[0])

SqlAlchemy fails the same way. SqlA "dumps" uses pickle. Pickle is very dangerous for the reason. It dumps everything. We need sanitation. It is the very good idea when we want to separate it to chunks. Now I need store/recover the databases. Why do we not use simple SQL dumps?

from alchemydumps.

The code you used only fails if the tables is empty (I'm opening another Issue for this):

In [1]: from sqlalchemy.ext.serializer import loads, dumps
In [2]: from ext.user.table.user import UserTable as User
In [3]: users = User.query.all()
In [4]: s = dumps(users[0])
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-4-9cf913d7f7a1> in <module>()
----> 1 s = dumps(users[0])
IndexError: list index out of range
In [5]: db = app.extensions['alchemydumps'].db
In [6]: db.session.add(User(email='a@a'))
In [7]: db.session.commit()
In [8]: users = User.query.all()
In [9]: s = dumps(users[0])
In [10]:

Regarding the use of dumps(): no special reason.

Feel free to start a branch, or send a PR changing the method for something else :)

from alchemydumps.

@VLikin I was thinking and I believe there are three good reasons to keep working with SQLAlchemy dumps() method:

1. AlchemyDumps is supposed to work alongside SQLAlchemy, and following their methods is a good way to keep this compatibility as close as possible.
2. Simple SQL can be problematic, because writing SQL code is exactly the problem SQLAlchemy try to avoid; in other words, PostgreSQL SQL code may be incompatible with MySQL, or with SQLite and so on — and to sort this issues we add an ORM.
3. Due to 1 and 2, I believe there's no secure way to generate safe SQL code from SQLAlchemy.

Anyway, that said, any PR is welcomed. I just believe if you try to write something different from dumps() the best choice would be to add it as an option, and not as replacement to the SQLAlchemy dumps().

from alchemydumps.