klen / peewee_migrate Goto Github PK

View Code? Open in Web Editor NEW

339.0 6.0 84.0 432 KB

Simple migration engine for Peewee

License: MIT License

Makefile 1.14% Python 98.86%

migrations peewee-orm peewee

peewee_migrate's Introduction

Peewee Migrate

Peewee Migrate -- A simple migration engine for Peewee ORM

Requirements

peewee >= 3.8

Dependency Note

For Peewee<3.0 please use Peewee-Migrate==0.14.0
For Python 3.0-3.6 please use Peewee-Migrate==1.1.6
For Python 3.7 please use Peewee-Migrate==1.6.6
For Python 3.8+ please use Peewee-Migrate>=1.12.1

Installation

Peewee Migrate should be installed using pip: :

pip install peewee-migrate

Usage

Do you want Flask integration? Look at Flask-PW.

From shell

Getting help: :

$ pw_migrate --help

Usage: pw_migrate [OPTIONS] COMMAND [ARGS]...

Options:
    --help  Show this message and exit.

Commands:
    create   Create migration.
    migrate  Run migrations.
    rollback Rollback migration.

Create migration: :

$ pw_migrate create --help

Usage: pw_migrate create [OPTIONS] NAME

    Create migration.

Options:
    --auto                  FLAG  Scan sources and create db migrations automatically. Supports autodiscovery.
    --auto-source           TEXT  Set to python module path for changes autoscan (e.g. 'package.models'). Current directory will be recursively scanned by default.
    --database              TEXT  Database connection
    --directory             TEXT  Directory where migrations are stored
    -v, --verbose
    --help                        Show this message and exit.

Run migrations: :

$ pw_migrate migrate --help

Usage: pw_migrate migrate [OPTIONS]

    Run migrations.

Options:
    --name TEXT       Select migration
    --database TEXT   Database connection
    --directory TEXT  Directory where migrations are stored
    -v, --verbose
    --help            Show this message and exit.

Rollback migrations: :

$ pw_migrate rollback --help

Usage: pw_migrate rollback [OPTIONS]

    Rollback a migration with given steps --count of last migrations as integer number

Options:
    --count INTEGER   Number of last migrations to be rolled back.Ignored in
                        case of non-empty name

    --database TEXT   Database connection
    --directory TEXT  Directory where migrations are stored
    -v, --verbose
    --help            Show this message and exit.

From python

from peewee_migrate import Router
from peewee import SqliteDatabase

router = Router(SqliteDatabase('test.db'))

# Create migration
router.create('migration_name')

# Run migration/migrations
router.run('migration_name')

# Run all unapplied migrations
router.run()

Migration files

By default, migration files are looked up in os.getcwd()/migrations directory, but custom directory can be given.

Migration files are sorted and applied in ascending order per their filename.

Each migration file must specify migrate() function and may specify rollback() function

def migrate(migrator, database, fake=False, **kwargs):
    pass

def rollback(migrator, database, fake=False, **kwargs):
    pass

Bug tracker

If you have any suggestions, bug reports or annoyances please report them to the issue tracker at https://github.com/klen/peewee_migrate/issues

Contributing

Development of starter happens at github: https://github.com/klen/peewee_migrate

peewee_migrate's People

Contributors

Stargazers

Watchers

Forkers

ericschles fuzeman nsavch judasnow dingus9 maplet lowks petrov-vova utomik tuukkamustonen kolanos qwivan juntatalor oalifdal digideskio lostdragon eentang ianlini shermin nilsjpwerner ezc catcuts eligundry binrush rafaelhrborges chekosoft patrickporto shayneoneill guanghuazeng jchiavacci channelcat smglab stlehmann kapucko ccssk alexandrukis zhyong10 msusik serg-i-o ryanb58 eggachecat keigohtr gridl irvind spumer logan-lu hieuttrieu oldsyang revolter moorefu grynko ishchenkodarina metallerok kenzoikeizume islomk rares-urdea norbertcyran chin-wag jimmyhli vinyk08 fmazan rpodgorny alteralt zed smoothgithub mulac gdassori prospect100 xiaodao90 enpaul cloudlinux lzgirlcat stungkit dmwyatt nixellion diegodg-01

peewee_migrate's Issues

Support raw SQL files

In my standard use case, I have no need for rollback and no need for peewee's abstractions, really, but I would like to keep that as an option. To reduce boilerplate ever so slightly, I would like to load raw .sql files from directory, as an alternative to python scripts.

Would you be willing to merge something like this if I created a PR:

class SqlRouter(Router):

    filemask = re.compile(r"[\d]{3}_[^\.]+\.(py|sql)$")

    @property
    def todo(self):
        """Scan migrations in file system."""
        if not os.path.exists(self.migrate_dir):
            self.logger.warn('Migration directory: %s does not exists.', self.migrate_dir)
            os.makedirs(self.migrate_dir)
        return sorted(
            f.rpartition('.')[0] for f in os.listdir(self.migrate_dir) if self.filemask.match(f))

    def read(self, name):
        """Read migration from SQL or python file."""
        sql_file = os.path.join(self.migrate_dir, name + '.sql')
        if os.path.isfile(sql_file):
            data = ''
            with open(sql_file) as f:
                for line in f.readlines():
                    if line and not line.startswith('--'):
                        data += line
            return lambda mg, db, *args, **kwargs: db.execute_sql(data), VOID
        return super().read(name)

Note that the only thing needing change in todo() is from ''.join(f[:-3]) to f.rpartition('.')[0] to support different length of file suffix. That could be improved in Router already.

Add possiblity to add constraints

Unable to fake a rollback

It is possible now to emulate a migration run through router.run(fake=True), but can't do it for rollbacks

Do not create abstract models

I have so-called base model for easy db linking:

class Model(peewee.Model):
    class Meta:
        abstract = True
        database = db

Creating migrations produces:

    @migrator.create_model
    class Model(pw.Model):
        id = pw.AutoField()

        class Meta:
            table_name = "model"

How should an SQLite database migration file look like?

The documentation is very lacking so it's not clear at all.

Firstly, in the def migrate(migrator, database, fake=False, **kwargs): method, migrator is of type peewee_migrate.migrator.Migrator, so I have to use migrator.migrator to access the peewee_migrate.migrator.SqliteMigrator so it doesn't crashes. Is this the intended behavior?

Secondly, I'm using migrator.migrator.drop_not_null('user', 'username') and nothing is happening. Is drop_not_null supported or not?

is not null but has no default

pw_migrate migrate
Add a foreign key to table ，produce the following error

role_id is not null but has no default

001_add_role.py as follows

def migrate(migrator, database, fake=False, **kwargs):
    """Write your migrations here."""
    try:
        migrator.add_fields(
            'host',
            role = pw.ForeignKeyField(db_column='role_id', rel_model='role', to_field='id'))
    except Exception as e:
        print(e)

`--auto` creates invalid Python file

I tried to use the --auto option to generate the migration files. The result is:

def migrate(migrator, database, fake=False, **kwargs):
    """Write your migrations here."""

    @migrator.create_model
    class BaseModel(pw.Model):

    @migrator.create_model
    class BasissetFamily(pw.Model):
        name = pw.CharField(max_length=255, unique=True)

    @migrator.create_model
    class MyModel1(pw.Model):
        family = pw.ForeignKeyField(db_column='family_id', rel_model=MyModel1, to_field='id')
        element = pw.CharField(max_length=255)
        basis = pw.TextField()

    @migrator.create_model
    class Model(pw.Model):

    @migrator.create_model
    class MyModel2(pw.Model):
        name = pw.CharField(max_length=255, unique=True)

The problem are the empty bodies for the BaseModel and Model classes which is invalid Python.

migrator.add_index fails with 'Relation ... already exists'

I am trying to add an index as follows:

migrator.add_index('structuresetstructure', 'structure', 'set', unique=True)

I get the following error:

[...]
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.5/pw_migrate", line 9, in <module>
    load_entry_point('peewee-migrate==0.5.10', 'console_scripts', 'pw_migrate')()
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/cli.py", line 57, in migrate
    migrations = router.run(name, fake=fake)
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 156, in run
    self.run_one(mname, migrator, fake=fake, force=fake)
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 128, in run_one
    migrator.run()
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/migrator.py", line 107, in run
    opn.run()
  File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 144, in run
    getattr(self.migrator, self.method)(*self.args, **kwargs))
  File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 133, in _handle_result
    self.execute(result)
  File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 129, in execute
    self.migrator.database.execute_sql(sql, params)
  File "/usr/lib64/python3.5/site-packages/playhouse/postgres_ext.py", line 380, in execute_sql
    self.commit()
  File "/usr/lib64/python3.5/site-packages/peewee.py", line 3316, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/usr/lib64/python3.5/site-packages/peewee.py", line 127, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib64/python3.5/site-packages/playhouse/postgres_ext.py", line 373, in execute_sql
    cursor.execute(sql, params or ())
peewee.ProgrammingError: FEHLER:  Relation „structuresetstructure_structure_id_set_id“ existiert bereits

I have another add_index call for a different table in an earlier (already applied) migration file.

What about the DB creation?

If we use Peewee without peewee_migrate, we would just execute this: db.create_tables([Model1, Model2])

So, if we use peewee_migrate, we should avoid this command and only use peewee_migrate, right?

Also, does Peewee_Migrate use Peewee (http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#migrate) for migrations, right? Any doc about how to create the migration files?

Thanks

Loosen dependency versions

In requirements.txt:

cached_property     == 1.3.0
click               == 6.4
mock                == 1.3.0
peewee              >= 2.8.0

Would it be possible to depend from a minimum only version (as with peewee)? Requiring exact versions is pretty annoying in projects, that happen to want to use a newer version.

Add support for multi-column index/unique

I added a "multi-column index unique" definition according to the PeeWee documentation: http://docs.peewee-orm.com/en/latest/peewee/models.html#indexes-and-constraints but when I run peewee_migrate it doesn't generate the statements for the multi-column index.

I was taking a look at your code (auto.py) and looks like you are not considering that case. It would be possible to add support for multicolumn index/unique ?

Adding manually the sentence using migrator.add_index(model, *col_names, unique=False) works but this means that I have to manually maintain autogenerated scripts and the other one for multi column indexes.

This is my model:

class Dog(BaseModel):
    id = PrimaryKeyField()
    user = ForeignKeyField(User, to_field='id', related_name='dogs', db_column='user_id')
    name = CharField(null=False)
    description = CharField(null=False)

    class Meta:
        indexes = (
            (('user', 'name'), True),  # This should generated a unique constraint of those two columns
        )

Use pw.SQL instead of SQL in migration files

Given the following simple model, generated migration has a typo:

class User(BaseModel):
    active = BooleanField(default=True)

Migration I get (the relevant part):

@migrator.create_model                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
class User(pw.Model):                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
    id = pw.AutoField()                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
    active = pw.BooleanField(constraints=[SQL("DEFAULT True")])

This fails, but replacing SQL with pw.SQL works like a charm. I can't find the code which generates it, so if you help me debug this issue, I'll be happy to send PR.

RuntimeError: Unrecognized or unsupported schema: "".

I have peewee version 3.7.1 installed and latest peewee-migrate
when I try to create a migration

pw_migrate create -v --database deploy_services --directory migrations/ --auto-source /path/to/model/new_table.py migration_name

/lib/python2.7/site-packages/playhouse/db_url.py line 97 in connect parsed.scheme)
RuntimeError: Unrecognized or unsupported schema: "".

I get the below error

`pw_migrate migrate` fails when changing fields against Postgresql

Happens with peewee-migrate version 1.0.0 (tested against multiple peewee versions from the latest one (3.6.4) backwards, didn't find any where this doesn't occur).
Error:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/peewee_migrate/router.py", line 144, in run_one
    migrator.run()
  File "/usr/local/lib/python2.7/site-packages/peewee_migrate/migrator.py", line 129, in run
    op.run()
  File "/usr/local/lib/python2.7/site-packages/playhouse/migrate.py", line 141, in run
    self._handle_result(method(*self.args, **kwargs))
  File "/usr/local/lib/python2.7/site-packages/playhouse/migrate.py", line 149, in inner
    return fn(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/peewee_migrate/migrator.py", line 36, in change_column
    operations = [self.alter_change_column(table, column_name, field)]
  File "/usr/local/lib/python2.7/site-packages/peewee_migrate/migrator.py", line 81, in alter_change_column
    field_clause = clause.nodes[-1]
AttributeError: 'Context' object has no attribute 'nodes'

Other details:
OS: Ubuntu 16.04
Setup: docker version 18.06.0-ce; docker-compose version 1.8.0; Python 2.7; postgresql version 9.6
Steps:

Create model with a few fields

class PgTest(Model):
    class Meta:
        database = psql_client.db
        table_function = lambda model_cls: '{model_name}'.format(model_name=model_cls.__name__.lower())

    id = AutoField()
    name = CharField(max_length=100, index=True)

Create migration, and migrate it
$ pw_migrate create --auto test.models --directory test/migrations --database postgresql://postgres@postgres:5432/postgres test
$ pw_migrate migrate --directory test/migrations --database postgresql://postgres@postgres:5432/datalake --name 001_test
Back to the model, alter max_length on the name field, generate new migration
Migrate the new migration, result is the traceback above.

BaseRouter.todo raises not implemented

So this appears to make things completely unusable as the NotImplementedError of BaseRouter.todo always raises NotImlementedError
router.py#L41-L43

Where are you going with all that? Do you need me to implement something there?
Nick

How are migrations supposed to be automatically created?

It always creates empty migrate and rollback methods.

Possibility for adding a not-null field with no default in a migration

I am trying to add a not-null field in a migration. The way to do it is to add the fill with null=True, fill it and then add the not-null constraint.

I came up with the following:

    # add the field with null=True
    migrator.add_fields(
        'mymodel1',
        format = pw.CharField(max_length=255, null=True)
    )

    # initialize the format attribute
    migrator.sql("""
        UPDATE mymodel1 SET format = CASE WHEN (somerelatedtable.name LIKE '%-FOO') THEN 'FOO' ELSE 'BAR' END
            FROM somerelatedtable WHERE mymodel1.somerelatedtable_id = somerelatedtable.id;""")

    # add the not-null constraint
    migrator.add_not_null('mymodel1', 'format')

which fails with the following message:


Starting migrations
Running "002_pseudo_format"
add_column ('pseudopotential', 'format', <peewee.CharField object at 0x7fe224bb8518>)
format is not null but has no default
Traceback (most recent call last):
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 128, in run_one
    migrator.run()
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/migrator.py", line 108, in run
    opn.run()
  File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 144, in run
    getattr(self.migrator, self.method)(*self.args, **kwargs))
  File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 152, in inner
    return fn(self, *args, **kwargs)
  File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 222, in add_column
    raise ValueError('%s is not null but has no default' % column_name)
ValueError: format is not null but has no default
Migration failed: 002_pseudo_format
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.5/pw_migrate", line 9, in <module>
    load_entry_point('peewee-migrate==0.5.9', 'console_scripts', 'pw_migrate')()
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib64/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/cli.py", line 57, in migrate
    migrations = router.run(name, fake=fake)
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 156, in run
    self.run_one(mname, migrator, fake=fake, force=fake)
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 128, in run_one
    migrator.run()
  File "/usr/lib64/python3.5/site-packages/peewee_migrate/migrator.py", line 108, in run
    opn.run()
  File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 144, in run
    getattr(self.migrator, self.method)(*self.args, **kwargs))
  File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 152, in inner
    return fn(self, *args, **kwargs)
  File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 222, in add_column
    raise ValueError('%s is not null but has no default' % column_name)
ValueError: format is not null but has no default

Removing the add_not_null makes it run through, but I should be able to have it completely in one migration, shouldn't I?

peewee.ProgrammingError: column of relation already exists

I have a model with a few fields. In the next version of my program, I want to add another field. I add it to the model class and write a migration with migrator.add_fields call. Everything is fine if I run the program with an empty database or if I migrate the database from the previous version. But I run migrations on the database of the last version which had no migrations I get the exception:

peewee.ProgrammingError: column of relation already exists

How do I handle this situation? Is there any way to suppress the exception or to check the field existence without executing SQL (I want my migrations to be DBMS independent)?

crashes with python 3

Traceback (most recent call last):
File "/Users/v/.venv/s/bin/pw_migrate", line 7, in
from peewee_migrate.cli import cli
File "/Users/v/.venv/s/lib/python3.6/site-packages/peewee_migrate/init.py", line 36, in
from .router import Migrator, Router # noqa
File "/Users/v/.venv/s/lib/python3.6/site-packages/peewee_migrate/router.py", line 14, in
from peewee_migrate.migrator import Migrator
File "/Users/v/.venv/s/lib/python3.6/site-packages/peewee_migrate/migrator.py", line 2, in
from playhouse.migrate import (
ImportError: cannot import name 'Clause'

SQL OperationalError due to missing quotation marks around default values

Error description

When using pw_migrate with the --auto flag to add a new CharField with a default value to a table it produces code like this:

@migrator.create_model
class Color(pw.Model):
    id = pw.AutoField()
    name = pw.CharField(constraints=[SQL("DEFAULT New Color")], max_length=255)

The last line will lead to an Operational error because there are quotation marks missing around the default values.

System configuration

Operarating System: Windows 10
Python Version: Python 3.7.0 64bit
Peewee Version: 3.6.4
peewee_migrate: develop branch
database engine: sqlite

Already run migrations are run again if there was error in migration

Lets say there are migrations 001 and 002. 001 succeeds but 002 does not. When you re-run migrations, library tries to re-run 001 even though it's already applied in the database.

In BaseRouter.migrator there is:

    for name in self.done:
        self.run_one(name, migrator)

I'm not sure why that is there, what is the logic, but removing the block fixes the issue.

Multiple Models.py files.

Hello, I currently have a situation where I have multiple models.py files separated into their own domains and folders. I was wondering how I can get pw_migrate to auto-generate the migrations. I have tried the following with no such luck.

File Structue:

 - api
 - - migrations
 - - auth
 - - - auth
 - - - - models.py
 - - groups
 - - - groups
 - - - - models.py

I am running the following command:

pw_migrate create --auto -v --database=${DB_URL} "initial"

from the /api/ folder.. and it comes back with this error message:

Can't import models module: True

Does anyone have any suggestions or ideas?

I would be okay having migration folders/files like so:

 - api
 - - auth
 - - - auth
 - - - - models.py
 - - - migrations
 - - groups
 - - - groups
 - - - - models.py
 - - - migrations

However, I would need all the migrations to be able to be run against the same database.

peewee.ProgrammingError: column is of type integer[] but expression is of type record

I want to change the default for ArrayField. I have following lines in my migration (simplified):

def migrate(migrator, db, fake=False, **kwargs):
    """Write your migrations here."""
    _, _ = fake, kwargs

    class Postfilter(BaseModel):
        actions = ArrayField(IntegerField, default=[2, 4, 11])

    db_proxy.initialize(db)
    migrator.orm['postfilter'] = Postfilter
    migrator.add_default('postfilter', 'actions', [10, 2, 4, 11])

Migration fails:

...
peewee.ProgrammingError: column "actions" is of type integer[] but expression is of type record
LINE 1: UPDATE "postfilter" SET "actions" = (10, 2, 4, 11)
                                            ^
HINT:  You will need to rewrite or cast the expression.

AttributeError: 'MySQLMigrator' object has no attribute 'change_column'

Hey, I'm trying to change a column like so...

migrator.change_fields('user', email=pw.CharField(max_length=100, null=True))

But I end up with the following error...

Traceback (most recent call last):
  File "scripts/reload_local.py", line 108, in <module>
    router.run()
  File "/usr/local/lib/python3.5/dist-packages/peewee_migrate/router.py", line 158, in run
    self.run_one(mname, migrator, fake=fake, force=fake)
  File "/usr/local/lib/python3.5/dist-packages/peewee_migrate/router.py", line 129, in run_one
    migrate(migrator, self.database, fake=fake)
  File "<string>", line 46, in migrate
  File "/usr/local/lib/python3.5/dist-packages/peewee_migrate/migrator.py", line 102, in wrapper
    return method(migrator, migrator.orm[model], *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/peewee_migrate/migrator.py", line 184, in change_columns
    self.ops.append(self.migrator.change_column(
AttributeError: 'MySQLMigrator' object has no attribute 'change_column'

I'm using MySQL.

Thanks for any help!

Cheers

Can't import models module

Hi there,

first of all thank you so much for taking the time and publishing this library, appreciate it a lot since peewee doesn't come with automatic migrations included.

I'm trying to use the cli to migrate my model, but load_models raises an Exception, saying that it can't import the models module (the one at the --auto path).

This is the failing command:
$ pw_migrate create --auto ~/myproject/model/ --database "postgres://zudibytvipthsa:c8c2d7f0blabla" --directory ~/myproject/model/ mymigration

Could it be that peewee_migrate doesn't resolve the database connection URL with playhouse.db_url.connect but expects something else instead? This isn't really clear from the help text.

My model module is structured as follows:

model/
    basemodel.py  <-- containing:
    class BaseModel(Model):
        class Meta:
             database = db

    .. more modules that inherit from BaseModel

I would appreciate getting some hints as to what might be causing this :)

Misbehaving logging config

Line https://github.com/klen/peewee_migrate/blob/develop/peewee_migrate/__init__.py#L21 adds a StreamHandler() to peewee_migrate logger. IMO it should be up to the user of the library to configure the logging as per platform requirements.
In our case we end up doing del(logging.getLogger('peewee_migrate').handlers[:]) to avoid having output on stderr which doesn't obey the logging configuration.

Proposed solution: remove lines adding handler and setting log level.

field is not null but has no default

I have a migration with the following code:

def migrate(migrator, database, fake=False, **kwargs):
    """Write your migrations here."""
    MyModel = migrator.orm['my_table']
    type_new = pw.CharField(null=True, constraints=[pw.Check(
        "type_new in ('type1', 'type2', 'type3')")])
    migrator.add_fields(MyModel, type_new=type_new)
    migrator.sql('UPDATE my_table SET type_new=type')
    migrator.remove_fields(MyModel, 'type')
    migrator.rename_field(MyModel, 'type_new', 'type')
    migrator.add_not_null(MyModel, 'type')

And when appllying it I get following error:

ValueError: type_new is not null but has no default

I'm explicitly passing null=True, but peewee-migrate claims this field to be non-null, how it is possible? Is it some bug or am I missing something?

--auto doesn't respect db_table meta

Invoking create --auto on the following model:

class Domain(peewee.Model):
	name = peewee.CharField()
	user = peewee.CharField()

	class Meta:
		db_table = 'domains'

generates these migrations:

def migrate(migrator, database, fake=False, **kwargs):
    """Write your migrations here."""

    @migrator.create_model
    class Domain(pw.Model):
        name = pw.CharField(max_length=255)
        user = pw.CharField(max_length=255)



def rollback(migrator, database, fake=False, **kwargs):
    """Write your rollback migrations here."""

    migrator.remove_model('domains')

As expected, this applies normally (creating domain table despite db_table meta option) but fails to rollback.

Automatically create migrations/.editorconfig

Since migrations/ directory will typically contain only Python files generated by the migrator (with certain formatting rules), it would be helpful if migrations/.editorconfig was automatically created with formatting rules explicitly stated:

[*.py]
indent_style = space
indent_size = 4

More on the format: http://editorconfig.org/

Customize migration template and include custom fields

How I can customize migration template and include my custom fields?

Do not catch exceptions

pw_migrate migrate --name THIS_FILES_DOES_NOT_EXIST.py --database $POSTGRES_URI

No error message at all. I believe there is some try/except somewhere. We should see an error like "this file does not exist"

AttributeError: type object 'Model' has no attribute '_meta'

python 3.6
peewee == 2.9.0
Peewee_migrate == 0.11.0

Command: pw_migrate create --auto my.models --database postgresql://localhost:5432/name auto-generated.py

    migrations = diff_many(models, source, migrator, reverse=reverse)
  File "env/lib/python3.6/site-packages/peewee_migrate/auto.py", line 115, in diff_many
    models1 = pw.sort_models_topologically(models1)
  File "playhouse/_speedups.pyx", line 345, in playhouse._speedups.sort_models_topologically (playhouse/_speedups.c:7541)
  File "stringsource", line 67, in cfunc.to_py.__Pyx_CFunc_tuple____object___to_py.wrap (playhouse/_speedups.c:8690)
  File "playhouse/_speedups.pyx", line 327, in playhouse._speedups._sort_key (playhouse/_speedups.c:7101)
AttributeError: type object 'Model' has no attribute '_meta'

Is this a Python 3 issue?

Or it's related to inheritance issues?

class BaseModel(Model):
    class Meta:
        database = db

class User(BaseModel):
    pass

`ForeignKeyField` nonsupport 'self'

models.py

class Group(BaseModel):
    name = pw.CharField(max_length=255, null=False, unique=True)
    parent = pw.ForeignKeyField('self', null=True, related_name='children')
    description = pw.CharField(max_length=255, null=True)

    def __str__(self):
        return self.name

migrations/001_auth.py

def migrate(migrator, database, fake=False, **kwargs):

    @migrator.create_model
    class Group(pw.Model):
        name = pw.CharField(max_length=255, unique=True)
        parent = pw.ForeignKeyField(db_column='parent_id', rel_model=migrator.orm['group'], related_name='children', to_field='id')
        description = pw.CharField(max_length=255, null=True)

        class Meta:
            db_table = "group"

errors

➜  ams-api git:(master) ✗ python manage.py migrate
Traceback (most recent call last):
  File "manage.py", line 71, in <module>
    cli()
  File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "manage.py", line 32, in migrate
    migrations = router.run(name, fake=False)
  File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/peewee_migrate/router.py", line 170, in run
    self.run_one(mname, migrator, fake=fake, force=fake)
  File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/peewee_migrate/router.py", line 140, in run_one
    migrate(migrator, self.database, fake=fake)
  File "<string>", line 94, in migrate
  File "<string>", line 96, in Group
KeyError: 'group'

Use separate SQL Files

Hello

I am using peewee to run raw MySQL queries. Sometimes, I have to use some multi-lines and they take some space on python code files. May I read an external SQL file? Or a Python file with multiliners?

Thank you

No documentation of the migration's name

It must start with 3 numbers, like 001_something.py

Import settings.py in conf.py

I noticed, we can write conf file in migration directory and setup settings about DB for example - https://github.com/klen/peewee_migrate/blob/develop/peewee_migrate/cli.py#L26

If I create migrations/conf.py with
DATABASE = 'postgresql://user:pass@host/db'
it works fine.
But if I want to confugure DATABASE variable from settings.py (for example) like this

import settings
DATABASE = 'postgresql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}/{DB_NAME}'.format(
    DB_HOST=settings.DB_HOST,
    DB_NAME=settings.DB_NAME,
    DB_USER=settings.DB_USER,
    DB_PASSWORD=settings.DB_PASSWORD,
)

I got

ModuleNotFoundError: No module named 'settings'

Are there any solutions of this problem?

Migrations not importing rounding enumeration from decimal

So in one of my models I have a field like this;-

bank_balance= DecimalField(max_digits=20,decimal_places=2,auto_round=True,null=True)

The migration creates a line as follows;-

bank_balance = pw.DecimalField(auto_round=True, decimal_places=2, max_digits=20, null=True, rounding=ROUND_HALF_EVEN)

Which is correct. However the migration needs a line like this;-

from decimal import ROUND_HALF_EVEN

to import the correct enumeration symbol. I presume if one of the other rounding options is chosen, that would need to be imported too.

On a side note, I notice the default= option doesnt seem to perculate across to the migrations (Specifically in my created_on=DateTimeField(datetime.datetime.now) field) Not sure if thats documented, will perhaps investigate later if I get a chance.

Best regards, and thanks for this great little library.

Relation "migratehistory" already exists

When I try to run all unapplied migrations, I get the following error:

peewee.ProgrammingError: relation "migratehistory" already exists

My python script to run all migrations:

db_connection.init_database()
db_connection.connect()
database = db_connection.get_database()

router = Router(database)

# Run all unapplied migrations
router.run()

After quick investigation to the source of peewee_migrate I found this from BaseRouter:

    @cached_property
    def model(self):
        """Initialize and cache MigrationHistory model."""
        MigrateHistory._meta.database = self.database
        MigrateHistory._meta.db_table = self.migrate_table
        MigrateHistory.create_table(True)
        return MigrateHistory

Looks like it tries to create the table always because of the last line before return which then fails if I have already migrated something to the db and the table exists. Why is that? Am I using this wrong somehow?

My peewee and peewee_migrate versions:

peewee==2.8.5
peewee-migrate==0.12.3

Using Python 3.5

And full stack trace:

Traceback (most recent call last):
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3768, in execute_sql
    cursor.execute(sql, params or ())
psycopg2.ProgrammingError: relation "migratehistory" already exists


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/tuomas/Code/KarjalaProject/karelian-db/tasks/migrate.py", line 12, in <module>
    router.run()
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee_migrate/router.py", line 163, in run
    diff = self.diff
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee_migrate/router.py", line 57, in diff
    done = set(self.done)
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee_migrate/router.py", line 52, in done
    return [mm.name for mm in self.model.select()]
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/cached_property.py", line 26, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee_migrate/router.py", line 42, in model
    MigrateHistory.create_table(True)
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 4975, in create_table
    db.create_table(cls)
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3852, in create_table
    return self.execute_sql(*qc.create_table(model_class, safe))
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3775, in execute_sql
    self.commit()
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3598, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 135, in reraise
    raise value.with_traceback(tb)
  File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3768, in execute_sql
    cursor.execute(sql, params or ())
peewee.ProgrammingError: relation "migratehistory" already exists

SQLite migrate [peewee.OperationalError: unable to open database file]

I made init migration file and i tried to run migrate on it but there was error: peewee.OperationalError: unable to open database file.

I used this command: pw_migrate migrate --name=001_mymigration.py --database=sqlite:C:\Users\zurek\Documents\BackBone\Alto\afs\data\database.db --directory=C:\Users\zurek\Documents\BackBone\Alto\afs\data\migrations

where --name is name of migration file, database is directory to database.db sqlite file and directory where migrations are. I use windows 10. Do you know where the problem might be? Some proposals? Thank you very much.

Migration `--auto` creation requiring `example` and `tests` module.

The --auto flag to scaffold a set of models and generate a migration appears broken when a solution lacks an example and tests module.

Based on what I've read in router.py it appears that the culprit is here: https://github.com/klen/peewee_migrate/blob/develop/peewee_migrate/router.py#L81

In this case, I was missing an example module; once I added this, then --auto appears to work. Currently it appears both a example and tests module are required in order for --auto to work.

I'm happy to submit a PR and remove this line if it's desired, but, I'm wondering whether there's something I'm missing concerning this line of code.

Thanks.

IntegrityError: Cannot add foreign key constraint

Migration generated using pw_migrate create --auto models --database mysql://user:pass@localhost/test --directory migrations test fails to apply with the error peewee.IntegrityError: (1215, 'Cannot add foreign key constraint').

Database: MySQL

mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+-------------------------+
| Variable_name           | Value                   |
+-------------------------+-------------------------+
| innodb_version          | 5.7.22                  |
| protocol_version        | 10                      |
| slave_type_conversions  |                         |
| tls_version             | TLSv1,TLSv1.1           |
| version                 | 5.7.22-0ubuntu0.16.04.1 |
| version_comment         | (Ubuntu)                |
| version_compile_machine | x86_64                  |
| version_compile_os      | Linux                   |
+-------------------------+-------------------------+

models.py

class BaseModel(Model):
    class Meta:
        database = database


class Model1(BaseModel):
    id = BigAutoField()

    class Meta:
        table_name = 'tb_name1'


class Model2(BaseModel):
    id = BigAutoField()
    site = ForeignKeyField(column_name='model1_id', field='id', model=Model1, null=True)

    class Meta:
        table_name = 'tb_name2'

migrations/001_test.py

import datetime as dt
import peewee as pw

try:
    import playhouse.postgres_ext as pw_pext
except ImportError:
    pass


def migrate(migrator, database, fake=False, **kwargs):
    """Write your migrations here."""

    @migrator.create_model
    class BaseModel(pw.Model):
        id = pw.AutoField()

        class Meta:
            table_name = "basemodel"

    @migrator.create_model
    class Model1(pw.Model):
        id = pw.BigAutoField()

        class Meta:
            table_name = "tb_name1"

    @migrator.create_model
    class Model2(pw.Model):
        id = pw.BigAutoField()
        site = pw.ForeignKeyField(backref='model2_set', column_name='model1_id', field='id', model=migrator.orm['tb_name1'], null=True)

        class Meta:
            table_name = "tb_name2"



def rollback(migrator, database, fake=False, **kwargs):
    """Write your rollback migrations here."""

    migrator.remove_model('tb_name2')

    migrator.remove_model('tb_name1')

    migrator.remove_model('basemodel')

Output of SHOW ENGINE INNODB STATUS;:

------------------------
LATEST FOREIGN KEY ERROR
------------------------
2018-05-30 10:27:24 0x7fdfd835c700 Error in foreign key constraint of table bench_test/tb_name2:
FOREIGN KEY (`model1_id`) REFERENCES `tb_name1` (`id`)):
Cannot find an index in the referenced table where the
referenced columns appear as the first columns, or column types
in the table and the referenced table do not match for constraint.
Note that the internal storage type of ENUM and SET changed in
tables created with >= InnoDB-4.1.12, and such columns in old tables
cannot be referenced by such columns in new tables.
Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-foreign-key-constraints.html for correct foreign key definition.

Bug: hidden error message

peewee_migrate/peewee_migrate/router.py

Line 73 in 99e7de8

return self.logger.error("Can't import models module: %s", auto)

It is better to raise this exception (ImportError) to show the real error, instead of showing a beautiful message but that is hard to debug.

Nice initiative of the library by the say! it is pretty cool!

Alter column with added default results in invalid PSQL query

If you have a model with some field that does not have a default value, and you then modify that model to include a default value, the generated migration for that will fail.

It fails because the resulting PSQL query is an invalid ALTER statement.

Following the formatting in #87

Steps

Example model in models.py:

class Test(Model):
    class Meta:
        database = DB
        only_save_dirty = True

    id = AutoField()
    name = CharField(max_length=123, index=True)

Create initial migration, and migrate it

$ pw_migrate create --auto --directory test/migrations --database $DATABASE_URL --auto-source test test1

$ pw_migrate migrate --directory test/migrations --database $DATABASE_URL --name 001_test1

Change the model in models.py so that the name field now has a default:

name = CharField(max_length=123, index=True, default='D')

Repeat step 2 to create a new migration and then attempt to migrate it.

Result: the generated PSQL query will be this invalid statement:

ALTER TABLE "test" ALTER COLUMN "name" VARCHAR(123) TYPE DEFAULT 'D'

Something like this PSQL statement would work and is would be more appropriate.

ALTER TABLE "test" ALTER COLUMN "name" SET DEFAULT 'D';

Version:

$ pip list | grep peewee
peewee                           3.9.3
peewee-migrate                   1.1.5

Thanks!

Peewee version dependency

Hello!

In the docs you state as only dependency: "python 2.7,3.3,3.4", which I accomplish.

But there's also a dependency of peewee >= 2.8.5, as specified in requirements.txt. I'm using peewee 2.6 so I can't use this library.
Do you think you could lower your peewee requirement to 2.6.0 or it is a must to have it set on 2.8.5?

In that case, perhaps it would be a good idea to write it down on the requirements section of the docs page :).

Thank you very much,
Miguel.

add_fields ignore default value

Migration doesn't set default value for text field if use add_fields

    migrator.add_fields(
        'table_name',

       text=pw.CharField(default='N', constraints=[SQL('DEFAULT "N"')]))

But if I create a model with same field always is good :-/

    @migrator.create_model
    class TableName(pw.Model):
            text=pw.CharField(default='N', constraints=[SQL('DEFAULT "N"')]))

Where is a mistake?

Can not create table in sqlite

I run functions:
`router = Router(chat_app_db)

# Create migration
router.create('migration_name')

# Run migration/migrations
router.run('migration_name')

# Run all unapplied migrations
router.run()`

but it just create table migratehistory

`--auto` does not know fields from the Peewee Playhouse

We are using the BinaryJSONField from the Playhouse module for PostgreSQL (see http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#postgres-ext).

The migrations file contains then the following:

    @migrator.create_model
    class Result(pw.Model):
        energy = pw.DoubleField()
        task = pw.ForeignKeyField(db_column='task_id', rel_model=Task, to_field='id')
        filename = pw.CharField(max_length=255, null=True)
        data = pw.BinaryJSONField(index=True, null=True)

while it should maybe be something like:

import playhouse.postgres_ext as pw_pext

[...]

    @migrator.create_model
    class Result(pw.Model):
        energy = pw.DoubleField()
        task = pw.ForeignKeyField(db_column='task_id', rel_model=Task, to_field='id')
        filename = pw.CharField(max_length=255, null=True)
        data = pw_pext.BinaryJSONField(index=True, null=True)

In the original models file I have:

from playhouse.postgres_ext import BinaryJSONField

peewee_migrate/router.py: module has no attribute 'path'

File "/Users/anaconda3/lib/python3.6/site-packages/peewee_migrate/router.py", line 277, in _import_submodules
if set(package.path) & passed:

AttributeError: module 'model' has no attribute 'path'

rename_field doesn't work

The rename_field method fails with a KeyError on trying to get the old field from model._meta_fields, i.e. field = model._meta.fields[old_name]. As far as I can see, this will always fail since the model field has been renamed already and the old field name key is gone, or am I missing something?

Version: 1.0.0

Steps to reproduce:

rename the field of a model
create a migration that includes migrator.rename_field(FooModel, 'old_foo', 'new_bar')
run the migration

Expected: Success
Actual: the following exception:

  File "/usr/local/lib/python3.6/site-packages/peewee_migrate/migrator.py", line 260, in rename_column
    field = model._meta.fields[old_name]
KeyError: 'old_foo'

Auto-generated ForeignKeyField invalid syntax for rel_model

When auto-generating a migration the ForeignKeyField rel_model argument appears to be performing an str() on the related model class instead of an name.

Start migrations
invalid syntax (<string>, line 50)
Traceback (most recent call last):
  File ".../venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 111, in run_one
    migrate, rollback = self.read(name)
  File ".../venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 209, in read
    exec_in(code, scope)
  File "<string>", line 3, in exec_in
  File "<string>", line 50
    organization = pw.ForeignKeyField(db_column='organization_id', rel_model=<class 'clams.blueprints.organization.models.Organization'>, related_name='organizationuser_set')
                                                                             ^
SyntaxError: invalid syntax
Migration failed: 001_initial
Traceback (most recent call last):
  File "manage.py", line 63, in <module>
    manager.run()
  File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])
  File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/flask_script/__init__.py", line 383, in handle
    res = handle(*args, **config)
  File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "manage.py", line 41, in migrate
    migrations = router.run(name, fake=fake)
  File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 156, in run
    self.run_one(mname, migrator, fake=fake, force=fake)
  File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 111, in run_one
    migrate, rollback = self.read(name)
  File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 209, in read
    exec_in(code, scope)
  File "<string>", line 3, in exec_in
  File "<string>", line 50
    organization = pw.ForeignKeyField(db_column='organization_id', rel_model=<class 'models.Organization'>, related_name='organizationuser_set')
                                                                             ^
SyntaxError: invalid syntax