I use pgmigrate and I have about 100 migrations in one of my projects. Is it possible to group migrations in subfolders? For instance "2017", "2018" and so on. Now list of migration files looks too long and it makes me feel uncomfortable.

It is impossible to run one migration set on different databases without editing migrations.yml. The simple case: developer and production environments.

I have made a pull request with implementation of this issue: #6

Consider the situation where we want to run the migrations as part of application startup. If we have multiple application containers running concurrently, is pgmigrate taking care of this internally, e.g. by locking, to make it safe to run concurrent migrations? Flyway does it, so I was wondering if it's also safe here.

I am implementing pgmigrate inside a docker container. It is executing all version script perfectly but when using the --verbose flag I don't get any additional logging.

I am using the environment variables to pass in the connection details and use the following command.
pgmigrate --verbose -c '' -t latest migrate

Using 1.0.6

bash-5.1# pip show yandex-pgmigrate
Name: yandex-pgmigrate
Version: 1.0.6
Summary: PostgreSQL migrations made easy
Home-page: https://github.com/yandex/pgmigrate/
Author: Yandex LLC
Author-email: [email protected]
License: PostgreSQL License
Location: /usr/local/lib/python3.9/site-packages
Requires: PyYAML, sqlparse, psycopg2
Required-by:

usage

echo "executing migration"
pgmigrate --verbose -c '' -t latest migrate

echo 
echo "----------- 📋 results -----------"

results

executing migration

----------- 📋 results -----------

Hello!
After setting the base line value, pgmigrator tries to create schema_version_type. Pgmigrate was never used before setting baseline
https://github.com/yandex/pgmigrate/blob/master/doc/tutorial.md#baseline

File "<...>/pgmigrate-runner.py", line 504, in <module>
    main()
  File "/<...>/pgmigrate-runner.py", line 84, in main
    migrations_db_name=args.migrations_db_name,
  File "<...>/pgmigrate-runner.py", line 286, in migrate_to_revision
    command=command,
  File "<...>/pgmigrate-runner.py", line 302, in migrate
    pgmigrate.COMMANDS[args.cmd](config)
  File "<...>/pgmigrate.py", line 781, in migrate
    config.schema, config.cursor)
  File "<...>/pgmigrate.py", line 542, in _migrate_step
    _init_schema(schema, cursor)
  File "<...>pgmigrate.py", line 402, in _init_schema
    ('auto', 'manual'))
  File "<...>/python3.7/site-packages/psycopg2/extras.py", line 448, in execute
    return super(LoggingCursor, self).execute(query, vars)
psycopg2.errors.DuplicateObject: type "schema_version_type" already exists

What could be the cause of this problem?

В версии 1.0.5 появилась ошибка

psycopg2.OperationalError: FATAL: database "test application_name=pgmigrate-266dd3f1-081e-4ef1-ae86-5124a32" does not exist

Может вы хотели в (https://github.com/yandex/pgmigrate/blob/master/pgmigrate.py#L162) добавить: ?
conn = _create_raw_connection('{dsn}?application_name={conn_id}'.format(

Hi!

We use set role and reset role statements in beforeEach callback and afterEach callback accordingly.

For example we run pgmigrate with user A and do set role B in the beforeEach callback.
Function _init_schema executed with user A but _apply_version executed with user B because afterEach callback is not called.

Is it possible to change order of functions and do callbacks.afterEach before _apply_version?

When I install yandex-pgmigrate via pip, it tries to build psycopg2 from source, which breaks the virtualenv setup for anyone who doesn't have a proper C toolchain set up. psycopg2 is the source package, and you have to use psycopg2-binary if you want to use the pre-built binary. I tried manually install psycopg2-binary first but pgmigrate insists on using the source package. I think this just needs to be changed in setup.py. Also, any plans to upgrade to psycopg3? psycopg2 is terrible for more reasons than just this.

What we're trying to do:

We have a mature postgres database that we are in the process of revamping. Unfortunately thus far we don't have the capability to build the database schema from scratch. So we use a recent "gold" backup, and then use pgmigrate to deploy future migrations.

So far we are getting a smattering of issues that appear to be a limitation of the tool itself.

First was multi-schema support. I've tried disabling the schema check, but the DDL will only proceed past the schema error if I specify -m $schema. It is currently unknown if the next statement (that operates on a different schema) will function since now pgmigrate fails to run because it cannot find the view we wanted to drop in the first schema.

So here's the sql in question:

set schema 'schema1';
drop view schema1.some_view;

set schema 'schema2';
drop view schema2.some_other_view;

Right now as long as I specify -m schema1 I can get past the error:

psycopg2.errors.InvalidSchemaName: schema "schema1" does not exist

but now I'm getting:

psycopg2.errors.UndefinedTable: view "some_view" does not exist

I've confirmed that the user in question can see that view and perform a drop on it using PSQL and Dbeaver - even using the same environment variables. The only thing that makes sense is if it's checking for previous migrations on a table that this tool creates.

So my question is:

1. Does pgmigrate support "taking over" an already mature, multi-schema database? Or does the application require it create the full DDL in the first place in order to function?

Hi, thank you for creating this tool, it's super useful!
The tool errors for the following migrations.yml files. A connection string is provided on the CLI in all cases.

callbacks:
    beforeAll:
    beforeEach:
    afterEach:
    afterAll:

Traceback (most recent call last):
  File "/usr/local/bin/pgmigrate", line 8, in <module>
    sys.exit(_main())
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 829, in _main
    config = get_config(args.base_dir, args)
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 767, in get_config
    callbacks=_get_callbacks(conf.callbacks, conf.base_dir))
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 505, in _get_callbacks
    return _parse_dict_callbacks(callbacks, ret, base_dir)
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 482, in _parse_dict_callbacks
    for j in callbacks[i]:
TypeError: 'NoneType' object is not iterable

and for

callbacks:

Traceback (most recent call last):
  File "/usr/local/bin/pgmigrate", line 8, in <module>
    sys.exit(_main())
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 829, in _main
    config = get_config(args.base_dir, args)
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 767, in get_config
    callbacks=_get_callbacks(conf.callbacks, conf.base_dir))
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 506, in _get_callbacks
    return _parse_str_callbacks(callbacks, ret, base_dir)
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 457, in _parse_str_callbacks
    callbacks = callbacks.split(',')
AttributeError: 'NoneType' object has no attribute 'split'

If the migrations.yml is present, but completely empty, the error is

Traceback (most recent call last):
  File "/usr/local/bin/pgmigrate", line 8, in <module>
    sys.exit(_main())
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 829, in _main
    config = get_config(args.base_dir, args)
  File "/usr/local/lib/python3.8/dist-packages/pgmigrate.py", line 747, in get_config
    if i in base:
TypeError: argument of type 'NoneType' is not iterable

Use case: you have production and development environments.
Development db requires some test data to be inserted after certain migrations applied (like new table created) e.g., for (auto)testing purposes, whereas production environment definitely doesn't need this data.
How's it possible to handle this case?

Возможно ли добавить параметр командной строки для отключения проверки на ascii? В моем случае есть большое количество кода на pl/pgsql с русским текстом, десятки и сотни файлов. Прописать в каждый файл специальный комментарий затруднительно.

Migration fails on German or Russian letters in SQL comments with this message:

pgmigrate.MalformedStatement: Non ascii symbols in file: migrations/V0005__initial_companies.sql, 'ascii' codec can't encode characters in position 447-451: ordinal not in range(128)

When comments are removed it works again.

When I start pgmigrate like this:

pgmigrate -l 1 -t 1000 -d /app/migrations -c "dbname=artdb host=host1.my.net,host2.my.net,host3.my.net port=5432 user=art" migrate

It would be great if pgmigrate will add target_session_attrs=read-write to the connection string if it is not already there.

Since you are instructing the user to explicitly create a group foo with password foo, you should include that in the connection string in migrations.yml file like so:

conn: host=localhost dbname=foodb user=foo password=foo

Some times info command shows versions in wrong order.

"32": {
"description": "",
"transactional": true,
"version": 32,
"installed_on": "2017-03-29 18:58:34",
"type": "auto",
"installed_by": ""
},
"12": {
"description": "",
"transactional": true,
"version": 12,
"installed_on": "2017-03-02 21:23:48",
"type": "manual",
"installed_by": ""
},
"13": {
"description": "",
"transactional": true,
"version": 13,
"installed_on": "2017-03-02 21:24:17",
"type": "auto",
"installed_by": ""
},
....