Test django schema and data migrations, including migrations' order and best practices.
Home Page: https://pypi.org/project/django-test-migrations/
License: MIT License
Hi,
I've just discovered this package and wanted to test a new migration that I am writing. So far, the project consists of a few apps, and the app I want to write migration tests for has currently > 100 applied migrations.
For the sake of simplicity, I set migrate_from and migrate_to to an already applied migration (~101 -> 102, only a change in a field type) just to fiddle arround. Curiously, it failed.
Traceback:
`MySQLdb._exceptions.OperationalError: (3730, "Cannot drop table 'wagtailcore_collectionviewrestriction' referenced by a foreign key constraint 'wagtailcore_collecti_collectionviewrestri_47320efd_fk_wagtailco' on table 'wagtailcore_collectionviewrestriction_groups'.")
env/lib/python3.6/site-packages/MySQLdb/connections.py:239: OperationalError
The above exception was the direct cause of the following exception:
env/lib/python3.6/site-packages/django_test_migrations/contrib/unittest_case.py:36: in setUp
self.migrate_from,
env/lib/python3.6/site-packages/django_test_migrations/migrator.py:46: in apply_initial_migration
sql.drop_models_tables(self._database, style)
env/lib/python3.6/site-packages/django_test_migrations/sql.py:32: in drop_models_tables
get_execute_sql_flush_for(connection)(database_name, sql_drop_tables)
env/lib/python3.6/site-packages/django/db/backends/base/operations.py:405: in execute_sql_flush
cursor.execute(sql)
env/lib/python3.6/site-packages/django/db/backends/utils.py:68: in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
env/lib/python3.6/site-packages/django/db/backends/utils.py:77: in _execute_with_wrappers
return executor(sql, params, many, context)
env/lib/python3.6/site-packages/django/db/backends/utils.py:86: in _execute
return self.cursor.execute(sql, params)
env/lib/python3.6/site-packages/django/db/utils.py:90: in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
env/lib/python3.6/site-packages/django/db/backends/utils.py:84: in _execute
return self.cursor.execute(sql)
env/lib/python3.6/site-packages/django/db/backends/mysql/base.py:74: in execute
return self.cursor.execute(query, args)
env/lib/python3.6/site-packages/MySQLdb/cursors.py:209: in execute
res = self._query(query)
env/lib/python3.6/site-packages/MySQLdb/cursors.py:315: in _query
db.query(q)
and a lot more other errors related to drop queries. As I understand it, during setup, django-test-migrations takes the current test_db and deletes all models in the database, and then reapplies all migrations up and including to migrate_from. In my case, this seems to fail. I cannot imagine what I could have done wrong in my testcase, as the error occurs during the setUp function of the MigratorTestCase.
I don't think that it has anything to do with wagtail, as the testcase seems to fail with different errors on each run:
self = <_mysql.connection closed at 0x35419e8>
query = b'DROP TABLE `auth_group` CASCADE'
def query(self, query):
# Since _mysql releases GIL while querying, we need immutable buffer.
if isinstance(query, bytearray):
query = bytes(query)
> _mysql.connection.query(self, query)
E django.db.utils.OperationalError: (3730, "Cannot drop table 'auth_group' referenced by a foreign key constraint 'auth_group_permissions_group_id_b120cbf9_fk_auth_group_id' on table 'auth_group_permissions'.")
Testcase:
class TestMigrations(MigratorTestCase):
migrate_from = ('myapp', None)
migrate_to = ('myapp', '001_initial')
def prepare(self):
pass
def test_migration001(self):
self.assertTrue(True)Maybe I am missing something crucial and obvious, so here is what I did:
install django-test-migrations with pip
wrote a unittest testcase, as provided in the example (the actual tc is just an assertTrue(True)
ran manage.py test
I did not add anything to INSTALLED_APPS whatsoever.
Any ideas?
DB: MYSQL 8.0.19 mysqlclient (1.4.6)
Django: 3.0.2
django-test-migrations: 1.0.0
These two methods do the same thing.
Why not call it migrate and reload the graph each time?
From my understanding before(('app', '0003')) should migrate to dependencies of 0003 but not the 0003 itself.
Calling tests with via pytest with --migrations or via Django's standard test runner (which apply migrations by default when using TransactionTestCase) can slow down tests a bit and when running tests in CI/CD pipeline we want to get feedback as fast as possible and often it's good idea to run standard and fast unit tests in parallel with migrations tests etc.
Currently it's really hard to run/skip only migrations tests and the easy solution is to add:
marker to every node that requested migrator_factory fixturetag to every subclass of MigrationTestCase - https://docs.djangoproject.com/en/3.0/topics/testing/tools/#tagging-testsI am preloading some test data via a migration, and once my sole migration test has run, it wipes that data out and other tests fail the next time i do a test run.
The culprit is
@pytest.fixture()
def migrator_factory(request, transactional_db, django_db_use_migrations):If I remove transactional_db then tests pass again
My conftest looks like this:
@pytest.fixture(autouse=True)
def enable_db_access_for_all_tests(db):
pass
@pytest.fixture()
def migrator():
migrator = Migrator(database='default')
yield migrator
migrator.reset()My custom migrator fixture gets tests passing again.
Related: #34 (review)
The previous approach proposed in #11 uses a fixture to wrap the test which has the desired effect of muting signals before they are first emitted (pre_migrate) and un-muting them after the last emission (post_migrate).
The implementation in ab393b2 however performs the signal muting inside the transaction where it has no effect (Signals are muted after first emission and un-muted before last emission).
The following PoC/hack demonstrates a working approach master...GertBurger:move-signal-muting-to-fixture
I am not very familiar with pytest and its fixture ordering so the above approach may not be sufficient to ensure that the signal muting fixture always wraps the transaction fixture.
Right now it fails:
poetry run pytest --nomigrations --cov-fail-under=0
Traceback (most recent call last):
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pytest_django/plugin.py", line 179, in _handle_import_error
yield
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pytest_django/plugin.py", line 351, in pytest_load_initial_conftests
dj_settings.DATABASES
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/django/conf/__init__.py", line 79, in __getattr__
self._setup(name)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/django/conf/__init__.py", line 66, in _setup
self._wrapped = Settings(settings_module)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/django/conf/__init__.py", line 157, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'django_test_app.settings'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/bin/pytest", line 8, in <module>
sys.exit(console_main())
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 188, in console_main
code = main()
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 146, in main
config = _prepareconfig(args, plugins)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 325, in _prepareconfig
config = pluginmanager.hook.pytest_cmdline_parse(
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
gen.send(outcome)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/_pytest/helpconfig.py", line 102, in pytest_cmdline_parse
config: Config = outcome.get_result()
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1013, in pytest_cmdline_parse
self.parse(args)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1301, in parse
self._preparse(args, addopts=addopts)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1203, in _preparse
self.hook.pytest_load_initial_conftests(
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pytest_django/plugin.py", line 350, in pytest_load_initial_conftests
with _handle_import_error(_django_project_scan_outcome):
File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/home/runner/work/django-test-migrations/django-test-migrations/.venv/lib/python3.10/site-packages/pytest_django/plugin.py", line 183, in _handle_import_error
raise ImportError(msg)
ImportError: No module named 'django_test_app.settings'
pytest-django could not find a Django project (no manage.py file could be found). You must explicitly add your Django project to the Python path to have it picked up.
make: *** [Makefile:11: unit] Error 1
Link: https://github.com/wemake-services/django-test-migrations/runs/5087399817?check_suite_focus=true
I know Django 3.2 is not officially supported, but I'm getting the following error importing MigratorTestCase with Django 3.2.0.
I'm on MacOS 10.15.6.
>>> env/bin/python
Python 3.7.0 (default, Oct 8 2020, 10:13:01)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
>>> from django_test_migrations.contrib.unittest_case import MigratorTestCase
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "[repo_root]/env/lib/python3.7/site-packages/django_test_migrations/contrib/unittest_case.py", line 7, in <module>
from django_test_migrations.migrator import Migrator
File "[repo_root]/env/lib/python3.7/site-packages/django_test_migrations/migrator.py", line 9, in <module>
from django_test_migrations import sql
File "[repo_root]/env/lib/python3.7/site-packages/django_test_migrations/sql.py", line 5, in <module>
from django.db import DefaultConnectionProxy, connections, transaction
ImportError: cannot import name 'DefaultConnectionProxy' from 'django.db' ([repo_root]/env/lib/python3.7/site-packages/django/db/__init__.py)
I've removed the absolute path for brevity.
The same error occurs on Django 3.2.1.
This works correctly with Django 3.1.9.
Is there any chance of this package being updated for compatibility with Django 3.2?
Thanks in advance for any help.
To use fewer GitHub actions' minutes, we should run the test workflow only once for all PRs.
Possible solution: https://github.community/t/how-to-trigger-an-action-on-push-or-pull-request-but-not-both/16662/10
Today I found a code like this:
class Migration(migrations.Migration):
# initial = True
dependencies = []And this was an initial migration for an app.
This suprised me:
What am I concerned about? I have an impression that all migrations with dependencies = [] must be initial, because otherwise they should be dependent on a previous migration. Am I correct?
So, I guess we can check this pretty easily in case initial really works this way.
Hello 👋
last year I have encountered problems with Django’s migrations similar to these described by you on your blog and after that, I also have decided to create a library to test that kind of code properly - http://github.com/skarzi/django-test-migrations.
We were both inspired by the same articles/code snippets so our code is almost identical.
I wanted to release my code after some polishing and refactoring, but you did it first. I think we could start working together on one project t's always more profitable for the community. Below I list some issues me and other developers (it has been used in several side and commercial projects) have spotted in my implementation:
--migrations cli option) and then migrations are reverted to these specified in .before(). Such an approach can cause a really ugly tracebacks.migrate command is called but it's run only on default DB (check Django’s test case source code for proper solution - https://github.com/django/django/blob/master/django/test/testcases.py#L1015)settings.MIGRATION_MODULES for migration testsI also wanted to add intergration with factory_boy - creating models' instances manually (with ModelClass.objects.create() or ModelClass.objects.bulk_create()) is really tiresome, but I need to do more experiments to decide on the best API for that.
New LTS version is out!
In #91 we are introducing new Django check that validate system timeout settings on following database:
postgresql -statement_timeoutmysql - max_execution_timeThe idea behind this group of checks is to help developers configure databases according to best practices.
If you have any ideas of such rules/checks, please shere in comments to this issue!
Some nice articles/sites about database configuration/settings:
It would be useful to have special key for ignoring all migrations inside an app.
Consider case when we have some third party apps and want to ignore all migrations or for all third party apps at all.
Right now we should do something like this:
DTM_IGNORED_MIGRATIONS = [
('taggit', '0002_auto_20150616_2121'),
('two_factor', '0002_auto_20150110_0810'),
('two_factor', '0003_auto_20150817_1733'),
('two_factor', '0004_auto_20160205_1827'),
('two_factor', '0005_auto_20160224_0450'),
('waffle', '0002_auto_20161201_0958'),
]With the special key, for example, it'll get much better:
THIRD_PARTY_APPS = (
'taggit',
'two_factor',
'waffle',
)
# Apps specific for this project go here.
LOCAL_APPS = (
'app_1',
'app_2',
)
INSTALLED_APPS = THIRD_PARTY_APPS + LOCAL_APPS
DTM_IGNORED_MIGRATIONS = [
(app, '*') for app in THIRD_PARTY_APPS
]Hi, I just wanted to give this project a go as I want to run a rather complex data migration.
However, as soon as I add the migrator fixture to my test case it's being skipped.
@pytest.mark.django_db
def test_datamig(migrator):
pass
This is the test output I get:
Test session starts (platform: linux, Python 3.8.4, pytest 5.4.3, pytest-sugar 0.9.4)
cachedir: .pytest_cache
django: settings: xyz.settings (from option)
rootdir: /code, inifile: setup.cfg
plugins: django-test-migrations-1.0.0, xdist-1.34.0, celery-4.4.7, sugar-0.9.4, django-3.9.0, mock-3.3.0, Faker-4.1.2, socket-0.3.5, cov-2.10.1, forked-1.3.0
[gw0] linux Python 3.8.4 cwd: /code/sales_cockpit_be
[gw0] Python 3.8.4 (default, Jul 14 2020, 02:56:59) -- [GCC 8.3.0]
gw0 [1]
scheduling tests via LoadScheduling
xyz/tests/test_migration.py::test_datamig s 100% ██████████
[gw0] SKIPPED xyz/tests/test_migration.py
Results (14.02s):
1 skipped
I also tried creating migrator manually inside the test with migrator = Migrator() or migrator_factory, but the issue persists.
I use the 1.0.0 version.
Does any1 have an idea what the reason could be?
Dependabot can't resolve your Python dependency files.
As a result, Dependabot couldn't update your dependencies.
The error Dependabot encountered was:
Creating virtualenv django-test-migrations-8iGcF5E9-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...
[PackageNotFound]
Package pytest-pythonpath (0.7.3) not found.
If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.
Dependabot can't resolve your Python dependency files.
As a result, Dependabot couldn't update your dependencies.
The error Dependabot encountered was:
Creating virtualenv django-test-migrations-BQFQEyOO-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...
[PackageNotFound]
Package safety (1.8.5) not found.
If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.
https://github.com/wemake-services/django-test-migrations/blob/master/pyproject.toml#L55
Related: #82 (comment)
Thanks to @lee-vg for the report
My migration is doing something like the following:
Note: Parent and Child are both subclasses of Activity, and there is a generic relation between them (not sure if this is related to the issues I'm seeing)
@pytest.mark.django_db
def test_activity_migration(migrator: Migrator):
old_state = migrator.apply_tested_migration(("app", an_old_migration_name))
Parent = old_state.apps.get_model("app", "Parent")
Child = old_state.apps.get_model("app", "Child")
# Data setup
parent = Parent.objects.create(...)
Child.objects.create(..., parent=parent)
orig_parents_values = list(Parent.objects.values())
orig_children_values = list(Child.objects.values())
# Finish all migrations on app
state_0007 = migrator.apply_tested_migration(("app", the_newer_migration_name))
state_0007_Parent = state_0007.apps.get_model("app", "Parent")
state_0007_Child = state_0007.apps.get_model("app", "Child")
state_0007_Activity = state_0007.apps.get_model("app", "Activity")
assert models.Activity.objects.count() == 2
# do not make assertion for the parent_activity_id
orig_children_values[0].pop("parent_id")
parent_0007 = state_0007_Parent.objects.values().first()
child_0007 = state_0007_Child.objects.values().first()
assert (
state_0007_Activity.objects.filter(id=parent["activity_ptr_id"])
.values(*orig_parents_values[0].keys())
.first()
== orig_parents_values[0]
)
assert (
state_0007_Activity.objects.filter(id=childActivity["activity_ptr_id"])
.values(*orig_children_values[0].keys())
.first()
== orig_children_values[0]
)After squashing 2 unrelated migrations together (using manage.py squashmigrations, this test no longer runs. It gives me this traceback:
______________________________________________________________ ERROR at teardown of test_activity_migration ______________________________________________________________
[snipped]
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django_test_migrations/migrator.py:76:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/core/management/__init__.py:181: in call_command
return command.execute(*args, **defaults)
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/core/management/base.py:398: in execute
output = self.handle(*args, **options)
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/core/management/base.py:89: in wrapped
res = handle_func(*args, **kwargs)
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/core/management/commands/migrate.py:95: in handle
executor.loader.check_consistent_history(connection)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[snipped]
E django.db.migrations.exceptions.InconsistentMigrationHistory: Migration app.0015_squashed_migration_name is applied before its dependency app.0014_previous_migration_name on database 'default'.
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/db/migrations/loader.py:306: InconsistentMigrationHistory
___________________________________________________________ ERROR at teardown of test_automatic_child_activity ___________________________________________________________
[snipped]
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django_test_migrations/migrator.py:76:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/core/management/__init__.py:181: in call_command
return command.execute(*args, **defaults)
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/core/management/base.py:398: in execute
output = self.handle(*args, **options)
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/core/management/base.py:89: in wrapped
res = handle_func(*args, **kwargs)
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/core/management/commands/migrate.py:95: in handle
executor.loader.check_consistent_history(connection)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.migrations.loader.MigrationLoader object at 0x7f808bbd0fd0>, connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f8091eaaa30>
[snipped]
E django.db.migrations.exceptions.InconsistentMigrationHistory: Migration app.0015_squashed_migration_name is applied before its dependency app.0014_previous_migration_name on database 'default'.
/home/joel/.cache/pypoetry/virtualenvs/relate-en3S9GuS-py3.8/lib/python3.8/site-packages/django/db/migrations/loader.py:306: InconsistentMigrationHistoryAfter this there is a second failure on the same test, related to the migrations not having applied successfully.
This line is in poetry.lock
[package.dependencies]
attrs = ">=20"
typing_extensions = {version = "*", markers = "python_version >= \"3.7\" and python_version < \"3.8\""}
However, on a fresh Ubuntu 20.04 system, after I install pytest and pytest-django, I got this error when I ran my tests:
traceback snipped
File "/home/vagrant/.local/share/virtualenvs/redacted-ikMrF094/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
exec(co, module.__dict__)
File "/home/vagrant/.local/share/virtualenvs/redacted/lib/python3.8/site-packages/django_test_migrations/contrib/pytest_plugin.py", line 6, in <module>
from django_test_migrations.constants import MIGRATION_TEST_MARKER
File "/home/vagrant/.local/share/virtualenvs/redacted/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
exec(co, module.__dict__)
File "/home/vagrant/.local/share/virtualenvs/redacted/lib/python3.8/site-packages/django_test_migrations/constants.py", line 1, in <module>
from typing_extensions import Final
ModuleNotFoundError: No module named 'typing_extensions'
It would appear this module still needs to depend on typing_extensions.
Ubuntu 20.04
Python 3.8.6
Dependabot can't resolve your Python dependency files.
As a result, Dependabot couldn't update your dependencies.
The error Dependabot encountered was:
Creating virtualenv django-test-migrations-OjLT_eJI-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...
[PackageNotFound]
Package safety (1.8.5) not found.
If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.
Error message
____________________________________________________ ERROR at teardown of test _____________________________________________________
self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f8187ac3588>
def _commit(self):
if self.connection is not None:
with self.wrap_database_errors:
> return self.connection.commit()
E psycopg2.errors.ForeignKeyViolation: insert or update on table "auth_permission" violates foreign key constraint "auth_permission_content_type_id_2f476e4b_fk_django_co"
E DETAIL: Key (content_type_id)=(385) is not present in table "django_content_type".
../../.pyenv/versions/_/lib/python3.7/site-packages/django/db/backends/base/base.py:236: ForeignKeyViolation
The above exception was the direct cause of the following exception:
self = <django.test.testcases.TransactionTestCase testMethod=__init__>
def _post_teardown(self):
"""Performs any post-test things. This includes:
* Flushing the contents of the database, to leave a clean slate. If
the class has an 'available_apps' attribute, post_migrate isn't fired.
* Force-closing the connection, so the next test gets a clean cursor.
"""
try:
> self._fixture_teardown()
../../.pyenv/versions/_/lib/python3.7/site-packages/django/test/testcases.py:925:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../.pyenv/versions/_/lib/python3.7/site-packages/django/test/testcases.py:960: in _fixture_teardown
inhibit_post_migrate=inhibit_post_migrate)
../../.pyenv/versions/_/lib/python3.7/site-packages/django/core/management/__init__.py:131: in call_command
return command.execute(*args, **defaults)
../../.pyenv/versions/_/lib/python3.7/site-packages/django/core/management/base.py:330: in execute
output = self.handle(*args, **options)
../../.pyenv/versions/_/lib/python3.7/site-packages/django/core/management/commands/flush.py:88: in handle
emit_post_migrate_signal(verbosity, interactive, database)
../../.pyenv/versions/_/lib/python3.7/site-packages/django/core/management/sql.py:53: in emit_post_migrate_signal
**kwargs
../../.pyenv/versions/_/lib/python3.7/site-packages/django/dispatch/dispatcher.py:193: in send
for receiver in self._live_receivers(sender)
../../.pyenv/versions/_/lib/python3.7/site-packages/django/dispatch/dispatcher.py:193: in <listcomp>
for receiver in self._live_receivers(sender)
../../.pyenv/versions/_/lib/python3.7/site-packages/django/contrib/auth/management/__init__.py:83: in create_permissions
Permission.objects.using(using).bulk_create(perms)
../../.pyenv/versions/_/lib/python3.7/site-packages/django/db/models/query.py:449: in bulk_create
obj_without_pk._state.db = self.db
../../.pyenv/versions/_/lib/python3.7/site-packages/django/db/transaction.py:223: in __exit__
connection.commit()
../../.pyenv/versions/_/lib/python3.7/site-packages/django/db/backends/base/base.py:262: in commit
self._commit()
../../.pyenv/versions/_/lib/python3.7/site-packages/django/db/backends/base/base.py:236: in _commit
return self.connection.commit()
../../.pyenv/versions/_/lib/python3.7/site-packages/django/db/utils.py:94: in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
../../.pyenv/versions/_/lib/python3.7/site-packages/django/utils/six.py:685: in reraise
raise value.with_traceback(tb)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f8187ac3588>
def _commit(self):
if self.connection is not None:
with self.wrap_database_errors:
> return self.connection.commit()
E django.db.utils.IntegrityError: insert or update on table "auth_permission" violates foreign key constraint "auth_permission_content_type_id_2f476e4b_fk_django_co"
E DETAIL: Key (content_type_id)=(385) is not present in table "django_content_type".
../../.pyenv/versions/_/lib/python3.7/site-packages/django/db/backends/base/base.py:236: IntegrityError
Workaround
@pytest.fixture(autouse=True)
def _mute_post_migrate_signal():
from django.db.models.signals import post_migrate
restore, post_migrate.receivers = post_migrate.receivers, []
yield
post_migrate.receivers = restoreHello,
I would like to know if it's possible to upgrade the dependency to typing_extensions >= 4.0.0.
As of now we cannot introduce some external dependencies like pydantic in our project because of dependency conflicts with django-test-migrations. I don't exactly know if some breaking changes have been introduce in v4, however on the project README we can read the following:
Starting with version 4.0.0, typing_extensions uses Semantic Versioning. The major version is incremented for all backwards-incompatible changes, including dropping support for older Python versions. Therefore, it's safe to depend on typing_extensions like this: typing_extensions >=x.y, <(x+1), where x.y is the first version that includes all features you need.
In #91 we are introducing more Django checks and according to plans this number of checks will grown, so it's time to establish some rules of check messages levels and it's usage.
Django checks messages has following levels (they are similar to logging levels):
DEBUGINFOSUCCESSWARNINGERRORReference: https://docs.djangoproject.com/en/3.0/topics/checks/#messages
Dependabot couldn't authenticate with https://pypi.python.org/simple/.
You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.
See wemake-services/wemake-django-template#1064 for more details.
I'm running into a problem when I apply a migration with Migrator.apply_initial_migration() and then try to construct model instances using the returned app state. One model has a foreign key to another model, but the foreign key field does not accept an instance of that model (see below for the exact output and exception).
I tried to reduce the setup as much a possible. In the end, two additional models and 2 indexes were necessary to trigger the issue. I tested it with django-test-migrations 1.2.0 and with Django 3.2.13 and 4.0.4. The attached ZIP archive contains a complete project which reproduces the issue:
To run the test, use the following couple of commands:
python3.10 -m venv venv
venv/bin/pip install django~=3.2 pytest-django django-test-migrations
venv/bin/pytest test_bad.py
When running this, you should see the following output:
___________________________________ test_bad ___________________________________
migrator = <django_test_migrations.migrator.Migrator object at 0x105c44430>
def test_bad(migrator):
state = migrator.apply_initial_migration(('my_app', '0002_foo'))
A = state.apps.get_model('my_app', 'A')
B = state.apps.get_model('my_app', 'B')
print(id(A), id(B.a_fk.field.related_model))
> B.objects.create(a_fk=A.objects.create(foo=1))
test_bad.py:9:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv/lib/python3.10/site-packages/django/db/models/manager.py:85: in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
venv/lib/python3.10/site-packages/django/db/models/query.py:512: in create
obj = self.model(**kwargs)
venv/lib/python3.10/site-packages/django/db/models/base.py:541: in __init__
_setattr(self, field.name, rel_obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.models.fields.related_descriptors.ForwardManyToOneDescriptor object at 0x105c47520>
instance = <B: B object (None)>, value = <A: A object (1)>
def __set__(self, instance, value):
"""
Set the related instance through the forward relation.
With the example above, when setting ``child.parent = parent``:
- ``self`` is the descriptor managing the ``parent`` attribute
- ``instance`` is the ``child`` instance
- ``value`` is the ``parent`` instance on the right of the equal sign
"""
# An object must be an instance of the related class.
if value is not None and not isinstance(
value, self.field.remote_field.model._meta.concrete_model
):
> raise ValueError(
'Cannot assign "%r": "%s.%s" must be a "%s" instance.'
% (
value,
instance._meta.object_name,
self.field.name,
self.field.remote_field.model._meta.object_name,
)
E ValueError: Cannot assign "<A: A object (1)>": "B.a_fk" must be a "A" instance.
venv/lib/python3.10/site-packages/django/db/models/fields/related_descriptors.py:235: ValueError
I wrote a test for the migration I am writing by hand, followed the guide for unittest and so on. Now, when I run pytest to run the actual test (this one alone) I get into the followin error:
E django.db.utils.ProgrammingError: extension "ltree" already exists
I searched here as best as I could but could not find anything.
Worth mentioning we run our tests with pytest and we are using the pytest-django plugin to manage django setup and so on.
Thank you in advance for your help!
Many of existing Django projects use factory-boy or other similar solution to create models instances before actual tests, so it will be really handy to provide some integration with such tools to make migrations testing easier and less verbose than Model.objects.create() etc.
Currently multiple databases are supported by creating instance of Migrator for each database, but only default database is cleaned at the end of test - https://github.com/wemake-services/django-test-migrations/blob/master/django_test_migrations/migrator.py#L77
Also using migrate management command is not the best idea when it comes to performance, because there is no need to apply all unapplied migrations just before the next test, which should start with clean state.
Django's TestCase uses flush management command to clean databases after test.
We need to reuse Django's code responsible for cleaning database on test teardown (or establish our own way to do that) and fix Migrator.reset() method to clean Migrator.database instead of default one.
Hi, is it possible to use an sql extract to test the migration procedure? ie
class TestDirectMigration(MigratorTestCase,_TestUtils):
"""This class is used to test direct migrations."""
migrate_from = ('app', '0016_auto_20200416_1527')
migrate_to = ('app', '0017_auto_20200511_1641')
def prepare(self):
"""Prepare some data before the migration."""
test_data = self.resource_path('migration_test_data/0016_auto_20200416_1527.sql')
with connection.cursor() as cursor:
for l in test_data.read_text().strip().split(';'):
cursor.execute(l)Dependabot can't resolve your Python dependency files.
As a result, Dependabot couldn't update your dependencies.
The error Dependabot encountered was:
Creating virtualenv django-test-migrations-Q_RgL_FZ-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...
[PackageNotFound]
Package safety (1.8.5) not found.
If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.
django_test_migration is growing, so it's pytest plugin also, e.g. in #138 more logic was added to it.
Currently, most tests use subprocess to run pytest and make some assertions on its output.
The following approach is limited, it requires quite much code in each test and it runs pytest on all project's tests by default.
pytest has pytester plugin that is really useful for testing custom pytest plugins, so it will be nice to add more detailed tests of our django_test_migration plugin using pytester.
We could also try to remove pytest_plugin from the coverage:run omit option, because it should be covered by tests then.
Current django-test-migrations tests setup runs all migrations forward (it's done either by Django’s test case or pytest's --migrations cli option) and then migrations are reverted by .before() to these specified in its arguments. Such an approach can cause a really ugly and unintuitive tracebacks.
Proper solution will be defer migrations running until .before() is called.
This way we can avoid applying unnecessary migrations' operations that will be reverted anyway in .before() and migrations's exceptions will be raised only when particular migration will be tested (currently if any migration even the last one in the plan is incorrect and will raise some exception then all migrations tests will be interrupted by this exception which will produce really unintuituve tracebacks).
If anyone has some other ideas or is against it, please raise your hand, so we can figure out the best solution for this issue.
Hi, I'm working on a medium-sized Django codebase and recently introduced django-test-migrations to test our migrations. It's a great project and solves a need that many serious Django apps have. We'd like to see it improved by fixing a blocking issue we're seeing.
Unfortunately, when our test suite isn't ordered in the usual fashion, I'm seeing failures. This is true for random ordering, and predictable but non-standard ordering (see below).
I'm careful to draw conclusions, but to me it seems likely that the issue is with our use of django-test-migrations. It doesn't always appear, but it only appears when those tests are run.
I'm a bit at loss about what might be causing the issue, and don't know how to debug further. I do have some work-time available to help debug this issue. I'd be interested to know if you've heard of this issue before. And please let me know how I can assist.
ContentType cache before and after each test: failures. I did this because it was documented as possible solution here. Also see #10827.The two tests that seem to be causing the unrelated failures are here: https://gist.github.com/maikhoepfel/0c4c52c86f047c79b85d8b4e98af194f I don't think they do anything special.
The tests I added aren't the first tests for migrations. We've been employing something very similar to the blog post where it all started. Only when we added django-test-migrations did we see the issue. This is surprising to me, because the actual code executed is similar.
django.db.utils.IntegrityError: insert or update on table "core_stock" violates foreign key constraint "core_stock_content_type_id_d8f48c0c_fk_django_content_type_id" DETAIL: Key (content_type_id)=(1) is not present in table "django_content_type".
self = <django.db.backends.utils.CursorWrapper object at 0x7f4b5f881dd8>
sql = 'SET CONSTRAINTS ALL IMMEDIATE', params = None
ignored_wrapper_args = (False, {'connection': <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f4b675bd4a8>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x7f4b5f881dd8>})
def _execute(self, sql, params, *ignored_wrapper_args):
self.db.validate_no_broken_transaction()
with self.db.wrap_database_errors:
if params is None:
> return self.cursor.execute(sql)
E psycopg2.errors.ForeignKeyViolation: insert or update on table "core_stock" violates foreign key constraint "core_stock_content_type_id_d8f48c0c_fk_django_content_type_id"
E DETAIL: Key (content_type_id)=(1) is not present in table "django_content_type".
venv/lib/python3.6/site-packages/django/db/backends/utils.py:82: ForeignKeyViolation
The above exception was the direct cause of the following exception:
self = <tests.ondemand.views.test_views.StoreCartTestCase testMethod=test_checkout_cart_pricing_rule_limit_add_occurences>
result = <TestCaseFunction test_checkout_cart_pricing_rule_limit_add_occurences>
def __call__(self, result=None):
"""
Wrapper around default __call__ method to perform common Django test
set up. This means that user-defined Test Cases aren't required to
include a call to super().setUp().
"""
testMethod = getattr(self, self._testMethodName)
skipped = (
getattr(self.__class__, "__unittest_skip__", False) or
getattr(testMethod, "__unittest_skip__", False)
)
if not skipped:
try:
self._pre_setup()
except Exception:
result.addError(self, sys.exc_info())
return
super().__call__(result)
if not skipped:
try:
> self._post_teardown()
venv/lib/python3.6/site-packages/django/test/testcases.py:274:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv/lib/python3.6/site-packages/django/test/testcases.py:1009: in _post_teardown
self._fixture_teardown()
venv/lib/python3.6/site-packages/django/test/testcases.py:1177: in _fixture_teardown
connections[db_name].check_constraints()
venv/lib/python3.6/site-packages/django/db/backends/postgresql/base.py:246: in check_constraints
self.cursor().execute('SET CONSTRAINTS ALL IMMEDIATE')
venv/lib/python3.6/site-packages/sentry_sdk/integrations/django/__init__.py:487: in execute
return real_execute(self, sql, params)
venv/lib/python3.6/site-packages/django/db/backends/utils.py:67: in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
venv/lib/python3.6/site-packages/django/db/backends/utils.py:76: in _execute_with_wrappers
return executor(sql, params, many, context)
venv/lib/python3.6/site-packages/django/db/backends/utils.py:82: in _execute
return self.cursor.execute(sql)
venv/lib/python3.6/site-packages/django/db/utils.py:89: in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.backends.utils.CursorWrapper object at 0x7f4b5f881dd8>
sql = 'SET CONSTRAINTS ALL IMMEDIATE', params = None
ignored_wrapper_args = (False, {'connection': <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f4b675bd4a8>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x7f4b5f881dd8>})
def _execute(self, sql, params, *ignored_wrapper_args):
self.db.validate_no_broken_transaction()
with self.db.wrap_database_errors:
if params is None:
> return self.cursor.execute(sql)
E django.db.utils.IntegrityError: insert or update on table "core_stock" violates foreign key constraint "core_stock_content_type_id_d8f48c0c_fk_django_content_type_id"
E DETAIL: Key (content_type_id)=(1) is not present in table "django_content_type".
venv/lib/python3.6/site-packages/django/db/backends/utils.py:82: IntegrityError
Dependabot can't resolve your Python dependency files.
As a result, Dependabot couldn't update your dependencies.
The error Dependabot encountered was:
Creating virtualenv django-test-migrations-A4eaEADC-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...
[PackageNotFound]
Package safety (1.8.5) not found.
If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.
Our library is closely connected with databases, so it will be really nice to have tests that run on each database engine supported by Django (especially when #91 will be merged), so I'm opening this issue to open some discussion about adding other database engines to our CI
Dependabot can't resolve your Python dependency files.
As a result, Dependabot couldn't update your dependencies.
The error Dependabot encountered was:
Creating virtualenv django-test-migrations-aTEqW9gF-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...
[PackageNotFound]
Package safety (1.8.5) not found.
If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.
Related #13
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.