Comments (12)
skarzi
commented on May 24, 2024
8
@Hafnernuss thank you for the great and detailed investigation!
I will try to fix issues related to MySQL in the following days.
Fix for django>=3.1 is already on master, but we need to fix a few other issues before releasing the new version.
from django-test-migrations.
Hafnernuss
commented on May 24, 2024
3
Sorry for the late reply. I had a little bit of time and played around.
What I did:
- fresh mysql db
- removed all wagtail references from the app
- ran all migrations
- created following testcase (same as above):
class TestMigrations(MigratorTestCase):
migrate_from = ('myapp', None)
migrate_to = ('myapp', '001_initial')
def prepare(self):
pass
def test_migration001(self):
self.assertTrue(True)Here is the error I get:
self.migrate_from,
env/lib/python3.6/site-packages/django_test_migrations/migrator.py:46: in apply_initial_migration
sql.drop_models_tables(self._database, style)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
database_name = 'default'
style = <django.core.management.color.Style object at 0x7fcc0ed13518>
def drop_models_tables(
database_name: str,
style: Optional[Style] = None,
) -> None:
"""Drop all installed Django's models tables."""
style = style or no_style()
connection = connections[database_name]
tables = connection.introspection.django_table_names(
only_existing=True,
include_views=False,
)
sql_drop_tables = [
connection.SchemaEditorClass.sql_delete_table % {
'table': style.SQL_FIELD(connection.ops.quote_name(table)),
}
for table in tables
]
if sql_drop_tables:
> get_execute_sql_flush_for(connection)(database_name, sql_drop_tables)
E TypeError: execute_sql_flush() takes 2 positional arguments but 3 were given
env/lib/python3.6/site-packages/django_test_migrations/sql.py:32: TypeError
I am completly clueless.
Edit: Unfortunately, I had to migrate to django 3.1.1 in the meantime.
from django-test-migrations.
Hafnernuss
commented on May 24, 2024
3
Apparently, the error above is indeed caused by Django 3.1. I have created a MRE that runs on Django 3.0. The problem does not occur when usig SQLITE, however, it does when using MYSQL. The migration does not even have to be applied.
All you have to do is create a env (requirements provided) and change the database credentials in the settings.py file.
The error I recieve:
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
E
======================================================================
ERROR: test_migration001 (sample.tests.TestPopulatePlayerPositions)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 74, in execute
return self.cursor.execute(query, args)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (3730, "Cannot drop table 'django_content_type' referenced by a foreign key constraint 'auth_permission_content_type_id_2f476e4b_fk_django_co' on table 'auth_permission'.")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django_test_migrations/contrib/unittest_case.py", line 35, in setUp
self.old_state = self._migrator.apply_initial_migration(
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django_test_migrations/migrator.py", line 46, in apply_initial_migration
sql.drop_models_tables(self._database, style)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django_test_migrations/sql.py", line 32, in drop_models_tables
get_execute_sql_flush_for(connection)(database_name, sql_drop_tables)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/base/operations.py", line 405, in execute_sql_flush
cursor.execute(sql)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 74, in execute
return self.cursor.execute(query, args)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/home/tms/migrate_test/migrate_test/env/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
_mysql.connection.query(self, query)
django.db.utils.OperationalError: (3730, "Cannot drop table 'django_content_type' referenced by a foreign key constraint 'auth_permission_content_type_id_2f476e4b_fk_django_co' on table 'auth_permission'.")
----------------------------------------------------------------------
Ran 1 test in 0.510s
FAILED (errors=1)
Tested on python 3.6 and 3.8.
from django-test-migrations.
skarzi
commented on May 24, 2024
1
@skarzi my wild guess is that it can be mysql related failure. What do you think?
The issue described by @Hafnernuss in the first comment is related to MySQL.
In django.db.backends.sql.operations.DatabaseOperations.sql_flush django disables FOREIGN_KEY_CHECKS before executing SQL FLUSH and the same behaviour should be used to trigger SQL DELETE queries.
Currently, we have django_test_migrations.db module, where we can move django_test_migrations.sql module and refactor it a bit to make adding such features per DB backend easier.
That's also a great time to focus on #107, so performing above mentioned refactoring will be easier.
from django-test-migrations.
Hafnernuss
commented on May 24, 2024
1
@skarzi sure, I'll try ;)
from django-test-migrations.
sobolevn
commented on May 24, 2024
@skarzi my wild guess is that it can be mysql related failure. What do you think?
from django-test-migrations.
Hafnernuss
commented on May 24, 2024
I did a quick test with the same project on another system that uses sqlite as backend. The error is different there and only occurs in the tearDown of the testcase:
self = <django.db.backends.utils.CursorWrapper object at 0x7f71205fd198>
sql = 'CREATE TABLE "wagtailsearch_editorspick" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "sort_order" integer NULL,...ription" text NOT NULL, "page_id" integer NOT NULL REFERENCES "wagtailcore_page" ("id") DEFERRABLE INITIALLY DEFERRED)'
params = None
ignored_wrapper_args = (False, {'connection': <django.db.backends.sqlite3.base.DatabaseWrapper object at 0x7f7122080780>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x7f71205fd198>})
def _execute(self, sql, params, *ignored_wrapper_args):
self.db.validate_no_broken_transaction()
with self.db.wrap_database_errors:
if params is None:
# params default might be backend specific.
> return self.cursor.execute(sql)
env/lib/python3.6/site-packages/django/db/backends/utils.py:84:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.backends.sqlite3.base.SQLiteCursorWrapper object at 0x7f711fe83d38>
query = 'CREATE TABLE "wagtailsearch_editorspick" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "sort_order" integer NULL,...ription" text NOT NULL, "page_id" integer NOT NULL REFERENCES "wagtailcore_page" ("id") DEFERRABLE INITIALLY DEFERRED)'
params = None
def execute(self, query, params=None):
if params is None:
> return Database.Cursor.execute(self, query)
E sqlite3.OperationalError: table "wagtailsearch_editorspick" already exists
env/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py:394: OperationalError
The above exception was the direct cause of the following exception:
env/lib/python3.6/site-packages/django_test_migrations/contrib/unittest_case.py:50: in tearDown
self._migrator.reset()
env/lib/python3.6/site-packages/django_test_migrations/migrator.py:69: in reset
call_command('migrate', verbosity=0, database=self._database)
env/lib/python3.6/site-packages/django/core/management/__init__.py:168: in call_command
return command.execute(*args, **defaults)
env/lib/python3.6/site-packages/django/core/management/base.py:369: in execute
output = self.handle(*args, **options)
env/lib/python3.6/site-packages/django/core/management/base.py:83: in wrapped
res = handle_func(*args, **kwargs)
env/lib/python3.6/site-packages/django/core/management/commands/migrate.py:233: in handle
fake_initial=fake_initial,
env/lib/python3.6/site-packages/django/db/migrations/executor.py:117: in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
env/lib/python3.6/site-packages/django/db/migrations/executor.py:147: in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
env/lib/python3.6/site-packages/django/db/migrations/executor.py:245: in apply_migration
state = migration.apply(state, schema_editor)
env/lib/python3.6/site-packages/django/db/migrations/migration.py:124: in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
env/lib/python3.6/site-packages/django/db/migrations/operations/models.py:92: in database_forwards
schema_editor.create_model(model)
env/lib/python3.6/site-packages/django/db/backends/base/schema.py:324: in create_model
self.execute(sql, params or None)
env/lib/python3.6/site-packages/django/db/backends/base/schema.py:142: in execute
cursor.execute(sql, params)
env/lib/python3.6/site-packages/django/db/backends/utils.py:68: in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
env/lib/python3.6/site-packages/django/db/backends/utils.py:77: in _execute_with_wrappers
return executor(sql, params, many, context)
env/lib/python3.6/site-packages/django/db/backends/utils.py:86: in _execute
return self.cursor.execute(sql, params)
env/lib/python3.6/site-packages/django/db/utils.py:90: in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
env/lib/python3.6/site-packages/django/db/backends/utils.py:84: in _execute
return self.cursor.execute(sql)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.backends.sqlite3.base.SQLiteCursorWrapper object at 0x7f711fe83d38>
query = 'CREATE TABLE "wagtailsearch_editorspick" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "sort_order" integer NULL,...ription" text NOT NULL, "page_id" integer NOT NULL REFERENCES "wagtailcore_page" ("id") DEFERRABLE INITIALLY DEFERRED)'
params = None
def execute(self, query, params=None):
if params is None:
> return Database.Cursor.execute(self, query)
E django.db.utils.OperationalError: table "wagtailsearch_editorspick" already exists
env/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py:394: OperationalError
but unlike with mysql, the error does not change on successive runs. implementing a dummy tearDown causes the test to pass.
from django-test-migrations.
skarzi
commented on May 24, 2024
@Hafnernuss could you please try to reproduce error described in your second comment without depending on wagtail?
It's quite big lib with a lot of migrations and it will make debugging much more complicated, so simpler and smaller example will be very welcome!
from django-test-migrations.
Hafnernuss
commented on May 24, 2024
Great to hear that! I suspected that the error was related to 3.1.x ;)
I think it is somehow related to foreign keys and the order in which models are deleted. I wonder especially how this can be solved if someone uses on_delete=PROTECT
from django-test-migrations.
Hafnernuss
commented on May 24, 2024
Maybe this could also help?
SET FOREIGN_KEY_CHECKS=0; since the whole db is cleared, doesn't matter if the checks fail or not. Has to be reenabled though ;)
from django-test-migrations.
tmm
commented on May 24, 2024
@skarzi anything I can help with to get the next release out? I'm getting hit by this also.
from django-test-migrations.
skarzi
commented on May 24, 2024
hi @tmm,
sure, thank you so much for your help! 👍
You can prepare PR with changes mentioned by me in #122 (comment).
Let's create some base class like BaseDatabaseConfiguration, but for database operations, so we can move there functions from django_test_migrations.sql and simply subclass and extend it for all supported vendors (currently only MySQL needs a different implementation of drop_models_table - we need to disable and then enable FOREIGN_KEY_CHECKS).
If something is not clear or you have some other ideas, please share it here
from django-test-migrations.
Related Issues (20)
- Django 3.2 compatibility HOT 1
- Upgrade dependecy of typing_extensions to ^4.0.0 HOT 5
- Add `[email protected]` to the test matrix
- Python3.10 support
- Support `[email protected]`
- Can't handle squashed migrations HOT 1
- typing_extensions is still required in newer Python versions HOT 5
- Models from app state aren't compatible (Cannot assign "<A: A object (1)>": "B.a_fk" must be a "A" instance) HOT 5
- Migrations unrelated to the migration I want to test force the migration I want to test to run before I can test state differences HOT 1
- Migrations unrelated to the migration I want to test force the migration I want to test to run before I can test state differences HOT 2
- Checks failing on MariaDB HOT 1
- Inconsistent behaviour with data migration HOT 8
- Error when testing multiple schemas in django+postgres HOT 1
- Allow using shorthand names & __latest__ for MigrationTargets
- Deadlock when using pants, pytest and django-test-migrations HOT 1
- Cannot test migrations if there is a collation in the migrations HOT 8
- "Relation {table} does not exist" when creating using prototype model HOT 2
- django.db.utils.OperationalError: (1050, "Table 'bar' already exists") HOT 6
- truncate_plan should truncate each app separately HOT 3
- `through_fields` not reflected in test models
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from django-test-migrations.