Deterministic fixture import order still missing in v14 and v15 about frappe HOT 8 OPEN

According to https://docs.python.org/3/library/os.html#os.listdir:

The list is in arbitrary order

If it were truly random, then I would disagree that forcing it into a deterministic order could break things (because it would already break every now and again due to random behaviour. At least now it would be either breaking all the time or not break at all).
But the order is not truly random, it depends on the underlying OS. This means, that when installing the same app on a different OS or even after an OS upgrade, things could break potentially.
I feel like I would rather have a consistent behavior than one determined by the OS on which the app happens to be running.

@ankush how about implementing a feature toggle? A flag in the hooks.py "import_fixture_in_sorted_order" or something like that? Then people aware of the issue could start implementing the toggle and the behavior would be documented.

from frappe.

There is one major possibility of breaking things when porting where random order that used to work will now no longer work because of deterministic order. - @ankush

If it were truly random, then I would disagree that forcing it into a deterministic order could break things (because it would already break every now and again due to random behaviour. - @Tom-Finke

I agree with Tom. I cannot think of any situation where randomly importing fixtures is okay for a Custom App. But importing in a predefined way is somehow worse.

Fixtures were never supposed to be "insert any random document to DB from code"... for that you have to write code and handle all the scenarios. @ankush

Yet the framework allows this. It does not error or stop you. If fixtures were not supposed to do this, then the developers shouldn't have allowed for it. No one is asking for new features: just improvements to what already exists today.

In your case only, you're inserting "Item" as a fixture, which was NEVER intended use for this. Item is a user/site-specific doctype that will almost never have to be hardcoded in code. @ankush

What "intended use"? Where documented? I've never read any article that says some DocTypes are acceptable for fixtures, but others are not.

Assuming for a moment that's true. If fixtures are not the answer, then what is? What other feature(s) can people use to automatically handle data updates when deploying? The key word here is automatically. Certainly data updates can be performed manually after a code deployment. No one is questioning that.

I agree with @tgraupne that this seems trivial: take an unsorted array and make it sorted. 🤷‍♂️

Will this solve every possible use case of fixtures? No. Of course not. You are correct @ankush that some data imports will still be problematic.

That doesn't mean fixtures cannot, or should not, be made incrementally better than they are today.

from frappe.

Fixtures have long outlived their intended use. Custom field, property setters, custom doctypes can all be inserted without any dependency on each other. In absence of dependencies they'll just not work. Fixtures were never supposed to be "insert any random document to DB from code"... for that you have to write code and handle all the scenarios.

In your case only, you're inserting "Item" as a fixture, which was NEVER intended use for this. Item is a user/site-specific doctype that will almost never have to be hardcoded in code.

from frappe.

Imho, this is a trivial basic function of custom apps

It's far from "trivial".

Export Item, Customer, Company, Sales order, sales invoice, delivery note and try writing a sane importer for it without assuming anything about those doctypes.

Even a known import order doesn't really work because ERPNext can require invoice before delivery sometimes and delivery before invoice some other times and both can be parallel inserted too in some other case. 🤷

Fixtures should just be limited to very dumb static data.

from frappe.

There is one major possibility of breaking things when porting where random order that used to work will now no longer work because of deterministic order 😬 main reason why I held off port.

TBH fixtures have very limited use case. When you start requiring order of insertion you're already entering into "code" from "data", so you might as well just write code instead.

Most complex apps that customize core behaviours eventually do this.

E.g: https://github.com/resilient-tech/india-compliance/blob/develop/india_compliance/gst_india/constants/custom_fields.py

from frappe.

TBH fixtures have very limited use case. When you start requiring order of insertion you're already entering into "code" from "data", so you might as well just write code instead.

🤔 I find this statement quite interesting, because I don't "require order". I was only hoping, that fixtures exported with bench can actually be imported again.

Imho, this is a trivial basic function of custom apps and bench export-fixtures. What other option do people have who need to package and ship Custom Fields and a handful of initialisations?

This all started, trying to find one working approach which enables developers to develop locally, checkin and deploy their changes to testing environments and eventually to production. And after hours of research and discussions the result was, a custom app and fixtures are your best bet. 🤷‍♀️

from frappe.

@ankush I really appreciate your time and knowledge sharing, but I have a hard time following.

Why are you talking about all those things? I have never mentioned those and I don't understand how they are relevant?

Export Item, Customer, Company, Sales order, sales invoice, delivery note and try writing a sane importer for it without assuming anything about those doctypes.

You say,

Fixtures should just be limited to very dumb static data.

And this is all I want to do, I just want to import 2 Item Groups and 2 basic Items which belong to those groups. And even this simple import of static data does not work, because frappe is complaining that the Item Groups do not exist when trying to import the Items. That is all I want to achieve.

You say,

In your case only, you're inserting "Item" as a fixture, which was NEVER intended use for this. Item is a user/site-specific doctype that will almost never have to be hardcoded in code.

Maybe there is a misunderstanding, I am not trying to "import" a new DocType. I am just trying to import documents of the type: DocType Item.

You say,

Fixtures have long outlived their intended use.

And then you mention Custom fields etc. If fixtures are not intended to be used to do exactly this, why is the official frappe documentation still stating this:

https://frappeframework.com/docs/user/en/guides/app-development/how-to-create-custom-fields-during-app-installation

And above all, discuss.frappe.io and the whole ERPNext-Youtube is full of people using Custom Apps and Fixtures to deploy and ship Custom Fields to Standard DocTypes. If those approaches are just plain wrong, I believe hundreds of people would be very happy to find out what the intended feature to use would be.

How can people make Changes and Create Custom Fields and add very dumb static data and at the end, ship everything as Code?

from frappe.

I use Fixtures to set up Workflows, Roles, Email templates, Price lists (just their existence, not any prices) and a few property setters (but not many of the latter any more), but nothing more complicated than that.

As an alternative to Fixtures, presumably any custom data needed by a custom app can be handled by a combination of install.py for new installations and patches for existing ones?

from frappe.