sp_allocate_products does not create unique codes for allocated datasets, resulting in NonuniqueCode errors about brightway2-io HOT 5 CLOSED

Original comment by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Well, that is embarrassing...

This is part of a broader discussion on when existing codes should be used or not. I am leaning towards the following:

• Existing codes are never used, unless 1) a flag is set, or 2) the import comes from the manual CSV where nice human codes are more or less expected. In either exception, code uniqueness should be checked.
• Internal linking should follow the same algorithm, i.e. linking by dataset attributes, which would fix the InvalidExchange error (though I have to figure out why linking wouldn't work after allocation, this is a bit confusing).

Big thanks for providing a sample dataset, this is a huge help. In general, we need much better testing and test input data in brightway2-io; the comparison with Ocelot, which has ~100% coverage for IO and transformation functions, is not pretty.

from brightway2-io.

Original comment by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

The reason Pascal says

Creating and assigning a new code when allocating fixes this problem, but creates a new one: 
activities that have an input of one pf the products from the multifunctional dataset no longer 
can link to the dataset, resulting in an `InvalidExchange` error, which is not better. 

is that even if we fix the code, we don't fix the linking. So, from his example dataset:

Process name
Rice, at farm (WFLDB 3.0)/kg/IN

Products
Rice, at farm (WFLDB 3.0)/IN U;kg;6250;95.8;not defined;_WFLDB 3.0\Plant products\Arable\Rice;INDIA�
Rice straw, at farm (WFLDB 3.0)/IN U;kg;3125.0;4.2;not defined;_WFLDB 3.0\Plant products\Arable\Rice;

Process name
Something else

Materials/fuels
Rice straw, at farm (WFLDB 3.0)/IN U;kg;0.771;Undefined;0;0;0;Default humidity of 13.1%�

Even if we adjust the code, the thing that would change would be the reference product, not the process name, so neither the default linking strategy, or the Simapro-specific link_technosphere_based_on_name_unit_location, would match Rice straw, at farm against a process named Rice, at farm with the reference product Rice straw, at farm.

This should be fixable with a new linking strategy that would use the reference product, but we need to make sure this doesn't raise non-uniqueness problems. In the end, I think this is an inherent problem with trying to link only by process name or product name, but not both. Alternatively, we could change the name of the allocated processes to their reference products.

from brightway2-io.

Original comment by Pascal Lesage (Bitbucket: MPa, ).

Thanks for looking into this.
How about something inspired by how LCIA method names are stored: first characters of the process name, first characters of the product name, and then the MD5 hash of the entire identifier (or, alternatively, simply a UUID)?

from brightway2-io.

Original comment by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

The codes should now be fixed, but we still have the problem of names not matching the products.

from brightway2-io.

Original comment by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Should be fixed in 41276a8. Added a sanitized version of the sample data as a test case.

from brightway2-io.