brightway-lca / brightway2-io Goto Github PK

View Code? Open in Web Editor NEW

24.0 24.0 39.0 113.61 MB

Importing and exporting for the Brightway LCA framework

License: BSD 3-Clause "New" or "Revised" License

Python 98.38% Jupyter Notebook 1.62%

bw2 data life-cycle-assessment python

brightway2-io's People

Contributors

Stargazers

Watchers

Forkers

pjamesjoyce yangqiu91 pascallesage tngtudor aleksandra-kim juweya dhetting dsavelief stlozhaw sc-gcoste tsgfan07 nicolito beeysian greenpass-venture renaud macmribo agneta20 trellixvulnteam shivika-iim m-rossi marc-vdm ccomb writesbytes shanikanishadhi cerebrovinny sapiologie astudillo20lca guidoca patquem loisel romainsacchi tfardet stew-mcd qtu-ubc oie-mines-paristech mfastudillo noahebrooks-cb sl0wiiio

brightway2-io's Issues

Importer for OLCA JSON LD schema

https://github.com/GreenDelta/olca-schema

This should be the default input and output schema for Brightway2.

Can't import Ecoinvent 3.2

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

#!python

Extracting ecospold2 files:
0%                          100%
[                              ]---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-5-8b91d0601dbe> in <module>()
----> 1 ei = SingleOutputEcospold2Importer(".", "ei 3.2")

/Users/cmutel/local34/bw3dev/lib/python3.4/site-packages/bw2io/importers/ecospold2.py in __init__(self, dirpath, db_name)
     47
     48         start = time()
---> 49         self.data = Ecospold2DataExtractor.extract(dirpath, db_name)
     50         print(u"Extracted {} datasets in {:.2f} seconds".format(
     51             len(self.data), time() - start))

/Users/cmutel/local34/bw3dev/lib/python3.4/site-packages/bw2io/extractors/ecospold2.py in extract(cls, dirpath, db_name)
     74         data = []
     75         for index, filename in enumerate(filelist):
---> 76             data.append(cls.extract_activity(dirpath, filename, db_name))
     77             pbar.update(item_id = filename[:15])
     78

/Users/cmutel/local34/bw3dev/lib/python3.4/site-packages/bw2io/extractors/ecospold2.py in extract_activity(cls, dirpath, filename, db_name)
    131             "activity type": ACTIVITY_TYPES[int(
    132                 stem.activityDescription.activity.get('specialActivityType')
--> 133                 or 0
    134             )],
    135             'activity':  stem.activityDescription.activity.get('id'),

KeyError: 10

sp_allocate_products does not create unique codes for allocated datasets, resulting in NonuniqueCode errors

Original report by Pascal Lesage (Bitbucket: MPa, ).

sp_allocate products creates one dataset per product in cases where there are multiple products.

However, the codes of the new, single-ouput datasets are not updated, resulting in a NonuniqueCode error upon sp.write_database()

Creating and assigning a new code when allocating fixes this problem, but creates a new one: activities that have an input of one pf the products from the multifunctional dataset no longer can link to the dataset, resulting in an InvalidExchange error, which is not better.

New parameters approach doesn't work with SimaPro importer

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

See email from Giuseppe. We already have the code to parse these values, just need to adapt it. Need to:

Distinguish between project and database parameters
Use his file to create a test fixture

We also probably need to close #14.

Make reset of activity codes during importing optional

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

See discussion. Existing codes are not always desired.

Importing ecospold 1 (like the full ecoinvent 2.2) produces datasets with integer ids.

Original report by Tomas Navarrete Gutierrez (Bitbucket: tomas_navarrete, ).

I think the expected behavior is to have hash ids to identify the activities.
Here is how to reproduce with python 3.5:

appdirs==1.4.0
asteval==0.9.7
brightway2==2.0.2
bw2analyzer==0.9.1
bw2calc==1.4
bw2data==2.3.2
bw2io==0.5.3
bw2parameters==0.5.2
bw2speedups==2.1
click==6.6
decorator==4.0.10
docopt==0.6.2
eight==0.3.3
fasteners==0.14.1
Flask==0.11.1
future==0.15.2
ipython==5.1.0
ipython-genutils==0.1.0
itsdangerous==0.24
Jinja2==2.8
lxml==3.6.4
MarkupSafe==0.23
monotonic==1.2
nose==1.3.7
numpy==1.11.1
peewee==2.8.3
pexpect==4.2.1
pickleshare==0.7.4
powerline-status==2.5
prompt-toolkit==1.0.7
psutil==4.3.0
ptyprocess==0.5.1
Pygments==2.1.3
PyPrind==2.9.8
requests==2.11.1
scipy==0.18.0
simplegeneric==0.8.1
six==1.10.0
stats-arrays==0.4.1
traitlets==4.2.2
unicodecsv==0.14.1
Unidecode==0.4.19
voluptuous==0.9.3
wcwidth==0.1.7
Werkzeug==0.11.10
Whoosh==2.7.4
wrapt==1.10.8
xlrd==1.0.0
XlsxWriter==0.9.3

import brightway2 as bw2
Applying automatic update: 2.3 processed data format

bw2.projects.set_current("ei22")
bw2.bw2setup()

Creating default biosphere

Applying strategy: normalize_units
Applying strategy: drop_unspecified_subcategories
Applied 2 strategies in 0.01 seconds
Writing activities to SQLite3 database:
0%                          100%
[##############################] | ETA: 00:00:00
Total time elapsed: 00:00:01
Title: Writing activities to SQLite3 database:
  Started: 08/30/2016 08:14:39
  Finished: 08/30/2016 08:14:41
  Total time elapsed: 00:00:01
  CPU %: 64.90
  Memory %: 1.57
Created database: biosphere3
Creating default LCIA methods

Applying strategy: normalize_units
Applying strategy: set_biosphere_type
Applying strategy: drop_unspecified_subcategories
Applying strategy: link_iterable_by_fields
Applied 4 strategies in 1.92 seconds
Wrote 665 LCIA methods with 169551 characterization factors
Creating core data migrations

ei22importer = bw2.SingleOutputEcospold1Importer('./2.2/', 'ecoinvent22')
Extracting XML data from 4087 datasets
Extracted 4087 datasets in 5.24 seconds

ei22importer.statistics()
4087 datasets
135892 exchanges
135892 unlinked exchanges
  Type biosphere: 1613 unique unlinked exchanges
  Type production: 4087 unique unlinked exchanges
  Type technosphere: 3015 unique unlinked exchanges


 ⓔ  ei22ecospold1  In [9]   ei22importer.strategies
                    Out[9]   
[<function bw2io.strategies.generic.normalize_units>,
 <function bw2io.strategies.generic.assign_only_product_as_production>,
 <function bw2io.strategies.ecospold1_allocation.clean_integer_codes>,
 <function bw2io.strategies.biosphere.drop_unspecified_subcategories>,
 <function bw2io.strategies.biosphere.normalize_biosphere_categories>,
 <function bw2io.strategies.biosphere.normalize_biosphere_names>,
 <function bw2io.strategies.biosphere.strip_biosphere_exc_locations>,
 <function bw2io.strategies.generic.set_code_by_activity_hash>,
 functools.partial(<function link_iterable_by_fields at 0x7f5eac298b70>, other=Brightway2 SQLiteBackend: biosphere3, kind='biosphere'),
 <function bw2io.strategies.generic.link_technosphere_by_activity_hash>]

 ⓔ  ei22ecospold1  In [10]   ei22importer.apply_strategies()
Applying strategy: normalize_units
Applying strategy: assign_only_product_as_production
Applying strategy: clean_integer_codes
Applying strategy: drop_unspecified_subcategories
Applying strategy: normalize_biosphere_categories
Applying strategy: normalize_biosphere_names
Applying strategy: strip_biosphere_exc_locations
Applying strategy: set_code_by_activity_hash
Applying strategy: link_iterable_by_fields
Applying strategy: link_technosphere_by_activity_hash
Applied 10 strategies in 4.46 seconds

 ⓔ  ei22ecospold1  In [11]   ei22importer.statistics()
4087 datasets
135892 exchanges
0 unlinked exchanges
  
                    Out[11]   (4087, 135892, 0)

 ⓔ  ei22ecospold1  In [12]   ei22importer.write_database()
Writing activities to SQLite3 database:
0%                          100%
[##############################] | ETA: 00:00:00
Total time elapsed: 00:01:01
Title: Writing activities to SQLite3 database:
  Started: 08/30/2016 08:19:46
  Finished: 08/30/2016 08:20:47
  Total time elapsed: 00:01:01
  CPU %: 60.60
  Memory %: 4.40
Created database: ecoinvent22
                    Out[12]   Brightway2 SQLiteBackend: ecoinvent22

 ⓔ  ei22ecospold1  In [13]   rds = bw2.Database('ecoinvent22').random()

 ⓔ  ei22ecospold1  In [14]   rds
                    Out[14]   'laser machining, metal, with YAG-laser, 500W power' (hour, RER, ['metals', 'chipless shaping'])

 ⓔ  ei22ecospold1  In [15]   rds.as_dict()
                    Out[15]   
{'authors': [{'address': 'Kanzleistrasse 4, 8610 Uster',
   'company': 'ESU',
   'country': 'CH',
   'email': '[email protected]',
   'name': 'Roland Steiner'}],
 'categories': ['metals', 'chipless shaping'],
 'code': '10137',
 'comment': "The reference for laser machining is its operation at 100% power for 1 hour. It does not include the input of the material processed. This need to be added separately. The dataset can be used when metals are treated with a YAG laser of the capacity indicated. Factory infrastructure needs to be added. Data are based on manufacturers' data (weight and power consumption) and literature (air emissions).\nThis dataset includes work piece feeder, laser system, cooling and control system. Any additional equipment such as possibly necessary ventilation or additional security installations are not included. It includes the input of energy, of cooling water (where needed) and of the laser equipment. Further factory infrastructure (halls, buildings) are not included. The dataset includes process specific air emissions.\nLocation:  Geographical coverage encompasses the industrialised countries.\nTechnology:  HL series of YAG Lasers and Lasma 584R processing machine\nProduction volume:  unknown\nSampling:  unknown\nExtrapolations:  none\nUncertainty:  none",
 'database': 'ecoinvent22',
 'filename': '/opt/db/2.2/Process_infra_roh/10137.XML',
 'location': 'RER',
 'name': 'laser machining, metal, with YAG-laser, 500W power',
 'production amount': 1.0,
 'type': 'process',
 'unit': 'hour'}

`strip` added to excel extractor breaks extraction ()

Original report by Pascal Lesage (Bitbucket: MPa, ).

Strip added to extractor in commit 3390d0f results in AttributeError when importing Excel:

c:\mypy\code\brightway2-io\bw2io\extractors\excel.py in <lambda>(x)
     17     def extract_sheet(cls, wb, name, strip=True):
     18         ws = wb.sheet_by_name(name)
---> 19         _ = lambda x: x.strip() if strip else x
     20         return [[_(ws.cell(row, col).value) for col in range(ws.ncols)] for row in range(ws.nrows)]
     21 
AttributeError: 'float' object has no attribute 'strip'

Tested on file "sample_activities_with_variables.xlsx"

Need SimaPro migration for ecoinvent 3.2 names

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

CSVImporter `write_database` should accept **kwargs

Original report by Pascal Lesage (Bitbucket: MPa, ).

The CSVImporter class method write_database does not accept any **kwargs.
See here.
We should be able to pass data, name, overwrite and backend, all acceptable kwargs in the class parent LCIImporter

Unit conversion in the CF for ReCiPe Midpoint Water Depletion wrong

Original report by Tomas Navarrete Gutierrez (Bitbucket: tomas_navarrete, ).

Looking at the characterization factors for ReCiPe Midpoint (H), water depletion, and found a problem with the CF for "water natural unspecified origin".

Here is the output of bw2-browser to see the cfs of the method:


default [ReCiPe Midpoint (H)/water depletion/WDP] >> cfs 
CFS 
---------------------------------  --------------------------------  ---------------  ---  -----  ----------- 
Water, in air                      ('natural resource', 'in air')    water depletion  WDP  1      cubic meter 
Water, lake                        ('natural resource', 'in water')  water depletion  WDP  1      cubic meter 
Water, river                       ('natural resource', 'in water')  water depletion  WDP  1      cubic meter 
Water, unspecified natural origin  ('natural resource', 'in water')  water depletion  WDP  0.001  cubic meter 
Water, well, in ground             ('natural resource', 'in water')  water depletion  WDP  1      cubic meter 
---------------------------------  --------------------------------  ---------------  ---  -----  -----------

Looking quickly at the xlsx files from recipe website (http://www.lcia-recipe.net/file-cabinet/ReCiPe111.xlsx?attredirects=0&d=1 [tab wdp] or http://www.lcia-recipe.net/file-cabinet/ReCiPe112Midpoints.CSV?attredirects=0&d=1, rows 79102 and 79103) , we found that:

there are two flows for water, unspecified natural origin:

the CF for WDP is of 1 m3 or 0.001 kg

Rework uncertainty with pedigree matrices

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Instead of current approach, we should read the basic uncertainty and pedigree matrix values, and calculate the final uncertainty only when processing.

There is an implementation of the pedigree matrix here: https://bitbucket.org/cmutel/pedigree-matrix. We can use functools.lru_cache to make evaluations quick.

This idea and methodology comes from Guillaume Audard.

Default strategies shouldn't assume that base data is installed

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Implement ecoinvent 3.2 changes to LCIA methods

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

And also add variant where negative CFs are included.

File is already added to hg repository.

Will also need to adapt method descriptions from categoryUUIDs.csv.

Need ecoinvent 2.2 -> 3.01 migration

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Separate migration for each system model

ecoSpold1 files no longer import due to 'reference product' in activities but not in exchanges

Original report by Pascal Lesage (Bitbucket: MPa, ).

Ever since this commit, the ecoinvent v2 database (in ecoSpold 1 format) no longer imports.
This is due fields 'reference product' bing added to activities, but not to the exchanges that are looking for those activities.

#!python

for f in ('name', 'categories', 'unit', 'reference product', 'location'):
    print(f, x.get(f) == y.get(f))

name True
categories False
unit True
reference product False
location True

BW2Package _create_obj fails sometimes

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

package.py, line 118:

#!python

    @classmethod
    def _create_obj(cls, data):
        instance = data['class'](data['name'])

        if data['name'] not in instance._metadata:
            instance.register(**data['metadata'])
        else:
            instance.backup()
            instance.metadata = data['metadata']
            instance._metadata.flush()

        instance.write(data['data'])
        return instance

instance.metadata can't set metadata for some reason. Need to be investigated more.

Migrations need to be cleaned up

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Documentation should state that they are only for unit conversions, not changing actual amounts
Need tests
Need some validation of format when saving

Production exchanges from ecospold 1 not linked (Py 2.7 only)

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

Hey Chris,
while trying to install all the ecoinvent dbs in my py2 installation I discovered it is not working with ei 2.2.
After I discovered I had bw2io 0.2.dev4 in py3 and dev5 in py2 enviro (the other packages same versions) thought this was the prob but then I tried to use also dev 6 and 4 and still it is not working.

It basically does not extract correctly ei 2.2. (used the same file you sent me for both py2 and 3 and the latter gives no prob) leaving several unlinked exchanges

I tried to import also ei 3.1 cutoff and it works...it seems something is wrong with the the ecospold 1 extractor when working with py2
I attached also the notebook
Hope this info help.
G

SimaPro import misimports "scale" as "shape"

Original report by Pascal Lesage (Bitbucket: MPa, ).

In create_distribution of SimaProCSVExtractor, the scale lognormal and normal distributions are imported as shape parameters.
Line 237:

'shape': math.log(math.sqrt(to_number(field1)))
should read
'scale': math.log(math.sqrt(to_number(field1)))

Line 245:
'shape': math.sqrt(to_number(field1))
should read
'scale': math.sqrt(to_number(field1))

Prevents Monte Carlo from being done on imported datasets.

SimaPro CSV should extract and apply unit conversions when possible

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Comparison chart for each major/freely available database

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

USDA
US LCI
GreenDelta nexus website

psutil missing requirement prevent importing to work correctly

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

HI Chris,
this is new and seems that prob is due to pyprind.
I checked the package repo and even if it says it require psutil the requirements.txt does not exist
In fact doing:

#!python

 pip install pyprind -r requirements.txt
Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

maybe a temporary fix could be to add psutil to bw2io requirements (anyway, I created the issue on the pyprind repo)

#!python


bw2setup()
bw2setup()
Creating default biosphere

Writing activities to SQLite3 database:
Applying strategy: normalize_units
Applying strategy: drop_unspecified_subcategories
Applied 2 strategies in 0.01 seconds
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-5-c9438012d7c9> in <module>()
----> 1 bw2setup()

/home/giuseppe/miniconda2/envs/bw2/lib/python3.4/site-packages/bw2io/__init__.py in bw2setup()
     90         return
     91     print("Creating default biosphere\n")
---> 92     create_default_biosphere3()
     93     print("Creating default LCIA methods\n")
     94     create_default_lcia_methods()

/home/giuseppe/miniconda2/envs/bw2/lib/python3.4/site-packages/bw2io/__init__.py in create_default_biosphere3()
     77     eb = Ecospold2BiosphereImporter()
     78     eb.apply_strategies()
---> 79     eb.write_database()
     80 
     81 def create_default_lcia_methods():

/home/giuseppe/miniconda2/envs/bw2/lib/python3.4/site-packages/bw2io/importers/base_lci.py in write_database(self, data, name, overwrite, backend, **kwargs)
     96         data = {(ds['database'], ds['code']): ds for ds in data}
     97         existing.update(data)
---> 98         db.write(existing)
     99         print("Created database: {}".format(db.name))
    100         return db

/home/giuseppe/miniconda2/envs/bw2/lib/python3.4/site-packages/bw2data/project.py in writable_project(wrapped, instance, args, kwargs)
    316     if projects.read_only:
    317         raise ReadOnlyProject(READ_ONLY_PROJECT)
--> 318     return wrapped(*args, **kwargs)

/home/giuseppe/miniconda2/envs/bw2/lib/python3.4/site-packages/bw2data/backends/peewee/database.py in write(self, data, process)
    228         if data:
    229             try:
--> 230                 self._efficient_write_many_data(data)
    231             except:
    232                 # Purge all data from database, then reraise

/home/giuseppe/miniconda2/envs/bw2/lib/python3.4/site-packages/bw2data/backends/peewee/database.py in _efficient_write_many_data(self, data, indices)
    150                     len(data),
    151                     title="Writing activities to SQLite3 database:",
--> 152                     monitor=True
    153                 )
    154 

/home/giuseppe/miniconda2/envs/bw2/lib/python3.4/site-packages/pyprind/progbar.py in __init__(self, iterations, track_time, width, bar_char, stream, title, monitor, update_interval)
     46                  stream=2, title='', monitor=False, update_interval=None):
     47         Prog.__init__(self, iterations, track_time,
---> 48                       stream, title, monitor, update_interval)
     49         self.bar_width = width
     50         self._adjust_width()

/home/giuseppe/miniconda2/envs/bw2/lib/python3.4/site-packages/pyprind/prog_class.py in __init__(self, iterations, track_time, stream, title, monitor, update_interval)
     42 
     43         if self.monitor:
---> 44             import psutil
     45             self.process = psutil.Process()
     46         if self.track:

ImportError: No module named 'psutil'

Need ecoinvent 3.01 -> 3.1 migration

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Separate migration for each system model

Unlinked exchange when importing ecoinvent 3.2 cutoff

Original report by Pascal Lesage (Bitbucket: MPa, ).

The following biosphere flow is "unlinked" when the ecoinvent v3.2 cutoff database is imported:
Carbon dioxide, non-fossil, from calcination
See attached log file.
This results in an import failure (unable to save the database).

Error in `loc` field for Ecoinvent 3.1 import

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

What is up with this exchange?

{u'activity': u'23f2e6c4-5155-43d7-a9a0-ee8dd1d584f8',
 u'amount': 0.10683617468301,
 u'comment': u'Transport distance based on US BTS Commodity Flow Surveys 1993, 1997, 2002, 2007, US Dep. Of Transportation, Bureau of Transportation Statistics.',
 u'flow': u'0ace02fa-eca5-482d-a829-c18e46a52db4',
 u'input': (u'ecoinvent 3.1 cutoff', u'2e292a12188f6121c4b6752889328ca4'),
 u'loc': 1.0, <= **This is wrong**
 u'name': u'transport, freight train',
 u'output': (u'ecoinvent 3.1 cutoff', u'3c7ee2d96b61ba8c11a5dddee6927068'),
 u'pedigree': {
     u'completeness': 1,
     u'further technological correlation': 4,
     u'geographical correlation': 5,
     u'reliability': 1,
     u'temporal correlation': 4
 },
 u'production volume': 0.0,
 u'scale': 0.17,

In market for aluminium, wrought alloy, GLO, ei 3.1 cutoff

Need clever approach to replace formula parameters names that conflict with Python reserved words

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

SimaPro LCIA importer: Waste types seem to be handled incorrectly

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Needs more detail before it can be addressed, though...

ExcelImporter doesn't respect manually specified codes

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Default biosphere social flows seem weird

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

They seem to include the following:
"residual hardwood, wet"
"venting of nitrogen, liquid"
"venting of argon, crude, liquid"
"residual softwood, wet"
"residual wood, dry"

(Report from Pascal Lesage)

Excel IO should support local and global parameters

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Also create `backup_project_directory`

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

SingleOutputEcospold1Importer not importing ecoinvent 2.2

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

In a new project, after installing base data (i.e. bw2setup() ) the installation of ecoinvent 2.2 (the version you sent me) return the following error

#!python


ei22 = SingleOutputEcospold1Importer(
    "Ecoinvent2filepath",
    "ecoinvent 2.2"
)
ei22.apply_strategies()
ei22.statistics()


Extracting ecospold1 files:
0%                          100%
[############################# ] | ETA[sec]: 59.156 | Item ID: /mnt/D092A60B92
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-14-c4d4efc73cc8> in <module>()
      1 ei22 = SingleOutputEcospold1Importer(
      2     "/mnt/D092A60B92A5F65E/DATA/Work/LCA_db-software/ecoinvent/Ecoinvent2",
----> 3     "ecoinvent 2.2"
      4 )
      5 ei22.apply_strategies()

/home/giuseppec/miniconda3/envs/bw2_py3/lib/python3.4/site-packages/bw2io/importers/ecospold1.py in __init__(self, filepath, db_name)
     63         self.db_name = db_name
     64         start = time()
---> 65         self.data = Ecospold1DataExtractor.extract(filepath, db_name)
     66         print(u"Extracted {} datasets in {:.2f} seconds".format(
     67               len(self.data), time() - start))

/home/giuseppec/miniconda3/envs/bw2_py3/lib/python3.4/site-packages/bw2io/extractors/ecospold1.py in extract(cls, path, db_name)
     54                 if dataset.tag == 'comment':
     55                     continue
---> 56                 data.append(cls.process_dataset(dataset, filename, db_name))
     57 
     58             pbar.update(item_id = filename[:15])

/home/giuseppec/miniconda3/envs/bw2_py3/lib/python3.4/site-packages/bw2io/extractors/ecospold1.py in process_dataset(cls, dataset, filename, db_name)
     75             ref_func.get("generalComment"),
     76             ref_func.get("includedProcesses"),
---> 77             ("Location: ", dataset.metaInformation.processInformation.geography.get("text")),
     78             ("Technology: ", dataset.metaInformation.processInformation.technology.get("text")),
     79             ("Time period: ", getattr2(dataset.metaInformation.processInformation, "timePeriod").get("text")),

lxml.objectify.pyx in lxml.objectify.ObjectifiedElement.__getattr__ (src/lxml/lxml.objectify.c:3497)()

lxml.objectify.pyx in lxml.objectify._lookupChildOrRaise (src/lxml/lxml.objectify.c:5947)()

AttributeError: no such child: {http://www.EcoInvent.org/EcoSpold01}geography

LCI methods imported in `create_default_methods`

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

The following shouldn't be imported:

#!python

   ('selected LCI results', 'air', 'CO2, fossil'),
   ('selected LCI results', 'air', 'NMVOC'),
   ('selected LCI results', 'air', 'nitrogen oxides'),
   ('selected LCI results', 'air', 'particulates, < 2.5 um'),
   ('selected LCI results', 'air', 'sulphur dioxide'),
   ('selected LCI results', 'resource', 'land occupation'),
   ('selected LCI results', 'soil', 'cadmium'),
   ('selected LCI results', 'water', 'BOD'),
   ('selected LCI results, additional', 'air', 'N20'),
   ('selected LCI results, additional', 'air', 'carbon monoxide'),
   ('selected LCI results, additional', 'air', 'lead'),
   ('selected LCI results, additional', 'air', 'methane'),
   ('selected LCI results, additional', 'air', 'particulates'),
   ('selected LCI results, additional', 'air', 'particulates >10 um'),
   ('selected LCI results, additional', 'air', 'particulates, >2.5 um and <10'),
   ('selected LCI results, additional', 'air', 'zinc'),
   ('selected LCI results, additional', 'air, radioactive', 'actinides'),
   ('selected LCI results, additional', 'air, radioactive', 'aerosole'),
   ('selected LCI results, additional', 'air, radioactive', 'noble gas'),
   ('selected LCI results, additional', 'air, radioactive', 'radon (+ radium)'),
   ('selected LCI results, additional', 'resource', 'carbon, biogenic, fixed'),
   ('selected LCI results, additional', 'resource', 'water'),
   ('selected LCI results, additional', 'total', 'heat, waste'),
   ('selected LCI results, additional', 'total', 'oils, unspecified'),
   ('selected LCI results, additional', 'water, radioactive', 'actinides'),
   ('selected LCI results, additional', 'water, radioactive', 'nuclides'),
   ('selected LCI results, additional', 'water, radioactive', 'radium'),
   ('selected LCI results, additional', 'water, radioactive', 'tritium')

Documentation should be more clear on how to remove strategies from default list.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

CapitalizationError when importing from simapro

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

Just see the question on stackoverflow

to complement this.

I checked and realized that despite I installed bw2 with pip install bw2-dev a couple of weeks ago I the version of bw2io was 0.3.
After upgrading to 0.4 the number of capitalization error returned seems to be less but still there

I am sending you the file via email (no idea on how to anonymise and better not to make public the work done form others)

Inconsistencies in biosphere between 3.2 and 3.3 (ElementaryExchanges.xml)

Original report by Adrian Haas (Bitbucket: haasad, GitHub: haasad).

I have mentioned this some days ago in a comment and now experienced the same thing in another place.

I made a short notebook to illustrate this.

There are some strange biosphere exchanges (social, unspecified), that appear to be unused
There is one biosphere exchange from 3.2 that is missing in 3.3 (Carbon dioxide, non-fossil, from calcination)
One biosphere exchange (Water, unspecified natural origin) is missing in 'ecoinvent elementary flows 3.3.xml', but is a valid flow in cutoff 3.3.

As far as I understand it, this is caused by ecoinvent, but I still think a workaround in brightway would be nice. I could create a pull-request that contains the following: Either
a) manually update bw2io/importers/ecospold2_biosphere.py and bw2io/data/lci/ecoinvent 33 new biosphere.json with the two missing processes or
b) make bw2io/importers/ecospold2_biosphere.py load both xml files and return the union of the two and dynamically create bw2io/data/lci/ecoinvent 33 new biosphere.json

In case of b) this could be reused for the next version of ecoinvent. I'm not sure what the effect of such a fix would be for the LICA methods, maybe they also need to be adapted.

Excel export/import should be round trippable

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

nuncertainty strategy override

Original report by Pascal Lesage (Bitbucket: MPa, ).

The nuncertainty strategy in ecospold2.py makes the calculation of uncertainty with Brightway2 "unique" and prevents the results from being comparable to uncertainties calcualted with other software.
While there may be a good reason to force these changes, there should be an easy way (e.g. during the apply_strategies call) to override this strategy.
Furthermore, the nuncertainty strategy actually has two strategies: dealing with negative numbers defined by lognormal distributions, and a "capping" of uncertainty at scale of 2.5. There may be reason to do one, but not the other, of these. They should be split in two distinct strategies.

US LCI database linking

Original report by Shubhankar Upasani (Bitbucket: Shubh1995, GitHub: Shubh1995).

I have imported US LCI database downloaded from NREL into bw2 by following instructions mentioned [here](Linhttp://nbviewer.jupyter.org/urls/bitbucket.org/cmutel/brightway2/raw/default/notebooks/IO%20-%20Importing%20the%20US%20LCI%20database.ipynbk URL).

I have several unlinked exchanges. What should be the next step?

If I go ahead with the unlinked exchanges, how do I access the database object?

bw.databases only returns biosphere3 and does not return the US LCI database that I defined earlier.

Need SimaPro to ecoinvent biosphere migration

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Needs complete list of SimaPro biosphere flows...

Need ecoinvent 3.1 -> 3.2 migration

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Separate for each system model

biosphere consistency check (for bw2_setup())

Original report by Niko Heeren (Bitbucket: nheeren, GitHub: nheeren).

Right now bw2_setup() checks for

if "biosphere3" in databases:
    print("Biosphere database already present!!! No setup is needed")
    return

Maybe we should include a check_biosphere_db_consistency() function, that will count the number of biosphere flows, do some sort of database hash, or whatnot. Right now it may happen that the biosphere database does not get properly initialized or otherwise destroyed and the user will not necessarily realize that.

If you hint me to a sensible method to check consistency, I could contribute some code... My first guess would be to compare against a hash that is specific to each database version.

migration simapro-water missing

Original report by Pascal Lesage (Bitbucket: MPa, ).

Upon import of SimaPro CSV, apply_strategies() fails because migration simapro-water missing.
Version: bw2io-0.5.7

Traceback:

AssertionError Traceback (most recent call last)
in ()
6 #correctUncertaintyInformation(sp)
7 sp.migrate('simapro-ecoinvent-3')
----> 8 sp.apply_strategies()
9 sp.match_database("ecoinvent 2.2",ignore_categories=True)
10 #sp.write_database()

C:\bw2-python\envs\bw2\lib\site-packages\bw2io\importers\base.py in apply_strategies(self, strategies, verbose)
70 func_list = self.strategies if strategies is None else strategies
71 for func in func_list:
---> 72 self.apply_strategy(func, verbose)
73 if verbose:
74 print("Applied {} strategies in {:.2f} seconds".format(

C:\bw2-python\envs\bw2\lib\site-packages\bw2io\importers\base.py in apply_strategy(self, strategy, verbose)
50 print("Applying strategy: {}".format(func_name))
51 try:
---> 52 self.data = strategy(self.data)
53 self.applied_strategies.append(func_name)
54 except StrategyError as err:

C:\bw2-python\envs\bw2\lib\site-packages\bw2io\strategies\migrations.py in migrate_exchanges(db, migration)
33
34 def migrate_exchanges(db, migration):
---> 35 assert migration in migrations, u"Can't find migration {}".format(migration)
36 migration_data = Migration(migration).load()
37

AssertionError: Can't find migration simapro-water

Don't create `ds['products']`

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Production exchanges are just like other exchanges.

Also, remove products from migrations and strategies, e.g. unit normalization.

Import fails

Original report by Pascal Lesage (Bitbucket: MPa, ).

Import fails on line 5 from init:
from bw2data import Database, databases, config, parameters
Cannot import name parameters

ecoinvent 3.1 cutoff has unlinked exchanges in bw2 dev / win 7 /py3

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

Hey Chris,

Hi am installing bw2 on a conda enviro in windows 7 32 bit (py 3.4.4).
Just followed the advanced installation (https://docs.brightwaylca.org/advanced-installation.html#advanced-installation) and installed the development version (pip install --no-cache-dir --pre brightway2) but while there are no prob with ei 2.2 the 3.1 cutoff , that should work smoothly,returns several unlinked exchanges, see below

#!python


In [8]: ei31cutoff = SingleOutputEcospold2Importer(
    "C:\\Users\\Giuseppec\\Downloads\\cutoff\\datasets",
    "ecoinvent 3.1 cutoff")
Extracting ecospold2 files:
0%                          100%
[##############################] | ETA: 00:00:00 | Item ID: fff527b1-0fe4-4
Total time elapsed: 00:01:52
Title: Extracting ecospold2 files:
  Started: 02/04/2016 15:50:18
  Finished: 02/04/2016 15:52:11
  Total time elapsed: 00:01:52
  CPU %: 87.200000
  Memory %: 19.460622
Extracted 11301 datasets in 114.01 seconds

In [9] ei31cutoff.apply_strategies()
Applying strategy: normalize_units
Applying strategy: remove_zero_amount_coproducts
Applying strategy: remove_zero_amount_inputs_with_no_activity
Applying strategy: remove_unnamed_parameters
Applying strategy: es2_assign_only_product_with_amount_as_reference_product
Applying strategy: assign_single_product_as_activity
Applying strategy: create_composite_code
Applying strategy: drop_unspecified_subcategories
Applying strategy: link_biosphere_by_flow_uuid
Applying strategy: link_internal_technosphere_by_composite_code
Applying strategy: delete_exchanges_missing_activity
Applying strategy: delete_ghost_exchanges
Applying strategy: nuncertainty
Applied 13 strategies in 4.51 seconds

In [10]: ei31cutoff.statistics()
11301 datasets
521712 exchanges
780 unlinked exchanges
  Type biosphere: 12 unique unlinked exchanges
Out[7]:
(11301, 521712, 780)

Should be easy to recalculate `amount` field based on parameters in importers

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Handling of imported parameter with both `value` and `formula`

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

Using the excel example in the documentation I realized that when for a parameter both value and formula are passed the formula is evaluated and the parameter takes this value (obviously). I would suggest to either makes sure only one or the other field is passed (i.e. return an error when both) in the parameter or (better I think) rise a warming. I (my personal user experience) would prefer to be able to either enter an amount or a parameter (which calculates the amount) and makes sure I do make confusion between the two...A situation might occur that I see an amount which is afterwards superseded by another calculated from the formula and have results which differ from what I expected based on the previous amount

Incompatibility with python 2.7 multiprocessing

Original report by Tomas Navarrete Gutierrez (Bitbucket: tomas_navarrete, ).

After creating a virtual environment for a pure python 2.7, I tried installing latest brightway2, and then importing ecoinvent 22, but it fails.

from brightway2 import *
projects.current = "import ei22"
bw2setup()
ei22 = SingleOutputEcospold1Importer("/home/tomas/dbs/ei22/", "ecoinvent22")

with the following results:

Extracting XML data from 4087 datasets 
--------------------------------------------------------------------------- 
PicklingError                             Traceback (most recent call last) 
<ipython-input-8-30e3215734e0> in <module>() 
----> 1 ei22 = SingleOutputEcospold1Importer("/home/tomas/dbs/ei22/", "ecoinvent22") 

/home/tomas/virtualenvs/bw2ui-dev/venv/lib/python2.7/site-packages/bw2io/importers/ecospold1.pyc in __init__(self, filepath, db_name) 
    63         self.db_name = db_name 
    64         start = time() 
---> 65         self.data = Ecospold1DataExtractor.extract(filepath, db_name) 
    66         print(u"Extracted {} datasets in {:.2f} seconds".format( 
    67               len(self.data), time() - start)) 

/home/tomas/virtualenvs/bw2ui-dev/venv/lib/python2.7/site-packages/bw2io/extractors/ecospold1.pyc in extract(cls, path, db_name, use_mp) 
    55                 x 
    56                 for p in results 
---> 57                 for x in p.get() 
    58                 if x 
    59             ] 

/usr/lib64/python2.7/multiprocessing/pool.pyc in get(self, timeout) 
   565             return self._value 
   566         else: 
--> 567             raise self._value 
   568  
   569     def _set(self, i, obj): 

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

Missing biosphere flows for consequential 3.2 and 3.3

Original report by Adrian Haas (Bitbucket: haasad, GitHub: haasad).

It is currently not possible to import consequential 3.2 and 3.3, due to missing biosphere flows:

Writing activities to SQLite3 database:
0%                          100%

---------------------------------------------------------------------------
InvalidExchange                           Traceback (most recent call last)
etc...

The missing flows are:

residual wood, dry
residual hardwood, wet (3.2 only)
residual softwood, wet (3.2 only)
venting of argon, crude, liquid
venting of nitrogen, liquid

As far as I can tell, these flows are still present in the current biosphere, but with an altered ecoinvent uuid. I suppose adding these flows with the old code to brightway2-io / bw2io / data / lci / previous elementary flows.json could solve the problem. Please let me know if I should make a PR.

brightway-lca / brightway2-io Goto Github PK

brightway2-io's People

Contributors

Stargazers

Watchers

Forkers

brightway2-io's Issues

Recommend Projects

Recommend Topics

Recommend Org