brightway-lca / brightway2-data Goto Github PK

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

bw2data.backends.json.sync_json_dict.SynchronousJSONDict object at 0x105ccebd0> is not JSON serializable

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

BlitzDB is an interesting file-based (JSON) database with indices and advanced queries.

Example code to import a database:

#!python

from brightway2 import *
import blitzdb
ei = Database("ecoinvent 2.2").load()

be = blitzdb.FileBackend(config.request_dir("blitz"))

def reformatter(key, value):
	value[u"!key"] = key[1]
	value[u"!db"] = key[0]
	return value

class Dataset(blitzdb.Document):
    pass

for key, value in ei.items():
    Dataset(reformatter(key, value)).save(be)

be.commit()

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

SimaPro has a new naming convention for ecoinvent 3 processes.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Enhancement request from Pascal Lesage

Original report by Tomas Navarrete Gutierrez (Bitbucket: tomas_navarrete, ).

Hi,

I want to be able to get the names of the databases from each project, and started with:

from brightway2 import *

for p in projects:
projects.current = p.name
print(p.name, databases)

but I get the following error:

  ...:     
default Databases dictionary with 2 object(s):
        biosphere3
        ecoinvent22
---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
<ipython-input-3-9166ed4a7809> in <module>()
----> 1 for p in projects:
      2     projects.current = p.name
      3     print(p.name, databases)
      4 

/home/tomas/virtualenvs/bw2/lib/python3.4/site-packages/bw2data/project.py in __iter__(self)
     63 
     64     def __iter__(self):
---> 65         for project_ds in ProjectDataset.select():
     66             yield project_ds
     67 

/home/tomas/virtualenvs/bw2/lib/python3.4/site-packages/peewee.py in next(self)
   2045             obj = self.qrw._result_cache[self._idx]
   2046         elif not self.qrw._populated:
-> 2047             obj = self.qrw.iterate()
   2048             self.qrw._result_cache.append(obj)
   2049             self.qrw._ct += 1

/home/tomas/virtualenvs/bw2/lib/python3.4/site-packages/peewee.py in iterate(self)
   2096 
   2097     def iterate(self):
-> 2098         row = self.cursor.fetchone()
   2099         if not row:
   2100             self._populated = True

ProgrammingError: Cannot operate on a closed database.

Then of course, if I don't iterate over the projects, I could do:

from brightway2 import *

names = [p.name for p in projects]

for name in names:
    projects.current = name
    print(name, databases)

... and that works fine.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Some instances of corruption in JSON files (even with atomic writes) have been observed in the wild, so this transition needs to happen ASAP...

Original report by Tomas Navarrete Gutierrez (Bitbucket: tomas_navarrete, ).

Hi,

When using python 2.7, printing the databases (bw2data.meta.Databases) goes wrong.

In [64]: type(databases)
Out[64]: bw2data.meta.Databases

In [65]: print databases
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-65-5a55644a77b6> in <module>()
----> 1 print databases

/home/tomas/virtualenvs/ve1/lib/python2.7/site-packages/future/utils/__init__.pyc in <lambda>(self)
    100     if not PY3:
    101         cls.__unicode__ = cls.__str__
--> 102         cls.__str__ = lambda self: self.__unicode__().encode('utf-8')
    103     return cls
    104 

... last 1 frames repeated, from the frame below ...

/home/tomas/virtualenvs/ve1/lib/python2.7/site-packages/future/utils/__init__.pyc in <lambda>(self)
    100     if not PY3:
    101         cls.__unicode__ = cls.__str__
--> 102         cls.__str__ = lambda self: self.__unicode__().encode('utf-8')
    103     return cls
    104 

RuntimeError: maximum recursion depth exceeded in cmp

With an origin in:

/home/tomas/virtualenvs/ve1/lib/python2.7/site-packages/bw2data/serialization.pyc in <lambda>(x)
    168             )
    169 
--> 170     __repr__ = lambda x: str(x)
    171 
    172     def __delitem__(self, name):

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Only reasonable response to Whoosh bug 428: Search index files not closed if filter is used (Python 3 only), given the Whoosh author's focus on the "next generation".

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Using BW2IO to import datasets - if there is an error, the database is locked, and can't be used in the same process. The Python interpreter must be restarted.

Original report by Niko Heeren (Bitbucket: nheeren, GitHub: nheeren).

When I try to do data = bw2.Database("ecoinvent 3.2 cutoff").load() my Mac is busy loading data for ages. I killed the process after it used 40+GB of RAM (!).

Is that the documentation that is contained in ecoinvent 3.2 now? Probably it would be smart to offer a way to load the database without the documentation.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

See this bug report in a SO comment.

Need to investigate further (with actual test case).

Original report by Pascal Lesage (Bitbucket: MPa, ).

Two cases were found:

1. Filtering on location using uppercase (as they are found in the activity) returns onthing (i.e. an empty list). Changing to lowercase works, even though in reality e.g. 'DE' != 'de'

2. Filtering on locations with dashes (as they are found in the activities) returns nothing (i.e. an empty list). Changing to lowercase in this case doesn't change anything, e.g. 'ca-qc', CA-QC', 'caqc', CAQC' all return nothing.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Current error is e.g.

#!python

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-30-8e5992c4fb52> in <module>()
      1 db = Database("biosphere2")
----> 2 db.make_searchable()

/Users/cmutel/local35/bw3/lib/python3.5/site-packages/bw2data/project.py in writable_project(wrapped, instance, args, kwargs)
    298     if projects.read_only:
    299         raise ReadOnlyProject(READ_ONLY_PROJECT)
--> 300     return wrapped(*args, **kwargs)

/Users/cmutel/local35/bw3/lib/python3.5/site-packages/bw2data/backends/peewee/database.py in make_searchable(self)
    276             print("This database is already searchable")
    277             return
--> 278         databases[self.name]['searchable'] = True
    279         databases.flush()
    280         IndexManager(self.filename).add_datasets(self)

/Users/cmutel/local35/bw3/lib/python3.5/site-packages/bw2data/serialization.py in __getitem__(self, key)
    140         if isinstance(key, list):
    141             key = tuple(key)
--> 142         return self.data[key]
    143 
    144     @writable_project

KeyError: 'biosphere2'

This should be clearer that the database doesn't exist, therefore can't be made searchable.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Some databases (ESU, US LCI) don't list categories for technological inputs. We currently search the inventory datasets in the database being imported, but we also need to search the dependent databases in a relatively intelligent way (i.e. including whatever attributes are included, e.g. unit, location) to get trouble-free imports.

Here is an example exchange:

#!python

{'amount': 0.020673,
 'code': 2,
 'comment': '(2,2,1,1,1,3); Estimation 20l/m2 panel',
 'group': 5,
 'loc': -3.8789267779389562,
 'matching': {'categories': (),
              'location': 'CH',
              'name': 'tap water, at user',
              'unit': u'kilogram'},
 'negative': False,
 'scale': 0.043409845976280094,
 'uncertainty type': 2}

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Based on tests on my laptop, it is ~33% faster and uses less space. The two issues are:

1. Strings would have to be decoded from raw bytes to Unicode
2. Another dependency, which would have to be installed using easy_install on windows (because it is a binary egg)

The advantages are:

1. Speed (especially on Windows, where a fast JSON parser might not be installed)
2. A safe file format to move between computers (as opposed to pickles)

See also:

• message pack
• HN discussion
• YAJL
• Recursive string decoding

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Current procedure with "Copy of ..." is prone to multiple failures.

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

During first installation bw2 create and sets automatically the data directory with appdirs but doing so, when different version of brightway in different virtual environments are installed (e.g. would like to keep the working installation I have and install the dev also), they all point to the same data_dir (in linux ~/.local/share/BRIGHTWAY2_or_3) and making impossible to have installed different version in separate virtual env.
If not possible yet, would be good to leave possible to choose the data directory during the installation....unless there is already a way to install multiple version of bw2 in different environment.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Appears especially bad on Windows, more than 2x total memory needed.

https://pypi.python.org/pypi/ijson/

Original report by Tomas Navarrete Gutierrez (Bitbucket: tomas_navarrete, ).

If I want to update all the amounts of the inputs of an activity (with different values), I would have to iterate over the technosphere of the activity, update the amount field per exchange, and call "save()" on each exchange. With n technosphere exchanges, this requires n transactions, in current implementation.

I would like to group the updates, so that there is only one transaction, and hopefully less I/O to hardrive.

The idea from: http://docs.peewee-orm.com/en/latest/peewee/querying.html#atomic-updates

something in the line of:

"""
exchanges is a dictionary of oldExchange:newExchange
"""
def bulk_update_exchanges(activity, exchanges):
    with db.atomic as txn:
        for old, new in exchanges.items():
            old.update(data = new.data)

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Need new datastore for calculation log files (see BW2Calc issue).
Also need function to prepare inputs for brightway-calc (next generation), based both of "normal" LCA inputs and parsed log files.

Original report by Tomas Navarrete Gutierrez (Bitbucket: tomas_navarrete, ).

It would be useful to have a way to specify that we want to access a project in "read only" mode, so that a write lock does not get created automatically.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Need a series of tests to make sure that this works consistently with drive letters and accents.

See also http://stackoverflow.com/questions/13232108/python-overriding-os-path-supports-unicode-filenames-on-ubuntu

Original report by Pascal Lesage (Bitbucket: MPa, ).

All projects (even new projects) default to read_only.
Deleting the write_lock does not help, it is replaced at every time the project is set to current.

Here is the error message upon attempting enable_writes:

C:\bw2-python\envs\bw2\lib\site-packages\bw2data\project.py in enable_writes(self, force) 219 """Enable writing for the current project.""" 220 if force: --> 221 os.remove(os.path.join(self.dir, "write-lock")) 222 self.read_only = not self._lock.acquire(timeout = 0.05) 223 if not self.read_only: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\pasca\AppData\Local\pylca\Brightway3\copy-of-ecoinvent_32_cutoff_US.546fa5af9888b4304f69c885ebe82de7\write-lock'

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

#!python

In [47]: lca = LCA({elec: 1})

In [48]: lca.lci(factorize=True)

In [49]: lca.lci(factorize=True)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-49-81d0786914c4> in <module>()
----> 1 lca.lci(factorize=True)

/Users/cmutel/local35/bw3/lib/python3.5/site-packages/bw2calc/lca.py in lci(self, factorize, builder)
    254         """
    255         self.load_lci_data(builder)
--> 256         self.build_demand_array()
    257         if factorize:
    258             self.decompose_technosphere()

/Users/cmutel/local35/bw3/lib/python3.5/site-packages/bw2calc/lca.py in build_demand_array(self, demand)
     88         for key in demand:
     89             try:
---> 90                 self.demand_array[self.product_dict[key]] = demand[key]
     91             except KeyError:
     92                 if key in self.activity_dict:

KeyError: 'electricity, low voltage, production CH, at grid' (kilowatt hour, CH, ['electricity', 'production mix'])

Something about fixing dictionaries and then reloading them...

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Currently trying to change these just raises an error

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

It would be nice to be able to do e.g.

#!python

Filter("name", "in", "foo") & Filter("unit", "is", "kilogram")

or

#!python

Filter("name", "in", "foo") | Filter("unit", "is", "kilogram")

This would require overriding and and or.

Also, the Result class should make it easier to see the individual dataset details.

(See http://www.rafekettler.com/magicmethods.html)

Original report by Caroline Nadel (Bitbucket: cazcazn, ).

The database has changed, so there was a bit of debugging needed. I followed the tutorial to the letter, aside from setting the relevant data directory.

Firstly, the subdir is no longer required so zip import needs to be amended (I did it manually).

Secondly there are a few entries which throw errors whilst being imported, namely that their category doesn't match values contained in BIOSPHERE. Without fully knowing what this means, I worked around that by changing line 21 in import_ecospold to BIOSPHERE = ("air", "water", "soil", "resource", "final-waste-flow", "raw", "others", "non-material"), although I don't know if that's the correct fix.

Also, L431 changed to: assert exc[u"matching"][u"categories"][0].lower() in BIOSPHERE, as some categories are capitalised in the XML.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Here are some libraries:

• https://github.com/fatiherikli/dictdiffer
• https://github.com/stefankoegl/python-json-patch

Original report by Anonymous.

Values are saved correctly, but not loaded in the web page.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Including:

• Activity.exchanges
• Activity.technosphere,
• Activity.biosphere

Plus raw=True

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

After the introduction of the read-only mode for the projects (before was ok) it is not possible anymore to run calculation in parallel even if the projects.enable_writes() is called.
attached an example of the file used

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Suggestion from Tomas - helpful for practitioners who work with common libraries.

Probably easiest to just export database and then re-import it in each project.

Needs a new data structure to track what should be mirrored where.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

These are exchanges and should only be available via ds.production()

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

For when you revert, but don't then don't want to overwrite previous versions.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

#!python

(bw3)cmutel🚲  ~ ➔ ipython
WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
Python 3.5.1 (default, Dec 10 2015, 12:03:53)
Type "copyright", "credits" or "license" for more information.

IPython 4.1.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from brightway2 import *
/Users/cmutel/local35/bw3/lib/python3.5/site-packages/bw2data/project.py:125: UserWarning:
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

  warnings.warn(READ_ONLY_PROJECT)
/Users/cmutel/local35/bw3/lib/python3.5/site-packages/bw2data/project.py:125: UserWarning:
***Read only project***

This project is being used by another process and no writes can be made until:
    1. You close the other program, or switch to a different project, *and*
    2. You call `projects.enable_writes` *and* get the response `True`.

  warnings.warn(READ_ONLY_PROJECT)
Warning: bad escape \s

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

It seems that something is wrong with activity.copy()

#!python

In [3]: fur=Database('mydb')

In [4]: unit=fur.random()

In [5]: unit.copy(code='test',name='test')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-14f9fd67177c> in <module>()
----> 1 unit.copy(code='test',name='test')

/home/giuseppec/miniconda3/envs/bw2_LU/lib/python3.4/site-packages/bw2data/project.py in writable_project(wrapped, instance, args, kwargs)
    368     if projects.read_only:
    369         raise ReadOnlyProject(READ_ONLY_PROJECT)
--> 370     return wrapped(*args, **kwargs)

/home/giuseppec/miniconda3/envs/bw2_LU/lib/python3.4/site-packages/bw2data/backends/peewee/proxies.py in copy(self, code, **kwargs)
    233             activity[key] = value
    234         for k, v in kwargs.items():
--> 235             setattr(activity._data, k, v)
    236         activity._data[u'code'] = str(code or uuid.uuid4().hex)
    237         activity.save()

AttributeError: 'dict' object has no attribute 'name'

also passing other keyword the error is the same

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

e.g. (u'ecological scarcity 2013', u'total', u'total') (but (u'ecological scarcity 2013', u'total', u'total', u'ecoinvent3')) is correct.

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

this should return also exchanges of type substitution being part of the technosphere matrix.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Choose one for consistency.

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Instead you just get None.

Original report by giuseppe cardellini (Bitbucket: cardosan, GitHub: cardosan).

delete_project delete the project from projects.db but not its folder

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Or maybe change input format completely to lists...

Original report by Chris Mutel (Bitbucket: cmutel, GitHub: cmutel).

Using order by ? takes a while, as SQLite has to order all rows, especially at startup. Better to leave ordering undefined, but still leave a iteration randomly ordered. So just need to change .random(), but check this!

See also: http://stackoverflow.com/questions/4114940/select-random-rows-in-sqlite