iiasa / message_ix Goto Github PK
View Code? Open in Web Editor NEWThe integrated assessment and energy systems model MESSAGEix
Home Page: https://docs.messageix.org
License: Apache License 2.0
The integrated assessment and energy systems model MESSAGEix
Home Page: https://docs.messageix.org
License: Apache License 2.0
It seems like in message_ix
, scen
is used as the kwarg while in ixmp
, it is scenario
. We should harmonize those two.
When running the Austria tutorial and running the following piece of code:
# launch the IX modeling platform using the local default database mp = ix.Platform(dbtype='HSQLDB')
I get the following error:
INFO:root:launching ixmp.Platform with local HSQLDB database at '/Users/b1017579/.local/ixmp/localdb/default' INFO:root:Could not launch the JVM for the ixmp.Platform.Make sure that all dependencies of ixmp.jarare included in the 'ixmp/lib' folder.
I have followed the readme tutorial to install imxp, but am struggling to get this working.
Any help would be greatly appreciated.
Thanks,
Alex
The decision variables concerning capacity CAP
and CAP_NEW
are in the same unit, but concern a different temporal dimension.
CAP_NEW
is the amount newly installed in one year.
CAP
is the amount installed at the end of a (usually multi-year) period.
So in the first year of installation, it usually holds that CAP = CAP_NEW x period_duration
(assuming newly built capacity is not immediately decommissioned, which is mathematically possible).
The difference needs to be better explained in the documentation and tutorials, because it has created some confusion...
It has to do with trying to write the json file. Will add more details asap.
When a input/output combination is defined for historical vintage years of a technology but no hist_new_cap is defined for all the historical vintage combinations, the model is not restricted to the CAPACITY_MAINTENANCE_HIST bound. This can lead to capacity beeing built where not allowed (as in the picture)
It should be possible to “report” or “post-process” a message_ix.Scenario
(given a sufficient amount of configuration metadata) to generate output (IAMC-compliant pd.DataFrame
or file) that can be directly submitted to IIASA databases for either single- or multi-model assessments.
message_ix.Reporting
class for early release.I noticed that after the most recent updates the model folder with the gams files has been moved inside a sub-folder message_ix/message_ix/model.
the solve
function under the MESSAGE scheme is not working anymore because of this change.
So I was wondering, is it a definitive structure we want to have for the repository? should I update the package accordingly?
We should add some documentation on how to change GAMS and solver options. Some of these are currently "hard-coded" in the MESSAGEix code.
This proposal contributes to #149. It builds on #150, but generalizes the conceptual framing. It also uses comments by @khaeru in https://github.com/iiasa/message_ix/wiki/Reporting.
The idea behind this proposal is to develop specific nomenclature/taxonomy of the data such that clearly defined processing steps as well as relationships/dependencies between steps and/or data can be identified.
We require a solution to turn the raw GAMS output (dataframes of variables and equations, e.g., ACT
) into an IAMC-style timeseries table.
Note that there isn't necessarily a hierarchical order in which these operations are called.
Primitives are data accessible directly from model results without any processing required.
Examples: activity
, input
, new_capacity
Derivatives are data that can be calculated directly from primitives or other derivatives. There is a plethora of possible functions here (multiplication, division, shares, growth rates).
Examples: gdp per capita
Aggregates are data that must be aggregated (using any method, e.g., sum, weighted average, etc.) from primitives or derivatives from one or more technologies, or where data is aggregated over regions.
The final operation, doing some clean-up, building up the variable tree, aggregating over regions, and committing (a selection of) the reporting output to the ixmp platform (aka database).
Given the similarities of features like 'derivatives' and 'aggregates' to pyam, it makes sense to use a pyam.IamDataFrame
as the internal data model. Every operation computes a timeseries (identified by an IAMC-style variable, one series for each relevant region) and appends it to the data model.
pp = message_ix.PostProcess(scenario)
pp.activity('Primary Energy|Wind', technology='wind_ppl')
...
pp.aggregate('Primary Energy)
pp.finalize()
- activity
- Primary Energy|Wind
- technology: wind_ppl
- aggregate
- Primary Energy
message_ix.Scenario.reporting()
electricity|secondary
as an ouputIn Firefox and EDGE the MESSAGEix documentation (public version at http://messageix.iiasa.ac.at) theme renders using an italics font which is difficult to read (screenshot from Firefox attached). Making sure that the theme renders the same across browsers (in a more readable way than the attached example) would be good.
This will require a few concrete steps:
My one concern is how our CI examines its OS state (through $TRAVIS_OS_NAME
. But perhaps we can make this new install target only available on linux to reduce headaches.
@khaeru anything else I'm missing?
The solvers for running MESSAGEix are currently specified in the file message_ix/model/MESSAGE/auxiliary_settings.gms
- this causes issues for users who do not have a CPLEX license. It would be prefereable that GAMS defaults to the solver selection made during GAMS installation, and only make the MESSAGE-specific selection of solver optional.
I was building my model based on CD_Links_SSP2 | baseline. I see it has the parameter 'renewable_tec' as empty in the input GDX (version 18). But when I run
Sc_hydro.add_set("renewable_tec", allhydrotech),
it complains:
at.ac.iiasa.ixmp.exceptions.IxException: No Set 'renewable_tec' exists in this Scenario!
So I did instead
Sc_hydro.init_set("renewable_tec")
Sc_hydro.add_set("renewable_tec", allhydrotech)
It runs ok, but then I get another error when I try to solve the model:
com.gams.api.GAMSExceptionPyRaisable: com.gams.api.GAMSException: Symbol renewable_tec exists already
I was able to expand other sets like 'commodity', 'technology', etc. without errors, but only this renewable_tec doesn't work.
If we set fixed_new_capacity to a certain value, the model reacts by setting the CAP_NEW parameter to the desired value. The CAP however seems to be unaffected by this, thus, remains zero. Any ideas on why this is happening / is this the desired behaviour?
Hey there, @tzipperle and I noticed that the equation OPERATING_RESERVE_CONSTRAINT is missing in the GAMS model_core and that the flexibility_factor does not show up in any other equation. Is this on purpose or is this a bug?
The documentation currently links to 'bare' IPython notebooks, as rendered by Github's built-in version of nbviewer. 'Bare' means they do not contain cell output, which makes them difficult to read.
Improve on this, so that tutorials with output are rendered and hosted in the same place as the documentation.
The strategy to do so is:
setup.py
.ci/travis-install.sh
script and invoke this when setup.py install message_ix[gams]
is given.(per a 2019-01-25 discussion with @gidden)
There are certain technologies/levels where an inequality for the COMMODITY_BALANCE
(i.e., SUPPLY > DEMAND) is not applicable, like electricity or pollutants.
We need to define a new equation in the GAMS formulation that implements strict equality as an option.
The question at this point is how to "activate" the strict equality. Should this be contingent on the commodity
only, or (commodity, level)
, or also be specific on node
, year
or sub-annual time
.
Previously, we tried to import the package to get the version number in setup.py
. This broke our conda build. It appears that versioneer can fix this. See, e.g., xarray.
It seems technology mapping sets, e.g., "map_tec_mode" and "map_tec_lifetime" don't get updated when the respective parameters get modified.
For example, I have had some changes in a scenario as follows:
1- removing technology A from all the parameters in one region, or
2- reducing the lifetime of technology B, resulting in less combination of vintage-active years in parameters "output" and "input" for that technology.
However, technology mapping sets are not being updated during the generation of input GDX file. As a result:
I hope I'm not missing something out.
We would like to support porting message_ix.Scenario
objects into appropriate payloads for running multi criteria analyses with the IIASA MCMA tool. To do so requires a number of steps:
For example, one existing criteria is
CUMGHG =E=
SUM( (node,type_emission,type_tec,type_year) $ (SAMEAS(type_emission, 'GHGs') AND SAMEAS(type_year,'cumulative')),
SUM( (year_all2,emission)$( cat_year(type_year,year_all2) AND cat_emission(type_emission,emission) ),
duration_period(year_all2) * emission_scaling(type_emission,emission) *
( EMISS(node,emission,type_tec,year_all2)$( year(year_all2) )
+ historical_emission(node,emission,type_tec,year_all2) )
)
/ SUM(year_all2$( cat_year(type_year,year_all2) ), duration_period(year_all2) )
)
This could be updated to
def cumulative_emissions(type_emission):
return
"""
CUM{type_emission} =E=
SUM( (node,type_emission,type_tec,type_year) $ (SAMEAS(type_emission, '{type_emission}') AND SAMEAS(type_year,'cumulative')),
SUM( (year_all2,emission)$( cat_year(type_year,year_all2) AND cat_emission(type_emission,emission) ),
duration_period(year_all2) * emission_scaling(type_emission,emission) *
( EMISS(node,emission,type_tec,year_all2)$( year(year_all2) )
+ historical_emission(node,emission,type_tec,year_all2) )
)
/ SUM(year_all2$( cat_year(type_year,year_all2) ), duration_period(year_all2) )
)
""".format(type_emission=type_emission)
Initial entries could be seeded with input from @volker-krey, @adrivinca, @scparkinson, @MarekMakowski and others.
generate a gdx file from an existing scenario (this can be done using available tools in the solve
function)
generate a new gms
file filled with criteria definitions and a "config" file. Existing config file is below:
CUMCOST
CUMINV
1000 rows
1000 cols
@MarekMakowski should advise on how best to do this automatically.
specifically should overrule cloning from ixmp
to return a message_ix.Scenario
The tutorial and docs should explain how to deal with different types of GAMS errors, like missing/invalid license or execution errors (infeasibility).
The debugging features also need to be explained.
When I want to clone a scenario, if I input a new scenario name using 'scenario' argument, it doesn't clone with the new scenario name. But when I use 'scen', it works fine, while it gives a deprecation warning. I mean:
this works: sc_new = sc_ref.clone(model='My_MESSAGE', scen='test')
this doesn't work: sc_new = sc_ref.clone(model='My_MESSAGE', scenario='test')
Hi guys,
Just as suggestion you might expose that we need to install the ixmp model before isntal the message_ix model.
Regards
Fabio Diuana
For #138, the Circle CI test fails as follows:
$ #!/bin/sh -eo pipefail
docker run \
-e MESSAGE_IX_CI_USER=$MESSAGE_IX_CI_USER \
-e MESSAGE_IX_CI_PW=$MESSAGE_IX_CI_PW \
message_ix/message_ix:py2 \
/bin/bash -c 'source activate root && cd /message_ix/tests/ci && ./run_on_ci.sh'
++ set -e
++ python generate_test_file.py
++ python download_license.py
Traceback (most recent call last):
File "download_license.py", line 29, in <module>
/gams24.8_linux_x64_64_sfx/gams
'Failed download with user/pass: {}/{}'.format(username, password))
IOError: Failed download with user/pass: /
Exited with code 1
However, it does not fail for, e.g. #142, which is from a different fork than mine.
Is there a specific action needed to add these credentials for each fork? If so, these should be documented, and it should be noted (perhaps in the PR template, or elsewhere) when this test is not required; e.g. if a non IIASA developer makes a PR and we cannot share with them a ENE GAMS license file.
Issue arises in second cell of tutorial when running
mp = ix.Platform(dbtype='HSQLDB')
Error is:
No JVM shared library file (jvm.dll) found. Try setting up the JAVA_HOME environment variable properly.
This can occur when message_ix
has been installed from command line using regular command prompt or WIndows PowerShell
Try installing message_ix
package using Anaconda Prompt. This can solve the issue
Suggestion: update installation instructions to use Anaconda Prompt.
We want to implement some tests that confirm changes in the code base do not affect basic baseline scenarios. This will require a number of steps.
First, we will want to make a ci
folder inside of message_ix/tests
. Inside, we want the following files:
scenarios.csv
: lists models, scenarios, expected objective function valuegenerate_scenario_db.[py|sh]
: connects to production database, creates new local database, copies models/scenarios in scenarios.csv
to local database, uploads local database to internal IIASA hosted filesystem accessible from outside. This will require the ixmp
branch gidden:newjar
.test_scenarios.py
:
README.md
which explains what each file is for, how to regenerate the data, etc.We will want to have all passwords required stored in environment variables. This can then be updated in circleci
.
Finally, we will want to update .circleci/config.yaml
to update the line which does pytest tests --ci
Assigning to @adrivinca but perhaps also useful for some YSSPers to take a look? cc: @byersiiasa @scparkinson
When loading ixmp.Platform() there is an error migrating database:
import os
import pandas as pd
import ixmp as ix
import message_ix
dbprops = "ixmp.properties"
mp = ix.Platform() #dbtype='HSQLDB'
INFO:root:launching ixmp.Platform using config file at 'C:\Users\alexk\Documents\GitHub\ixmp\config\ixmp.properties'
Traceback (most recent call last):
File "", line 19, in
mp = ix.Platform() #dbtype='HSQLDB'
File "C:\Miniconda3\lib\site-packages\ixmp\core.py", line 82, in init
self._jobj = java.ixmp.Platform("Python", dbprops)
File "C:\Miniconda3\lib\site-packages\jpype_jclass.py", line 111, in _javaInit
*args)
org.flywaydb.core.api.FlywayExceptionPyRaisable: org.flywaydb.core.api.FlywayException: Validate failed: Detected resolved migration not applied to database: 1.2
It should explain what each tutorial is and what order they are needed in.
This proposal contributes to #149
The idea behind this proposal is to develop specific nomenclature/taxonomy of the data such that clearly defined processing steps as well as relationships/dependencies between steps and/or data can be identified.
I propose (prior to any other suggestions/inputs) four classifications of data:
Their relationships are shown in the below figure:
I am trying to install message_ix following the Install from Conda steps. Everything was fine until I ran the config model files command, i.e.
messageix-config --model_path /path/to/model
The following error message is printed:
Traceback (most recent call last):
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\Scripts\messageix-config-script.py", line 10, in
sys.exit(config())
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\site-packages\message_ix\cli.py", line 65, in config
do_config(model_path=args.model_path, overwrite=args.overwrite)
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\site-packages\message_ix\cli.py", line 53, in do_config
with open(CONFIG_PATH, mode='w') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\wangy\.local\message_ix\config.json'
I could not find a .local folder in C:\Users\wangy.
After creating the relevant folder and an empty "config.json", running the command still creates error message:
Traceback (most recent call last):
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\Scripts\messageix-config-script.py", line 6, in
from message_ix.cli import config
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\site-packages\message_ix_init_.py", line 10, in
model_file = os.path.join(default_paths.model_path(), '{model}run.gms')
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\site-packages\message_ix\default_paths.py", line 12, in model_path
data = json.load(f)
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\json_init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\json_init_.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Installing from source copies all gdx files from the cloned folder to the installed location, which is a pain if you have run large model instances previously.
When explicitily using the keyword arguments model/scenario, and only when the model name is kept the same as in the scenario to be cloned (scenario: base - in the example below), then the clone command makes a a copy and assigns this the base scenario name "base" scenario instead of making a copy and assigning the copied scenario the "new scenario name".
import message_ix
import ixmp
mp = ixmp.Platform(dbtype='HSQLDB')
base = message_ix.Scenario(mp, 'Westeros Electrified', 'baseline', cache=True)
__
2018-11-26 10:33:35,663 INFO at.ac.iiasa.ixmp.objects.Scenario:215 - loading default Scenario 'Westeros Electrified|baseline' (version: 7, runid: 86)...
2018-11-26 10:33:35,744 INFO at.ac.iiasa.ixmp.objects.Scenario:241 - done loading Scenario from the database!
2018-11-26 10:33:35,744 INFO at.ac.iiasa.ixmp.objects.MsgScenario:457 - spatial and temporal disaggregation maps assignment
2018-11-26 10:33:35,760 INFO at.ac.iiasa.ixmp.objects.MsgScenario:573 - period mapping assignment
2018-11-26 10:33:35,760 INFO at.ac.iiasa.ixmp.objects.MsgScenario:599 - duration period assignment
2018-11-26 10:33:35,760 INFO at.ac.iiasa.ixmp.objects.Scenario:1826 - committing changes of Scenario 'Westeros Electrified|baseline' to the database (runid: 86)...
2018-11-26 10:33:35,856 INFO at.ac.iiasa.ixmp.objects.Scenario:601 - done updating Scenario 'Westeros Electrified|baseline' to the database (runid: 86, version: 7)!
2018-11-26 10:33:35,856 INFO at.ac.iiasa.ixmp.objects.MsgScenario:457 - spatial and temporal disaggregation maps assignment
2018-11-26 10:33:35,872 INFO at.ac.iiasa.ixmp.objects.MsgScenario:573 - period mapping assignment
2018-11-26 10:33:35,872 INFO at.ac.iiasa.ixmp.objects.MsgScenario:599 - duration period assignment
__
With the explicit use of keyword arguments yields the following
scen = base.clone(model='Westeros Electrified', scenario='baseline_1',)
--> the base scenario is cloned and assigned a new version
__
2018-11-26 10:33:35,948 INFO at.ac.iiasa.ixmp.objects.Scenario:215 - loading default Scenario 'Westeros Electrified|baseline' (version: 0, runid: 86)...
2018-11-26 10:33:35,963 INFO at.ac.iiasa.ixmp.objects.Scenario:241 - done loading Scenario from the database!
2018-11-26 10:33:35,963 INFO at.ac.iiasa.ixmp.objects.Scenario:1896 - saving Scenario 'Westeros Electrified|baseline' to database (runid: 111)...
2018-11-26 10:33:36,087 INFO at.ac.iiasa.ixmp.objects.Scenario:604 - done saving Scenario 'Westeros Electrified|baseline' to the database (runid: 111, version: 10)
__
withOUT the explicit use of keyword arguments yields the following
scen = base.clone('Westeros Electrified', 'baseline_2',)
--> the base scenario is cloned with a new scenario name and assigned a new version
__
2018-11-26 10:33:36,109 INFO at.ac.iiasa.ixmp.objects.Scenario:215 - loading default Scenario 'Westeros Electrified|baseline_2' (version: 0, runid: 86)...
2018-11-26 10:33:36,125 INFO at.ac.iiasa.ixmp.objects.Scenario:241 - done loading Scenario from the database!
2018-11-26 10:33:36,125 INFO at.ac.iiasa.ixmp.objects.Scenario:1896 - saving Scenario 'Westeros Electrified|baseline_2' to database (runid: 112)...
2018-11-26 10:33:36,217 INFO at.ac.iiasa.ixmp.objects.Scenario:604 - done saving Scenario 'Westeros Electrified|baseline_2' to the database (runid: 112, version: 4)!
__
Some functions for which slack variables are required:
@Jihoon (in #62) and @ClaraLuisa (in #63) separately ran into this issue. The basic problem is that messageix-dl
will download the current version of tutorials, which are designed to be used with the current version of the master branch. However, when installing from anaconda (or otherwise), they should download the version of the tutorials aligned with the version installed.
This will likely require also #29
Some parameters are described inaccurately on http://messageix.iiasa.ac.at/. eg "historical_emission" should be something like:
instead it reads:
tried to download the tutorials from command line using
messageix-dl --local_path c:/github/msgix_tutorials
Files downloaded correctly to the folder, but threw this error.
INFO:root:Retrieving https://github.com/iiasa/message_ix/archive/master.zip
INFO:root:Unzipping C:\Temp\dvile4p7\master.zip to C:\Temp\dvile4p7
INFO:root:Copying C:\Temp\dvile4p7/message_ix-master/tutorial to c:/github/msgix_tutorials/tutorial
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\README.md (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria.png (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_load_scenario.ipynb (overwrite is
ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_load_scenario_R.ipynb (overwrite
is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_multiple_policies-answers.ipynb (
overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_multiple_policies.ipynb (overwrit
e is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_reticulate.ipynb (overwrite is ON
)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_single_policy.ipynb (overwrite is
ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\tools.py (overwrite is ON)
Traceback (most recent call last):
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 101, in do_dl
shutil.rmtree(tmp)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Temp
\\dvile4p7\\master.zip'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\Scripts\messageix-dl-script.py", line 10, in <module>
sys.exit(dl())
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 119, in dl
local_path=args.local_path)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 103, in do_dl
shutil.rmtree(tmp)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Temp
\\dvile4p7\\master.zip'
PS H:\> messageix-dl --local_path c:/github/msgix_tutorials
INFO:root:Retrieving https://github.com/iiasa/message_ix/archive/master.zip
INFO:root:Unzipping C:\Temp\2net5aru\master.zip to C:\Temp\2net5aru
INFO:root:Copying C:\Temp\2net5aru/message_ix-master/tutorial to c:/github/msgix_tutorials/tutorial
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\README.md (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria.png (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_load_scenario.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_load_scenario_R.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_multiple_policies-answers.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_multiple_policies.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_reticulate.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_single_policy.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\tools.py (overwrite is ON)
Traceback (most recent call last):
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 101, in do_dl
shutil.rmtree(tmp)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Temp\\2net5aru\\master.zi
p'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\Scripts\messageix-dl-script.py", line 10, in <module>
sys.exit(dl())
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 119, in dl
local_path=args.local_path)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 103, in do_dl
shutil.rmtree(tmp)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Temp\\2net5aru\\master.zi
p'
The idea here is to easily rename functional components of a model. It should go something like
for item in scenario.set_list():
value = scenario.set(item)
if old_tech in value:
scenario.add_set(item, new_tech)
for item in scenario.par_list():
df = scenario.par(item)
if 'technology' in df and old_tech in df['technology']:
df = df[df['technology'] == old_tech]
df['technology'] = new_tech
scenario.add_par(item, df)
scenario.rm_set('technology', old_tech)
Ok, I am a beginner here, and it is also likely that I didn't do it right.
I installed MESSAGEix through Anaconda and got the tutorial through messageix-dl. Then, when I ran the whole thing without any changes, I see this at the line "scenario.solve()":
TypeError Traceback (most recent call last)
in ()
----> 1 scenario.solve()
Is it supposed to run ok? Following Matt's suggestion, when I do scenario.solve('MESSAGE'), it runs fine.
Currently, mappings between sets and the category-mapping sets have to be added explicitly. This should be made "more automatic", so that a bound on a set element (e.g., an emission) can be added without the need to define the category mapping (to itself) explicitly.
If I add a further spatial level (eg. state: Vienna) is it not possible to add a timeseries. I have added a short code example with the austrian tutorial: https://github.com/tzipperle/message_ix/blob/test-timeseries/tutorial/Austrian_energy_system/austria-add_timeseries.ipynb
So I added the spatial level:
country = "Austria"
state = "Vienna"
scenario.add_set("node", [country, state, "state"])
scenario.add_set("lvl_spatial", ["country", "state"])
scenario.add_set("map_spatial_hierarchy", ["state", state, "World"])
scenario.add_set("map_spatial_hierarchy", ["country", country, "state"])
at.ac.iiasa.ixmp.exceptions.IxExceptionPyRaisable
Traceback (most recent call last)
<ipython-input-44-991cd1855e1c> in <module>()
----> 1 scenario.add_timeseries(ts)
~\AppData\Local\Continuum\anaconda3\envs\message\lib\site-packages\ixmp\core.py in add_timeseries(self, df, meta)
348 java.Double(float(df.value[i])))
349 # add the final iteration of the loop
--> 350 self._jobj.addTimeseries(region, variable, time, jData, unit, meta)
351
352 # if in 'IAMC-style' format
at.ac.iiasa.ixmp.exceptions.IxExceptionPyRaisable: at.ac.iiasa.ixmp.exceptions.IxException: The node 'Vienna' does not exist in the database!'
It seems the link to conda is not working. I don't know if this is a permanent change, but some students couldn't install the package before the course only for this minor issue.
Currently, one cannot use the solve()
function to read non-default variables and equations from the solution of a MESSAGE-scheme scenario. We should allow to pass args through the solve function to read_sol_from_gdx()
.
cplex.opt is configured as follows:
_* CPLEX LP solution method
lpmethod = 4
threads = 4
epopt 1e-6_
When message initially solves, the barrier method is applied and solves quickly. After macro solves and runs message with updated demands, the primal simplex method is applied despite being configured otherwise. Maybe i dont understand how the barrier algorithm works, but the current solving of macro-initiated solving is very slow and may benefit from using a different solution method.
See log below:
_--- MESSAGE-MACRO_run.gms(4019) 778 Mb
--- LOOPS iteration = 2
--- 469,900 rows 398,378 columns 3,171,485 non-zeroes
--- Executing CPLEX (Solvelink=1): elapsed 0:02:32.683
IBM ILOG CPLEX 24.9.1 r63795 Released Aug 30, 2017 WEI x86 64bit/MS Windows
--- GAMS/Cplex Link licensed for continuous and discrete problems.
Cplex 12.7.1.0
Reading parameter(s) from "C:\Repo\message_ix\message_ix\model\cplex.opt"
- CPLEX LP solution method
- see https://www.gams.com/help/topic/gams.doc/solvers/cplex/index.html#CPLEXlpmethod
- 0 Automatic
- 1 Primal Simplex
- 2 Dual Simplex
- 3 Network Simplex
- 4 Barrier
- 5 Sifting
- 6 Concurrent
lpmethod = 4
threads = 4
epopt 1e-6
Finished reading from "C:\Repo\message_ix\message_ix\model\cplex.opt"
*** Basis supplied by GAMS will be used.
*** Use option 'advind 0' to discard basis information.
Reading data...
Starting Cplex...
Space for names approximately 47.34 Mb
Use option 'names no' to turn use of names off
CPXPARAM_Advance 2
CPXPARAM_TimeLimit 1000000
CPXPARAM_LPMethod 4
CPXPARAM_Threads 4
CPXPARAM_Parallel 1
CPXPARAM_Tune_TimeLimit 200000
CPXPARAM_Barrier_Limits_Iteration 100000000
Parallel mode: deterministic, using up to 4 threads for concurrent optimization.
Iteration log . . .
Iteration: 1 Dual objective = 1889662.827424
Markowitz threshold set to 0.1
Iteration: 24 Dual objective = 1889664.236562
Iteration: 667 Dual objective = 1890028.298649
...._
I have a strange observation, which I want to clarify.
By mistake, I didn't put any 'renewable_potential' values for the historical hydro tec I defined ("hydro_hist") for the entire time horizon, which means its input commodity ("hydro_c_hist") dedicated to the tec has no ground. And input and output parameters were defined only from 1965 and on. But there was historical_new_capacity starting from 1950.
Then in the result, I found that ACT has a value for 2020 for the hydro capacity of year_vtg 1960. (lifetime 60 years). I guess there is a loophole where ACT can still show up when all the input/output/renewable_potential are not defined.
Shouldn't it still give zero ACT if there is no potential defined for modeling years?
When generating a model, a sanity check is performed as part of data_load.gms, to ensure "that the economic and technical lifetime are defined and consistent for all investment technologies". When removing all vintage specific data for a technology after the year for which it should not be available anymore e.g. the technology should only be built up until 2020, therefore vintage specific parameters after 2020 are removed, this results in an error when the sanity check is executed.
More details:
In the global message model, there are several technologies which should not be built/available after a certain time period. For these technologies, there is year_vtg specific data for years after which there is no input/output data specified, such as inv_cost, fix_cost etc. E.g. the technology has not input/defined for vintage_years after 2020, but there are still vintage specific parameters defined for 2030 and 2040, the years when the technology can still be active based on the technical lifetime for the vintage 2020 (in this case the technical lifetime is 30 years). Currently, a bound_new_capacity up is also used for 2030/2040, but if this wouldnt be the case, and someone uses the function vintage_and_active_years() to insert data, then this could result in issues because the vintage_and_active_years() function will return activity years all years for which a technical lifetime is provided including those years for which no input/output data is specified.
In order to resolve this issue, either the sanity check needs to be corrected, or the parameter interdependencies need to be looked at to understand why vintage specific parameters are required for years where a technology cant be built.
It would be great to have some explanation added to the messageix documentation on the handling of the years in periods. e.g. an explanation as given by @danielhuppmann in the hackathon chanell:
The period 700
includes the years [691, .. , 700]
. Thus the duration_period_sum(year1, year2)
parameter counts the number of years from the first year of period year1
to the first year of period year2
including the first year when counting. So duration_period_sum(689, 720)
goes from January 1, 689 until December 31, 710, which is 22 years.
We need a new section to cover "drivers", like population or gdp, which can have direct or indirect impacts on emissions. The GDP trajectory in particular should be linked to the variable returned from MACRO to ensure consistency.
The formulation should be extended to incorporate diffusion constraints that include multiple technologies or work across spatial or temporal scales.
See the following article for a previous implementation in MESSAGE:
Benjamin D. Leibowicz, Volker Krey, and Arnulf Grubler. Representing spatial technology diffusion in an energy system optimization model. Technological Forecasting and Social Change 103:350-363, 2015. 10.1016/j.techfore.2015.06.001
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.