Some parameters are described inaccurately on http://messageix.iiasa.ac.at/. eg "historical_emission" should be something like:

• node | type_emission | type_tec | type_year

instead it reads:

• node | emission | type_tec | year

I noticed that after the most recent updates the model folder with the gams files has been moved inside a sub-folder message_ix/message_ix/model.
the solve function under the MESSAGE scheme is not working anymore because of this change.
So I was wondering, is it a definitive structure we want to have for the repository? should I update the package accordingly?

Hey there, @tzipperle and I noticed that the equation OPERATING_RESERVE_CONSTRAINT is missing in the GAMS model_core and that the flexibility_factor does not show up in any other equation. Is this on purpose or is this a bug?

There are certain technologies/levels where an inequality for the COMMODITY_BALANCE (i.e., SUPPLY > DEMAND) is not applicable, like electricity or pollutants.

We need to define a new equation in the GAMS formulation that implements strict equality as an option.

The question at this point is how to "activate" the strict equality. Should this be contingent on the commodity only, or (commodity, level), or also be specific on node, year or sub-annual time.

In Firefox and EDGE the MESSAGEix documentation (public version at http://messageix.iiasa.ac.at) theme renders using an italics font which is difficult to read (screenshot from Firefox attached). Making sure that the theme renders the same across browsers (in a more readable way than the attached example) would be good.

It would be great to have some explanation added to the messageix documentation on the handling of the years in periods. e.g. an explanation as given by @danielhuppmann in the hackathon chanell:

The period 700 includes the years [691, .. , 700] . Thus the duration_period_sum(year1, year2) parameter counts the number of years from the first year of period year1 to the first year of period year2
including the first year when counting. So duration_period_sum(689, 720) goes from January 1, 689 until December 31, 710, which is 22 years.

Currently, one cannot use the solve() function to read non-default variables and equations from the solution of a MESSAGE-scheme scenario. We should allow to pass args through the solve function to read_sol_from_gdx().

Hi guys,

Just as suggestion you might expose that we need to install the ixmp model before isntal the message_ix model.

Regards
Fabio Diuana

It seems the link to conda is not working. I don't know if this is a permanent change, but some students couldn't install the package before the course only for this minor issue.

I am trying to install message_ix following the Install from Conda steps. Everything was fine until I ran the config model files command, i.e.

messageix-config --model_path /path/to/model

The following error message is printed:

Traceback (most recent call last):
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\Scripts\messageix-config-script.py", line 10, in
sys.exit(config())
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\site-packages\message_ix\cli.py", line 65, in config
do_config(model_path=args.model_path, overwrite=args.overwrite)
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\site-packages\message_ix\cli.py", line 53, in do_config
with open(CONFIG_PATH, mode='w') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\wangy\.local\message_ix\config.json'

I could not find a .local folder in C:\Users\wangy.

After creating the relevant folder and an empty "config.json", running the command still creates error message:

Traceback (most recent call last):
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\Scripts\messageix-config-script.py", line 6, in
from message_ix.cli import config
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\site-packages\message_ix_init_.py", line 10, in
model_file = os.path.join(default_paths.model_path(), '{model}run.gms')
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\site-packages\message_ix\default_paths.py", line 12, in model_path
data = json.load(f)
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\json_init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\json_init_.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\wangy\AppData\Local\Continuum\Anaconda2\envs\py36\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

When generating a model, a sanity check is performed as part of data_load.gms, to ensure "that the economic and technical lifetime are defined and consistent for all investment technologies". When removing all vintage specific data for a technology after the year for which it should not be available anymore e.g. the technology should only be built up until 2020, therefore vintage specific parameters after 2020 are removed, this results in an error when the sanity check is executed.

More details:

In the global message model, there are several technologies which should not be built/available after a certain time period. For these technologies, there is year_vtg specific data for years after which there is no input/output data specified, such as inv_cost, fix_cost etc. E.g. the technology has not input/defined for vintage_years after 2020, but there are still vintage specific parameters defined for 2030 and 2040, the years when the technology can still be active based on the technical lifetime for the vintage 2020 (in this case the technical lifetime is 30 years). Currently, a bound_new_capacity up is also used for 2030/2040, but if this wouldnt be the case, and someone uses the function vintage_and_active_years() to insert data, then this could result in issues because the vintage_and_active_years() function will return activity years all years for which a technical lifetime is provided including those years for which no input/output data is specified.

In order to resolve this issue, either the sanity check needs to be corrected, or the parameter interdependencies need to be looked at to understand why vintage specific parameters are required for years where a technology cant be built.

When I want to clone a scenario, if I input a new scenario name using 'scenario' argument, it doesn't clone with the new scenario name. But when I use 'scen', it works fine, while it gives a deprecation warning. I mean:
this works: sc_new = sc_ref.clone(model='My_MESSAGE', scen='test')
this doesn't work: sc_new = sc_ref.clone(model='My_MESSAGE', scenario='test')

I have a strange observation, which I want to clarify.

By mistake, I didn't put any 'renewable_potential' values for the historical hydro tec I defined ("hydro_hist") for the entire time horizon, which means its input commodity ("hydro_c_hist") dedicated to the tec has no ground. And input and output parameters were defined only from 1965 and on. But there was historical_new_capacity starting from 1950.

Then in the result, I found that ACT has a value for 2020 for the hydro capacity of year_vtg 1960. (lifetime 60 years). I guess there is a loophole where ACT can still show up when all the input/output/renewable_potential are not defined.
Shouldn't it still give zero ACT if there is no potential defined for modeling years?

If I add a further spatial level (eg. state: Vienna) is it not possible to add a timeseries. I have added a short code example with the austrian tutorial: https://github.com/tzipperle/message_ix/blob/test-timeseries/tutorial/Austrian_energy_system/austria-add_timeseries.ipynb

So I added the spatial level:

country = "Austria"
state = "Vienna"
scenario.add_set("node", [country, state, "state"])
scenario.add_set("lvl_spatial", ["country", "state"])
scenario.add_set("map_spatial_hierarchy", ["state", state, "World"])
scenario.add_set("map_spatial_hierarchy", ["country", country, "state"])

at.ac.iiasa.ixmp.exceptions.IxExceptionPyRaisable
Traceback (most recent call last)
<ipython-input-44-991cd1855e1c> in <module>()
----> 1 scenario.add_timeseries(ts)

~\AppData\Local\Continuum\anaconda3\envs\message\lib\site-packages\ixmp\core.py in add_timeseries(self, df, meta)
    348                           java.Double(float(df.value[i])))
    349             # add the final iteration of the loop
--> 350             self._jobj.addTimeseries(region, variable, time, jData, unit, meta)
    351 
    352         # if in 'IAMC-style' format

at.ac.iiasa.ixmp.exceptions.IxExceptionPyRaisable: at.ac.iiasa.ixmp.exceptions.IxException: The node 'Vienna' does not exist in the database!'

The solvers for running MESSAGEix are currently specified in the file message_ix/model/MESSAGE/auxiliary_settings.gms - this causes issues for users who do not have a CPLEX license. It would be prefereable that GAMS defaults to the solver selection made during GAMS installation, and only make the MESSAGE-specific selection of solver optional.

When running the Austria tutorial and running the following piece of code:
# launch the IX modeling platform using the local default database mp = ix.Platform(dbtype='HSQLDB')

I get the following error:
INFO:root:launching ixmp.Platform with local HSQLDB database at '/Users/b1017579/.local/ixmp/localdb/default' INFO:root:Could not launch the JVM for the ixmp.Platform.Make sure that all dependencies of ixmp.jarare included in the 'ixmp/lib' folder.

I have followed the readme tutorial to install imxp, but am struggling to get this working.

Any help would be greatly appreciated.

Thanks,
Alex

We would like to support porting message_ix.Scenario objects into appropriate payloads for running multi criteria analyses with the IIASA MCMA tool. To do so requires a number of steps:

1. Develop a library of customizeable optimization criteria

For example, one existing criteria is

    CUMGHG =E=
    SUM( (node,type_emission,type_tec,type_year) $ (SAMEAS(type_emission, 'GHGs') AND SAMEAS(type_year,'cumulative')),
         SUM( (year_all2,emission)$( cat_year(type_year,year_all2) AND cat_emission(type_emission,emission) ),
        duration_period(year_all2) * emission_scaling(type_emission,emission) *
            ( EMISS(node,emission,type_tec,year_all2)$( year(year_all2) )
                + historical_emission(node,emission,type_tec,year_all2) )
      )
    / SUM(year_all2$( cat_year(type_year,year_all2) ), duration_period(year_all2) )
       )

This could be updated to

def cumulative_emissions(type_emission):
return
"""
    CUM{type_emission} =E=
    SUM( (node,type_emission,type_tec,type_year) $ (SAMEAS(type_emission, '{type_emission}') AND SAMEAS(type_year,'cumulative')),
         SUM( (year_all2,emission)$( cat_year(type_year,year_all2) AND cat_emission(type_emission,emission) ),
        duration_period(year_all2) * emission_scaling(type_emission,emission) *
            ( EMISS(node,emission,type_tec,year_all2)$( year(year_all2) )
                + historical_emission(node,emission,type_tec,year_all2) )
      )
    / SUM(year_all2$( cat_year(type_year,year_all2) ), duration_period(year_all2) )
       )
""".format(type_emission=type_emission)

Initial entries could be seeded with input from @volker-krey, @adrivinca, @scparkinson, @MarekMakowski and others.

2. generate a gdx file from an existing scenario (this can be done using available tools in the solve function)

3. generate a new gms file filled with criteria definitions and a "config" file. Existing config file is below:

CUMCOST
CUMINV

1000 rows
1000 cols

@MarekMakowski should advise on how best to do this automatically.

4. zip all relevant files

The decision variables concerning capacity CAP and CAP_NEW are in the same unit, but concern a different temporal dimension.

CAP_NEW is the amount newly installed in one year.
CAP is the amount installed at the end of a (usually multi-year) period.

So in the first year of installation, it usually holds that CAP = CAP_NEW x period_duration (assuming newly built capacity is not immediately decommissioned, which is mathematically possible).

The difference needs to be better explained in the documentation and tutorials, because it has created some confusion...

We want to implement some tests that confirm changes in the code base do not affect basic baseline scenarios. This will require a number of steps.

First, we will want to make a ci folder inside of message_ix/tests. Inside, we want the following files:

• scenarios.csv: lists models, scenarios, expected objective function value
• generate_scenario_db.[py|sh]: connects to production database, creates new local database, copies models/scenarios in scenarios.csv to local database, uploads local database to internal IIASA hosted filesystem accessible from outside. This will require the ixmp branch gidden:newjar.
• test_scenarios.py:
  • all tests should have a pytest CLI decorator to skip unless a specific flag is provided (e.g., pytest --ci)
  • all tests should have a check that downloads the localdb if not already there
  • single test for each scenario (either explicitly or generated on the fly, we can talk about the best strategy here)
• a README.md which explains what each file is for, how to regenerate the data, etc.

We will want to have all passwords required stored in environment variables. This can then be updated in circleci.

Finally, we will want to update .circleci/config.yaml to update the line which does pytest tests --ci

Installing from source copies all gdx files from the cloned folder to the installed location, which is a pain if you have run large model instances previously.

This proposal contributes to #149

The idea behind this proposal is to develop specific nomenclature/taxonomy of the data such that clearly defined processing steps as well as relationships/dependencies between steps and/or data can be identified.

I propose (prior to any other suggestions/inputs) four classifications of data:

• primitives
• derivatives
• aggregates
• variables

Their relationships are shown in the below figure:

Issue arises in second cell of tutorial when running
mp = ix.Platform(dbtype='HSQLDB')

Error is:
No JVM shared library file (jvm.dll) found. Try setting up the JAVA_HOME environment variable properly.
This can occur when message_ix has been installed from command line using regular command prompt or WIndows PowerShell

Try installing message_ix package using Anaconda Prompt. This can solve the issue

Suggestion: update installation instructions to use Anaconda Prompt.

When a input/output combination is defined for historical vintage years of a technology but no hist_new_cap is defined for all the historical vintage combinations, the model is not restricted to the CAPACITY_MAINTENANCE_HIST bound. This can lead to capacity beeing built where not allowed (as in the picture)

The documentation currently links to 'bare' IPython notebooks, as rendered by Github's built-in version of nbviewer. 'Bare' means they do not contain cell output, which makes them difficult to read.

Improve on this, so that tutorials with output are rendered and hosted in the same place as the documentation.

The strategy to do so is:

• Add a new "gams" extra target to setup.py.
• Modify/move the ci/travis-install.sh script and invoke this when setup.py install message_ix[gams] is given.
• Ensure that RTD builds install this new target.
• Ensure that the Sphinx build also renders the tutorials.

(per a 2019-01-25 discussion with @gidden)

Currently, mappings between sets and the category-mapping sets have to be added explicitly. This should be made "more automatic", so that a bound on a set element (e.g., an emission) can be added without the need to define the category mapping (to itself) explicitly.

It should explain what each tutorial is and what order they are needed in.

Assigning to @adrivinca but perhaps also useful for some YSSPers to take a look? cc: @byersiiasa @scparkinson

cplex.opt is configured as follows:
_* CPLEX LP solution method

• see https://www.gams.com/help/topic/gams.doc/solvers/cplex/index.html#CPLEXlpmethod
• 0 Automatic
• 1 Primal Simplex
• 2 Dual Simplex
• 3 Network Simplex
• 4 Barrier
• 5 Sifting
• 6 Concurrent

lpmethod = 4
threads = 4
epopt 1e-6_

When message initially solves, the barrier method is applied and solves quickly. After macro solves and runs message with updated demands, the primal simplex method is applied despite being configured otherwise. Maybe i dont understand how the barrier algorithm works, but the current solving of macro-initiated solving is very slow and may benefit from using a different solution method.

See log below:
_--- MESSAGE-MACRO_run.gms(4019) 778 Mb
--- LOOPS iteration = 2
--- 469,900 rows 398,378 columns 3,171,485 non-zeroes
--- Executing CPLEX (Solvelink=1): elapsed 0:02:32.683

IBM ILOG CPLEX 24.9.1 r63795 Released Aug 30, 2017 WEI x86 64bit/MS Windows
--- GAMS/Cplex Link licensed for continuous and discrete problems.
Cplex 12.7.1.0

Reading parameter(s) from "C:\Repo\message_ix\message_ix\model\cplex.opt"

• CPLEX LP solution method
• see https://www.gams.com/help/topic/gams.doc/solvers/cplex/index.html#CPLEXlpmethod
• 0 Automatic
• 1 Primal Simplex
• 2 Dual Simplex
• 3 Network Simplex
• 4 Barrier
• 5 Sifting
• 6 Concurrent

lpmethod = 4
threads = 4
epopt 1e-6
Finished reading from "C:\Repo\message_ix\message_ix\model\cplex.opt"
*** Basis supplied by GAMS will be used.
*** Use option 'advind 0' to discard basis information.
Reading data...
Starting Cplex...
Space for names approximately 47.34 Mb
Use option 'names no' to turn use of names off
CPXPARAM_Advance 2
CPXPARAM_TimeLimit 1000000
CPXPARAM_LPMethod 4
CPXPARAM_Threads 4
CPXPARAM_Parallel 1
CPXPARAM_Tune_TimeLimit 200000
CPXPARAM_Barrier_Limits_Iteration 100000000
Parallel mode: deterministic, using up to 4 threads for concurrent optimization.

Iteration log . . .
Iteration: 1 Dual objective = 1889662.827424
Markowitz threshold set to 0.1
Iteration: 24 Dual objective = 1889664.236562
Iteration: 667 Dual objective = 1890028.298649
...._

Some functions for which slack variables are required:

• share_commodity_lo/up
• share_mode_lo/up
• addon_lo/up

For #138, the Circle CI test fails as follows:

$ #!/bin/sh -eo pipefail
docker run \
-e MESSAGE_IX_CI_USER=$MESSAGE_IX_CI_USER \
-e MESSAGE_IX_CI_PW=$MESSAGE_IX_CI_PW \
message_ix/message_ix:py2 \
/bin/bash -c 'source activate root && cd /message_ix/tests/ci && ./run_on_ci.sh'
++ set -e
++ python generate_test_file.py
++ python download_license.py
Traceback (most recent call last):
  File "download_license.py", line 29, in <module>
/gams24.8_linux_x64_64_sfx/gams
    'Failed download with user/pass: {}/{}'.format(username, password))
IOError: Failed download with user/pass: /
Exited with code 1

However, it does not fail for, e.g. #142, which is from a different fork than mine.

Is there a specific action needed to add these credentials for each fork? If so, these should be documented, and it should be noted (perhaps in the PR template, or elsewhere) when this test is not required; e.g. if a non IIASA developer makes a PR and we cannot share with them a ENE GAMS license file.

This proposal contributes to #149. It builds on #150, but generalizes the conceptual framing. It also uses comments by @khaeru in https://github.com/iiasa/message_ix/wiki/Reporting.

The idea behind this proposal is to develop specific nomenclature/taxonomy of the data such that clearly defined processing steps as well as relationships/dependencies between steps and/or data can be identified.

Aim

We require a solution to turn the raw GAMS output (dataframes of variables and equations, e.g., ACT) into an IAMC-style timeseries table.

Classifications of operation:

Note that there isn't necessarily a hierarchical order in which these operations are called.

Primitives

Primitives are data accessible directly from model results without any processing required.

Examples: activity, input, new_capacity

Derivatives

Derivatives are data that can be calculated directly from primitives or other derivatives. There is a plethora of possible functions here (multiplication, division, shares, growth rates).

Examples: gdp per capita

Aggregates

Aggregates are data that must be aggregated (using any method, e.g., sum, weighted average, etc.) from primitives or derivatives from one or more technologies, or where data is aggregated over regions.

Finalization

The final operation, doing some clean-up, building up the variable tree, aggregating over regions, and committing (a selection of) the reporting output to the ixmp platform (aka database).

Architecture

Given the similarities of features like 'derivatives' and 'aggregates' to pyam, it makes sense to use a pyam.IamDataFrame as the internal data model. Every operation computes a timeseries (identified by an IAMC-style variable, one series for each relevant region) and appends it to the data model.

Workflow

• Programmatic:

  pp = message_ix.PostProcess(scenario)
  pp.activity('Primary Energy|Wind', technology='wind_ppl')
  ...
  pp.aggregate('Primary Energy)
  pp.finalize()
  

• File-based, using YAML or other syntax.

  - activity
      - Primary Energy|Wind
          - technology: wind_ppl
  - aggregate
      - Primary Energy
  

  called using message_ix.Scenario.reporting()
• Introspective, by examining the sets and category mappings in the model/scenario definition
  Example: every technology having electricity|secondary as an ouput

The formulation should be extended to incorporate diffusion constraints that include multiple technologies or work across spatial or temporal scales.

See the following article for a previous implementation in MESSAGE:
Benjamin D. Leibowicz, Volker Krey, and Arnulf Grubler. Representing spatial technology diffusion in an energy system optimization model. Technological Forecasting and Social Change 103:350-363, 2015. 10.1016/j.techfore.2015.06.001

Previously, we tried to import the package to get the version number in setup.py. This broke our conda build. It appears that versioneer can fix this. See, e.g., xarray.

When loading ixmp.Platform() there is an error migrating database:

import os
import pandas as pd
import ixmp as ix
import message_ix

dbprops = "ixmp.properties"

mp = ix.Platform() #dbtype='HSQLDB'

INFO:root:launching ixmp.Platform using config file at 'C:\Users\alexk\Documents\GitHub\ixmp\config\ixmp.properties'
Traceback (most recent call last):

File "", line 19, in
mp = ix.Platform() #dbtype='HSQLDB'

File "C:\Miniconda3\lib\site-packages\ixmp\core.py", line 82, in init
self._jobj = java.ixmp.Platform("Python", dbprops)

File "C:\Miniconda3\lib\site-packages\jpype_jclass.py", line 111, in _javaInit
*args)

org.flywaydb.core.api.FlywayExceptionPyRaisable: org.flywaydb.core.api.FlywayException: Validate failed: Detected resolved migration not applied to database: 1.2

We need a new section to cover "drivers", like population or gdp, which can have direct or indirect impacts on emissions. The GDP trajectory in particular should be linked to the variable returned from MACRO to ensure consistency.

It seems technology mapping sets, e.g., "map_tec_mode" and "map_tec_lifetime" don't get updated when the respective parameters get modified.
For example, I have had some changes in a scenario as follows:
1- removing technology A from all the parameters in one region, or
2- reducing the lifetime of technology B, resulting in less combination of vintage-active years in parameters "output" and "input" for that technology.

However, technology mapping sets are not being updated during the generation of input GDX file. As a result:

• In case 1, the model leads to an error during the GAMS compilation phase, stating that there is a problem with "technical_lifetime" of technology A.
• In case 2, the model uses the old vintage-active year mapping and solves, resulting in ACT in some years not desired with the new shortened lifetime of technology B.

I hope I'm not missing something out.

The idea here is to easily rename functional components of a model. It should go something like

for item in scenario.set_list():
    value = scenario.set(item)
    if old_tech in value:
        scenario.add_set(item, new_tech)

for item in scenario.par_list():
    df = scenario.par(item)
    if 'technology' in df and old_tech in df['technology']:
        df = df[df['technology'] == old_tech]
        df['technology'] = new_tech
        scenario.add_par(item, df)

scenario.rm_set('technology', old_tech)

It should be possible to “report” or “post-process” a message_ix.Scenario (given a sufficient amount of configuration metadata) to generate output (IAMC-compliant pd.DataFrame or file) that can be directly submitted to IIASA databases for either single- or multi-model assessments.

Tasks

• Identify requirements — done as of 2019-02-20, see https://github.com/iiasa/message_ix/wiki/Reporting.
• Develop enhancement/strategy proposal (similar to PEPs or NEPs) —done as of 2019-03-01. In two stages (see wiki page for further details):
  1. For version 1.2.0 (April 2019): pyam-based message_ix.Reporting class for early release.
  2. For version 1.3 or 2.0 (later): dask/graph-based architecture using pyam, with deprecation.
• Implementation & examples:
  • #142 —led by @danielhuppmann, includes tutorial updates & examples.
  • iiasa/ixmp#150 and #206.

specifically should overrule cloning from ixmp to return a message_ix.Scenario

It seems like in message_ix, scen is used as the kwarg while in ixmp, it is scenario. We should harmonize those two.

We should add some documentation on how to change GAMS and solver options. Some of these are currently "hard-coded" in the MESSAGEix code.

If we set fixed_new_capacity to a certain value, the model reacts by setting the CAP_NEW parameter to the desired value. The CAP however seems to be unaffected by this, thus, remains zero. Any ideas on why this is happening / is this the desired behaviour?

It has to do with trying to write the json file. Will add more details asap.

tried to download the tutorials from command line using
messageix-dl --local_path c:/github/msgix_tutorials

Files downloaded correctly to the folder, but threw this error.

INFO:root:Retrieving https://github.com/iiasa/message_ix/archive/master.zip
INFO:root:Unzipping C:\Temp\dvile4p7\master.zip to C:\Temp\dvile4p7
INFO:root:Copying C:\Temp\dvile4p7/message_ix-master/tutorial to c:/github/msgix_tutorials/tutorial
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\README.md (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria.png (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_load_scenario.ipynb (overwrite is
 ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_load_scenario_R.ipynb (overwrite
is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_multiple_policies-answers.ipynb (
overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_multiple_policies.ipynb (overwrit
e is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_reticulate.ipynb (overwrite is ON
)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_single_policy.ipynb (overwrite is
 ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\tools.py (overwrite is ON)
Traceback (most recent call last):
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 101, in do_dl
    shutil.rmtree(tmp)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Temp
\\dvile4p7\\master.zip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\Scripts\messageix-dl-script.py", line 10, in <module>
    sys.exit(dl())
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 119, in dl
    local_path=args.local_path)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 103, in do_dl
    shutil.rmtree(tmp)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Temp
\\dvile4p7\\master.zip'
PS H:\> messageix-dl --local_path c:/github/msgix_tutorials
INFO:root:Retrieving https://github.com/iiasa/message_ix/archive/master.zip
INFO:root:Unzipping C:\Temp\2net5aru\master.zip to C:\Temp\2net5aru
INFO:root:Copying C:\Temp\2net5aru/message_ix-master/tutorial to c:/github/msgix_tutorials/tutorial
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\README.md (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria.png (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_load_scenario.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_load_scenario_R.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_multiple_policies-answers.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_multiple_policies.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_reticulate.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\austria_single_policy.ipynb (overwrite is ON)
INFO:root:Writing to c:/github/msgix_tutorials/tutorial\Austrian_energy_system\tools.py (overwrite is ON)
Traceback (most recent call last):
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 101, in do_dl
    shutil.rmtree(tmp)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Temp\\2net5aru\\master.zi
p'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\Scripts\messageix-dl-script.py", line 10, in <module>
    sys.exit(dl())
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 119, in dl
    local_path=args.local_path)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\message_ix\cli.py", line 103, in do_dl
    shutil.rmtree(tmp)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Temp\\2net5aru\\master.zi
p'

The tutorial and docs should explain how to deal with different types of GAMS errors, like missing/invalid license or execution errors (infeasibility).

The debugging features also need to be explained.

This will require a few concrete steps:

• move tutorials to doc folder
• render tutorials when building docs with their own dedicated page, a la pyam
• make a new install target which installs gams
• remove explicit gams install on CI and use this target
• confirm everything works as expected on RTD

My one concern is how our CI examines its OS state (through $TRAVIS_OS_NAME. But perhaps we can make this new install target only available on linux to reduce headaches.

@khaeru anything else I'm missing?

Ok, I am a beginner here, and it is also likely that I didn't do it right.

I installed MESSAGEix through Anaconda and got the tutorial through messageix-dl. Then, when I ran the whole thing without any changes, I see this at the line "scenario.solve()":

TypeError Traceback (most recent call last)
in ()
----> 1 scenario.solve()

TypeError: solve() missing 1 required positional argument: 'model'

Is it supposed to run ok? Following Matt's suggestion, when I do scenario.solve('MESSAGE'), it runs fine.

@Jihoon (in #62) and @ClaraLuisa (in #63) separately ran into this issue. The basic problem is that messageix-dl will download the current version of tutorials, which are designed to be used with the current version of the master branch. However, when installing from anaconda (or otherwise), they should download the version of the tutorials aligned with the version installed.

This will likely require also #29

I was building my model based on CD_Links_SSP2 | baseline. I see it has the parameter 'renewable_tec' as empty in the input GDX (version 18). But when I run

Sc_hydro.add_set("renewable_tec", allhydrotech),

it complains:

at.ac.iiasa.ixmp.exceptions.IxException: No Set 'renewable_tec' exists in this Scenario!

So I did instead

Sc_hydro.init_set("renewable_tec")
Sc_hydro.add_set("renewable_tec", allhydrotech)

It runs ok, but then I get another error when I try to solve the model:

com.gams.api.GAMSExceptionPyRaisable: com.gams.api.GAMSException: Symbol renewable_tec exists already

I was able to expand other sets like 'commodity', 'technology', etc. without errors, but only this renewable_tec doesn't work.

When explicitily using the keyword arguments model/scenario, and only when the model name is kept the same as in the scenario to be cloned (scenario: base - in the example below), then the clone command makes a a copy and assigns this the base scenario name "base" scenario instead of making a copy and assigning the copied scenario the "new scenario name".

import message_ix
import ixmp
mp = ixmp.Platform(dbtype='HSQLDB')
base = message_ix.Scenario(mp, 'Westeros Electrified', 'baseline', cache=True)

__
2018-11-26 10:33:35,663 INFO at.ac.iiasa.ixmp.objects.Scenario:215 - loading default Scenario 'Westeros Electrified|baseline' (version: 7, runid: 86)...
2018-11-26 10:33:35,744 INFO at.ac.iiasa.ixmp.objects.Scenario:241 - done loading Scenario from the database!
2018-11-26 10:33:35,744 INFO at.ac.iiasa.ixmp.objects.MsgScenario:457 - spatial and temporal disaggregation maps assignment
2018-11-26 10:33:35,760 INFO at.ac.iiasa.ixmp.objects.MsgScenario:573 - period mapping assignment
2018-11-26 10:33:35,760 INFO at.ac.iiasa.ixmp.objects.MsgScenario:599 - duration period assignment
2018-11-26 10:33:35,760 INFO at.ac.iiasa.ixmp.objects.Scenario:1826 - committing changes of Scenario 'Westeros Electrified|baseline' to the database (runid: 86)...
2018-11-26 10:33:35,856 INFO at.ac.iiasa.ixmp.objects.Scenario:601 - done updating Scenario 'Westeros Electrified|baseline' to the database (runid: 86, version: 7)!
2018-11-26 10:33:35,856 INFO at.ac.iiasa.ixmp.objects.MsgScenario:457 - spatial and temporal disaggregation maps assignment
2018-11-26 10:33:35,872 INFO at.ac.iiasa.ixmp.objects.MsgScenario:573 - period mapping assignment
2018-11-26 10:33:35,872 INFO at.ac.iiasa.ixmp.objects.MsgScenario:599 - duration period assignment
__

With the explicit use of keyword arguments yields the following
scen = base.clone(model='Westeros Electrified', scenario='baseline_1',)
--> the base scenario is cloned and assigned a new version
__
2018-11-26 10:33:35,948 INFO at.ac.iiasa.ixmp.objects.Scenario:215 - loading default Scenario 'Westeros Electrified|baseline' (version: 0, runid: 86)...
2018-11-26 10:33:35,963 INFO at.ac.iiasa.ixmp.objects.Scenario:241 - done loading Scenario from the database!
2018-11-26 10:33:35,963 INFO at.ac.iiasa.ixmp.objects.Scenario:1896 - saving Scenario 'Westeros Electrified|baseline' to database (runid: 111)...
2018-11-26 10:33:36,087 INFO at.ac.iiasa.ixmp.objects.Scenario:604 - done saving Scenario 'Westeros Electrified|baseline' to the database (runid: 111, version: 10)
__

withOUT the explicit use of keyword arguments yields the following
scen = base.clone('Westeros Electrified', 'baseline_2',)
--> the base scenario is cloned with a new scenario name and assigned a new version
__
2018-11-26 10:33:36,109 INFO at.ac.iiasa.ixmp.objects.Scenario:215 - loading default Scenario 'Westeros Electrified|baseline_2' (version: 0, runid: 86)...
2018-11-26 10:33:36,125 INFO at.ac.iiasa.ixmp.objects.Scenario:241 - done loading Scenario from the database!
2018-11-26 10:33:36,125 INFO at.ac.iiasa.ixmp.objects.Scenario:1896 - saving Scenario 'Westeros Electrified|baseline_2' to database (runid: 112)...
2018-11-26 10:33:36,217 INFO at.ac.iiasa.ixmp.objects.Scenario:604 - done saving Scenario 'Westeros Electrified|baseline_2' to the database (runid: 112, version: 4)!
__