Comments (10)
@SongminYu, thanks for these two suggestions.
First, when opening an issue on this repository you should have seen an a template (this one) that asks you to give certain information—including "the output of message-ix show-versions
". We need this information—including which version(s) of the tutorials you are using—in order to understand the nature of the issue.
[In] The emission bound (cumulative) tutorial, it seems one step is missing: add category for year, so "type_year" has "cumulative".
The latest versions of the tutorials are automatically tested daily; see here. These tests all pass, which means that the code in each notebook cell runs without any error. When you say "one step is missing", what specifically happens without that step? Do you see an error message? If so, please paste the full text.
the following line is not clear enough … the users do not know the keys in the
key_or_data
.
This is a good suggestion. Although (a) this is a documented and supported usage of .add_par()
and (b) the dimensions of bound_emission
are documented, here —in general we try to use make_df()
in our tutorials. The reason is to be clear and explicit about the data that is being entered, as you suggest, without the reader needing to consult the documentation.
@behnam-zakeri @glatterf42 perhaps we can include a small change to this line into #815.
from message_ix.
Hi @khaeru, thanks for your fast reply. I will remember to use the template next time.
Regarding the tutorial error I saw, first I need to say that I am not running the Jupyter file but using PyCharm that I am more used to. I just pushed my code to GitHub and you can try it by running the main.py
in the westeros_tutorial
folder.
If you comment out line 91 in the main.py file and run it, you can see the error message:
ValueError: The index set 'type_year' does not have an element 'cumulative'!
The error was fixed after I added this line 91, setting the category (or type_year
) "cumulative" to "700, 710, 720".
# Line 91:
scenario.add_category(set_name="year", file_name="Category_Year")
The solved OBJ is 206280.146577.
The excel file Category_Year
and function add_category
are as shown below:
![Screenshot 2024-04-20 at 20 02 24](https://private-user-images.githubusercontent.com/16057131/324175918-4bb12df9-4e99-44bd-ade8-14ae839c674a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTU5NjA2NDYsIm5iZiI6MTcxNTk2MDM0NiwicGF0aCI6Ii8xNjA1NzEzMS8zMjQxNzU5MTgtNGJiMTJkZjktNGU5OS00NGJkLWFkZTgtMTRhZTgzOWM2NzRhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTE3VDE1MzkwNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVkZWExZmFiYjU3ZmUxMDZjNGQ4MGNmN2RkYjI1NzYyMzJjZDY0NjQyOWE5NTc1ZWVlMGQyNWFkYTIyOGI3MjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Yom9M3bgoxZDg1VgGJvgc277OcbQcCh7FiK3uv6o3Nw)
def add_category(self, set_name: str, file_name: str):
df = self.load_xlsx(file_name)
for _, row in df.iterrows():
self.scenario.add_cat(set_name, row["category"], row["name"])
Output of message-ix show-versions
<!--
ixmp: 3.8.0
message_ix: 0.0.0
message_ix_models: None
message_data: None
click: 8.1.7
dask: 2023.11.0
genno: installed
graphviz: None
jpype: 1.4.1
… JVM path: /Users/songminyu/miniconda3/envs/MESSAGE/lib/jli/libjli.dylib
openpyxl: 3.1.2
pandas: 2.1.3
pint: 0.22
xarray: 2023.10.1
yaml: 6.0.1
iam_units: installed
jupyter: installed
matplotlib: 3.8.1
plotnine: 0.12.4
pyam: 0.7.0
GAMS: 40.4.0
python: 3.10.13 (main, Sep 11 2023, 08:39:02) [Clang 14.0.6 ]
python-bits: 64
OS: Darwin
OS-release: 22.1.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: (None, 'UTF-8')
-->
from message_ix.
After replying above, I continued implementing the other two emission bound examples in the tutorial:
- emission_bound_year
- emission_bound_cumulative_tax
Now they are also in the repo. I noticed that,
First, in the emission_bound_year
example, I met another similar error:
ValueError: The index set 'type_year' does not have an element '700'!
This error can be solved by using either of the following two lines:
scenario.add_category(set_name="year", file_name="Category_Year_EmissionBoundYear") # Line 120 in main.py
scenario.add_set(set_name="type_year", file_name="Set_TypeYear") # Line 121 in main.py
The two excel files are as below:
Actually, the second line is used in the emission_bound_cumulative_tax
example in the tutorial. Adding type_year
info seems necessary in this example, but not mentioned in the other two emission bound examples.
Second, I tried running the baseline example again, and surprisingly found:
In the "cat_year" sheet generated (and exported) by the reporter, the type_year
info exists... But I didn't add such info by using any code. So, my guess is, that such information was somehow added to the scenario when solving it. In the tutorial of emission_bound
and emission_bound_year
, the scenario was cloned from baseline
, so the info was there. However, in my PyCharm workflow, the scenario was newly created so I need to add it by myself, or I will see the error message. Is this right?
![Screenshot 2024-04-20 at 23 38 25](https://private-user-images.githubusercontent.com/16057131/324187790-06723988-70e0-483f-9bf5-e02e2b0e5461.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTU5NjA2NDYsIm5iZiI6MTcxNTk2MDM0NiwicGF0aCI6Ii8xNjA1NzEzMS8zMjQxODc3OTAtMDY3MjM5ODgtNzBlMC00ODNmLTliZjUtZTAyZTJiMGU1NDYxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTE3VDE1MzkwNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJiYWQ5NWM1MTg4ZWQ0MjczNTI0ZDY5OWRmNTQxYTZjMWM1YjBiZGU5MGZkYmYwMzAxYTVmNmJiODUyMDE1NGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.gADv1QkenstrRRzCDcdnOrt-tP-HldGSQtv1cfYjvGU)
from message_ix.
Hi @SongminYu and thanks for your extensive report! There are a few things that jump out to me here:
- Your output of
message-ix show-versions
shows that you're using v3.8.0 of ixmp, but v0.0.0 of message_ix. This is not how it should be; with current versions, both packages should have an identical version. So in your case, we would expect message_ix to also have v3.8.0.
It looks like you installed via miniconda. Do you still have the output of theconda install
command or can you reproduce it? Maybe from that we'll get a hint which version of the tutorial files you tried to use. - The output also shows you're using v0.7.0 of pyam, which was released in 2020, so is severely outdated by now (they just released v2.2.2). This could be related to the reason why you're message_ix version is not 3.8.0.
- In the code you shared with us, you create a
Scenario
withScenario(config=config)
and then add acategory
for theset
namedyear
. However, the currentmessage_ix.Scenario
doesn't take aconfig
keyword, but it would be passed on toixmp.Scenario
. This doesn't use aconfig
keyword either, but passes it on to the GAMSModel class (most likely), which doesn't use it, either. So I'm not sure what's going on here. Based on the above, it seems like the values you set inconfig
would not be used at all, butScenario
s typically need at least amodel
name, which would be given with themodel
key, though, notmodel_name
. Also,Scenario
s are stored in so-calledPlatform
s, which provide access to underlying databases via ixmp. For example in the westeros_baseline tutorial, you would callscenario = message_ix.Scenario(mp, model="Westeros Electrified", scenario="baseline", version="new")
to create aScenario
(wheremp
is the name of aPlatform
object). Yourmain.py
is missingPlatform
s completely, so that seems severely outdated, too.
Finally, also from the westeros_baseline tutorial, here's how we currently addyear
s to aScenario
:
history = [690]
model_horizon = [700, 710, 720]
scenario.add_horizon(year=history + model_horizon, firstmodelyear=model_horizon[0])
The add_horizon()
function is a convenience function that automatically adds required category
s, too.
So I think the first thing we need to help you is to figure out which version of MESSAGE you're using. Some of the aspects that are not in your main.py
(like ixmp.Platform
) seem so fundamental to MESSAGEix that I'm beginning to wonder: are you perhaps still using MESSAGE-V?
from message_ix.
I'm beginning to wonder: are you perhaps still using MESSAGE-V?
Looking at the repo shared in the earlier comment, we see e.g. https://github.com/SongminYu/message_westeros/blob/main/message/scenario.py —so it seems the user has written their own code including classes with similar names to Scenario that somehow wrap the actual message_ix.Scenario
class.
@SongminYu, to be clear, we can provide support for the features of message_ix
per se, but your own code is your own code, and we obviously can't be responsible for errors you've made in writing it.
from message_ix.
the user has written their own code including classes with similar names to Scenario that somehow wrap the actual message_ix.Scenario class.
I see, that makes more sense :)
from message_ix.
Referring to our own westeros_baseline.ipynb, I modify the following cell:
scenario.solve()
To the following
from icecream import ic
# Prior to solve
ic(scenario.set("type_year"))
ic(scenario.set("cat_year"))
scenario.solve()
# After solve
ic(scenario.set("type_year"))
ic(scenario.set("cat_year"))
And see this output:
ic| scenario.set("type_year"): 0 firstmodelyear
1 lastmodelyear
2 initializeyear_macro
dtype: object
ic| scenario.set("cat_year"): type_year year
0 firstmodelyear 700
…
ic| scenario.set("type_year"): 0 firstmodelyear
1 lastmodelyear
2 initializeyear_macro
3 cumulative
4 700
5 710
6 720
dtype: object
ic| scenario.set("cat_year"): type_year year
0 firstmodelyear 700
1 cumulative 700
2 700 700
3 cumulative 710
4 710 710
5 cumulative 720
6 720 720
This tells me a few things:
- The elements in the
type_year
set named "cumulative" and "700" etc. (the latter corresponding to elements "700" etc. in the setyear
) seem to be magically created when solve() is called. - Looking more closely, this happens in the ixmp_source/JDBCBackend code. This violates stack separation: this behaviour (the existence of a
type_year
member named "cumulative") should be implemented withinmessage_ix
itself, and notixmp
or other backends (for instance, ixmp4). So this is indeed a bug that needs fixing—not in the tutorial, but within message_ix itself. - In @SongminYu's custom code, I suspect (haven't looked) that
Scenario.solve()
is not called before they try to.add_par()
with a reference to "cumulative". This is different from the message_ix tutorials and test suite, and that's why they encounter an error that we do not.
I will retitle this issue to cover the point in message_ix to be resolved.
@SongminYu, for your own code, I think the work-arounds already identified are fine:
- Call
.add_set()
manually. .solve()
the scenario before using type_year="cumulative".
from message_ix.
The fix should be to extend message_ix.models.MESSAGE.initialize()
:
message_ix/message_ix/models.py
Lines 304 to 327 in aabb789
from message_ix.
Hi @khaeru, hi @glatterf42, thanks for your reply! Yes, that's exactly how I suspect: the "cumulative" category info was added in the .solve()
step. So, when cloning baseline
from the database, this info was carried back into the emission_bound
model. That's why it works in the tutorial but not in my code. Thanks for the clarification!
from message_ix.
Hi @glatterf42, yes, I installed the packages via miniconda. It was a long time ago when I first want to play with MESSAGE but was interrupted by other work. I just checked the version problem and managed to update the pyam-iamc
package to V2.0.0.
Output of message-ix show-versions
<!--
ixmp: 3.8.0
message_ix: 0.0.0
message_ix_models: None
message_data: None
click: 8.1.7
dask: 2024.4.2
genno: installed
graphviz: None
jpype: 1.5.0
… JVM path: /Users/songminyu/miniconda3/envs/MESSAGE/lib/jvm/lib/libjli.dylib
openpyxl: 3.1.2
pandas: 2.2.2
pint: 0.23
xarray: 2024.3.0
yaml: 6.0.1
iam_units: installed
jupyter: installed
matplotlib: 3.6.2
plotnine: 0.13.4
pyam: 2.0.0
GAMS: 40.4.0
python: 3.10.13 (main, Sep 11 2023, 08:39:02) [Clang 14.0.6 ]
python-bits: 64
OS: Darwin
OS-release: 22.1.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: (None, 'UTF-8')
-->
But, the message_ix
version is still V0.0.0 printed here. However, it should be V3.8.0 as shown in the screenshot below. I am not sure why it is printed differently in the terminal. Anyway, it seems working fine... Thanks!
![Screenshot 2024-04-22 at 20 15 12](https://private-user-images.githubusercontent.com/16057131/324457342-32d02101-3ee0-4582-9d33-27f190d2b55b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTU5NjA2NDYsIm5iZiI6MTcxNTk2MDM0NiwicGF0aCI6Ii8xNjA1NzEzMS8zMjQ0NTczNDItMzJkMDIxMDEtM2VlMC00NTgyLTlkMzMtMjdmMTkwZDJiNTViLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTE3VDE1MzkwNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY4MjhlNmFkZmJkYWQ1YzkzZDRlYjM3YjE0NjUxNzgwY2E1Mjg0YTdlMmM0OWQ4MDA1ZGVlMGM2MjUyYTk4MzYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.CGq4rLyCf6nlj36qA0Ag-XVXeiF6l5nYs_MF1WAWYG8)
from message_ix.
Related Issues (20)
- Track/adjust for test failure with genno 1.23.0 HOT 5
- NoneType' object has no attribute 'replace HOT 8
- Make more tests robust & independent HOT 8
- How to represent power transfer between nodes HOT 3
- How to represent a three-node energy system with one-way electricity transmission between the nodes?
- Comply with pandas 3.0 CoW default
- Electricity grid has no capacity-related parameters in Westeros tutorials
- Units are not correct and consistent in Westeros tutorials
- How to clone two single-node models into multiple nodes without success HOT 1
- Mention Z shell specifics in install instructions HOT 1
- scenario.solve ModelError HOT 10
- Loading Scenarios from Platform gets very slow over time
- ModuleNotFoundError: No module named 'utils' HOT 2
- Accidentally openend issue HOT 1
- ModuleNotFoundError: No module named 'utils' HOT 21
- Add `.report.key`
- Plotting functionality should join `historical_activity` to `ACT` HOT 1
- Update install instructions with note about using conda and pip together HOT 2
- LP status (22): dual objective limit exceeded HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from message_ix.