pik-piam / quitte Goto Github PK
View Code? Open in Web Editor NEWBits and pieces of code to use with quitte-style data frames
Bits and pieces of code to use with quitte-style data frames
@0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q There is a bug in calc_addVariable. Using only.new = F and setting variable to something else than "variable", might lead to additional unintended rows in the output dataframe.
Example:
test <- data.frame(
"efScen" = c("efCO2-00", "efCO2-00", "efCO2-05", "efCO2-05"),
"variable" = c("Emi|CO2|Land-Use Change", "PE|Biomass|Energy Crops",
"Emi|CO2|Land-Use Change", "PE|Biomass|Energy Crops"),
"unit" = c("Mt CO2/yr", "EJ/yr", "Mt CO2/yr", "EJ/yr"),
"value" = c(1952.2740, 383.4084, 1443.3521, 330.4681)
)
test_add_var <- test %>%
calc_addVariable("efCO2-diff" = "`efCO2-00` - `efCO2-05`",
variable = efScen)
I would expect the following:
> test
efScen variable unit value
1 efCO2-00 Emi|CO2|Land-Use Change Mt CO2/yr 1952.2740
2 efCO2-00 PE|Biomass|Energy Crops EJ/yr 383.4084
3 efCO2-05 Emi|CO2|Land-Use Change Mt CO2/yr 1443.3521
4 efCO2-05 PE|Biomass|Energy Crops EJ/yr 330.4681
> test_add_var
efScen variable unit value
1 efCO2-00 Emi|CO2|Land-Use Change Mt CO2/yr 1952.2740
4 efCO2-00 PE|Biomass|Energy Crops EJ/yr 383.4084
5 efCO2-05 Emi|CO2|Land-Use Change Mt CO2/yr 1443.3521
8 efCO2-05 PE|Biomass|Energy Crops EJ/yr 330.4681
9 efCO2-diff Emi|CO2|Land-Use Change <NA> 508.9219
10 efCO2-diff PE|Biomass|Energy Crops <NA> 52.9403
But this is what happens:
> test_add_var
efScen variable unit value
1 efCO2-00 Emi|CO2|Land-Use Change Mt CO2/yr 1952.2740
2 efCO2-00 Emi|CO2|Land-Use Change EJ/yr 1952.2740
3 efCO2-00 PE|Biomass|Energy Crops Mt CO2/yr 383.4084
4 efCO2-00 PE|Biomass|Energy Crops EJ/yr 383.4084
5 efCO2-05 Emi|CO2|Land-Use Change Mt CO2/yr 1443.3521
6 efCO2-05 Emi|CO2|Land-Use Change EJ/yr 1443.3521
7 efCO2-05 PE|Biomass|Energy Crops Mt CO2/yr 330.4681
8 efCO2-05 PE|Biomass|Energy Crops EJ/yr 330.4681
9 efCO2-diff Emi|CO2|Land-Use Change <NA> 508.9219
10 efCO2-diff PE|Biomass|Energy Crops <NA> 52.9403
That is, there are additional rows that did not exist before, where "Emi|CO2|Land-Use Change" is combined with "EJ/yr" and "PE|Biomass|Energy Crops" with "Mt CO2/yr". The value is equal to the row with the respective entry in "variable". If only.new = T, this does not happen.
magclass::write.report
supports appending to an existing .mif
-style output file. It would be nice to have this functionality also in write.mif
.
We have collected a list of minor recommendation for changes but none of these are important:
- Move loading of exemplary data further up and make it more clear how to read in exemplary data from quitte (i.e. write "data <- quitte_example_data")
- Always put explanation in front of code for function, e.g.
- inline.data.frame()
- order.levels()
- Add explanation „By settting only.new = TRUE“ to the text „It is also possible to only keep the newly calculated values for further manipulation.“
- Format the line „(Note: You don't have to call the mip-functions via the double colon (::) operator. This is only needed in this vignette, since quitte can't import mip.)“ as a Tip
One thing we noticed for the interpolation, is that the results are same for linear and spline interpolation. Is this how it is supposed to be?
Currently read.quitte("scenario.mif")
throws a (non-helpful) warning if the mif file contains Inf
or -Inf
as values. This can be prevented by adding "Inf", "-Inf"
to the argument na.strings
, e.g., read.quitte("scenario.mif", na.strings = c("UNDF", "NA", "N/A", "n_a", "Inf", "-Inf"))
. It would be nice if either this would be the default or if "Inf"
would be correctly parsed as the value Inf
.
It seem with the latest version of the tibble package (tibble_3.0.0) the quitte function character.data.frame()
returns malformed column names. With an older tibble version (tibble_2.1.1) it worked on my local machine. The issue occured after updating tibble on my local machine.
The result of:
character.data.frame(as.quitte(read.report("REMIND_generic_SSP2-Base.mif",as.list = FALSE)))
Result with tibble_2.1.1:
#A tibble: 356,174 x 7
model scenario region variable unit period value
<chr> <chr> <chr> <chr> <chr> <int> <dbl>
1 REMIND SSP2-Base CAZ Consumption billion US$2005/yr 2005 1122.
2 REMIND SSP2-Base CHA Consumption billion US$2005/yr 2005 1604.
3 REMIND SSP2-Base EUR Consumption billion US$2005/yr 2005 8753.
4 REMIND SSP2-Base IND Consumption billion US$2005/yr 2005 517.
5 REMIND SSP2-Base JPN Consumption billion US$2005/yr 2005 3032.
6 REMIND SSP2-Base LAM Consumption billion US$2005/yr 2005 1867.
7 REMIND SSP2-Base MEA Consumption billion US$2005/yr 2005 996.
8 REMIND SSP2-Base NEU Consumption billion US$2005/yr 2005 943.
9 REMIND SSP2-Base OAS Consumption billion US$2005/yr 2005 1300.
10 REMIND SSP2-Base REF Consumption billion US$2005/yr 2005 570.
#... with 356,164 more rows
Result with tibble_3.0.0:
# A tibble: 356,174 x 7
model[,"model"] [,"scenario"] [,"region"] [,"variable"] [,"unit"] scenario[,"mode~ [,"scenario"] [,"region"] [,"variable"] [,"unit"] region[,"model"]
<chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
1 REMIND SSP2-Base CAZ Consumption billion ~ REMIND SSP2-Base CAZ Consumption billion ~ REMIND
2 REMIND SSP2-Base CHA Consumption billion ~ REMIND SSP2-Base CHA Consumption billion ~ REMIND
3 REMIND SSP2-Base EUR Consumption billion ~ REMIND SSP2-Base EUR Consumption billion ~ REMIND
4 REMIND SSP2-Base IND Consumption billion ~ REMIND SSP2-Base IND Consumption billion ~ REMIND
5 REMIND SSP2-Base JPN Consumption billion ~ REMIND SSP2-Base JPN Consumption billion ~ REMIND
6 REMIND SSP2-Base LAM Consumption billion ~ REMIND SSP2-Base LAM Consumption billion ~ REMIND
7 REMIND SSP2-Base MEA Consumption billion ~ REMIND SSP2-Base MEA Consumption billion ~ REMIND
8 REMIND SSP2-Base NEU Consumption billion ~ REMIND SSP2-Base NEU Consumption billion ~ REMIND
9 REMIND SSP2-Base OAS Consumption billion ~ REMIND SSP2-Base OAS Consumption billion ~ REMIND
10 REMIND SSP2-Base REF Consumption billion ~ REMIND SSP2-Base REF Consumption billion ~ REMIND
# ... with 356,164 more rows, and 16 more variables: [,"scenario"] <chr>, [,"region"] <chr>, [,"variable"] <chr>, [,"unit"] <chr>, variable[,"model"] <chr>,
# [,"scenario"] <chr>, [,"region"] <chr>, [,"variable"] <chr>, [,"unit"] <chr>, unit[,"model"] <chr>, [,"scenario"] <chr>, [,"region"] <chr>, [,"variable"] <chr>,
# [,"unit"] <chr>, period <int>, value <dbl>
structure(
list(
region = c("US", "US", "US", "US", "EU28", "EU28", "EU28", "EU28", "EU28", "EU28"),
iso3c = c("PRI", "PRI", "USA", "USA", "ALA", "ALA", "AND", "AND", "AUT", "AUT"),
name = c("EEK", "kap", "EEK", "kap", "EEK", "kap", "EEK", "kap", "EEK", "kap"),
value = c(7.28297787369661e-07, 1.4825575525539, 5.47830041372924e-05,
196.700540269622, 7.1732435863781e-09, 0.00494520584427543,
1.98196459756622e-08, 0.0099356972852833, 9.80993573948498e-07,
0.955472798121879)),
row.names = c(NA, -10L),
class = c("tbl_df", "tbl", "data.frame")
) %>%
sum_total(group = iso3c)
Would it be possible to have a function to read only a subset of variables from one or multiple mif/csv files into a quitte dataframe? To make it fast, the best way is probably to pre-filter the file(s) using command line tools behind the scenes, and then to read only the filtered temporary file? (as suggested in previous discussions on Mattermost)
Hi,
Since 0.3110.0, quitte is using forcats::fct_na_value_to_level
, which is only available in forcats since version 1.0.0 according to the changelog. If my analysis is correct, it should be enough to include this requirement in the DESCRIPTION.
With an earlier version of forcats, installing quitte fails:
> install.packages("quitte")
Installing package into ‘/home/pflueger/R/x86_64-pc-linux-gnu-library/4.2’
(as ‘lib’ is unspecified)
trying URL 'https://pik-piam.r-universe.dev/src/contrib/quitte_0.3110.0.tar.gz'
Content type 'application/x-gzip' length 989055 bytes (965 KB)
==================================================
downloaded 965 KB
* installing *source* package ‘quitte’ ...
** using staged installation
** R
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
Error:
! object ‘fct_na_value_to_level’ is not exported by 'namespace:forcats'
Backtrace:
▆
1. └─tools:::makeLazyLoading(...)
2. └─tools:::code2LazyLoadDB(...)
3. ├─base::suppressPackageStartupMessages(...)
4. │ └─base::withCallingHandlers(expr, packageStartupMessage = function(c) tryInvokeRestart("muffleMessage"))
5. └─base::loadNamespace(...)
6. └─base::namespaceImportFrom(...)
7. └─base::importIntoEnv(impenv, impnames, ns, impvars)
Execution halted
ERROR: lazy loading failed for package ‘quitte’
* removing ‘/home/pflueger/R/x86_64-pc-linux-gnu-library/4.2/quitte’
* restoring previous ‘/home/pflueger/R/x86_64-pc-linux-gnu-library/4.2/quitte’
The downloaded source packages are in
‘/tmp/RtmpF4Ccfq/downloaded_packages’
Warning message:
In install.packages("quitte") :
installation of package ‘quitte’ had non-zero exit status
Cheers,
Mika
interpolate_missing_periods {quitte} "periods" in docu but expects "period"
I found an erratic behavior of interpolate_missing_periods
when the data has periods before the interpolation period but is missing periods at the end of the interpolation period.
> df <- data.frame(period = 2000:2004,
+ variable = rep("a", 5),
+ value = sqrt(1:5))
> df
period variable value
1 2000 a 1.000000
2 2001 a 1.414214
3 2002 a 1.732051
4 2003 a 2.000000
5 2004 a 2.236068
> interpolate_missing_periods(df, 2002:2005)
# A tibble: 6 x 3
variable period value
<chr> <int> <dbl>
1 a 2002 1.73
2 a 2003 2
3 a 2004 2.24
4 a 2005 1.41
5 a 2000 1
6 a 2001 1.41
To my understanding, the function should not extrapolate until 2005, as expand.values
is FALSE
. But instead, it assumes missing periods at the end of the interpolation period as the final value of the periods before the interpolation period.
I am still struggling with calc_addVariable
if completeMissing == TRUE
(related to this comment). The function will fill much more "missing data combinations" with zero than I expected.
> data
model variable period value
1 m1 A 1 1
2 m1 A 2 2
3 m1 A 3 3
4 m1 B 1 4
5 m1 B 2 5
6 m2 A 1 6
7 m2 A 2 7
8 m2 B 1 8
9 m2 B 2 9
> calc_addVariable(data, C = "A + B", completeMissing = TRUE)
model variable period value
1 m1 A 1 1
2 m1 B 1 4
3 m1 C 1 5
4 m1 A 2 2
5 m1 B 2 5
6 m1 C 2 7
7 m1 A 3 3
8 m1 B 3 0
9 m1 C 3 3
10 m2 A 1 6
11 m2 B 1 8
12 m2 C 1 14
13 m2 A 2 7
14 m2 B 2 9
15 m2 C 2 16
16 m2 A 3 0
17 m2 B 3 0
18 m2 C 3 0
In the case above, I don't see the need to fill values for period 3 in model m2 because none of the variables A and B are defined for this combination. So I would expect the result without rows 16-18. Row 8 on the other hand should be filled in order to calculate row 9.
If the current behavior should remain, is it possible to introduce an reduced version of the argument like completeGaps
that would only fill missing variables for data combination that exist for at least one of the variable used in the calculation?
I might have a special use case, but in EDGE-Buildings we often have the scenario history with values for historic periods and the scenarios SSP* for projected periods. calc_addVariable
with completeMissing == TRUE
introduces many unwanted zeros.
I can imagine this to require a bit more effort. So if there is no time for this feature (or my suggestion makes no sense at all), this issues is just to flag the behaviour I described. Thanks in any case!
replace_column
works silently even if provided with an ambiguous mask that maps values to multiple other values (i.e. columns of mask contain duplicates). In all my applications so far, this would be a mistake and the result with more rows than data
was not what I intended calculate.
Do you think it makes sense to give a warning or even throw an error if this kind of mask
is not explicitly allowed via a new argument? Or are my use cases just a very limited fraction of what this function should be able to do?
bla <-
tibble(model = "M",
scenario = c("A","A","B"),
region = "R",
variable = c("X","Y","X"),
unit = "U",
period = 2020,
value = c(1,2,4)) %>%
as.quitte()
#works
bla %>%
calc_addVariable("Z" = "X + Y", units = "U",
na.rm = FALSE, completeMissing = FALSE)
# doesn't work
bla %>%
calc_addVariable("Z" = "X + Y", units = "U",
na.rm = FALSE, completeMissing = TRUE)
Error: Join columns must be present in data.
x Problem with `cols` and `fill`.
Occured after a package update, so presumably related to upstream changes in dplyr/tidyr/... .
Package versions:
dplyr_1.0.1
tidyr_1.1.1
quitte_0.3084.3
The function calc_addVariable
does not allow a new variable to have the same name as an existing one. Is this intended or a bug? I would actually like the function to overwrite the old variable by the newly calculated one. Unit conversions would be a classic application of this functionality.
> df <- tibble(period = rep(2000:2002, 2),
+ variable = rep(c("a", "b"), each = 3),
+ value = 1:6)
> calc_addVariable(df, a = "a")
Error in `mutate()`:
! Problem while computing `a = f_eval(f = .dots[[i]], data = .)`.
Caused by error:
! object 'a' not found
Run `rlang::last_error()` to see where the error occurred.
building piamInterfaces fails with the following
══ Warnings ════════════════════════════════════════════════════════════════════
1. test summationFile without errors using AR6 ('test-checkSummations.R:22') - `fct_explicit_na()` was deprecated in forcats 1.0.0.
ℹ Please use `fct_na_value_to_level()` instead.
ℹ The deprecated feature was likely used in the quitte package.
Please report the issue at <https://github.com/pik-piam/quitte/issues>.
══ DONE ════════════════════════════════════════════════════════════════════════
Before submission you need to take care of the following warnings:
test "test summationFile without errors using AR6": `fct_explicit_na()` was deprecated in forcats 1.0.0.
ℹ Please use `fct_na_value_to_level()` instead.
ℹ The deprecated feature was likely used in the quitte package.
Please report the issue at <https://github.com/pik-piam/quitte/issues>.
You can find solutions to common problems at https://github.com/pik-piam/discussions/discussions/18
Error in lucode2::verifyTests(...) : The package tests produced warnings.
Error in check(cran = cran, config = cfg) : lucode2::check failed
Hey Michaja,
so Silvia and I happened upon this little bug/anomaly:
When reading in a mif file with read.quitte()
, the returned object is not recognized by pivot_wider()
as a tibble. Even though is_tibble()
returns TRUE, you have to pass the quitte object through as_tibble()
, before pivot_wider()
accepts it.
Below is some code describing the problem. Specifically a warning message that is only displayed every 8 hours(?). Does the problem maybe have something to do with the use of the deprecated data_frame()
function?
So I was wondering if this is something you were aware of and if it needs fixing. I am glad to work on it, I just wanted to get your input on this.
Thanks a lot!
x<- read.quitte("output/default_2020-03-03_09.38.01/REMIND_generic_default.mif")
Warning message:
`data_frame()` is deprecated as of tibble 1.1.0.
Please use `tibble()` instead.
This warning is displayed once every 8 hours.
Call `lifecycle::last_warnings()` to see where this warning was generated.
x
A tibble: 356,649 x 7
model scenario region variable unit period value
<fct> <fct> <fct> <fct> <fct> <int> <dbl>
1 REMIND default CAZ Consumption billion US$2005/yr 2005 972.
2 REMIND default CHA Consumption billion US$2005/yr 2005 3493.
3 REMIND default EUR Consumption billion US$2005/yr 2005 7846.
4 REMIND default IND Consumption billion US$2005/yr 2005 898.
5 REMIND default JPN Consumption billion US$2005/yr 2005 2671.
6 REMIND default LAM Consumption billion US$2005/yr 2005 1710.
7 REMIND default MEA Consumption billion US$2005/yr 2005 1086.
8 REMIND default NEU Consumption billion US$2005/yr 2005 858.
9 REMIND default OAS Consumption billion US$2005/yr 2005 1402.
10 REMIND default REF Consumption billion US$2005/yr 2005 660.
… with 356,639 more rows
x %>% pivot_wider(names_from = variable)
Error: `x` must be a vector, not a `tbl_df/tbl/data.frame/quitte` object.
Run `rlang::last_error()` to see where the error occurred.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.