merge_simfiles not working again about epimodelhpc HOT 9 CLOSED

Sorry for the delay, I finally found where the issue comes from!

Here in the merge_simfiles function, the call to merge has keep.other = FALSE.

When merging the first and second file, the temp sublist is dropped. So when loading the 3rd one, there is a temp sublist in it but not in the previously merged ones and merge complains that the structure is different.

this PR fixes the issue by adding the keep.other argument.

merge_simfiles(1, ftype = "max", keep.other = TRUE)

This manages to combine all simulations without errors

from epimodelhpc.

@AdrienLeGuillou : can you take a look at this when you get a chance? I am also surprised the param.error = FALSE flag in merge.netsim does not overlook this potential parameter mismatch.

from epimodelhpc.

The param.error = FALSE flag only check for the params. However, the message

x and y have different structure

occurs with this check:

  if (length(x) != length(y) || !identical(names(x), names(y))) {
    stop("x and y have different structure")
}

@sgoodreau, could you give me the output of

names(netsim_object)

for each of the 5 files you are trying to merge?

from epimodelhpc.

Oops, I saw the GitHub messages for @clchand23's similar issue, and mistakenly thought this one was resolved, so didn't follow through on this. Sorry, @AdrienLeGuillou!

Looks like all five are the same (see below). I'd share the objects, but they're each 1.2 Gb. I could upload to a Google Drive if you'd like, or elsewhere.

Thanks!!!

names(sims[[1334]][[1]])
[1] "param" "control" "nwparam" "last_timestep" "epi"
[6] "el.cuml" "_last_unique_id" "attr.history" "raw.records" "stats"
[11] "attr" "temp" "el"

names(sims[[1334]][[2]])
[1] "param" "control" "nwparam" "last_timestep" "epi"
[6] "el.cuml" "_last_unique_id" "attr.history" "raw.records" "stats"
[11] "attr" "temp" "el"

names(sims[[1334]][[3]])
[1] "param" "control" "nwparam" "last_timestep" "epi"
[6] "el.cuml" "_last_unique_id" "attr.history" "raw.records" "stats"
[11] "attr" "temp" "el"

names(sims[[1334]][[4]])
[1] "param" "control" "nwparam" "last_timestep" "epi"
[6] "el.cuml" "_last_unique_id" "attr.history" "raw.records" "stats"
[11] "attr" "temp" "el"

names(sims[[1334]][[5]])
[1] "param" "control" "nwparam" "last_timestep" "epi"
[6] "el.cuml" "_last_unique_id" "attr.history" "raw.records" "stats"
[11] "attr" "temp" "el"

from epimodelhpc.

@sgoodreau, no worries.

I am at a loss here. I think I would need the files to experiment. I am not using merge_simfiles personally so I have a hard time zeroing on where the error could come from.

If you could send me a link to download them I would experiment with them a bit and hopefully find were the error lies.

from epimodelhpc.

@AdrienLeGuillou I put them in the /gscratch/csde directory on Hyak -- I believe both of us should be able to access that, and that seemed easiest given their large size. I'm imaging you'll be playing around with them on Hyak or Union/Libra/Nori anyway. LMK if you have trouble or want me to download them somewhere else.

from epimodelhpc.

That will be perfect I think. I check that ASAP

from epimodelhpc.

Thank you!!!

from epimodelhpc.

#42 merged into main

from epimodelhpc.