sbmlteam / sbml-test-suite Goto Github PK

Results wrong.

Reported by: sarahkeating

The settings file for case 1281 specifies "p4", but the SBML file does not contain a p4.

Reported by Chris Myers direct to me.
"I'm still not matching on 00875 even when I decrease the time step
considerably, so I'm wondering about your results on this one. "

Reported by: sarahkeating

Sarah pointed out that in the Preferences panel, the option for overriding the number of threads at the bottom of the panel should be grayed out if the currently-selected wrapper does not allow multiple threads in the first place.

This is going to be a bit awkward to implement because of the separate objects used to implement the wrapper panel and the rest of the preferences panel.

Request by Ralph Gauges:

The testrunner could remember the last binary used.

Reported by: sarahkeating

Even though it is not the best idea converting ASTNode2 to ASTNode with FormulaParserLL3 we thought it would be working.
But the formula (A xor B) causes the following exception:
Caused by: org.sbml.jsbml.text.parser.ParseException: Encountered " "xor "" at line 1, column 3.
Was expecting one of:

"+" ...
"^" ...
"-" ...
"*" ...
"/" ...
"%" ...
...
<BOOLEAN_LOGIC> ...
...

Reported by Ralph Gauges:

He tried to not include Events in his selection but got models with events.

Reported by: sarahkeating

In Model 955, parameter P26 is provided as time^time which is an indeterminate expression and not 1 (as the reference result seem to indicate)

Reported by: *anonymous

I am currently working with test #41. I am simulating the system using CVODES using the provided tolerances. For all but time-points 24 and 47 the relative and absolute errors I observer are smaller than 100x(specified tolerances). I believe that the reason for the discrepancy is that the time-points at which the simulation results are provided are quite close the the timepoints at which the events fire. This results in a strong instability of the output at points 24 and 47. I would recommend to change the model such that there are no time-points close to event occurences.

Reported by: ffroehlich

When the tests are running in the testrunner the top panel output says

Passed: XXX/900 etc ...

- even if some models have been excluded.

This gives the impression that you passed less than you thought you would.
The '900' should be replaced with the number of tests you are going to run.

Reported by: sarahkeating

Any models using concentration with non-unity compartments where a rate rule is used to specify the rate of change of a species may have incorrect results.

These are models: 709, 710, 711, 686, 715, 716, 740, 741, 742

and possibly: 712, 713, 714 (use initial amt but with non-unit compartment)

Reported by: sarahkeating

From Chris Myers:

I've discovered what appears to be another problem in examples 310 to
318. In these examples, there is a rule:

compartment = p1 * S1

where S1 is a species with HasOnlySubstanceUnits equal to false.
Therefore, S1 in the equation above should be a concentration.
However, S1 is given a value of initialAmount of 1.5. Which means
that S1 above should be 1.5 divided by the size of the compartment
(initially 0.15), so 1.5 / 0.15 or 10 which times p1 (0.1) yields 1
for the initial compartment size. Then, S1 is not 10 but actually
1.5, which means the compartment size is 0.15. This process repeats,
and the compartment size cannot be set consistently.

These need to be changed :-)

Reported by: sarahkeating

The parameter P21 (arccot(-0.1)) must be 1,670 and not -1,47.

Reported by: rolandkel

Lucian has created a bunch of test cases for the SBML distrib package. These should be incorporated into the main test suite.

Steps include at least the following:

• incorporate the files from Lucian's repo
• rewrite the README file in the stochastic subdirectory because the contents will no longer be solely from the DSMTS
• add instructions for using the new cases
• create a new test cases release

The experimental data do not seem to fit the model. So in the model the initial amount of the species x is 3, whereas in the experimental data x starts with the value 0.

Reported by: rolandkel

This repo lacks a code of conduct.

Reported by Frank Bergmann and Henning Schmidt

Comparison code reports time step mismatches because the output contained
time points displayed as 0.8999999999999999999999999999 when the comparison code expects
0.9.

Reported by: sarahkeating

In test 942 the simulation program is supposed to output species x, but the settings file does not specify whether the amount or the concentration is requested.

Reported by: *anonymous

The presence of the DSMTS documentation file has confused people. We should remove it, and put information about the test cases (and how to use them) into the readme file in the cases/stochastic directory.

I believe there currently is no tests that test behavior for species that have no initial conditions (as in no initialAmount/initialConcentration/initialAssignment/Assignment). I am not sure whether such models are well defined in sbml standards.

sbml l3v2 states:

Missing initialAmount and initialConcentration values implies that their values either are unknown, or to be obtained from an external source, or determined by an initial assignment (Section 4.8 on p. 55) or other SBML construct elsewhere in the model.

This suggests that species without initial conditions might not be valid SBML, but libSBML (5.17.0) does not seem to complain about it. Libroadrunner seems to default these initial conditions to a numeric value of 0.0, which seems intuitive but I could not find anything that supports this behaviour in the documentation.

The window brought up by ⌘-i does not have a scrollbar.

Settings file says simulation time is 8.0 but results file only has data up to 7.5

Reported by: sarahkeating

It would be cool if the testrunner had a browse button for locating the binary.

Reported by: sarahkeating

Reported by Chris Myers direct to me

Reported by: sarahkeating

settings.txt:

steps: 100

results.csv:

time
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1

There may be new syntactic test cases that never got added to cases/syntactic.

The result files of the test cases (XXXXX-results.csv) have an inconsistent header line (in release sbml-test-cases-2010-10-06.zip). Some test cases denote the species values with a prefix "caseXXXXX`" and others don't.

For example the header lines of the first 20 test cases are:
00001: time,S1,S2
00002: time,S1,S2
00003: time,S1,S2
00004: time,S1,S2
00005: time,S1,S2
00006: time,S1,S2
00007: time,S1,S2
00008: time,S1,S2
00009: time,S1,S2
00010: time,case00010S1,case00010S2,case00010S3 00011: time,S1,S2,S3 00012: time,S1,S2,S3 00013: time,S1,S2,S3 00014: time,case00014S1,case00014S2,case00014S3
00015: time,case00015S1,case00015S2,case00015S3,case00015S4
00016: time,case00016S1,case00016S2,case00016S3,case00016S4
00017: time,case00017S1,case00017S2,case00017S3,case00017S4
00018: time,S1,S2,S3,S4
00019: time,case00019S1,case00019S2,case00019S3,case00019S4
00020: time,case00020S1,case00020S2,case00020S3,case00020S4

It would help a lot parsing these files, if one kind of referring to species would be used for all test cases.

Reported by: *anonymous

From Frank Bergmann

Today I came across a possible bug in RoadRunner, that has been slumbering there for eternities. So I was wondering whether a test case would have helped me (and perhaps a gazillion other SBML simulator developers). Do you currently have a test in the test suite that tests: hOSU together with non unit compartments and raterules on a non-constant boundary species? The idea would be to have to tests: one with hOSU true and one with it being false.

back when I implemented raterules I must have misread the description in the spec:

In the case of a species, a RateRule sets the rate of change of the species’ quantity to the value
determined by the formula in math.

to mean substance/time and thus came to different results!

Reported by: sarahkeating

Hello,

In a software I develop (which is tested using the semantic test cases of this testuite) I recently encountered problems with a SBML model which had multiple entries referring to the same species in the ListOfReactants of a Reaction. As I can imagine that others could run into similar problems with such models, I thought it might be worthwhile to include a corresponding test case.

If desired, I would contribute a minimal example where this occurs.

Best,
Fabian

Unfortunately, I won't have much time before end of April to play with adopting RoadRunner to fix remaining problems with L3V1, however you could make my live easier by providing all test cases. There is a bunch of files that pass right now for no better reason, that the L3V1 SBML file is not there:

Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00068\00068-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00069\00069-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00070\00070-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00129\00129-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00130\00130-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00131\00131-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00388\00388-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00391\00391-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00394\00394-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00445\00445-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00448\00448-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00451\00451-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00516\00516-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00517\00517-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00518\00518-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00519\00519-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00520\00520-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00521\00521-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00561\00561-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00562\00562-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00563\00563-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00564\00564-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00609\00609-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00610\00610-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00725\00725-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00726\00726-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00727\00727-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00728\00728-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00729\00729-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00730\00730-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00731\00731-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00827\00827-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00828\00828-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00829\00829-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00898\00898-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00899\00899-sbml-l3v1.xml'.
Error initializing the wrapper: Could not find file 'C:\Users\fbergmann.sbmltestrunner\2008-08-24\cases\semantic\00900\00900-sbml-l3v1.xml'.

P.S: don't mind the folder location ... I've so far tried my best to have the SBML TestRunner UI working for me ... and it seems to like this location.

Reported by: fbergmann

The test case uses symbols kr / kf but never defines them as parameters / local parameters

Reported by: fbergmann

A time point is quoted as 0.014999999999999999

Reported by: sarahkeating

The stp size for this case is 0.8 but the data uses
... 2.32, 2.44, 2.48 ...

Reported by: sarahkeating

This repo lacks a CONTRIBUTING.md file.

Requested by Ralph Gauges:

The testrunner command line requires the user to fill in a level and version each time - there could be %l %v flags that get filled by the Level Version selection page.

Reported by: sarahkeating

Hello,

I am currently running my software against the semantic testcases. The software uses the matlab bindings from libSBML (version 5.13) to parse SBML models.

For the semantic test case 0112 (l3v1) libSBML encounters the following error:

Error using CheckAndConvert>CorrectFormula (line 237)
Bracket mismatch
Error in CheckAndConvert>SortLogicals (line 396)
    Replace = CorrectFormula(Find{i}, 'and(');

Error in CheckAndConvert (line 77)
Formula = SortLogicals(Formula);

The problem occurs when SortLogicals(Formula) with Formula = piecewise(1, and(), 0)
which probably originates from the initial assignment of "c". This formula ends up as to and(),0)) in CorrectFormula() and correctly fails.

To me this looks like malformed sbml (and() without arguments) but the error could also be with libSBML.

Reported by Henning Schmidt.

Basically the ODEs used to generate the results are not correct.

Reported by: sarahkeating

In 536 and 537 the order of the data in the results file does not match the order in the sbml file - in all others it does
should be consistent
Thanks to Chris Myers for finally testing the models with algebraicRules and reporting this

Reported by: sarahkeating

Models where an initial assignment is used to assign an initial concentration may have inaccurate results:

These are models: 676, 677, 678, 688, 787, 788

Thanks to Chris Myers for digging these up.

Reported by: sarahkeating

The screenshots on http://sbml.org/Software/SBML_Test_Suite/SBML_Test_Suite_Screenshots are out of date.

Hello,

I've run into another problem with the L3 models and the sbml test suite. There are a couple of models, like 709, 710 ... that I can pass for L2V4 but not for L3 ... when I was debugging it it would seem that no spatial dimension is specified on the compartment

In L2V4: <compartment id="C" name="C" size="0.5" units="volume"/>
means spatialDimensions = 3

in L3V1: <compartment id="C" name="C" size="0.5" units="volume" constant="true"/>
means spatialDimensions = not defined!

In any case ... that changes this result for me! ... I believe there is a whole round of those models around ... (or at least I hope so otherwise something else is wrong)

Cheers
Frank

Reported by: fbergmann

The data for P16 (arccot(-time)) is wrong.
arccot (x) != -arccot(-x)

Reported by: rolandkel

In the archive sbml-test-cases-2014-10-22.zip:
XML files of Multiple Compartments test cases 54-56, 110-112, 286-294 put all species in compartment1 , instead of putting them in separate compartments.
This probably doesn't affect numerical results, but it is confusing.

Reported by: ftasb1

Reported by Chris Myers direct to me:

By the way, when I reduce the time step, I can get a match on 00684,
but I still cannot get a match on 00664 (the one with algebraic
rules). I thought I would send you my results on this one as well to
check against yours. It could, of course, be some accuracy issues on
my end.

Reported by: sarahkeating

Values used in some models are 1e-15 - which means that hitting the required tolerance becomes machine dependant.

Reported by: sarahkeating

Results from VCell dont match

Reported by: sarahkeating

Several un-rounded time points

Reported by: sarahkeating

Model: extras/sample-models/sbml-l1v1/from-gepasi-export/linear5.xml

This model uses a parameter rule to assign value to a parameter that is in fact declared as local to the reaction. Also the formula for the rule uses the id of another parameter that is local.

Both parameters are assigned values when declared; which in l1v1 they had to be.

Difficult to know whether the model intends the rule to override these ie should they be global parameters?

Need to think about how to fix!

Reported by: sarahkeating

The parameter P21 (arccot(-0.1)) must be 1,670 and not -1,47.

Reported by: *anonymous