Sensitivity Analysis of Ranking Results about aimss HOT 11 CLOSED

@smiths thanks for the explanations :)

from aimss.

Hi @smiths , I have created the requested sensitivity analysis in a46f9f6.
The grades are randomly adjusted by a float between -1 and 1.

Prior to the sensitivity analysis the top 10 LBM were, in order:

ESPResSo
Ludwig
Palabos
OpenLB
LUMA
pylbm
waLBerla
DL_MESO (LBE)
TCLB
Sailfish

With the sensitivity analysis the order is now:

Ludwig
ESPResSo
Palabos
LUMA
waLBerla
OpenLB
pylbm
DL_MESO (LBE)
Sailfish
lettuce

Please confirm that this issue can be closed.

from aimss.

Thank you @peter-michalski. The results look good. It looks like 9 of the top 10 are the same between the two lists. I think the only difference is TCLB in the first list in position 9 has been replaced by lettuce in position 10. This build confidence that the grading isn't too sensitive to errors. The top 7 are common between the two lists.

How difficult would it be to automate the steps you took? Ideally we would repeat this analysis for each project we measure, to build confidence that our top software projects are legitimate choices.

from aimss.

Hi @smiths,

I had updated the AHP README on how to run the code. Running it analyzes all of the projects (software packages) within a domain. If we need to change the code for a different domain then the steps are also outlined in the aforementioned README file.
At this point there are only a few steps required for such changes and they should only take a few minutes.

from aimss.

Great! We can make this sensitivity analysis part of our process.

from aimss.

@smiths, I have rerun the sensitivity analysis with a random number range of -2 to 2. The top ten results are:

ESPResSo
OpenLB
Ludwig
Palabos
LUMA
DL_MESO (LBE)
pylbm
TCLB
ESPResSo++
lettuce

We can see 8 of these in the original top 10.

from aimss.

Great! I think we are definitely safe with the selection of the top 8 packages. 😄

from aimss.

@smiths can we close this issue?

from aimss.

Yes.

from aimss.

@smiths Here is a risk of using the Sensitivity Analysis (SA).

Figure 1 shows the scores without SA. Please check the first 4 scores: BioImage Suite Web has the same score as INVESALIUS 3 and Slice:Drop.

Figure 2 shows the scores after applying SA with only [-0.5,0.5]. Now BioImage Suite Web has a much higher score than the other two.

Now, when writing a report, it's hard to explain why this is so.

Since SA generates the values randomly in [-0.5,0.5], it doesn't always cause the problem for all qualities. However, it almost always causes a problem somewhere.

from aimss.

@Ao99, what you show isn't a problem. Our goal with the sensitivity analysis is to show that the order (the ranking) remains unchanged, or only slightly changed. The absolute values of the measures don't matter. Those numbers themselves are fairly meaningless. What matters is that our relative ranking remains close to the same.

from aimss.