[perf] Why noDiff? about definitelytyped-tools HOT 3 CLOSED

Good question! Mainly because the values are already expressed in percentages, and expressing the percent difference between two percentages can be confusing/misleading—from dimensional analysis you can’t tell whether it’s a ratio or a simple arithmetic difference, and it’s somewhat unclear which would be the better measure of change. Secondly, since the CV only measures noise in the system, it’s not something the definition authors are responsible for, and a change in its value indicates nothing about the code change of the PR. A high value in either column should lead someone to view the other measurements in that column with a higher degree of skepticism. A significant difference between the CV before and after might lead you to believe that comparing other before and after values might be unreliable, but in fact, that comparison would be more reliable than if the CV before and after are both high. So paying attention to the difference at all is misleading, unless you are trying to debug the measurement infrastructure itself.

from definitelytyped-tools.

Thanks for the answer! Makes sense. Should it be added to quickInfoAvgCV as well then?

from definitelytyped-tools.

Oh, yes, good catch. Neither of those measurements should report a diff.

from definitelytyped-tools.