Bug? Size zero patches possible about gin HOT 12 CLOSED

from gin.

Yes, that's intentional IIRC. Do you think it shouldn't? As it's local search, I was thinking it's useful to be able to backtrack... consider that execution times are noisy.

from gin.

I guess that's basically asking for a re-evaluation of the original unaltered source? I was seeing empty Patches taking the "best" run time when I suspect they shouldn't be - i.e. after some other patches had been found that did offer an improvement - I will check this in the morning.

from gin.

I think that's probably due to noisy execution times. This is a problem generally with non-functional optimisation. I think doing something rigorous here would be great, e.g. take a similar approach to "racing" algorithms as in irace2.

from gin.

Got it. The JUnit tests were failing, so all program variants were running in about 1ms (I thought that the run was a bit fast!)). Due to noise this meant that the fastest patch varied.

The reason the JUnit tests were failing was down to a copy+paste error in the example test class I wrote. I'll raise a separate issue for that.

btw - agree wrt racing for handling noise.

from gin.

Thanks for chiming in, Markus.

I can see a few ways of implementing a racing approach:

• Racing two (or n) patches across a set of test cases (this fits well with local search or tournament selection)

• Repeatedly sampling the same set of tests with two patches until we can determine a difference between them with a certain statistical confidence.

Do we want to go there right away?

Not necessarily, but I think we need to work out some way of solving the execution time comparison problem. Alternatives include counting the number of bytecodes executed, and using a model to estimate expected execution time, for instance.

Immediate question is - how is this solved in other software? I know that some work takes the 3rd quartile as a conservative estimate of the execution time, for example.

from gin.

from gin.

from gin.

from gin.

This is delving in to territory that I'm less familiar with but we ought to be careful with counting bytecodes because of JIT. The approach that Nathan Burles developed for measuring energy using bytecode tracing ran into issues when JIT was enabled - aside from producing stochastic results I guess it would probably also vary per CPU and OS.

Certainly as a proxy it would be okay, particularly if the differences in run time are relatively big, but it would possibly need supplemented with something else. (though I'm not sure what!)

from gin.

Some super-useful comments here, although we've departed a long way from the original bug report!

@sandybrownlee - please could you confirm whether this was a bug? Otherwise we should close this!

@markuswagnergithub @sandybrownlee @codykenb - I'm going to start a channel on Slack for this conversation and summarise the above.

from gin.

We confirmed that size-zero patches are possible, and intended, so if anything it was a behaviour I didn't expect rather than a bug. I think we can safely close this.

from gin.