Comments (12)
from gin.
Yes, that's intentional IIRC. Do you think it shouldn't? As it's local search, I was thinking it's useful to be able to backtrack... consider that execution times are noisy.
from gin.
I guess that's basically asking for a re-evaluation of the original unaltered source? I was seeing empty Patches taking the "best" run time when I suspect they shouldn't be - i.e. after some other patches had been found that did offer an improvement - I will check this in the morning.
from gin.
I think that's probably due to noisy execution times. This is a problem generally with non-functional optimisation. I think doing something rigorous here would be great, e.g. take a similar approach to "racing" algorithms as in irace2.
from gin.
Got it. The JUnit tests were failing, so all program variants were running in about 1ms (I thought that the run was a bit fast!)). Due to noise this meant that the fastest patch varied.
The reason the JUnit tests were failing was down to a copy+paste error in the example test class I wrote. I'll raise a separate issue for that.
btw - agree wrt racing for handling noise.
from gin.
Thanks for chiming in, Markus.
I can see a few ways of implementing a racing approach:
-
Racing two (or n) patches across a set of test cases (this fits well with local search or tournament selection)
-
Repeatedly sampling the same set of tests with two patches until we can determine a difference between them with a certain statistical confidence.
Do we want to go there right away?
Not necessarily, but I think we need to work out some way of solving the execution time comparison problem. Alternatives include counting the number of bytecodes executed, and using a model to estimate expected execution time, for instance.
Immediate question is - how is this solved in other software? I know that some work takes the 3rd quartile as a conservative estimate of the execution time, for example.
from gin.
from gin.
from gin.
from gin.
This is delving in to territory that I'm less familiar with but we ought to be careful with counting bytecodes because of JIT. The approach that Nathan Burles developed for measuring energy using bytecode tracing ran into issues when JIT was enabled - aside from producing stochastic results I guess it would probably also vary per CPU and OS.
Certainly as a proxy it would be okay, particularly if the differences in run time are relatively big, but it would possibly need supplemented with something else. (though I'm not sure what!)
from gin.
Some super-useful comments here, although we've departed a long way from the original bug report!
@sandybrownlee - please could you confirm whether this was a bug? Otherwise we should close this!
@markuswagnergithub @sandybrownlee @codykenb - I'm going to start a channel on Slack for this conversation and summarise the above.
from gin.
We confirmed that size-zero patches are possible, and intended, so if anything it was a behaviour I didn't expect rather than a bug. I think we can safely close this.
from gin.
Related Issues (20)
- Testing
- Tutorial
- Re-design and tidy-up HOT 1
- Add a list of publications section to README
- Extend TestCaseGenerator to fully support Gradle projects
- FullyQualifiedNames.getClassName() confused by inner classes HOT 1
- Multiple CopyStatement to the same block are all applied at the same location HOT 1
- Make all classes Serializable
- Test index incorrect HOT 2
- Failure of RTS tests due to uninitiated .ekstazi directory HOT 1
- Only tests with annotations will be run with LocalSearch and their timeout modified HOT 1
- SourceFileTree.getStatement() and getNode() don't clone properly HOT 1
- Symbolic links cause problems with filenames HOT 1
- Malformation in methods names - Profiling Problem (probably) HOT 4
- Running Gin with a project with modules causes class to not be found during compilation
- Expensive test cases HOT 1
- Probable bug within the Project class HOT 2
- Update MemoryProfiler to JFR
- Profiler will not work with Windows and Java Flight Recorder on Gradle projects HOT 1
- Make sampling interval for Profiler and Memory Profiler configurable again
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gin.