sbt / sbt-jmh Goto Github PK

View Code? Open in Web Editor NEW

783.0 25.0 88.0 468 KB

"Trust no one, bench everything." - sbt plugin for JMH (Java Microbenchmark Harness)

License: Apache License 2.0

Scala 100.00%

openjdk scala benchmark

sbt-jmh's Introduction

sbt-jmh

SBT plugin for running OpenJDK JMH benchmarks.

JMH about itself:

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.

Please read nanotrusting nanotime and other blog posts on micro-benchmarking (or why most benchmarks are wrong) and make sure your benchmark is valid, before you set out to implement your benchmarks.

Versions

Plugin version	Default JMH version	Notes
`0.4.7` (sbt 1.3.0+)	`1.37`
`0.4.6` (sbt 1.3.0+)	`1.37`
`0.4.5` (sbt 1.3.0+)	`1.36`
`0.4.4` (sbt 1.3.0+)	`1.36`
`0.4.3` (sbt 1.3.0+)	`1.32`
`0.4.2` (sbt 1.3.0+)	`1.31`	JMH `-prof async` supports 2.x
`0.4.1` (sbt 1.3.0+)	`1.30`
`0.4.0` (sbt 1.3.0+)	`1.25`	profilers now in JMH core
`0.3.7` (sbt 0.13.17 / sbt 1.1.4)	`1.21`	support JDK 11
`0.3.6` (sbt 0.13.17 / sbt 1.1.4)	`1.21`	support JDK 11
`0.3.4` (sbt 0.13.17 / sbt 1.1.4)	`1.21`	support of GraalVM
`0.3.3` (sbt 0.13.17 / sbt 1.1.1)	`1.20`	JMH bugfix release
`0.3.2` (sbt 0.13.16 / sbt 1.0)	`1.19`	minor bugfix release
`0.3.1` (sbt 0.13.16 / sbt 1.0)	`1.19`	minor bugfix release
`0.3.0` (sbt 0.13.16 / sbt 1.0)	`1.19`	async profiler, flame-graphs
...	...

Not interesting versions are skipped in the above listing. Always use the newest which has the JMH version you need. You should stick to the latest version at all times anyway of course.

Adding to your project

Since sbt-jmh is an AutoPlugin all you need to do in order to activate it in your project is to add the below line to your project/plugins.sbt file:

// project/plugins.sbt
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.4.3")

and enable it in the projects where you want to (useful in multi-project builds, as you can enable it only where you need it):

// build.sbt
enablePlugins(JmhPlugin)

If you define your project in a Build.scala, you also need the following import:

import pl.project13.scala.sbt.JmhPlugin

You can read more about auto plugins in sbt on it's documentation page.

Write your benchmarks in src/main/scala. They will be picked up and instrumented by the plugin.

JMH has a very specific way of working (it generates loads of code), so you should prepare a separate project for your benchmarks. In it, just type Jmh/run in order to run your benchmarks. All JMH options work as expected. For help type Jmh/run -h. Another example of running it is:

Jmh/run -i 3 -wi 3 -f1 -t1 .*FalseSharing.*

Which means "3 iterations" "3 warmup iterations" "1 fork" "1 thread". Please note that benchmarks should be usually executed at least in 10 iterations (as a rule of thumb), but more is better.

For "real" results we recommend to at least warm up 10 to 20 iterations, and then measure 10 to 20 iterations again. Forking the JVM is required to avoid falling into specific optimisations (no JVM optimisation is really "completely" predictable)

If your benchmark should be a module in a multimodule project and needs access to another modules test classes then you might want to define your benchmarks in src/test as well (because Intellij does not support "compile->test" dependencies). While this is not directly supported it can be achieved with some tweaks. Assuming the benchmarks live in a module bench and need access to test classes from anotherModule, you have to define this dependency in your build.sbt:

lazy val bench = project
  .dependsOn(anotherModule % "test->test")
  .enablePlugins(JmhPlugin)
  .settings(
     Jmh / sourceDirectory := (Test / sourceDirectory).value,
     Jmh / classDirectory := (Test / classDirectory).value,
     Jmh / dependencyClasspath := (Test / dependencyClasspath).value,
     // rewire tasks, so that 'bench/Jmh/run' automatically invokes 'bench/Jmh/compile' (otherwise a clean 'bench/Jmh/run' would fail)
     Jmh / compile := (Jmh / compile).dependsOn(Test / compile).value,
     Jmh / run := (Jmh / run).dependsOn(Jmh / compile).evaluated
  )

Options

Please invoke Jmh/run -h to get a full list of run as well as output format options.

Useful hint: If you plan to aggregate the collected data you should have a look at the available output formats (-lrf). For example it's possible to keep the benchmark's results as csv or json files for later regression analysis.

Using Java Flight Recorder / async-profiler.

NOTE: sbt-jmh-s integration with async-profiler and Java Flight Recorder has been contributed to the JMH project as of JMH 1.25 and removed from this project. Please migrate to using -prof jfr / -prof async. Use -prof jfr:help / -prof async:help to list available options.

Examples

The examples are scala-fied examples from the original JMH repo, check them out, and run them!

The results will look somewhat like this:

...

[info] # Run progress: 92.86% complete, ETA 00:00:15
[info] # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre/bin/java
[info] # VM options: <none>
[info] # Fork: 1 of 1
[info] # Warmup: 2 iterations, single-shot each
[info] # Measurement: 3 iterations, single-shot each
[info] # Threads: 1 thread, will synchronize iterations
[info] # Benchmark mode: Single shot invocation time
[info] # Benchmark: org.openjdk.jmh.samples.JMHSample_02_BenchmarkModes.measureSingleShot
[info] # Warmup Iteration   1: 100322.000 us
[info] # Warmup Iteration   2: 100556.000 us
[info] Iteration   1: 100162.000 us
[info] Iteration   2: 100468.000 us
[info] Iteration   3: 100706.000 us
[info]
[info] Result : 100445.333 ±(99.9%) 4975.198 us
[info]   Statistics: (min, avg, max) = (100162.000, 100445.333, 100706.000), stdev = 272.707
[info]   Confidence interval (99.9%): [95470.135, 105420.532]
[info]
[info]
[info] # Run progress: 96.43% complete, ETA 00:00:07
[info] # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre/bin/java
[info] # VM options: <none>
[info] # Fork: 1 of 1
[info] # Warmup: 2 iterations, single-shot each, 5000 calls per batch
[info] # Measurement: 3 iterations, single-shot each, 5000 calls per batch
[info] # Threads: 1 thread, will synchronize iterations
[info] # Benchmark mode: Single shot invocation time
[info] # Benchmark: org.openjdk.jmh.samples.JMHSample_26_BatchSize.measureRight
[info] # Warmup Iteration   1: 15.344 ms
[info] # Warmup Iteration   2: 13.499 ms
[info] Iteration   1: 2.305 ms
[info] Iteration   2: 0.716 ms
[info] Iteration   3: 0.473 ms
[info]
[info] Result : 1.165 ±(99.9%) 18.153 ms
[info]   Statistics: (min, avg, max) = (0.473, 1.165, 2.305), stdev = 0.995
[info]   Confidence interval (99.9%): [-16.988, 19.317]
[info]
[info]
[info] Benchmark                                                 Mode   Samples         Mean   Mean error    Units
[info] o.o.j.s.JMHSample_22_FalseSharing.baseline               thrpt         3      692.034      179.561   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.baseline:reader        thrpt         3      199.185      185.188   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.baseline:writer        thrpt         3      492.850        7.307   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.contended              thrpt         3      706.532      293.880   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.contended:reader       thrpt         3      210.202      277.801   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.contended:writer       thrpt         3      496.330       78.508   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.hierarchy              thrpt         3     1751.941      222.535   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.hierarchy:reader       thrpt         3     1289.003      277.126   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.hierarchy:writer       thrpt         3      462.938       55.329   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.padded                 thrpt         3     1745.650       83.783   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.padded:reader          thrpt         3     1281.877       47.922   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.padded:writer          thrpt         3      463.773      104.223   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.sparse                 thrpt         3     1362.515      461.782   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.sparse:reader          thrpt         3      898.282      415.388   ops/us
[info] o.o.j.s.JMHSample_22_FalseSharing.sparse:writer          thrpt         3      464.233       49.958   ops/us

Advanced: Using custom Runners

It is possible to hand over the running of JMH to an App implemented by you, which allows you to programmatically access all test results and modify JMH arguments before you actually invoke it.

To use a custom runner class with runMain, simply use it: Jmh/runMain com.example.MyRunner -i 10 .* – an example for this is available in plugin/src/sbt-test/sbt-jmh/runMain (open the test file).

To replace the runner class which is used when you type Jmh/run, you can set the class in your build file – an example for this is available in plugin/src/sbt-test/sbt-jmh/custom-runner (open the build.sbt file).

Contributing

Yes, pull requests and opening issues is very welcome!

The plugin is maintained at an best-effort basis -- submitting a PR is the best way of getting something done :-)

You can locally publish the plugin with:

sbt 'project plugin; ^publishLocal'

Please test your changes by adding to the [scripted test suite][sbt-jmh/plugin/src/sbt-test/sbt-jmh/] which can be run with:

 sbt 'project plugin; ^scripted'

Special thanks

Special thanks for contributing async-profiler and flame-graphs support and other improvements go to @retronym of Lightbend's Scala team.

sbt-jmh's People

Contributors

Stargazers

Watchers

Forkers

mosesn alexander-myltsev ijuma imace paddymahoney auvik xuwei-k sokrahta 2m sergiotudela udl akkomar julien-truffaut ocadaruma bantonsson mallman narayana-glassbeam magro dp1310 calvinlfer jadireddi hminle retronym junix gkossakowski monkey-mas kxbmap blemale golem131 paplorinc epiphyllum yuvalitzchakov yanns som-snytt martijnhoekstra geekhuyang jiminhsieh lomigmegard jvican ikuo-suyama adam-singer chbatey dwijnand gortiz elliott-miller jozic gpoulin shanielh sullis tkrs plokhotnyuk duelist256 rintcius johnreedlol timsoethout helianthuslulu jxnu-liguobin viatsevskyi olka teazaid open-source-repository-archive yuriygagarin dmitry-timofeev kcrimson dineshdataengineer algobardo jatcwang ldrygala exoego d-ogxwx alexandrnikitin eternalerrors adpi2 sethtisue cybernetics akilegaspi isabella232 limansky catap phelps-sg pawel-wiejacha antonipp smarter i10416 guizmaii tmccombs despitedeath thiruselvaa

sbt-jmh's Issues

Upgrade to JMH 1.6

@shipilev is moving fast (I guess being a perf guy this is expected).
By now JMH 1.6 is already out again for a few days and sbt-jmh needs another upgrade.

Make sbt-jmh an AutoPlugin

Make it an autoplugin :-) http://www.scala-sbt.org/release/docs/Plugins.html

sbt-jmh-extras should be published scala-version-agnostic

There's only Java code in there

Add support for uploading a fat-jar to servers for benchmarking

Once we get fat-jars, we could look into "upload and run on server xxx"

Unable to add jmhSettings

Hi,

I tried adding jmhSettings to my build.sbt or build.scala but it never finds the reference even after importing pl.project13.scala.sbt.SbtJmh._

Any idea how to add this jmhSettings to my scala project?

Enabling the plugin in project definitions with settings makes run raise exception

I tried to use two separate project definitions n scala-streams (build file), and after switching to each project separately, invoking run produces the No main class detected. error.

Include tests

I'd suggest you to include any facilities (e. g. tests) that might verify I break nothing with my changes.

IncompatibleClassChangeError with custom runner

I just added object CustomRunnerApp extends App.
I do not use it.

By running:

jmh:run -i 1 -wi 1 -f1 -t1

I have the following exception:

Annotation generator had thrown the exception.
java.lang.IncompatibleClassChangeError: sphere.CustomRunnerApp and sphere.CustomRunnerApp$delayedInit$body disagree on InnerClasses attribute
    at java.lang.Class.getDeclaringClass0(Native Method)
    at java.lang.Class.getDeclaringClass(Class.java:1235)
    at java.lang.Class.getEnclosingClass(Class.java:1277)
    at java.lang.Class.getCanonicalName(Class.java:1392)
    at org.openjdk.jmh.generators.reflection.RFClassInfo.getQualifiedName(RFClassInfo.java:67)
    at org.openjdk.jmh.generators.core.BenchmarkGenerator.buildAnnotatedSet(BenchmarkGenerator.java:219)
    at org.openjdk.jmh.generators.core.BenchmarkGenerator.generate(BenchmarkGenerator.java:85)
    at org.openjdk.jmh.generators.bytecode.JmhBytecodeGenerator.main(JmhBytecodeGenerator.java:100)
    at pl.project13.scala.sbt.JmhPlugin$.pl$project13$scala$sbt$JmhPlugin$$internalGenerateBenchmarkSourcesAndResources(JmhPlugin.scala:81)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$2.apply(JmhPlugin.scala:64)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$2.apply(JmhPlugin.scala:62)

Should I remove this class from the jmh "transformations"?

Support for building benchmarks.jar not just running

We need to be able to create a far "benchmarks.jar" like the maven version does

Update to JMH 1.6.2

Minor update, but still

JMH version bump to 1.3.1

I am profiling a library written in scala and many of the hottest regions aren't reported (pretty printed) because they are unparseable.

I would like to see how 1.3.1 behaves because perfasm has been improved since 1.1.

Thx!

java.lang.InternalError: Malformed class name on jmh:run

Currently trying to run previously built/running benchmarks with sbt-jmh and appears to fail with a runtime error. I have tried turning the example into something simple, which may help in reproducing.

scala version: 2.11.4
sbt version: 0.13.7
sbt-jmh version: 0.2.6 (also tried with 0.2.4, and it fails there now also)

The project is a multi-module project, where there is a benchmarks module containing the jmh benchmarks and has the JmhPlugin enabled.

Below, is a simple benchmark that can be run, in isolation:

package foo.bar

import org.openjdk.jmh.annotations.Benchmark

class Benchmarks {
  @Benchmark
  def helloWorldBench(): Unit = {
  }
}

If we put this file in the same directory, then we get the error seen below:

package foo.bar
object Bar {
  object Hello {
    //Note: if we move the trait below out of Hello, but within Bar it "fixes"
    trait Something
  }
}

Here is the general error observed:

[info] Compiling 1 Scala source to ...
Processing 140 classes from /.../target/scala-2.11/classes with "reflection" generator
Writing out Java source to /.../target/scala-2.11/src_managed/jmh and resources to /.../target/scala-2.11/resource_managed/jmh
Annotation generator had thrown the exception.
java.lang.InternalError: Malformed class name
    at java.lang.Class.getSimpleName(Class.java:1330)
    at java.lang.Class.getCanonicalName(Class.java:1399)
    at org.openjdk.jmh.generators.reflection.RFClassInfo.getQualifiedName(RFClassInfo.java:67)
    at org.openjdk.jmh.generators.core.BenchmarkGenerator.buildAnnotatedSet(BenchmarkGenerator.java:226)
    at org.openjdk.jmh.generators.core.BenchmarkGenerator.generate(BenchmarkGenerator.java:85)
    at org.openjdk.jmh.generators.bytecode.JmhBytecodeGenerator.main(JmhBytecodeGenerator.java:100)
    at pl.project13.scala.sbt.JmhPlugin$.pl$project13$scala$sbt$JmhPlugin$$internalGenerateBenchmarkSourcesAndResources(JmhPlugin.scala:91)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$2.apply(JmhPlugin.scala:74)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$2.apply(JmhPlugin.scala:72)
    at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:186)
    at sbt.FileFunction$$anonfun$cached$1.apply(Tracked.scala:186)
    at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$anonfun$apply$4.apply(Tracked.scala:200)
    at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3$$anonfun$apply$4.apply(Tracked.scala:196)
    at sbt.Difference.apply(Tracked.scala:175)
    at sbt.Difference.apply(Tracked.scala:157)
    at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.apply(Tracked.scala:196)
    at sbt.FileFunction$$anonfun$cached$2$$anonfun$apply$3.apply(Tracked.scala:195)
    at sbt.Difference.apply(Tracked.scala:175)
    at sbt.Difference.apply(Tracked.scala:151)
    at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:195)
    at sbt.FileFunction$$anonfun$cached$2.apply(Tracked.scala:193)
    at pl.project13.scala.sbt.JmhPlugin$.pl$project13$scala$sbt$JmhPlugin$$generateBenchmarkSourcesAndResources(JmhPlugin.scala:76)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$projectSettings$9.apply(JmhPlugin.scala:42)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$projectSettings$9.apply(JmhPlugin.scala:42)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

DRY up JMH version definition

it's in build.sbt and the scala plugin file now.

JmhPlugin is using a deprecated method

Hi,

While I was upgrading to JMH 1.10, I found that JmhPlugin is using a deprecated method https://github.com/ktoso/sbt-jmh/blob/master/src/main/scala/pl/project13/scala/sbt/JmhPlugin.scala#L118

@deprecated("Use the compile method instead.", "0.13.8")

In case that SBT 0.13.6 was not supported anymore in this plugin, this warning should be fixed. So this issue should be considered in the future.

Keep getting compilation errors (after jmh generation) when bumped version of sbt-jmh to 0.2.0

Full stack trace of jmh:compile.

java.lang.RuntimeException: java.lang.AbstractMethodError: org.openjdk.jmh.generators.annotations.APGeneratorDestinaton.getResource(Ljava/lang/String;)Ljava/io/Reader;
    at com.sun.tools.javac.main.Main.compile(Main.java:553)
    at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
    at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
    at sbt.compiler.javac.LocalJavaCompiler.run(LocalJava.scala:69)
    at sbt.compiler.javac.JavaCompilerAdapter.compileWithReporter(JavaCompilerAdapter.scala:31)
    at sbt.compiler.javac.AnalyzingJavaCompiler$$anonfun$compile$1.apply$mcV$sp(AnalyzingJavaCompiler.scala:65)
    at sbt.compiler.javac.AnalyzingJavaCompiler$$anonfun$compile$1.apply(AnalyzingJavaCompiler.scala:65)
    at sbt.compiler.javac.AnalyzingJavaCompiler$$anonfun$compile$1.apply(AnalyzingJavaCompiler.scala:65)
    at sbt.compiler.javac.AnalyzingJavaCompiler.timed(AnalyzingJavaCompiler.scala:93)
    at sbt.compiler.javac.AnalyzingJavaCompiler.compile(AnalyzingJavaCompiler.scala:64)
    at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileJava$1$1.apply$mcV$sp(MixedAnalyzingCompiler.scala:61)
    at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileJava$1$1.apply(MixedAnalyzingCompiler.scala:61)
    at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileJava$1$1.apply(MixedAnalyzingCompiler.scala:61)
    at sbt.compiler.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:75)
    at sbt.compiler.MixedAnalyzingCompiler.compileJava$1(MixedAnalyzingCompiler.scala:60)
    at sbt.compiler.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:65)
    at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
    at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
    at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:66)
    at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:64)
    at sbt.inc.IncrementalCommon.cycle(IncrementalCommon.scala:31)
    at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:62)
    at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:61)
    at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:89)
    at sbt.inc.Incremental$.compile(Incremental.scala:61)
    at sbt.inc.IncrementalCompile$.apply(Compile.scala:54)
    at sbt.compiler.IC$.compileInternal(IncrementalCompiler.scala:160)
    at sbt.compiler.IC$.incrementalCompile(IncrementalCompiler.scala:138)
    at sbt.Compiler$.compile(Compiler.scala:128)
    at sbt.Compiler$.apply(Compiler.scala:109)
    at sbt.Compiler$.apply(Compiler.scala:94)
    at pl.project13.scala.sbt.JmhPlugin$.myCompile(JmhPlugin.scala:118)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$projectSettings$8.apply(JmhPlugin.scala:60)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$projectSettings$8.apply(JmhPlugin.scala:57)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AbstractMethodError: org.openjdk.jmh.generators.annotations.APGeneratorDestinaton.getResource(Ljava/lang/String;)Ljava/io/Reader;
    at org.openjdk.jmh.generators.core.BenchmarkGenerator.complete(BenchmarkGenerator.java:164)
    at org.openjdk.jmh.generators.BenchmarkProcessor.process(BenchmarkProcessor.java:54)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:794)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$200(JavacProcessingEnvironment.java:91)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator.runContributingProcs(JavacProcessingEnvironment.java:627)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1033)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1198)
    at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1170)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:856)
    at com.sun.tools.javac.main.Main.compile(Main.java:523)
    at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
    at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
    at sbt.compiler.javac.LocalJavaCompiler.run(LocalJava.scala:69)
    at sbt.compiler.javac.JavaCompilerAdapter.compileWithReporter(JavaCompilerAdapter.scala:31)
    at sbt.compiler.javac.AnalyzingJavaCompiler$$anonfun$compile$1.apply$mcV$sp(AnalyzingJavaCompiler.scala:65)
    at sbt.compiler.javac.AnalyzingJavaCompiler$$anonfun$compile$1.apply(AnalyzingJavaCompiler.scala:65)
    at sbt.compiler.javac.AnalyzingJavaCompiler$$anonfun$compile$1.apply(AnalyzingJavaCompiler.scala:65)
    at sbt.compiler.javac.AnalyzingJavaCompiler.timed(AnalyzingJavaCompiler.scala:93)
    at sbt.compiler.javac.AnalyzingJavaCompiler.compile(AnalyzingJavaCompiler.scala:64)
    at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileJava$1$1.apply$mcV$sp(MixedAnalyzingCompiler.scala:61)
    at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileJava$1$1.apply(MixedAnalyzingCompiler.scala:61)
    at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileJava$1$1.apply(MixedAnalyzingCompiler.scala:61)
    at sbt.compiler.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:75)
    at sbt.compiler.MixedAnalyzingCompiler.compileJava$1(MixedAnalyzingCompiler.scala:60)
    at sbt.compiler.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:65)
    at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
    at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
    at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:66)
    at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:64)
    at sbt.inc.IncrementalCommon.cycle(IncrementalCommon.scala:31)
    at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:62)
    at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:61)
    at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:89)
    at sbt.inc.Incremental$.compile(Incremental.scala:61)
    at sbt.inc.IncrementalCompile$.apply(Compile.scala:54)
    at sbt.compiler.IC$.compileInternal(IncrementalCompiler.scala:160)
    at sbt.compiler.IC$.incrementalCompile(IncrementalCompiler.scala:138)
    at sbt.Compiler$.compile(Compiler.scala:128)
    at sbt.Compiler$.apply(Compiler.scala:109)
    at sbt.Compiler$.apply(Compiler.scala:94)
    at pl.project13.scala.sbt.JmhPlugin$.myCompile(JmhPlugin.scala:118)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$projectSettings$8.apply(JmhPlugin.scala:60)
    at pl.project13.scala.sbt.JmhPlugin$$anonfun$projectSettings$8.apply(JmhPlugin.scala:57)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[error] (jmh:compile) java.lang.AbstractMethodError: org.openjdk.jmh.generators.annotations.APGeneratorDestinaton.getResource(Ljava/lang/String;)Ljava/io/Reader;

Ease data collection, like an append target (?)

This should allow Akka to build regression tests based on it

Compile only the needed-to-be-compiled java sources

Comments here: b4d7c7f#commitcomment-6798078

Filter/ignore javac warnings in generated benchmark sources

We know there's some unsafe stuff going on there, no need to warn us about generated code :)

FIX readme and activator after the plugin was made an AutoPlugin

As mentioned here https://github.com/ktoso/sbt-jmh/pull/35/files#r29582330

Evaluate whether `fork in (Compile, run) := true` is still required

runMain appears function ok without it.

Version 0.1.6 (from readme) isn't published

Looks to me like version 0.1.6 (from the readme) isn't published:
http://dl.bintray.com/sbt/sbt-plugin-releases/pl.project13.scala/sbt-jmh/scala_2.10/sbt_0.13/

How to run the benchmarks

Hi,

I was able to set up the Auto Plugin in my project and now I am not sure how to execute a simple benchmark test on scala method.

Here is the example method I wrote in Scala

BenchMarkSample.scala
import org.openjdk.jmh.annotations.Benchmark

/**

Created by a567223 on 5/7/2015.
*/
class HelloWorld {

@benchmark
def callBenchmark{
println("******************WILL THIS BENCHMARK")
}
}

Command I tried to use to run benchmark: sbt
"run -i 10 .BenchMarkSample."

Move this repo under sbt/sbt-jmh

The "more official place" for sbt plugins.
And add it to the list on the sbt site as well.

Remove generatorType "wont work" notice, once 0.9 fixes minor issue

See discussion at: https://twitter.com/ktosopl/status/480452428396257283

In general, the 4th argument to the benchmark generator is being ignored. It's a minor issue in JMH, and should be fixed soon.

The typo was in the args parsing here: http://hg.openjdk.java.net/code-tools/jmh/file/86072ed4c056/jmh-generator-bytecode/src/main/java/org/openjdk/jmh/generators/bytecode/JmhBytecodeGenerator.java

Fix examples link

Hello

Examples points to this link:
https://github.com/ktoso/sbt-jmh/tree/master/src/sbt-test/sbt-jmh/run/src/main/scala/org/openjdk/jmh/samples that does not work

can not found any benchmarks after compile the scala file

Hi,ktoso，I am using this plugin for test the benchmark of an akka-Serializer ,but it can not found any benchmarks with run -h

> run -l
[info] Updating {file:/home/kerr/devProject/i4joy/svn/u3d_server/server/LittleJianghu-Server/}serializers...
[info] Resolving jline#jline;2.12 ...
[info] Done updating.
[info] Compiling 1 Scala source and 2 Java sources to /home/kerr/devProject/i4joy/svn/u3d_server/server/LittleJianghu-Server/framework/serializers/target/scala-2.11/classes...
[warn] 注: /home/kerr/devProject/i4joy/svn/u3d_server/server/LittleJianghu-Server/framework/serializers/src/main/java/qgame/serializer/protostuff/ProtostuffRuntimeSerializer.java使用了未经检查或不安全的操作。
[warn] 注: 有关详细信息, 请使用 -Xlint:unchecked 重新编译。
[info] Generating JMH benchmark Java source files...
Processing 7 classes from /home/kerr/devProject/i4joy/svn/u3d_server/server/LittleJianghu-Server/framework/serializers/target/scala-2.11/classes with "reflection" generator
Writing out Java source to /home/kerr/devProject/i4joy/svn/u3d_server/server/LittleJianghu-Server/framework/serializers/target/scala-2.11/generated-sources/jmh and resources to /home/kerr/devProject/i4joy/svn/u3d_server/server/LittleJianghu-Server/framework/serializers/target/scala-2.11/classes
[info] Compiling generated JMH benchmarks...
[info] Compiling 1 Scala source and 12 Java sources to /home/kerr/devProject/i4joy/svn/u3d_server/server/LittleJianghu-Server/framework/serializers/target/scala-2.11/classes...
[warn] 注: 某些输入文件使用了未经检查或不安全的操作。
[warn] 注: 有关详细信息, 请使用 -Xlint:unchecked 重新编译。
[info] Running org.openjdk.jmh.Main -l
[info] Benchmarks:

there are no benchmark been found
the benchmark is so simple

@State(Scope.Benchmark)
@BenchmarkMode(Array(Mode.Throughput))
@Warmup(iterations = 10)
@Measurement(iterations = 20, timeUnit = TimeUnit.MILLISECONDS)
class ProtostuffRuntimeSerializerJMHBenckmark {

  var serializer:ProtostuffRuntimeSerializer = _
  var bean :Person =_
  var beanBytes :Array[Byte] =_
  var dataCount :Int =_

  @Setup
  def setup(): Unit = {
     serializer = new ProtostuffRuntimeSerializer
     bean = Person("me",13)
     beanBytes = serializer.toBinary(bean)
     dataCount = 10000
  }

  @TearDown
  def shutdown(): Unit = {

  }

  @Benchmark
  @OperationsPerInvocation(10000)
  def testSerializing(): Unit = {
    for( i <- 1 to dataCount){
      serializer.toBinary(bean)
    }
  }

  @Benchmark
  @OperationsPerInvocation(10000)
  def testDeserializing() {
    for (i <- 1 to dataCount){
      serializer.fromBinaryJava(beanBytes,Person.getClass)
    }
  }

  case class Person(@BeanProperty name:String, @BeanProperty age:Int)

}

the version is 0.1.6

Plugin version 2.4 depends on wrong version of JMH

According to

https://github.com/ktoso/sbt-jmh/blob/185131ce460f864e8e5e1344c895e8fccfd41338/build.sbt#L17

the plugin is built against JMH 1.11. However, according to

https://github.com/ktoso/sbt-jmh/blob/c49d49b564eb2d2dbbb2be3c41ac2197b977f574/src/main/scala/pl/project13/scala/sbt/JmhPlugin.scala#L32

the plugin adds JMH 1.10.3 to the plugged-in project's library dependencies.

Support JMH 0.9.2

http://mail.openjdk.java.net/pipermail/jmh-dev/2014-July/001137.html

Add sbt-tests using `scripted`

Update jmh version

http://mail.openjdk.java.net/pipermail/jmh-dev/2014-July/001168.html

Enable programmatic running of benchmarks

As discussed in #27 (comment) we sometimes may want to runMain com.example.MyRunner in order to get programatic access to the test results (send them over somewhere etc).

This does not currently work, and we'll get

[info] # JMH 1.5 (released 5 days ago)
[info] # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/bin/java
[info] # VM options: <none>
[info] # Warmup: 1 iterations, single-shot each
[info] # Measurement: 1 iterations, single-shot each
[info] # Timeout: 10 min per iteration
[info] # Threads: 1 thread
[info] # Benchmark mode: Single shot invocation time
[info] # Benchmark: org.openjdk.jmh.samples.JMHSample_02_BenchmarkModes.measureSingleShot
[info]
[info] # Run progress: 100.00% complete, ETA 00:00:00
[info] # Fork: 1 of 1
[info] <failure>
[info]
[info] java.lang.IllegalArgumentException: Benchmark does not match a class
[info]  at org.openjdk.jmh.util.ClassUtils.loadClass(ClassUtils.java:90)
[info]  at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:214)
[info]  at org.openjdk.jmh.runner.BaseRunner.runBenchmarks(BaseRunner.java:111)
[info]  at org.openjdk.jmh.runner.ForkedRunner.run(ForkedRunner.java:51)
[info]  at org.openjdk.jmh.runner.ForkedMain.main(ForkedMain.java:80)
[info] Caused by: java.lang.ClassNotFoundException: org.openjdk.jmh.samples.generated.JMHSample_02_BenchmarkModes_measureSingleShot
[info]  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
[info]  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
[info]  at java.security.AccessController.doPrivileged(Native Method)
[info]  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
[info]  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
[info]  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
[info]  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
[info]  at java.lang.Class.forName0(Native Method)
[info]  at java.lang.Class.forName(Class.java:191)
[info]  at org.openjdk.jmh.util.ClassUtils.loadClass(ClassUtils.java:72)
[info]  ... 4 more
[info]
[info]

Adding these:

./target/scala-2.10/classes/...

to the classpath should solve things.

jmh:compile doesnt work with 0.2.3 ?

We didn't change any code just upgraded to 0.2.3. Did we miss any step ?

java:9: package org.openjdk.jmh.annotations does not exist
[error] org.openjdk.jmh.annotations.CompilerControl

java:10: package org.openjdk.jmh.runner does not exist
[error] org.openjdk.jmh.runner.InfraControl

java:11: cannot find symbol
[error]   symbol:   class ThreadParams
[error]   location: package org.openjdk.jmh.infra
[error] org.openjdk.jmh.infra.ThreadParams

java:12: package org.openjdk.jmh.results does not exist
[error] org.openjdk.jmh.results.BenchmarkTaskResult

java:13: package org.openjdk.jmh.results does not exist
[error] org.openjdk.jmh.results.Result

java:15: package org.openjdk.jmh.results does not exist
[error] org.openjdk.jmh.results.AverageTimeResult

Use `crossTarget`

As discussed here: d344f21

Fails if also has Java benchmarks

Supposedly:

I have tested it now using 0.1.4, and it works. 
The only minor issue is that it fails if there is a Java-based benchmark in the 
same sbt project with an IllegalArgumentException in 
org.objectweb.asm.ClassReader/org.openjdk.jmh.generators.bytecode.ASMGeneratorS‌ource.processClass(ASMGeneratorSource.java:65)

Launching Akka ActorSystem

Hi guys,

I know that it is not the best place to ask about ActorSystem in JMH, but I believe the right people are here.

@BenchmarkMode(Array(Mode.AverageTime))
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Benchmark)
@Warmup(iterations = 8)
@Measurement(iterations = 4)
@Threads(1)
@Fork(1)
class HandlerBenchmark {
  implicit val actorServiceAskTimeout = Timeout(5.seconds)

  val config   = ConfigLoader.load
  implicit val system = ActorSystem("HandlerBenchmarkActorSystem", config)
  runActors(config)

  @TearDown
  def teardown() = this.synchronized {
    system.shutdown()
  }
}

But teardown does not work as I see those messages:

[info] <JMH had finished, but forked VM did not exit, are there stray running threads? Waiting 24 seconds more...>
[info] <JMH had finished, but forked VM did not exit, are there stray running threads? Waiting 19 seconds more...>
[info] <JMH had finished, but forked VM did not exit, are there stray running threads? Waiting 14 seconds more...>

Could you recommend a best practice in what way you do it?

The with-FlightRecorderProfiler scripted test has versions mixed up

If you modify jmh-extras and run the with-FlightRecorderProfiler scripted test, the local modifications won't be picked up. The reason is that the scripted test doesn't set the version of extras so the default (non snapshot) version is used.

The workaround is:

diff --git a/plugin/src/sbt-test/sbt-jmh/with-FlightRecorderProfiler/build.sbt b/plugin/src/sbt-test/sbt-jmh/with-FlightRecorderProfiler/build.sbt
index bb17cb1..601d9cc 100644
--- a/plugin/src/sbt-test/sbt-jmh/with-FlightRecorderProfiler/build.sbt
+++ b/plugin/src/sbt-test/sbt-jmh/with-FlightRecorderProfiler/build.sbt
@@ -1 +1,2 @@
 enablePlugins(JmhPlugin)
+extrasVersion in Jmh := sys.props("project.version")

Consider including JFR Recording profiler

@retronym has a nice JFR recording profiler over here: https://github.com/retronym/scala-jmh-suite/blob/master/src/main/java/scala/tools/nsc/benchmark/FlightRecordingProfiler.java
We could ship it as part of sbt-jmh I think

Clean up of previously generated benchmarks

When I created a benchmark, some classes are generated to output path. Then I deleted this benchmark from sources. But sbt benchmarkproject/run is still trying to find those classes causing error as follows:

[error]   symbol:   class JMHSample_01_HelloWorld
[error]   location: package org.openjdk.jmh.samples
[error] /Users/alex/projects/orca/benchmarks/target/scala-2.11/generated-sources/jmh/org/openjdk/jmh/samples/generated/JMHSample_01_HelloWorld_wellHelloThere.java:68: error: cannot find symbol
[error]                 l_jmhsample_01_helloworld0_0.wellHelloThere();
[error]

Is it a bug? Or am I doing something wrong?

> jmh:version
[info] 0.9
> consoleProject
[info] Starting scala interpreter...
[info]
import sbt._
import Keys._
import dsl._
import _root_.org.sbtidea.SbtIdeaPlugin._
import _root_.net.virtualvoid.sbt.graph.Plugin._
import _root_.com.typesafe.sbteclipse.plugin.EclipsePlugin._
import _root_.pl.project13.scala.sbt.SbtJmh._
import _root_.sbt.plugins.IvyPlugin
import _root_.sbt.plugins.JvmPlugin
import _root_.sbt.plugins.CorePlugin
import _root_.sbt.plugins.JUnitXmlReportPlugin
import currentState._
import extracted._
import cpHelpers._
Welcome to Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_25).
Type in expressions to have them evaluated.
Type :help for more information.

scala> pl.project13.scala.sbt.SbtJmh.getClass.getProtectionDomain
res0: java.security.ProtectionDomain =
ProtectionDomain  (file:/Users/jason/.ivy2/cache/scala_2.10/sbt_0.13/pl.project13.scala/sbt-jmh/jars/sbt-jmh-0.1.4.jar <no signer certificates>)
 sbt.PluginManagement$PluginClassLoader@3fc2a1d1
 <no principals>
 java.security.Permissions@130f7c25 (
 ("java.util.PropertyPermission" "java.specification.version" "read")
 ("java.util.PropertyPermission" "java.version" "read")
 ("java.util.PropertyPermission" "os.arch" "read")
 ("java.util.PropertyPermission" "java.specification.vendor" "read")
 ("java.util.PropertyPermission" "java.vm.specification.name" "read")
 ("java.util.PropertyPermission" "java.vm.vendor" "read")
 ("java.util.PropertyPermission" "path.separator" "read")
 ("java.util.PropertyPermission" "os.version" "read")
 ("java.util.PropertyP...
scala> pl.project13.scala.sbt.SbtJmh.jmhSettings.find(_.toString.contains("version")).get.init.evaluate(null)
res1: Any = 0.9

[info] Could not start recording, not able to write to file "/var/folders/2y/yrrnf9rd2ks0dk9jqkt_m5v80000gn/T/jmh5429315193288516734.jfrDatafilename=/var/folders/2y/yrrnf9rd2ks0dk9jqkt_m5v80000gn/T/jmh5429315193288516734.jfrData"

It turns out that if you let jmh run, the recordings do get saved. However, the message appears to be very alarming.