dotnet / crank Goto Github PK

Benchmarking infrastructure for applications

License: MIT License

Batchfile 0.04% PowerShell 12.12% Shell 8.60% C# 77.77% Lua 0.01% CMake 1.03% Dockerfile 0.21% JavaScript 0.23%

crank's Introduction

Microsoft.Crank

Crank is the benchmarking infrastructure used by the .NET team to run benchmarks including (but not limited to) scenarios from the TechEmpower Web Framework Benchmarks.

One of the goals of this system is to give developers a tool that lets them work on performance and measure potential improvements very easily. Some of the capabilities are:

Deploy and benchmark multi-tiered applications based on .NET or Docker containers
Store results in JSON or in SQL Server for charting
Customize the applications or their environment to test different settings
Collect traces

Want to learn more? Check out our documentation.

Components

The benchmarking infrastructure is made of these components:

crank-agent - A service that executes jobs that run as part of a benchmark
crank - A command line utility that can enqueue jobs and record results

There are also some built in jobs:

wrk - An http client benchmarking tool. This tool is used when benchmarking TechEmpower https://github.com/wg/wrk (Linux only).
wrk2 - An http client benchmarking tool optimized for latency testing https://github.com/giltene/wrk2 (Linux only).
bombardier - A go based http client benchmarking tool https://github.com/codesenberg/bombardier (cross-platform).

Get Started

Read the documentation
Install the crank controller dotnet tool: dotnet tool update Microsoft.Crank.Controller --version "0.2.0-*" --global
Use some predefined scenarios

How to Engage, Contribute, and Give Feedback

If you want a low-spam way to follow what the team is doing you should subscribe to the issue we use to post announcements here. Only team-members can post on this issue.

Some of the best ways to contribute are to try things out, file issues, join in design conversations, and make pull-requests.

Download our latest daily builds
Try tutorials and working with your own projects
Log issues if you find problems, or if you have suggestions.
Log an issue if you have feedback you want to share with the team.

Check out the contributing page to see the best places to log issues and start discussions.

Code of conduct

This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. For more information, see the .NET Foundation Code of Conduct.

crank's People

Contributors

Stargazers

Watchers

Forkers

manish2020 lhutyra zhaodongtian molinch thiagobarradas marioszmsft anouarattn anatolib rr-wfm kant2002 alefranz kevinmiles ironcev-forks mikeharder liuziyi1 devbox10 fanyang-mono shirhatti v-joesun awesomedotnetcore mmitche mgasparel layomia habbes benjaminpetit ievangelist tonyredondo brianrob zhamppx97 sywhang billwert l2 ezeji microestc isabella232 adityamandaleeka bhaan-vanco andreabrantes kchinburarat maketochen mattgal zhenlei520 rducom brunolins16 debasispaul yuyixg corrieriluca wolfoftotem giulianob robertpi abimbolaomotoso dongfo martincostello abkhan5 doddgu jxced dystudio tony-li-avepoint brucehu123 michalstrehovsky kunalspathak pyjcode pockees carnaviire kevingosse sleepybag kirankumarkolli jiakaidou jacobusmeintjes mofaggolhoshen aaronrobinsonmsft kirkaiya elinunez2 ikvm lvies mitchdenny eerhardt kasperk81 mrsharm gg-big-org ultimatemanas-swi drewscoggins mihazupan catcherwong drakantas stulzq trylek roberthenry6bev seanpm2001 caaavik-msft loopedbard3 shumpeihorimoto mikelle-rogers haoguanjun oinpentuls keithmifsud17 ks1990cn birojnayak sharpensols

crank's Issues

Websocket support

Hi,
in the past there was crank to benchmark SignalR. Are there plans on doing that in this version as well?
And what about plain old websockets?

regards
Riccardo

Support pushing results into a time series database

InfluxDb
Prometheus

Add limit on the amount of counter data collected / change time interval

if you run a test for many hours, counters will be collected for every second, leading to a very big dataset.

add the ability to limit the amount of counters collected and change the time interval.

cc: @sebastienros

Unable to build Platform benchmarks when uploading local .dll

I'm trying to measure the performance of a local build of the runtime, but am getting the following build error:

acovingt@DESKTOP-DOTNET-T2:~/projects/dotnet-benchmarking$ crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/platform.benchmarks.yml --config amd.benchmarks.yml --scenario plaintext --profile local --application.framework netcoreapp5.0 --application.options.outputFiles ~/projects/runtime/artifacts/bin/testhost/net6.0-Linux-Release-x64/shared/Microsoft.NETCore.App/6.0.0/System.Private.CoreLib.dll
[03:49:20.520] Running session 'd7c18df9ea6d4187aead247f460bd79e' with description ''
[03:49:20.563] Starting job 'application' ...
[03:49:20.566] Submitted job: http://localhost:5010/jobs/6
[03:49:21.582] Job has been selected by the server ...
[03:49:21.585] Uploading /home/acovingt/projects/runtime/artifacts/bin/testhost/net6.0-Linux-Release-x64/shared/Microsoft.NETCore.App/6.0.0/System.Private.CoreLib.dll
[03:49:21.665] Job is now building ... http://localhost:5010/jobs/6/buildlog
[03:49:31.676] Job failed on benchmark server, stopping...
Job failed at runtime:
Kestrel Platform-Level Application
/plaintext
/json
Args:

[03:49:31.682] Job has failed, interrupting benchmarks ...
[03:49:31.684] Stopping job 'application' ...
[03:49:32.692] Deleting job 'application' ...

| application         |        |
| ------------------- | ------ |
| Build Time (ms)     |  5,253 |
| Published Size (KB) | 94,305 |

If I drop the --application.options.outputFiles flag, then I get a normal run:

acovingt@DESKTOP-DOTNET-T2:~/projects/dotnet-benchmarking$ crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/platform.benchmarks.yml --config amd.benchmarks.yml --scenario plaintext --profile local --application.framework netcoreapp5.0
[03:50:54.670] Running session '8e9a5ede235943bb84dd17770b9e7681' with description ''
[03:50:54.714] Starting job 'application' ...
[03:50:54.717] Submitted job: http://localhost:5010/jobs/7
[03:50:55.729] Job has been selected by the server ...
[03:50:55.730] Job is now building ... http://localhost:5010/jobs/7/buildlog
[03:51:04.739] Job is running ... http://localhost:5010/jobs/7/output
[03:51:04.743] Starting job 'load' ...
[03:51:04.744] Submitted job: http://localhost:5010/jobs/8
[03:51:06.747] Job has been selected by the server ...
[03:51:06.748] Job is now building ... http://localhost:5010/jobs/8/buildlog
[03:51:13.755] Job is running ... http://localhost:5010/jobs/8/output
[03:51:32.770] Stopping job 'load' ...
[03:51:33.779] Deleting job 'load' ...
[03:51:33.780] Stopping job 'application' ...
[03:51:35.788] Deleting job 'application' ...

| application          |                                |
| -------------------- | ------------------------------ |
| CPU Usage (%)        |                             50 |
| Raw CPU Usage (%)    |                         604.39 |
| Working Set (MB)     |                             85 |
| Build Time (ms)      |                          4,883 |
| Start Time (ms)      |                            132 |
| Published Size (KB)  |                         94,305 |
| .NET Runtime Version | 5.0.0-rc.2.20475.5+c5a3f49c... |
| ASP.NET Core Version | 5.0.0-rc.2.20475.17+25bd01f... |

| load                   |            |
| ---------------------- | ---------- |
| CPU Usage (%)          |         51 |
| Raw CPU Usage (%)      |     609.60 |
| Working Set (MB)       |         50 |
| Build Time (ms)        |      3,914 |
| Start Time (ms)        |          0 |
| Published Size (KB)    |     76,404 |
| Requests/sec           |  5,799,441 |
| Requests               | 87,515,872 |
| Mean latency (ms)      |       6.52 |
| Max latency (ms)       |     246.88 |
| Bad responses          |          0 |
| Socket errors          |          0 |
| Read throughput (MB/s) |     696.88 |
| Latency 50th (ms)      |       3.32 |
| Latency 75th (ms)      |       8.52 |
| Latency 90th (ms)      |      15.42 |
| Latency 99th (ms)      |      43.58 |

Am I misunderstanding how to upload a local build of the runtime? Any help would be appreciated!

--iterations value ignored

--iterations has no effect on the running. The value acquired from the arguments seems to be ignored in this assignment

crank/src/Microsoft.Crank.Controller/Program.cs

Line 200 in 3b80dc5

var iterations = 1;

Fix the name pipe timeout error on the agent

This happens when I run the agent locally and schedule a job:

[09:38:44.232] [ERROR] System.TimeoutException: The operation has timed out.
   at System.IO.Pipes.NamedPipeClientStream.ConnectInternal(Int32 timeout, CancellationToken cancellationToken, Int32 startTime)
   at System.IO.Pipes.NamedPipeClientStream.Connect(Int32 timeout)
   at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.GetTransport(Int32 processId) in C:\dev\git\perf\src\Microsoft.Benchmarks.Age
nt\Microsoft.Diagnostics.Tools.RuntimeClient\DiagnosticsIpc\IpcClient.cs:line 38
   at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.SendMessage(Int32 processId, IpcMessage message, IpcMessage& response) in C:\
dev\git\perf\src\Microsoft.Benchmarks.Agent\Microsoft.Diagnostics.Tools.RuntimeClient\DiagnosticsIpc\IpcClient.cs:line 104
   at Microsoft.Diagnostics.Tools.RuntimeClient.EventPipeClient.CollectTracing(Int32 processId, SessionConfiguration configuration, UInt64& sessionId)
 in C:\dev\git\perf\src\Microsoft.Benchmarks.Agent\Microsoft.Diagnostics.Tools.RuntimeClient\Eventing\EventPipeClient.cs:line 52
   at Microsoft.Benchmarks.Agent.Startup.<>c__DisplayClass85_0.<StartMeasurement>b__0() in C:\dev\git\perf\src\Microsoft.Benchmarks.Agent\Startup.cs:l
ine 3316

When running hello firewall prompt pupping up

Since crank appears to download everything into unique temp folder, and compile code there, it constantly ask for permissions to run hello.exe from unique location each time. See below.

C:\users\user\appdata\local\temp\benchmarks-agent\benchmarks-server-24280\fye32qrb.n4c\crank\samples\hello\published\hello.exe

Maybe docs can be expanded to provide solution for that. Or crank itself should open firewall rules for benchmarks.

Add support for gather metrics from the benchmark app using a the Prometheus format

Today we collect metrics from a job via the following mechanisms:

docker stats
Process stats
Event pipe via a custom BenchmarkEventSource
Console output using a custom JSON format

Prometheus is becoming a standard format and we can add support to the agent to support collecting job metrics using that format. See more here https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-format-example

The it would need to be configured on the as part of the job definition (the path, and the poll interval)

When the controller is interrupted in the initializing state, the server never stops

If the controller dies before the server starts the job it'll hang forever.

Scenarios names are case sensitive?

I just ran into this:

The scenario `mvcjson` was not found. Possible values: 'MvcJson', 'MvcJsonNet', 'MvcJson2k', 'MvcJsonNet2k', 'MvcJsonOutput60k', 'MvcJsonNetOutput60k', 'MvcJsonOutput2M', 'MvcJsonNetOutput2M', 'MvcJsonInput2k', 'MvcJsonNetInput2k', 'MvcJsonInput60k', 'MvcJsonNetInput60k', 'MvcJsonInput2M', 'MvcJsonNetInput2M'

Is it by design?

Enable GC analysis

Based on the flags for collection on perfview in this blog series
https://devblogs.microsoft.com/dotnet/work-flow-of-diagnosing-memory-performance-issues-part-0/

Chart gets skewed when trace takes a long time to download

Use consistent time units

@RobertHenry6bev

Some times are reported in ms, and other times in us, and rates in per-second. I'd pick either ms or us, and stick with per-second.

user un-friendliness reporting errors building code for a job

I have an error in a .cs file in a localFolder as from
jobs: server: source: localFolder

The server agent says, many times over, to stdout:
[04:58:25.776] Processing job 'application' (4) in state Starting

but, as far as I can tell, there's no Log message to stdout/stderr saying there was an error building the job.

AFAICT, the invoking instance of crank doesn't tell me anything either.

I have to go look at the job server status page, such as http://localhost:5010/jobs/4 to see the error, and even then that's tedious.

I'd like to see at least an error message saying there was an error, with a URL to the job server status page.

[[apologies if I filed some thing like this before. My first action when running new software is to iinject errors into the source files to see what happens.]]

Cannot build locally due to missing permissions

I was trying to build this repo locally on my machine using the provided build.sh script. That script delegates to eng/common/build.sh, but that file isn't marked as executable. I ran chmod +x build.sh from eng/common and that fixes the issue. I can send a PR for that change, but I'm not sure whether that's desirable. I believe that script is coming from arcade, but I'm not entirely sure. Should I sent a PR for this change?

Build failed

I version from master of both agent and controller.

Run agent
Run controller with hello world.
Task failed.

Agent log

[08:47:14.741] Benchmarked Application in C:\Users\user\AppData\Local\Temp\benchmarks-agent\benchmarks-server-27520\awxprvto.nua\crank\samples\hello
[08:47:14.743] Downloading build tools
[08:47:14.751] Installing dotnet runtimes and sdk
[08:47:14.782] Detected target framework: 'netcoreapp3.1'
[08:47:14.785] Patching project file with Framework References
[08:47:18.744] Runtime: 3.1.7 (Current)
[08:47:18.877] SDK: 3.1.401 (Current)
[08:47:18.974] ASP.NET: 3.1.7 (Current)
[08:47:19.022] Patching existing global.json file
[08:47:19.286] Desktop: 3.1.7 (Current)
[08:47:19.338] Installing SDK '3.1.401' ...
========
[08:57:04.728] Build is taking too long. Halting build.
========
[08:57:04.729] Starting -> Failed
[08:57:04.836] Driver stopping job '1'
[08:57:05.794] Processing job 'application' (1) in state Failed

Controller output

[08:46:56.619] Running session '60ae7a14d595429fa582cc96466bf546' with description ''
[08:46:58.590] Starting job 'application' ...
[08:46:59.703] Submitted job: http://localhost:5010/jobs/1
[08:47:03.978] Job has been selected by the server ...
[08:47:05.144] Job is now building ... http://localhost:5010/jobs/1/buildlog
[08:57:04.795] Job failed on benchmark server, stopping...
dotnet-install could not install a component: SDK '3.1.401'
[08:57:04.807] Job has failed, interrupting benchmarks ...
[08:57:04.827] Stopping job 'application' ...
[08:57:07.222] Deleting job 'application' ...

CPU measurement precision

Printing of benchmarks/cpu/raw prints far more digits of precision than the underlying system is capable of measuring.

Some error

I follow tutorial and install crank-agent and start it.

Then I launch crank --config samples/hello/hello.benchmarks.yml --scenario hello --profile local from powershell core

First time it produce results

crank --config samples/hello/hello.benchmarks.yml --scenario hello --profile local
[06:35:46.635] Running session 'c1d5e040be52474c97d0c1ca42b0d634' with description ''
[06:35:46.690] Starting job 'application' ...
[06:35:46.961] Fetching job: http://localhost:5010/jobs/1
[06:36:39.408] Job has been selected by the server ...
[06:36:39.413] Job is now building ...
[06:42:08.029] Job is running...
[06:42:08.034] Starting job 'load' ...
[06:42:08.040] Fetching job: http://localhost:5010/jobs/2
[06:42:10.069] Job has been selected by the server ...
[06:42:10.071] Job is now building ...
[06:42:23.175] Job is running...
[06:44:41.675] Stopping job 'load' ...
[06:44:42.760] Deleting job 'load' ...
[06:44:42.804] Stopping job 'application' ...
[06:44:43.828] Deleting job 'application' ...

application
-------

## Host Process:
CPU Usage (%):        5
Raw CPU Usage (%):    38.53
Working Set (MB):     52
Build Time (ms):      14,211
Start Time (ms):      4,408
Published Size (KB):  86,819

load
-------

## Host Process:
CPU Usage (%):        3
Raw CPU Usage (%):    20.17
Working Set (MB):     33
Build Time (ms):      5,223
Start Time (ms):      41
Published Size (KB):  68,190

## Benchmark:
First Request (ms):   306
Requests:             512
Bad responses:        512
Mean latency (us):    10,096,008
Max latency (us):     10,609,015
Requests/sec:         17
Requests/sec (max):   3,350

I notice that there 512 failed requests, and attempt to run again with same command.
I have only partial results

[07:14:33.811] Running session 'd56a13d2506345abbf5ec7a4d38ae2df' with description ''
[07:14:33.860] Starting job 'application' ...
[07:14:33.903] Fetching job: http://localhost:5010/jobs/7
[07:14:34.926] Job has been selected by the server ...
[07:14:34.930] Job is now building ...
[07:14:50.047] Job is running...
[07:14:50.052] Starting job 'load' ...
[07:14:50.055] Fetching job: http://localhost:5010/jobs/8
[07:14:52.071] Job has been selected by the server ...
[07:14:52.073] Job is now building ...
[07:15:05.216] Job is running...

Crank-agent has following logs

[07:14:34.640] Processing job 'application' (7) in state New
[07:14:34.641] Acquiring Job 'application' (7)
[07:14:34.641] Checking requirements...
[07:14:34.642] New -> Initializing
[07:14:35.645] Processing job 'application' (7) in state Waiting
[07:14:35.645] Starting job 'application' (7)
[07:14:35.645] Waiting -> Starting
[07:14:35.646] Created temp directory 'C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr'
[07:14:35.647] Cloning https://github.com/dotnet/crank with branch 'master'
[07:14:35.755] [STDERR] Cloning into 'crank'...
[07:14:38.506] Benchmarked Application in C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr\crank\samples\hello
[07:14:38.507] Downloading build tools
[07:14:38.507] Installing dotnet runtimes and sdk
[07:14:38.510] Detected target framework: 'netcoreapp3.1'
[07:14:38.511] Patching project file with Framework References
[07:14:41.288] Runtime: 3.1.6 (Current)
[07:14:41.288] SDK: 3.1.302 (Current)
[07:14:41.289] ASP.NET: 3.1.6 (Current)
[07:14:41.289] Patching existing global.json file
[07:14:41.292] Desktop: 3.1.6 (Current)
[07:14:41.293] Publishing application in C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr\crank\samples\hello\published with:
 publish hello.csproj -c Release -o C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr\crank\samples\hello\published /p:MicrosoftNETCoreAppPackageVersion=3.1.6 /p:MicrosoftWindowsDesktopAppPackageVersion=3.1.6 /p:MicrosoftAspNetCoreAppPackageVersion=3.1.6 /p:BenchmarksNETStandardImplicitPackageVersion=3.1.6 /p:BenchmarksNETCoreAppImplicitPackageVersion=3.1.6 /p:BenchmarksRuntimeFrameworkVersion=3.1.6 /p:BenchmarksTargetFramework=netcoreapp3.1 /p:BenchmarksAspNetCoreVersion=3.1.6 /p:MicrosoftAspNetCoreAllPackageVersion=3.1.6 /p:NETCoreAppMaximumVersion=99.9 /p:MicrosoftNETCoreApp31PackageVersion=3.1.6 /p:MicrosoftNETPlatformLibrary=Microsoft.NETCore.App /p:RestoreNoCache=true --framework netcoreapp3.1 --self-contained -r win-x64
[07:14:47.451] Application published successfully in 6156.6666 ms
[07:14:47.452] Patching hello.runtimeconfig.json
[07:14:47.454] Application is self-contained, skipping runtimeconfig.json
[07:14:47.455] Published size: 86819
[07:14:47.455] Invoking executable: C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr\crank\samples\hello\published\hello.exe
[07:14:47.456]   Arguments:
[07:14:47.456]   Working directory: C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr\crank\samples\hello\published
[07:14:47.499] Process started: 29640
[07:14:47.499] Starting measurement
[07:14:48.739] Processing job 'application' (7) in state Starting
[07:14:49.564] AspNetCore location: C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr\crank\samples\hello\published\Microsoft.AspNetCore.Hosting.Abstractions.dll
[07:14:49.564] AspNetCore version: 3.1.6
[07:14:49.564] NETCoreApp location: C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr\crank\samples\hello\published\System.Private.CoreLib.dll
[07:14:49.564] NETCoreApp version: 3.1.5-servicing.20269.1+018cfd06dceb19b6eb1e9217a500fb1071946fcd
[07:14:49.689] info: Microsoft.Hosting.Lifetime[0]
[07:14:49.689]       Now listening on: http://[::]:5000
[07:14:49.703] info: Microsoft.Hosting.Lifetime[0]
[07:14:49.703]       Application started. Press Ctrl+C to shut down.
[07:14:49.704] Running job 'application' (7)
[07:14:49.704] info: Microsoft.Hosting.Lifetime[0]
[07:14:49.704]       Hosting environment: Production
[07:14:49.705] info: Microsoft.Hosting.Lifetime[0]
[07:14:49.705]       Content root path: C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr\crank\samples\hello\published
[07:14:49.750] Processing job 'application' (7) in state Running
[07:14:50.755] Adding job 'load' (8) to group
[07:14:50.755] Processing job 'application' (7) in state Running
[07:14:51.765] Processing job 'load' (8) in state New
[07:14:51.765] Acquiring Job 'load' (8)
[07:14:51.765] Checking requirements...
[07:14:51.766] New -> Initializing
[07:14:52.770] Processing job 'application' (7) in state Running
[07:14:53.779] Processing job 'load' (8) in state Waiting
[07:14:53.779] Starting job 'load' (8)
[07:14:53.780] Waiting -> Starting
[07:14:53.780] Created temp directory 'C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\1hfikjk4.jua'
[07:14:53.781] Cloning https://github.com/dotnet/crank.git with branch 'master'
[07:14:53.885] [STDERR] Cloning into 'crank'...
[07:14:56.843] Benchmarked Application in C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\1hfikjk4.jua\crank\src\Microsoft.Crank.Jobs.Bombardier
[07:14:56.843] Downloading build tools
[07:14:57.156] Installing dotnet runtimes and sdk
[07:14:57.158] Detected target framework: 'netcoreapp3.1'
[07:14:57.158] Patching project file with Framework References
[07:15:00.233] Runtime: 3.1.6 (Current)
[07:15:00.234] SDK: 3.1.302 (Current)
[07:15:00.234] ASP.NET: 3.1.6 (Current)
[07:15:00.234] Patching existing global.json file
[07:15:00.238] Desktop: 3.1.6 (Current)
[07:15:00.239] Publishing application in C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\1hfikjk4.jua\crank\src\Microsoft.Crank.Jobs.Bombardier\published with:
 publish Microsoft.Crank.Jobs.Bombardier.csproj -c Release -o C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\1hfikjk4.jua\crank\src\Microsoft.Crank.Jobs.Bombardier\published /p:MicrosoftNETCoreAppPackageVersion=3.1.6 /p:MicrosoftWindowsDesktopAppPackageVersion=3.1.6 /p:MicrosoftAspNetCoreAppPackageVersion=3.1.6 /p:BenchmarksNETStandardImplicitPackageVersion=3.1.6 /p:BenchmarksNETCoreAppImplicitPackageVersion=3.1.6 /p:BenchmarksRuntimeFrameworkVersion=3.1.6 /p:BenchmarksTargetFramework=netcoreapp3.1 /p:BenchmarksAspNetCoreVersion=3.1.6 /p:MicrosoftAspNetCoreAllPackageVersion=3.1.6 /p:NETCoreAppMaximumVersion=99.9 /p:MicrosoftNETCoreApp31PackageVersion=3.1.6 /p:MicrosoftNETPlatformLibrary=Microsoft.NETCore.App /p:RestoreNoCache=true --framework netcoreapp3.1 --self-contained -r win-x64
[07:15:05.081] Application published successfully in 4841.0721 ms
[07:15:05.082] Patching Microsoft.Crank.Jobs.Bombardier.runtimeconfig.json
[07:15:05.083] Application is self-contained, skipping runtimeconfig.json
[07:15:05.084] Published size: 68190
[07:15:05.084] Invoking executable: C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\1hfikjk4.jua\crank\src\Microsoft.Crank.Jobs.Bombardier\published\Microsoft.Crank.Jobs.Bombardier.exe
[07:15:05.085]   Arguments:  -c 256 -w 0 -d 15 -n 0 --insecure -l  --fasthttp   http://localhost:5000/
[07:15:05.085]   Working directory: C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\1hfikjk4.jua\crank\src\Microsoft.Crank.Jobs.Bombardier\published
[07:15:05.120] Running job 'load' (8)
[07:15:05.120] Process started: 24904
[07:15:05.120] Starting measurement
[07:15:06.871] Processing job 'application' (7) in state Running
[07:15:07.881] Processing job 'load' (8) in state Running
[07:15:08.890] Processing job 'application' (7) in state Running
[07:15:08.943] Bombardier Client
[07:15:08.943] args: -c 256 -w 0 -d 15 -n 0 --insecure -l --fasthttp http://localhost:5000/
[07:15:09.269] Measuring first request ... 306 ms
[07:15:09.279] Downloading bombardier from https://github.com/codesenberg/bombardier/releases/download/v1.2.4/bombardier-windows-amd64.exe to bombardier-windows-amd64.exe
[07:15:09.906] Processing job 'load' (8) in state Running
[07:15:10.917] Processing job 'application' (7) in state Running
......
[07:15:48.231] Processing job 'load' (8) in state Running
[07:15:49.246] Processing job 'application' (7) in state Running
[07:15:49.731] > bombardier  -d 15s -c 256 --insecure -l --fasthttp http://localhost:5000/ --print r --format json
[07:15:50.261] Processing job 'load' (8) in state Running
[07:15:50.375] [STDERR] ##ChildProcessId:27176
[07:15:51.262] Processing job 'application' (7) in state Running
[07:15:52.263] Processing job 'load' (8) in state Running
[07:15:53.269] Processing job 'application' (7) in state Running
[07:15:54.278] Processing job 'load' (8) in state Running
[07:15:55.285] Processing job 'application' (7) in state Running
[07:15:56.285] Processing job 'load' (8) in state Running
[07:15:57.294] Processing job 'application' (7) in state Running
[07:15:58.299] Processing job 'load' (8) in state Running
[07:15:59.314] Processing job 'application' (7) in state Running
[07:15:59.576] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'application' (7).
[07:15:59.576] Running -> Stopping
[07:16:00.326] Processing job 'load' (8) in state Running
[07:16:00.528] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:00.529] Running -> Stopping
[07:16:00.590] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'application' (7).
[07:16:01.327] Processing job 'application' (7) in state Stopping
[07:16:01.327] Stopping job 'application' (7)
[07:16:01.328] Stopping measurement event pipes for job 'application' (7)
[07:16:01.544] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:02.559] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:03.580] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:04.592] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:05.606] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:06.621] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:07.626] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:08.636] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:09.653] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:10.669] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:10.994] {"spec":{"numberOfConnections":256,"testType":"timed","testDurationSeconds":15,"method":"GET","url":"http://localhost:5000/","body":"","stream":false,"timeoutSeconds":2,"client":"fasthttp"},"result":{"bytesRead":0,"bytesWritten":158720,"timeTakenSeconds":20.596004,"req1xx":0,"req2xx":0,"req3xx":0,"req4xx":0,"req5xx":0,"others":512,"errors":[{"description":"the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection","count":512}],"latency":{"mean":1.0088474890625e+07,"stddev":65479.310496470105,"max":1.0576999e+07,"percentiles":{"50":10066038,"75":10121002,"90":10144001,"95":10148002,"99":10568014}},"rps":{"mean":17.368926103888505,"stddev":195.21480631463487,"max":3375.717339934736,"percentiles":{"50":0.000000,"75":0.000000,"90":0.000000,"95":0.000000,"99":45.608400}}}}
[07:16:11.684] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:16:11.684] Process has exited (0)
[07:16:11.685] Stopping -> Stopped
....
[07:17:26.358] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:17:26.359] Process has exited (0)
[07:17:26.359] Stopped -> Stopped
[07:17:27.372] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:17:27.373] Process has exited (0)
[07:17:27.373] Stopped -> Stopped
[07:17:28.374] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:17:28.375] Process has exited (0)
[07:17:28.375] Stopped -> Stopped
[07:17:29.388] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:17:29.389] Process has exited (0)
[07:17:29.389] Stopped -> Stopped
[07:17:30.390] Process has exited (0)
[07:17:30.390] Stopped -> Stopped
[07:17:31.316] Forcing process to stop ...
[07:17:31.322] Waiting for process 29640 to stop ...
[07:17:31.399] Process has exited (0)
[07:17:31.399] Stopped -> Stopped
[07:17:32.327] Process has stopped
[07:17:32.327] Process stopped (Stopped)
[07:17:32.410] Process has exited (0)
[07:17:32.410] Stopped -> Stopped
[07:17:33.339] Processing job 'load' (8) in state Stopped
[07:17:33.339] Job 'load' (8) has stopped, waiting for the driver to delete it
[07:17:33.422] Process has exited (0)
[07:17:33.422] Stopped -> Stopped
[07:17:34.345] Processing job 'application' (7) in state Stopped
[07:17:34.345] Job 'application' (7) has stopped, waiting for the driver to delete it
[07:17:34.346] Driver didn't communicate for 00:00:10. Halting job.
[07:17:34.346] Stopped -> Deleting
[07:17:34.435] Process has exited (0)
[07:17:34.435] Stopped -> Stopped
[07:17:35.355] Processing job 'load' (8) in state Stopped
[07:17:35.355] Job 'load' (8) has stopped, waiting for the driver to delete it
[07:17:35.445] Process has exited (0)
[07:17:35.445] Stopped -> Stopped
[07:17:36.364] Processing job 'application' (7) in state Deleting
[07:17:36.365] Deleting job 'application' (7)
[07:17:36.365] Deleting directory 'C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\uutkasr1.mmr'
[07:17:36.459] Process has exited (0)
[07:17:36.459] Stopped -> Stopped
[07:17:36.961] SUCCESS
[07:17:36.962] Deleting -> Deleted
[07:17:37.462] Process has exited (0)
[07:17:37.462] Stopped -> Stopped
[07:17:37.968] Processing job 'load' (8) in state Stopped
[07:17:37.968] Job 'load' (8) has stopped, waiting for the driver to delete it
[07:17:38.476] Process has exited (0)
[07:17:38.476] Stopped -> Stopped
[07:17:38.969] Processing job 'application' (7) in state Deleted
[07:17:39.489] Process has exited (0)
[07:17:39.490] Stopped -> Stopped
[07:17:39.981] Processing job 'load' (8) in state Stopped
[07:17:39.982] Job 'load' (8) has stopped, waiting for the driver to delete it
[07:17:40.499] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:17:40.499] Process has exited (0)
[07:17:40.500] Stopped -> Stopped
[07:17:40.983] Processing job 'application' (7) in state Deleted
[07:17:41.511] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:17:41.512] Process has exited (0)
[07:17:41.512] Stopped -> Stopped
[07:17:41.995] Processing job 'load' (8) in state Stopped
[07:17:41.996] Job 'load' (8) has stopped, waiting for the driver to delete it
[07:17:41.996] Driver didn't communicate for 00:00:10. Halting job.
[07:17:41.996] Stopped -> Deleting
[07:17:42.527] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:17:42.528] Process has exited (0)
[07:17:43.011] Processing job 'application' (7) in state Deleted
[07:17:43.544] [Heartbeat] Driver didn't communicate for 00:00:10. Halting job 'load' (8).
[07:17:43.544] Process has exited (0)
[07:17:44.026] Processing job 'load' (8) in state Deleting
[07:17:44.026] Deleting job 'load' (8)
[07:17:44.026] Stopping measurement event pipes for job 'load' (8)
[07:17:44.026] Process stopped (Deleting)
[07:17:44.027] Deleting directory 'C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-24280\1hfikjk4.jua'
[07:17:44.486] SUCCESS
[07:17:44.486] Deleting -> Deleted
[07:17:45.497] All jobs in group are finished

Any way to troubleshoot?

Mono llvm-aot got the wrong runtime package

Plaintext Platform - Mono - LLVM AOT CI run failed with

Could not find file '/tmp/benchmarks-agent/benchmarks-server-1/RuntimePackages/microsoft.netcore.app.runtime.mono.llvm.aot.linux-x64.5.0.0-rtm.20509.6.nupkg'.)

https://dev.azure.com/dnceng/internal/_build/results?buildId=884581&view=logs&j=21c28b0f-457a-52ab-b995-50ff67ad2c92&t=c4539f32-788e-5b8b-370b-d3724d25e11e

[Proposal] Add envoyproxy/nighthawk's client as a load generator

Inspired by envoyproxy/envoy#13412, I took a look at this repo, and at the surface
it seems to me that it would take a relatively small effort to introduce a wrapper for using Nighthawk's
load gen client [1].

If there's interest in a pull request, I'd be up for contributing this.

Nighthawk's offers some options and knobs that may be nice to have (for example explicit connection re-use strategies, open-loop mode, a gRPC service for remote load test execution).

[1] https://github.com/envoyproxy/nighthawk

ARM64 host execution of crank agent builds x86_64 executable

I manually invoke the crank agent from source on an ARM64 host machine, following https://github.com/dotnet/crank/blob/master/docs/development.md

cd dotnet/crank/src/Microsoft.Crank.Agent
~/dotnet/dotnet run --hardware Physical --hardware-version "QEMU" --no-cleanup

This eventually fails. It builds and then tries to run an x86_64 binary which does not work on ARM64 hardware.

11:03:47.117] Invoking executable: /tmp/benchmarks-agent/benchmarks-server-2913/4wticrvn.z0w/src/published/hello
[11:03:47.117] Arguments:
[11:03:47.117] Working directory: /tmp/benchmarks-agent/benchmarks-server-2913/4wticrvn.z0w/src/published
[11:03:47.451] An unexpected error occurred while building the job. System.AggregateException: One or more errors occurred. (Exec format error)
---> System.ComponentModel.Win32Exception (8): Exec format error
at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)

Handle canceled messaged

When an Azdo taks is cancelled, and the message was already sent to the queue, dequeuing this message from the worker results in this exception:

Processing message '{MessageId:9b197b2d06b4434fbcaf9df4f52c6f9a}'
Job failed: System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.Crank.AzureDevOpsWorker.DevopsMessage.PostDataAsync(String url, String requestBody) in /_/src/Microsoft.Crank.AzureDevOpsWorker/DevopsMessage.cs:line 162
   at Microsoft.Crank.AzureDevOpsWorker.Program.<>c.<<ProcessAzureQueue>b__2_0>d.MoveNext() in /_/src/Microsoft.Crank.AzureDevOpsWorker/Program.cs:line 81
Failed to abandon task: System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.Crank.AzureDevOpsWorker.DevopsMessage.PostDataAsync(String url, String requestBody) in /_/src/Microsoft.Crank.AzureDevOpsWorker/DevopsMessage.cs:line 162
   at Microsoft.Crank.AzureDevOpsWorker.Program.<>c.<<ProcessAzureQueue>b__2_0>d.MoveNext() in /_/src/Microsoft.Crank.AzureDevOpsWorker/Program.cs:line 142
Processing message '{MessageId:9b197b2d06b4434fbcaf9df4f52c6f9a}'
Job failed: System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.Crank.AzureDevOpsWorker.DevopsMessage.PostDataAsync(String url, String requestBody) in /_/src/Microsoft.Crank.AzureDevOpsWorker/DevopsMessage.cs:line 162
   at Microsoft.Crank.AzureDevOpsWorker.Program.<>c.<<ProcessAzureQueue>b__2_0>d.MoveNext() in /_/src/Microsoft.Crank.AzureDevOpsWorker/Program.cs:line 81
Failed to abandon task: System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.Crank.AzureDevOpsWorker.DevopsMessage.PostDataAsync(String url, String requestBody) in /_/src/Microsoft.Crank.AzureDevOpsWorker/DevopsMessage.cs:line 162
   at Microsoft.Crank.AzureDevOpsWorker.Program.<>c.<<ProcessAzureQueue>b__2_0>d.MoveNext() in /_/src/Microsoft.Crank.AzureDevOpsWorker/Program.cs:line 142

It is then ignored, but this exception should be handled and not throw.

Project jobs should use project settings by default

Today the benchmark-agent will download the latest .NET Core runtime instead of using what the project file specifies. The defaults should be flipped and the project file's target framework should be used by default.

Resolve relative paths in --bodyFile based on the location relative to the yml file.

I was playing with this today.

Had an action defined as follows:

    # Add product
  ApiCrudAddProduct:
    application:
      job: mvcServer
    load:
      job: bombardier
      variables:
        path: /Products
        verb: POST
        bodyFile: ./crudproductbody.json

and I kept getting errors because the file couldn't be found.

Upon investigation, it turned out that it was passing the value verbatim to bombardier, which didn't run in the same context.

Is it possible to resolve the file path relative to the location of the yml file?

When wrk job returns non-zero exit code, it should automagically show the output

I had to run with showOutput to see the following

[load] args: -c 10 http://10.0.0.102:5000/file.txt --latency -d 15s -w 0s -t 32
[load] Measuring first request ... 101 ms
[07:53:58.088] Job is running...
[load] > wrk -d 15s -c 10 http://10.0.0.102:5000/file.txt --latency -t 32
[load] [STDERR] number of connections must be >= threads
[load] [STDERR] ##ChildProcessId:10614
[load] Usage: wrk <options> <url>
[load]   Options:
[load]     -c, --connections <N>  Connections to keep open
[load]     -d, --duration    <T>  Duration of test
[load]     -t, --threads     <N>  Number of threads to use
[load]
[load]     -s, --script      <S>  Load Lua script file
[load]     -H, --header      <H>  Add header to request
[load]         --latency          Print latency statistics
[load]         --timeout     <T>  Socket/request timeout
[load]     -v, --version          Print version details
[load]
[load]   Numeric arguments may include a SI unit (1k, 1M, 1G)
[load]   Time arguments may include a time unit (2s, 2m, 2h)
[load] Failed to parse latency
[load] Failed to parse latency
[load] Failed to parse requests
[load] Failed to parse latency
[load] Failed to parse latency
[load] Failed to parse latency
[load] Failed to parse latency

multiple crank-agents

It seems that one can have multiple crank-agent running at the same time. Is this a good idea? Each one is listening on its own port, sequentially assigned.

Is there a log file or pid file that I can consult to see if crank-agent is up and running?

Differentiate between job definition and job status

Today the properties on job are a mix of the requested information and the state of the job (process id etc). We should split these properties into sections so it's clear what things were originally part of the request and what things are the current status.

buglets with documentation of units in results.json

The results.json file contains, after a fashion, documentation on all of the measured quantities. Phew!

The documentation is in places inconsistent or incomplete.

Printing of benchmarks/cpu/raw prints far more digits of precision than the underlying system is capable of measuring.

Some times are reported in ms, and other times in us, and rates in per-second. I'd pick either ms or us, and stick with per-second.

Pedantic: if you report in us, perhaps use UTF8 mu symbol rather than Latin u.

Many, but not all, longDescriptions have units. All should have units, replicating what's already in shortDescription. See benchmarks/build-time, benchmarks/start-time, benchmarks/published-size where the units are omitted.

Some shortDescriptions use '(ms)' but the longDescription says 'in ms'. Use the style '(...)' uniformly.

Are sizes (such as benchmarks/swap) reported in SI Mega (1e6) or Mibi (1024*1024) units. This may be tricky.

The "format" is always given as "n0". I had to look n0 format up. That's irrelevant, since this is json, and json doesn't support using ',' to group digits. I don't see the point in reporting "format".

What happens if the number being reported is mis-computed upstream as Inf or NaN. What happens when it is printed in the json file? (JSON doesn't support Inf or NaN...)

Validate benchmark manifest in the controller

Today invalid configuration show random exceptions in the code base. Once that succeeds, it's possible to have an invalid configuration model and we don't have much validation in place today (besides very basic required fields, like the name of services).

We need to do both input validation and model validation:

Input validation - Making sure the configuration schema is correct (type checking).
Model validation - Making sure the configuration is valid after it is correctly parsed.

Exception when application fails

When application failed to start.
The command should have stopped without exception.

Note: It was with --iterations 3

[09:13:05.929] Job has been selected by the server ...
[09:13:05.962] Job is now building ... http://asp-perf-lin:5001/jobs/2/buildlog
[09:16:14.002] Job failed on benchmark server, stopping...
Job didn't start during the expected delay. Check that it outputs a startup message on the log.
[09:16:14.075] Job has failed, interrupting benchmarks ...

| application           |                         |
| --------------------- | ----------------------- |
| Build Time (ms)       | 3,717                   |
| Published Size (KB)   | 87,776                  |
| .NET Core SDK Version | 6.0.100-alpha.1.20568.5 |

Unhandled exception. System.Collections.Generic.KeyNotFoundException: The given key 'load' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Microsoft.Crank.Controller.Program.Run(Configuration configuration, String[] dependencies, String session, Int32 iterations, Int32 exclude, TimeSpan span, IEnumerable`1 scripts) in /_/src/Microsoft.Crank.Controller/Program.cs:line 750
   at Microsoft.Crank.Controller.Program.<>c__DisplayClass39_0.<<Main>b__1>d.MoveNext() in /_/src/Microsoft.Crank.Controller/Program.cs:line 414
--- End of stack trace from previous location where exception was thrown ---
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Microsoft.Crank.Controller.Program.Main(String[] args)

MVP checklist

Split repository
- aspnet/benchmarks
  - We put the benchmark apps here
- aspnet/perf
  - Infrastructure for benchmarks (driver etc)
CI/CD
- Testing
  - Infrastructure
    - Start two local agents
    - Docker image and csproj
  - Apps
    - Each app should be tested
- Docker image for Agent
- Global tools (controller + agent)
Documentation for the infrastructure
- README
  - New driver only
  - docs, samples
Refactoring
- Naming
  - Benchmarks Agent
  - Benchmarks Controller
  - Benchmarks Relay
- Simpler arguments for Agent when running locally
  - No arguments necessary
  - Automatic available port detection
Plugins
- Wrk plugin
- Bombardier plugin

Override temp folder

Can some configuration option for temporary folder where everything would be stored?

I would like to remove pressure from disk C if possible.

Specifying OutputFiles also includes ref dll which causes issues

crank someArgs --application.outputFiles path/net5.0/a.dll

Both a.dll and ref/a.dll are uploaded and it looks like ref/a.dll gets written last and wins so the application tries to load a ref assembly and fails. Deleting the ref folder is the current workaround.

Zip nettrace files for download

We should zip the nettrace files and unzip them automagically on download to reduce transfer size. Taking a verbose gc trace > 15 seconds is death 😄

Crank Announcements

This issue contains status updates from the Crank team to provide insight into future plans, what the team is working on, and highlight helpful work done by the community.

If you want a low-spam way of tracking what's going on with Crank, then you should subscribe to this issue.

Comments are disabled on this issue to reduce noise. If you have questions or want to discuss something, please open a new issue and tell us what's on your mind.

local mode looses error messages and loops forever

Following https://github.com/dotnet/crank/blob/master/docs/local_application.md I malevolently put a syntax error into samples/hello/Program.cs and run the crank. (I am not using code to edit c#, but plain old vi.)

I don't see where the error messages end up. I can't find a log file. The error messages don't seem to come out on stderr.
The crank engine spits out this string once per second, for 3 minutes, before complaining that the job didn't start.

[09:51:30.228] Processing job 'application' (9) in state Starting

Compare numbers are off

| Gen 0 GC (#/s)           |            0 |          0 |   +4.17% |
| Gen 1 GC (#/s)           |            0 |          0 | +212.50% |
| Gen 2 GC (#/s)           |            0 |          0 |      +∞% |
| Time in GC (%)           |            0 |          0 |          |

Seems like the % difference between 0 and 0 can vary 😄

job server json output should be pretty printed

The json served by, for example, http://localhost:5010/Jobs/4/ should be pretty printed to make it easier to read or grep.

Use metric in descriptions

@RobertHenry6bev

Many, but not all, longDescriptions have units. All should have units, replicating what's already in shortDescription. See benchmarks/build-time, benchmarks/start-time, benchmarks/published-size where the units are omitted.

Support the ability to collect custom event counters

We should support specifying a set of event sources to collect counters from for a job.

Exec format error with bombardier job

I've been trying out crank on my machine to do some benchmarks on a project. I used the sample from the docs and then tweaked it slightly to match my scenario, but I'm only seeing a single request going to my app. Looking at the crank-agent logs I noticed this:

[09:02:18.802] > bombardier  -d 15s -c 256 --insecure -l --fasthttp http://localhost:5000/realization --print r --format json
[09:02:19.520] [STDERR] Unhandled exception. System.ComponentModel.Win32Exception (8): Exec format error
[09:02:19.520] [STDERR]    at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
[09:02:19.520] [STDERR]    at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
[09:02:19.520] [STDERR]    at System.Diagnostics.Process.Start()
[09:02:19.520] [STDERR]    at Microsoft.Crank.Jobs.Bombardier.Program.Main(String[] args) in /private/var/folders/9n/ghtcrrtx7pn2jdd7fjxf33580000gn/T/benchmarks-agent/benchmarks-server-23139/tyoiw5zr.2iz/crank/src/Microsoft.Crank.Jobs.Bombardier/Program.cs:line 131
[09:02:19.520] [STDERR]    at Microsoft.Crank.Jobs.Bombardier.Program.<Main>(String[] args)

Any idea what's going wrong here? It almost looks like the bombardier executable cannot be run on my machine. I'm running on a Mac, so maybe that's the case. Is that not supported?

Real world example

It would be nice to have some real world example - with separate agents for each job and how to wire those in complex scenarios.

Support deploying published applications as jobs

Today jobs need to be built from source by the agent, we should support a mode where the controller does a dotnet publish and pushes those assets to the server side. This means we also need a way for jobs to define their entrypoint (or have that as part of a package)

Now way to pass custom headers to load generators

Something like customHeaders instead of having to create a new entry in headers and set it for presetHeaders.
The difficulty is with not having to define --headers and also being able to use multiple values

Clean up command line arguments for controller and agent

Do a pass, make sure they are sane.

Cannot relaunch after failure

After I experience #89 the hello task does not work anymore. I complains about address in use. "Failed to bind to address http://[::]:5000: address already in use."

Not sure that this is crank issue, or hello example.

crank --config samples/hello/hello.benchmarks.yml --scenario hello --profile local --application.options.displayOutput true
[08:24:59.452] Running session '314abc03af4a432f88872e712807d495' with description ''
[08:24:59.516] Starting job 'application' ...
[08:24:59.567] Fetching job: http://localhost:5010/jobs/3
[08:25:00.593] Job has been selected by the server ...
[08:25:00.596] Job is now building ...
[application] AspNetCore location: C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-3504\bxtapxpz.wjm\crank\samples\hello\published\Microsoft.AspNetCore.Hosting.Abstractions.dll
[application] AspNetCore version: 3.1.6
[application] NETCoreApp location: C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-3504\bxtapxpz.wjm\crank\samples\hello\published\System.Private.CoreLib.dll
[application] NETCoreApp version: 3.1.5-servicing.20269.1+018cfd06dceb19b6eb1e9217a500fb1071946fcd
[application] crit: Microsoft.AspNetCore.Server.Kestrel[0]
[application]       Unable to start Kestrel.
[application] System.IO.IOException: Failed to bind to address http://[::]:5000: address already in use.
[application]  ---> Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted.
[application]  ---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.
[application]    at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
[application]    at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
[application]    at System.Net.Sockets.Socket.Bind(EndPoint localEP)
[application]    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
[application]    --- End of inner exception stack trace ---
[application]    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
[application]    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
[application]    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass21_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
[application] --- End of stack trace from previous location where exception was thrown ---
[application]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
[application]    --- End of inner exception stack trace ---
[application]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
[application]    at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context)
[application]    at Microsoft.AspNetCore.Server.Kestrel.Core.AnyIPListenOptions.BindAsync(AddressBindContext context)
[application]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context)
[application]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IServerAddressesFeature addresses, KestrelServerOptions serverOptions, ILogger logger, Func`2 createBinding)
[application]    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
[application] [STDERR] Unhandled exception. System.IO.IOException: Failed to bind to address http://[::]:5000: address already in use.
[application] [STDERR]  ---> Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted.
[application] [STDERR]  ---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.
[application] [STDERR]    at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
[application] [STDERR]    at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
[application] [STDERR]    at System.Net.Sockets.Socket.Bind(EndPoint localEP)
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
[application] [STDERR]    --- End of inner exception stack trace ---
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass21_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
[application] [STDERR] --- End of stack trace from previous location where exception was thrown ---
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
[application] [STDERR]    --- End of inner exception stack trace ---
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context)
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Core.AnyIPListenOptions.BindAsync(AddressBindContext context)
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context)
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IServerAddressesFeature addresses, KestrelServerOptions serverOptions, ILogger logger, Func`2 createBinding)
[application] [STDERR]    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
[application] [STDERR]    at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
[application] [STDERR]    at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
[application] [STDERR]    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
[application] [STDERR]    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
[application] [STDERR]    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
[application] [STDERR]    at hello.Program.Main(String[] args) in C:\Users\kant\AppData\Local\Temp\benchmarks-agent\benchmarks-server-3504\bxtapxpz.wjm\crank\samples\hello\Program.cs:line 21
[08:25:21.858] Job failed on benchmark server, stopping...

[08:25:21.861] Job has failed, interrupting benchmarks ...
[08:25:21.870] Stopping job 'application' ...
[08:25:23.185] Deleting job 'application' ...

application
-------

## Host Process:
CPU Usage (%):        2
Raw CPU Usage (%):    16.90
Working Set (MB):     34
Build Time (ms):      9,150
Published Size (KB):  86,819

Agent crashed

I run crank-agent from master.

I run crank job and by mistake press Enter instead of click on Allow. This trigger cancelling creation for firewall rule.
crank failed, but agent crashed as well too.

[08:10:49.313] {"spec":{"numberOfConnections":256,"testType":"timed","testDurationSeconds":15,"method":"GET","url":"http://localhost:5000/","body":"","stream":false,"timeoutSeconds":2,"client":"fasthttp"},"result":{"bytesRead":0,"bytesWritten":158720,"timeTakenSeconds":20.5850021,"req1xx":0,"req2xx":0,"req3xx":0,"req4xx":0,"req5xx":0,"others":512,"errors":[{"description":"the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection","count":512}],"latency":{"mean":1.006327409375e+07,"stddev":48776.50049402562,"max":1.0570021e+07,"percentiles":{"50":10047996,"75":10077999,"90":10107996,"95":10116000,"99":10566004}},"rps":{"mean":17.133376015997403,"stddev":203.08256544870855,"max":3473.336876838632,"percentiles":{"50":0.000000,"75":0.000000,"90":0.000000,"95":0.000000,"99":99.982503}}}}
Unhandled exception. System.InvalidOperationException: Process has exited, so the requested information is not available.
   at System.Diagnostics.Process.EnsureState(State state)
   at System.Diagnostics.Process.get_WorkingSet64()
   at Microsoft.Crank.Agent.Startup.<>c__DisplayClass60_5.<ProcessJobs>b__16(Object _) in c:\GitHub\dotnet\crank\src\Microsoft.Crank.Agent\Startup.cs:line 870
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()

Handle flaky network with uploads

When network over VPN is spotty, file uploads can fail and the error message might not be representative a bout the issue.
It might also be possible to detect that /touch was unsuccessful and retry it to keep the server job alive

[08:30:42.210] Uploading /home/roji/projects/npgsql/src/Npgsql/bin/Release/net5.0/Npgsql.dll
[08:30:43.416] GET http://asp-citrine-db:5001/jobs/175/touch...
[08:30:44.758] GET http://asp-citrine-lin:5001/jobs/255/touch...
[08:30:49.796] GET http://asp-citrine-db:5001/jobs/175/touch...
[08:30:49.945] GET http://asp-citrine-lin:5001/jobs/255/touch...
[08:30:58.509] GET http://asp-citrine-db:5001/jobs/175/touch...
[08:30:58.514] GET http://asp-citrine-lin:5001/jobs/255/touch...
[08:31:12.328] GET http://asp-citrine-db:5001/jobs/175/touch...
[08:31:12.328] GET http://asp-citrine-lin:5001/jobs/255/touch...
[08:31:24.748] GET http://asp-citrine-lin:5001/jobs/255/touch...
[08:31:24.748] GET http://asp-citrine-db:5001/jobs/175/touch...
[08:31:28.954] GET http://asp-citrine-lin:5001/jobs/255/touch...
[08:31:28.954] GET http://asp-citrine-db:5001/jobs/175/touch...
[08:31:40.513] 500 InternalServerError
Unhandled exception. System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.Crank.Controller.JobConnection.StartAsync(String jobName) in /_/src/Microsoft.Crank.Controller/JobConnection.cs:line 297

Add script hook on results

The goal is to be able to generate more results (and metadata?) from the measurements.
For instance compute the P95 of the GC metrics, which is based on an existing measure. Or render more than just one aggregating operation (max) without having to hard code all the potential variations.

This can be done by adding a new section that will contain some javascript to alter the results json payload. These scripts can be shared as they are part of the configuration files. The command line will then allow to run specific scripts to postprocess the data.

/cc @MarioszMSFT

All build settings should be moved to the Job.Source

This is because of organic growth but all of the settings related to building a project should be hoisted to the source part of the job.