jaredpar / runfo Goto Github PK

Runfo Creating Tracking Issue (data being generated)

The most common search filter today is for a build definition yet in order to execute that filter we have to join the ModelBuild with the ModelBuildDefinition table. That is proving to be expensive when it's nested within other joins (like filtering ModelBuildTestResults on definitions). Experiments show perf can improve significantly if we can just lookup the definition info in the ModelBuild table directly.

The reason it is not there today is that definitions can change names. The only part of a definition that is persistent is it's id. Yet users want to query by name because that is what we know. Hence I decided to centralize the name and make it easy to update.

Definition name change is extremely rare though. Really doesn't happen after the first day or two of creation. I think the better balance here is putting the definition info into the build and handle the rare case where the name change happens when it occurs.

https://runfo.azurewebsites.net/view/build/?number=843631

That build actually had three attempts but the runfo UI is only recording two. The last attempt was successful but not recorded.

Runfo Creating Tracking Issue

Today runfo will only issue an error about missing PAT tokens when running test based commands. That's because tests are the only place where tokens are absolutely required. For pretty much everything else tokens are not required if the underlying projects are public.

That being said the error messages produced when the user digs into internal projects are really not helpful. Also there are pressingly few searches which don't dive into either tests are internal projects. Much simpler to just require a PAT token all the time.

Runfo Creating Tracking Issue (data being generated)

Runfo Creating Tracking Issue

Runfo Creating Tracking Issue

Runfo Tracking Issue: OSX deprovision

Build Result Summary

Need to support a syntax for filtering builds by queue time: queued:2020-09-16. This is a more reliable metric than started time which can change during retries. This requires both the model to be updated as well as the search syntax to be added.

Essentially be able to seacrh for failures , passes, etc ...

The build log search does a fresh search every time. That means it re-downloads the timeline and re-downloads all of the log files. Consider caching that to improve successive search performance.

This variable represents when GitHub auth was being done with PAT. Now that it's using a client app there are better ways to do this.

It may need to stay in scratch for a while so I can test better locally but that's it.

These builds are returning full and partial timeline histories. Need to figure out the merge strategy here:

Error getting timeline for https://dev.azure.com/dnceng/internal/_build/results?buildId=831543: Multiple previous timelines with at least one full
Error getting timeline for https://dev.azure.com/dnceng/internal/_build/results?buildId=830088: Multiple previous timelines with at least one full
Error getting timeline for https://dev.azure.com/dnceng/internal/_build/results?buildId=822962: Response status code does not indicate success: 503 (Service Unavailable).
Error getting timeline for https://dev.azure.com/dnceng/internal/_build/results?buildId=818952: Multiple previous timelines with at least one full
Error getting timeline for https://dev.azure.com/dnceng/internal/_build/results?buildId=816681: Multiple previous timelines with at least one full

e.g. when build 791688 was mentioned by the runfo tool in dotnet/roslyn#47231, it would be nice to add a tag dotnet/roslyn#47231 to the build itself so users can see the cross link. Here is a screenshot following manual addition of this tag.

Runfo Creating Tracking Issue

Runfo Creating Tracking Issue

Can't easily search for new failures of issues until we can filter by the test run / job name

dotnet/runtime#37896

The current test failure page makes it difficult to see which failures should have been covered by a fix and which ones were simply on an older branch. It would help if the tests page on the website and filed issues contain two more columns:

• Name of Target Branch for the pull request (or rolling build)
• Commit hash which was the head commit of the target branch at the time of the build

Runfo Creating Tracking Issue

Runfo Tracking Issue: testsnapshottranslateto

Build Result Summary

Build Result Summary

Runfo Creating Tracking Issue (data being generated)

Runfo Creating Tracking Issue

Overview

Please use this queries to discover issues

• Blocking CI
• Blocking Official Build
• Blocking CI Optional
• Blocking Outerloop

Blocking CI

Blocking Official Build

Blocking CI Optional

Blocking Outerloop

Goals

1. A minimum 95% passing rate for the runtime pipeline

Resources

1. runtime pipeline analytics

Build Result Summary

Runfo Tracking Issue: Agents canceling during Roslyn build

Build Result Summary

As more people use this, we'll have more duplicate issues like dotnet/roslyn#47233. When an issue already exists for a test, we should link to it instead of creating a new one.

Runfo Creating Tracking Issue

Build Result Summary

It would be nice if we could include dumps in output table when available in the attachments for search-helix action.

cc: @jaredpar

Need total hit count, week hit count and month hit count.

Need to support a filter syntax like branch:master in the build queries. This can't be done today because the information isn't being recorded in the DB model. Need to update that model and then add the query syntax into the website.

Runfo Creating Tracking Issue (data being generated)

Issue used for testing. Please don't update

Most recent failure 4/27/2020 6:44:00 AM

Today you need to join a ModelBuildDefinition to a ModelBuild in order to get basic information like the URI of the build. This is excessive. Should adjust the ModelBuild so that it maintains the organization and project thus making it possible to get BuildInfo without a join.

Build Result Summary

Right now the -pr option just includes pull requests in the results, it doesn't include only pull requests. Need a way to filter to just pull requests.

When passing down an invalid option to the action, the tool prints out the usage summary for that action and the error is at the end which can be lost in the cmd. I didn't notice that was the error case until I was reading the usage a couple times.

Maybe add coloring, or print the error summary at the top of the output and then the usage.
Example:

  -d, --definition=VALUE     build definition (name|id)(:project)?
  -p, --project=VALUE        default project to search (public)
  -c, --count=VALUE          count of builds to show for a definition
      --pr                   include pull requests
      --before=VALUE         filter to builds before this date
      --after=VALUE          filter to builds after this date
  -r, --repository=VALUE     filter to repository
      --br, --branch=VALUE   filter to builds against this branch
  -b, --build=VALUE          build id to print tests for
  -e, --exclude=VALUE        exclude build ids from the results
  -g, --grouping=VALUE       output grouping: tests*, builds, jobs
  -m, --markdown             output in markdown
  -n, --name=VALUE           name regex to match in results
  -v, --verbose              verobes output
Extra arguments: exit code -1073740791

Probably would be better to have something like:

ERROR: Unexpected arguments: [list of unexpected args] (red color)

Usage:
-d, --definition=VALUE     build definition (name|id)(:project)?
  -p, --project=VALUE        default project to search (public)
  -c, --count=VALUE          count of builds to show for a definition
      --pr                   include pull requests
      --before=VALUE         filter to builds before this date
      --after=VALUE          filter to builds after this date
  -r, --repository=VALUE     filter to repository
      --br, --branch=VALUE   filter to builds against this branch
  -b, --build=VALUE          build id to print tests for
  -e, --exclude=VALUE        exclude build ids from the results
  -g, --grouping=VALUE       output grouping: tests*, builds, jobs
  -m, --markdown             output in markdown
  -n, --name=VALUE           name regex to match in results
  -v, --verbose              verobes output

cc: @jaredpar

Context: dotnet/roslyn#47231

The code to extract GitHub repository names doesn't work for rolling builds. It's only working for pull request builds.

Right now the Repository Info for builds is hard coded to GitHub. Need to support the following in the Repository section of a Build

   "repository":{
      "id":"a2f9a77f-0d37-4eaf-aecc-5b3ff7457ad6",
      "type":"TfsGit",
      "name":"dotnet-runtime",
      "url":"https://dev.azure.com/dnceng/internal/_git/dotnet-runtime",
      "clean":null,
      "checkoutSubmodules":false

Runfo Creating Tracking Issue (data being generated)