Arcade Engineering Services
License: MIT License
This repo is home of the services that help us construct .NET. Mainly, you can find the Maestro dependency flow system and the Darc CLI tool.
See instructions on how to debug Maestro here.
We welcome contributions! Please follow the Code of Conduct.
This repo should contain issues that are tied to Maestro and Darc.
.NET (including this repo) is licensed under the MIT license.
I've disabled this for now to unblock a prod rollout. Stable versions cannot be generated because of prerelease Microsoft.TeamFoundationServer.Client dependencies.
If you add a dependency to a repo using darc add...
> darc add -n Microsoft.DotNet.PlatformAbstractions -t toolset -r https://github.com/dotnet/core-setup
... it creates an entry in $(RepositoryEngineeringDir)Versions.props...
<PropertyGroup>
...
<MicrosoftDotNetPlatformAbstractionsVersion></MicrosoftDotNetPlatformAbstractionsVersion>
</PropertyGroup>If the repo already had an entry, there are now two entries, with the latter entry being empty...
<PropertyGroup>
<MicrosoftDotNetPlatformAbstractionsVersion>2.1.0</MicrosoftDotNetPlatformAbstractionsVersion>
...
<MicrosoftDotNetPlatformAbstractionsVersion></MicrosoftDotNetPlatformAbstractionsVersion>
</PropertyGroup>Now, when you run darc update-dependencies, you will always update the first entry but the value will resolve as empty (no version) because of the second entry.
@dougbu commented on Wed Jul 03 2019
darc get-subscriptions confusingly shows non-empty policies for batchable subscriptions. If this command is going to display repository policies, it should be clear the policies aren't actually part of the subscription definition. An alternative would be to show no policies or (perhaps best) to display just an indication that repo policies exist for the target branch.
For example,
darc get-subscriptions --target-repo https://github.com/aspnet/AspNetCore$ --regex --target-branch master --channel '.NET Tools - Latest'
https://github.com/dotnet/arcade (.NET Tools - Latest) ==> 'https://github.com/aspnet/AspNetCore' ('master')
- Id: 4409590c-0c9e-4f51-7dbf-08d6ba16a224
- Update Frequency: EveryWeek
- Enabled: True
- Batchable: True
- Merge Policies:
AllChecksSuccessful
ignoreChecks =
[
"AspNetCore-helix-test",
"license/cla"
]
NoRequestedChanges
darc get-repository-policies --repo aspnet/AspNetCore --branch masterhttps://github.com/aspnet/AspNetCore @ master
- Merge Policies:
AllChecksSuccessful
ignoreChecks =
[
"AspNetCore-helix-test",
"license/cla"
]
NoRequestedChanges
https://github.com/aspnet/AspNetCore-Tooling @ master
- Merge Policies:
Standard
@garath commented on Thu Jul 11 2019
@mmitche @jcagme Where's the best place to track this?
@mmitche commented on Thu Jul 11 2019
I this repo is fine. Need to create a p8 or GA epic for dependency flow to put this in.
Suggestion from @mmitche here: dotnet/arcade#562 (comment)
Currently Darc outputs the results of some commands in YAML. There has been several request, including build from source, to change the output to JSON.
FYI: @mmitche @chcosta @crummel
Rough stab at what could easily-ish be supported by json:
When Maestro creates a PR, the only reviewers added are the default ones in the target repo (like CODEOWNERS in github, for example).
This has a couple drawbacks:
Therefore, it would be nice if Maestro could add reviewers when the PR is created.
This work probably requires two parts:
This needs to work in both github and AzDO.
Is this planned? For example, anonymously seeing the latest core-setup build with this URL:
Right now, I get a 401 anonymously, and the results I want if I provide my auth.
In particular, Tizen builds used to be triggered by changes on dotnet/versions. How can we accomplish the same with BAR with dotnet/versions publishing turned off?
If you assign a build to the same channel twice, the second time will fail. This is just a no-op and shouldn't fail.
Subscription history says useful things like "Checking merge policy for XXXXX". In cases where merge polciies fail, we need more info to be presented in the API. For example:
At release time, expiration date needs to be extended from 180 days to 10950 days (30 years). In earlier infra, this was done by publishing the same symbol packages again with the longer expiration date.
How is this done when some bits aren't in the symbol packages?
I imagine you could by downloading all the bits and pushing them back up, but how is that done in a release context? Is there tooling?
It looks, cursorily, like darc makes use of whatever dotnet is on the machine.
D:\demo\hello-demo>darc
A fatal error occurred, the required library hostfxr.dll could not be found.
If this is a self-contained application, that library should exist in [C:\Users\chcosta\.dotnet\tools\.store\microsoft.dotnet.darc\1.0.0-beta.18517.1\microsoft.dotnet.darc\1.0.0-beta.18517.1\tools\netcoreapp2.1\any\].
If this is a framework-dependent application, install the runtime in the default location [C:\Program Files\dotnet] or use the DOTNET_ROOT environment variable to specify the runtime location.
Arcade is supposed to be fully bootstrappable, and installs a local copy of the cli under the .\.dotnet\ folder. Yes, Arcade will actually use the machine wide install of the CLI if available, but I think there's lots of scenarios where a dev might not have / want one of those. This is useful...
If this is a framework-dependent application, install the runtime in the default location [C:\Program Files\dotnet] or use the DOTNET_ROOT environment variable to specify the runtime location.
… but it would be nice if, on a clean machine (with no cli installed machine-wide), if you just restored darc and it started working.
While working on refactoring some post-build PS scripts - see this folder I noticed that I had seen similar routines on other places (for instance). Wouldn't it make sense to create a package with PS/"Bash" routines to interface with Darc / MaestroAPI?
We agreed during the discussion around dotnet/arcade#1059 that verifying the files in eng/common to be "untampered" with is important.
The update and add dependencies operations on the git file manager use different handling for:
This should be consistent.
In addition, this is a good time to change how we specify 'well known' types. These should be able to be specified in the Version.details.xml file. So darc would read that and choose an updater based on the type. The default should be Versions.props.
Enable an audit trail of changes made to various tables in the DB. Track:
Should be able to query on branch (github or azdo).
Without this build lookup in Darc is hard.
Appears to be a bug introduced with the long path fix - the exception for the commit not being found is being treated as a long path exception. Consequently, we never try to fetch new commits.
Workaround: delete the "master" copy of the repos (.git/modules in source-build). Then everything will be recloned and you'll get all the commits.
Assigning to myself so I fix this properly.
According to @ViktorHofer the instructions detailed in https://github.com/dotnet/core-setup/blob/master/Documentation/project-docs/how-to-track-changes.md are out of date. Could you possibly update them? cc @ericstj
Issues that are not a current business priority, but are still worth considering for publishing and dependency flow infrastructure.
All issues in this section should be triaged by the v-team into one of their business objectives or features.
@dougbu commented on Wed Jul 03 2019
Today, "repository merge policies" are actually branch-specific. At least for ASP.NET Core, we have no reason to vary the policies per-branch. Suggest making the --branch option of the darc set-repository-policies command optional. If that can't be done for some reason, fall back to supporting branch wildcards e.g. (for our case) --branch *.
For example,
darc get-repository-policies --repo aspnet/aspdoes not list a repo policy set that I just added using
darc set-repository-policies --repo https://github.com/aspnet/AspNetCore --all-checks-passed --no-requested-changes --quiet --ignore-checks AspNetCore-helix-test, license/cla --branch *I suspect this means the --branch * policy set will match no branch instead of all.
Enable bulk editing of subscriptions
After some review / discussions about these issues: dotnet/arcade#167, dotnet/arcade#173, dotnet/arcade#1874 it was clear that issue dotnet/arcade#1874 encompasses both dotnet/arcade#167 and dotnet/arcade#173. Moreover, it was decided that we would use symbol-tool to perform validation and availability check of symbols.
A temporary solution for both of these problems was provided in this PR: dotnet/arcade#2492 . However, there are a few things that the script in that PR intentionally doesn't do and that we should make sure symbol tool does. Moreover, there are a few things we should implement in symbol tool as well:
For the record:
Binlog file is actually produced but before we start the official part of the build we copy the outputs to a validatesdk folder in the artifacts folder which we are not copying.
We need to copy eng/common/templates/job/job.yml into a location outside eng/common so no other repos are affected by this change and add a task similar to https://github.com/dotnet/arcade/blob/6f33f02efeafb578b3f99da08cb2670e0470c917/eng/common/templates/job/job.yml#L162 but updating PathtoPublish and ArtifactName
Since ValidateSdk only runs in official builds the task we will add should just run in official builds
In an auto-update scenario, when we determine what dependencies need to be updated in a repo+branch we are doing case sensitive comparison on the asset/dependency name, we should make this insensitive.
Change should happen around https://github.com/dotnet/arcade-services/blob/master/src/Microsoft.DotNet.Darc/src/DarcLib/Actions/Remote.cs#L290
FYI @mmitche
When pushing new updates to a PR, Maestro/DarcLib should update the PR branch with the target branch (if possible). While GitHub/AzDO will do this when creating the merge branch, if a user has to make modifications on top of the PR branch, they will likely download and modify an out of date branch. Typically this would happen when an update causes a failure which sits for a few days.
Once AzDO supports proper flow of variables across stages (i.e., set a variable in one stage and consume it in another) use that to communicate variables across publishing stages.
@alexperovich commented on Thu Jan 10 2019
Roslyn hasn't gotten any updates because their eng\Versions.props starts with a byte order mark.
@dougbu commented on Wed Jul 03 2019
Currently the only darc commands related to repository policies are get-repository-policies and set-repository-policies. While that covers CRU because set-repository-policies both creates and updates (well, overwrites) policies, there is no command to delete repo policies.
For example, we have old policies for non-existent branches that won't be serviced. They just add noise to darc get-repository-policies --all output. I know --all isn't the default but it's very useful when creating policies for branches that don't yet exist.
For example
darc get-repository-policies --repo https://github.com/aspnet/Extensions --allhttps://github.com/aspnet/Extensions @ master
- Merge Policies:
Standard
https://github.com/aspnet/Extensions @ release/3.0-preview3
- Merge Policies:
Standard
https://github.com/aspnet/Extensions @ release/3.0-preview4
- Merge Policies:
Standard
https://github.com/aspnet/Extensions @ release/3.0-preview5
- Merge Policies:
Standard
https://github.com/aspnet/Extensions @ release/3.0-preview53
- Merge Policies:
Standard
https://github.com/aspnet/Extensions @ release/3.0-preview6
- Merge Policies:
Standard
https://github.com/aspnet/Extensions @ release/3.0-preview7
- Merge Policies:
Standard
Hmm, that output includes policies for a branch that never existed. The 'release/3.0-preview53' repo policies must have been accidentally created (probably by me
In the past we have accidentally included links to dnceng instead of github. As of now, someone has to do a manual check to ensure all of these are correct. This check should be something that we can automate and track continually
cc: @mmitche, @markwilkie , @danmosemsft, @Pilchie
@dougbu commented on Wed Jul 03 2019
darc get-repository-policies lacks options that could / should be copied over from darc get-subscriptions. The one I miss the most is --regex. --exact is also applicable but I've never needed that one.
For example, the following command displays information about policies in AspNetCore-Tooling as well as the hoped-for AspNetCore information
darc get-repository-policies --repo https://github.com/aspnet/AspNetCore --branch masterDarc should not push new branches to the main repo. It introduces a cleanup issue over time (if branches are not deleted) and increased number of refs.
Enable duplication of subscriptions
I noticed issues about using Octokit #2580 and VSTS C# libs #2579 in Darc. This sounds like a great idea, but I want to make sure it's implemented in a way that leaves Darc able to build from source.
If Octokit and VSTS APIs can build from source, ok, but it would be preferable not to build them at all. The builds take time, and the capabilities won't be used in source-build.
/cc @alexperovich @jcagme
I haven't looked into every one, but these are unknowns when it comes to source-build:
An easy way I see to do this is to split code that needs to use these dependencies into separate csproj's, condition the dependency, and condition some small part of the code to link in (or not) web client capability. (Microsoft.DotNet.Maestro.Client is an example of something that's already split off!) The property ExcludeFromSourceBuild is used to skip a project during source-build.
I can imagine something fancier being done, where there's a real extensibility point on Darc. The user could choose to download some extra packages (or something) to enable web client capabilities.
Rather than polling every 5 minutes or so, we should receive github webhooks to control subscription updates.
Internal URL is currently: https://maestro-int.westus2.cloudapp.azure.com/
We should have a nicer URL for production.
/cc @davidfowl @Chrisboh @mmitche
We have the data for this along with A visual representation, but the context here is a quick glance picture that highlights problems in order to expedite investigation into issues. Think of a screen in a room that is always showing the state of maestro flow and flashes red when something needs attention. This is something I've had in my head, but also got a similar request from David Fowler this morning. Since we have an explicit customer ask for this, I'd like to start with someone reaching out to him and understanding the exact use case and details.
David, please feel free to add/correct anything I have in here.
Add an API which allows for a default channel to be updated, rather than having to delete and recreate the default channel.
The basic rule is that a channel may only be produced from one branch, so updating or adding a default channel association where the channel was previously produced from another branch should update the existing entry.
Add some validation to commands such as darc create-subscription.
Is there a way to tell the bot to hold-off merging a PR? I figured the "request changes" gesture in GitHub reviews would have been honored, but apparently not: dotnet/efcore#14506
if you move all targets logic to Publish.proj and only expose tasks from the package you'll have to define and maintain only the tasks API. It this thus going to be simpler.
I agree that the current state is not desirable. I'd suggest that instead of importing the targets file defined in package here we actually remove it from the package and move all the logic here.
It looks like this is because Darc assumes that the parent of the .git directory is always the root of a submodule: https://github.com/dotnet/arcade-services/blob/master/src/Microsoft.DotNet.Darc/src/DarcLib/Actions/Local.cs#L27 and https://github.com/dotnet/arcade-services/blob/master/src/Microsoft.DotNet.Darc/src/DarcLib/Helpers/LocalHelpers.cs#L45
I think Darc should be doing git rev-parse --show-toplevel in order to get the root of the Git repo.
Repro steps:
git clone https://github.com/dotnet/source-build
git submodule update --init
cd src/core-sdk
dotnet Microsoft.DotNet.Darc.dll get-dependency-graph
Expected results: Get the dependency graph based on core-sdk's Version.Details.xml.
Actual results:
Getting root dependencies from local repository...
info: Microsoft.DotNet.Darc.Operations.Operation[0]
Getting a collection of dependencies from 'eng/Version.Details.xml' in repo '/home/chris/dotnet_source-build/.git/modules/src'...
info: Microsoft.DotNet.Darc.Operations.Operation[0]
Reading 'eng/Version.Details.xml' in repo '/home/chris/dotnet_source-build/.git/modules/src' and branch ''...
fail: Microsoft.DotNet.Darc.Operations.Operation[0]
Something failed while getting the dependency graph.
System.IO.DirectoryNotFoundException: Could not find a part of the path '/home/chris/dotnet_source-build/.git/modules/src/eng/Version.Details.xml'.
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize)
at Microsoft.DotNet.DarcLib.LocalGitClient.GetFileContentsAsync(String relativeFilePath, String repoUri, String branch) in /home/chris/dotnet_source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/LocalGitClient.cs:line 58
at Microsoft.DotNet.DarcLib.GitFileManager.ReadXmlFileAsync(String filePath, String repoUri, String branch) in /home/chris/dotnet_source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/Helpers/GitFileManager.cs:line 412
at Microsoft.DotNet.DarcLib.GitFileManager.ReadVersionDetailsXmlAsync(String repoUri, String branch) in /home/chris/dotnet_source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/Helpers/GitFileManager.cs:line 37
at Microsoft.DotNet.DarcLib.GitFileManager.ParseVersionDetailsXmlAsync(String repoUri, String branch, Boolean includePinned) in /home/chris/dotnet_source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/Helpers/GitFileManager.cs:line 84
at Microsoft.DotNet.DarcLib.Local.GetDependenciesAsync(String name, Boolean includePinned) in /home/chris/dotnet_source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/Actions/Local.cs:line 114
at Microsoft.DotNet.Darc.Operations.GetDependencyGraphOperation.ExecuteAsync() in /home/chris/dotnet_source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/GetDependencyGraphOperation.cs:line 73
Because Maestro is an app today, it only has R/W permissions to repos. If a repository has required reviewers on the repo or has restricted the users who can push to a branch, it is unable to merge automatically. Find a way to enable auto-merging even in these cases. Some ideas:
@dotnet-mc-bot commented on Sat Mar 16 2019
There were a set of failures during this build. Here is a summary of these:
@robertborr commented on Mon Mar 18 2019
The build timed out - you may wish to allocate more time.
F:\workspace_work\1\s.dotnet\sdk\2.1.504\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.targets(129,5): error MSB4220: Waiting for the currently executing task "GenerateDepsFile" to cancel. [F:\workspace_work\1\s\src\Maestro\SubscriptionActorService\SubscriptionActorService.csproj]
F:\workspace_work\1\s.dotnet\sdk\2.1.504\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.targets(129,5): error MSB4220: Waiting for the currently executing task "GenerateDepsFile" to cancel. [F:\workspace_work\1\s\src\Maestro\SubscriptionActorService\SubscriptionActorService.csproj]
##[error]The operation was canceled.
This is a fairly common scenario for dev work, and I don't see a way to do this with darc yet. With my understanding of the current design, the Maestro bot will leave behind stale references to deprecated packages when either of these scenarios occur. In ProdCon v1, we had a protection against this -- a strict check on incoming version variables -- which helped us know when packages were dropped or renamed. Unless I've missed something, I don't see a replacement for this in darc/maestro.
Team A was producing previews of System.Banana and pushing to channel ".NET Core Dev", but decides they would rather ship this as System.Strawberry. Consumers of need to update to the new package ID.
Team A was producing previews of System.Titanic, but due to reasons, has decided this package should not ship (pun intended). Consumers should stop using this package as it will not have a production release.
Some of our arcade-services libraries use LibGit2Sharp which in turn uses a package called LibGit2Sharp.NativeBinaries which contains some *.so files that don't have a BuildID note (see example here) in them. This causes our symbol uploader to fail for those arcade-services packages.
We need to figure out what's the right thing moving forward here. Whether it is patching microsoft.symboluploader.build.task to not fail in such cases or something else.
We don't have any docs regarding the usage of the maestro API.
We expect most users to use darc to interact with the API rather than call into it directly, but we should still provide docs around it.
The dry run ability today stops just short of downloading files. Location information is verified, but not much else. Dry run should be improved so:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.