Setup artifacts-credprovider in TFS CI/CD with Docker about artifacts-credprovider HOT 22 CLOSED

Solution !!!!

@randyridge @shubham90 I've found a solution that seems to be working, and found out what I did wrong, below is an updated version of the dockerfile that I'm using (I bolded the changes that I've made)

• I've copied the file the nuget.config
• Most important I've added DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0 this makes the difference in working or not working
• also **very important ** you need to have the same address in nuget.config as in your build definition !!!
• also very important if you're using $(System.AccessToken) make sure the address to nuget is not HTTPS, otherwise it won't work
• if your address is HTTPS you need to create a PAT for an User who has access externally

What would be nice is that when you add the Feel URL (VSS_NUGET_EXTERNAL_FEED_ENDPOINTS) to also add it automatically in the default nuget.config so you don't have to drag after you another nuget.config file

FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/dotnet:2.2-sdk AS build
ARG FEED_URL
ARG PAT

WORKDIR /src
**COPY nuget.config ./**
COPY ["Web/Web.csproj", "Web/"]

# download and install latest credential provider. Not required after https://github.com/dotnet/dotnet-docker/issues/878
**RUN wget -O - https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh  | bash**

# Environment variable to enable seesion token cache. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
**ENV DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0**

# Environment variable for adding endpoint credentials. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
# Add "FEED_URL" AND "PAT" using --build-arg in docker build step. "endpointCredentials" field is an array, you can add multiple endpoint configurations.
# Make sure that you *do not* hard code the "PAT" here. That is a sensitive information and must not be checked in.
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS {\"endpointCredentials\": [{\"endpoint\":\"${FEED_URL}\", \"username\":\"ArtifactsDocker\", \"password\":\"${PAT}\"}]}

COPY nuget.config ./
RUN dotnet restore "Web/Web.csproj" --configfile nuget.config -nowarn:msb3202,nu1503 --verbosity diag
COPY . .
WORKDIR "/src/Web"
RUN dotnet build "eb.csproj" -c Release -o /app

FROM build AS publish
RUN dotnet publish "Web.csproj" -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Web.dll"]

from artifacts-credprovider.

Kudos to @moisoiu and @shubham90 for helping to figure out how to do it. Just as another improvement iteration, I separated all the instructions specific to the feed auth so that it can be easily reused, especially in combination with VS-generated Dockerfiles. The following snippet should be put before RUN dotnet restore ...:

### enable nuget restore from private repo
ARG FEED_URL
ARG PAT
COPY nuget.config ./
# download and install latest credential provider. Not required after https://github.com/dotnet/dotnet-docker/issues/878
RUN wget -O - https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh  | bash
# Environment variable to enable seesion token cache. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER 0
# Environment variable for adding endpoint credentials. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
# Add "FEED_URL" AND "PAT" using --build-arg in docker build step. "endpointCredentials" field is an array, you can add multiple endpoint configurations.
# Make sure that you *do not* hard code the "PAT" here. That is a sensitive information and must not be checked in.
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS {\"endpointCredentials\": [{\"endpoint\":\"${FEED_URL}\", \"username\":\"ArtifactsDocker\", \"password\":\"${PAT}\"}]}

from artifacts-credprovider.

Thank you for the feedback, I did end up finding a solution similar to what you mentioned.

I stumbled across this post
"https://developercommunity.visualstudio.com/content/problem/530691/response-status-code-does-not-indicate-success-401.html"

Change the PAT Organization to "All accessible organizations" AND scroll through the permission\security sections and set the following:

Build (Artifacts, definitions, requests, queue a build, and updated build properties): Read
Connected server (Access endpoints): Connected server
Packaging (Create, read, update, and delete feeds and packages): Read

(Not sure if all of these are required.)

Docker File:

    FROM mcr.microsoft.com/dotnet/core/aspnet:2.2-stretch-slim AS base
    WORKDIR /app
    
    FROM mcr.microsoft.com/dotnet/core/sdk:2.2-stretch AS build
    WORKDIR /src
    
    # The Personal Access Token arg
    ARG PAT
    
    # Set environment variables
    ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
    ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS '{"endpointCredentials":[{"endpoint":"https://pkgs.dev.azure.com/ORG/_packaging/FEEDNAME/nuget/v3/index.json","username":"USERNAME","password":"'${PAT}'"}]}'
    
    # Get and install the Artifact Credential provider
    RUN wget -O - https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh  | bash
    
    COPY ["PROJECTNAME.csproj", "./"]
    RUN dotnet restore -s "https://pkgs.dev.azure.com/ ORG /_packaging/ FEEDNAME /nuget/v3/index.json" -s "https://api.nuget.org/v3/index.json"
    
    COPY . ./
    WORKDIR /src
    RUN dotnet build " PROJECTNAME.csproj" -c Release -o /app
    
    FROM build AS publish
    RUN dotnet publish --no-restore " PROJECTNAME.csproj" -c Release -o /app
    
    FROM base AS final
    WORKDIR /app
    COPY --from=publish /app .
    ENTRYPOINT ["dotnet", " PROJECTNAME.dll"]

docker command:
docker build -t BUILDNAME:local . --build-arg PAT=xxxxxxxxxxxxxxxxxxxxxxxxx

from artifacts-credprovider.

Can you try something like this https://gist.github.com/shubham90/ad85f2546a72caa20d57bce03ec3890f ?

Specially, make sure that the Environment variable "VSS_NUGET_EXTERNAL_FEED_ENDPOINTS" is set using ENV. Also, the [PAT] you are using is passed as Args and matches the password field.
I see that you are setting SYSTEM_ACCESSTOKEN but using [PAT] in the password field.

from artifacts-credprovider.

@shubham90 Thanks a lot for the example, it guide me to see where I did wrong, but unfortunately still it doesn't help and I start to think if this is not an issue regarding the TFS or something, even though I have no idea where to start to look first.

I've tried in 2 ways,

First way without --configfile nuget.config
Result: can't find in nuget.org the package SDCC.Core (the one which is in private nugget)

Tried with --configfile nuget.config
GSSAPI operation failed with error - An invalid status code was supplied (SPNEGO cannot find mechanisms to negotiate)

Also in nuget.config I've removed the PAT when I tried with --configfile nuget.config

I have no idea where to start or what to do next, and what's more frustrating is that if I make the build / restore in TFS build agent, everything is fine and is working fine.

nuget.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
 <packageSources>
    <!--To inherit the global NuGet package sources remove the <clear/> line below -->
    <clear />
    <add key="nuget" value="https://api.nuget.org/v3/index.json" />
	<add key="SDCC" value="http://[Redacted]/tfs/SDCC/_packaging/SDCC/nuget/v3/index.json" />
 </packageSources>
</configuration>

Docker File

FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/dotnet:2.2-sdk AS build
ARG FEED_URL
ARG PAT

WORKDIR /src
COPY ["Web/Web.csproj", "Web/"]

# download and install latest credential provider. Not required after https://github.com/dotnet/dotnet-docker/issues/878
RUN wget -qO- https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | bash

# Environment variable to enable seesion token cache. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true

# Environment variable for adding endpoint credentials. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
# Add "FEED_URL" AND "PAT" using --build-arg in docker build step. "endpointCredentials" field is an array, you can add multiple endpoint configurations.
# Make sure that you *do not* hard code the "PAT" here. That is a sensitive information and must not be checked in.
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS {\"endpointCredentials\": [{\"endpoint\":\"${FEED_URL}\", \"username\":\"ArtifactsDocker\", \"password\":\"${PAT}\"}]}

COPY nuget.config ./
RUN dotnet restore "Web/Web.csproj" --configfile nuget.config -nowarn:msb3202,nu1503 --verbosity diag
COPY . .
WORKDIR "/src/Web"
RUN dotnet build "eb.csproj" -c Release -o /app

FROM build AS publish
RUN dotnet publish "Web.csproj" -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Web.dll"]

from artifacts-credprovider.

@moisoiu Did you pass the Feed_Url and PAT as build-arg? For ex: docker build should have --build-arg PAT={YOUR_PAT_HERE} and --build-arg FEED_URL={Your feed Url}

Also, the error message that you are getting is suspicious. Can you try with dotnet 2.1 instead of 2.2 ? This might be related https://github.com/dotnet/corefx/issues/28961

[Update] The issue looks similar to https://github.com/dotnet/corefx/issues/34369

from artifacts-credprovider.

Just to keep it updated, I'm still investigating, and yes I've passed the builg-arg in TFS

Those are my next steps and will come back with feedback. I'm thinking if is not a security policy from TFS or something, because is weird, my TFS agents are on Linux, also the docker container is on linux, why it shouldn't work ?

from artifacts-credprovider.

I believe I'm experiencing same issue, wanting to restore from on-prem tfs package management within a linux docker build and failing with the negotiate/ntlm not correctly working as mentioned above in dotnet/corefx#34369. It doesn't appear that there's any workaround except disable negotiate on tfs (which i've not actually tried, as I have no idea what chaos would befall across the enterprise), wait until the httpclient bug is fixed, it looks like maybe in 3.0, or give the docker nuget restore the packages via a local directory (which is what I ended up doing and is a manual mess).

from artifacts-credprovider.

That's the approach I have taken, doing the regular Restore, Build, Publish in TFS and after copy all the publish in a docker image, it's an ugly work around but till now, have no idea how I could achieve in some other way

from artifacts-credprovider.

Thanks! Your workaround worked for me as well.

from artifacts-credprovider.

We're having the same issue and went so far as to publish everything to a different Nuget repository for Docker builds. I'll try your solution shortly.

from artifacts-credprovider.

Alright, I'm curious if is working and maybe someone can tell me who is generating the nuget.config, because maybe a solution for this problem would be to add a variable for DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER that can be set on False (0) and also for each endpoint you add in VSS_NUGET_EXTERNAL_FEED_ENDPOINTS, to also add it in the nuget.config

from artifacts-credprovider.

Thank you, Leon, I've used your code in a couple of my projects with success. Today I suddenly found myself with one project that would not build, failing with the same 401 error as before. In my case, for this particular project, I found that the Microsoft.VisualStudio.Azure.Containers.Tools.Targets package had been upgraded to v1.7.9. I figured I'd share my findings to hopefully save someone else some time: I downgraded that package to v1.5.4, and the issue went away.

from artifacts-credprovider.

Has anybody had a resurgence of this issue since the workarounds posted above? I'm at my wits end trying to resolve the Response status code does not indicate success: 401 error when trying to dotnet restore from a container job in a pipeline.

Docker file for the container we're trying to restore in:

FROM mcr.microsoft.com/dotnet/core/sdk:2.1

# Accept feed URL and personal access token as arguments
ARG FEED_URL
ARG PAT

# Get and install the Artifact Credential provider
RUN wget -O - https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh  | bash

# Set environment variables
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER 0
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS {\"endpointCredentials\": [{\"endpoint\":\"${FEED_URL}\", \"username\":\"ArtifactsDocker\", \"password\":\"${PAT}\"}]}

And here's the pipeline of the application we're trying to restore:

resources:
  containers:
  - container: <<organization>>NetCoreSDK21
    image: <<organization>>.azurecr.io/build/netcore/production/sdk:2.1
    endpoint: <<organization>> ACR
container: <<organization>>NetCoreSDK21
trigger:
  - master
steps:
- task: DotNetCoreCLI@2
  displayName: 'dotnet restore'
  inputs:
    command: restore
    feedsToUse: config
    nugetConfigPath: 'TestApplication/TestApplication/nuget.config'
    projects: '**/TestApplication.csproj'
- task: DotNetCoreCLI@2
  displayName: 'dotnet publish'
  inputs:
    command: publish
    publishWebProjects: false
    projects: '**/*.csproj'
    arguments: '--no-restore'
    workingDirectory: TestApplication/TestApplication

If I log on to the agent machine that's picking up the pipeline runs and manually create an instance of this image and try restoring the same application it works fine. It only breaks when executing as part of a pipeline.

from artifacts-credprovider.

@buck-mbeer, Yes, we have started experiencing this over the last couple of weeks. Builds that were working are now failing. I am not sure what has changed to cause this. And we have followed all suggested solutions for private nuget feeds.

Has anyone found a solution?

from artifacts-credprovider.

@clintstedman The fix for us when using the Microsoft .Net Core SDK image as a base was to force NuGet to use basic authentication by setting an additional environment variable -NUGET_AUTHENTICATION_TYPES=basic

Edit: Also, as always - if you're using a PAT to authenticate, check it hasn't expired :)

from artifacts-credprovider.

Thanks @buck-mbeer and @clintstedman for your investigations. I am glad that you are unblocked now. 401's are usually caused because of expired PATs or some change in environment which caused basic auth to fail.

from artifacts-credprovider.

Solution !!!!

@randyridge @shubham90 I've found a solution that seems to be working, and found out what I did wrong, below is an updated version of the dockerfile that I'm using (I bolded the changes that I've made)

• I've copied the file the nuget.config
• Most important I've added DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0 this makes the difference in working or not working
• also **very important ** you need to have the same address in nuget.config as in your build definition !!!
• also very important if you're using $(System.AccessToken) make sure the address to nuget is not HTTPS, otherwise it won't work
• if your address is HTTPS you need to create a PAT for an User who has access externally

What would be nice is that when you add the Feel URL (VSS_NUGET_EXTERNAL_FEED_ENDPOINTS) to also add it automatically in the default nuget.config so you don't have to drag after you another nuget.config file

FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/dotnet:2.2-sdk AS build
ARG FEED_URL
ARG PAT

WORKDIR /src
**COPY nuget.config ./**
COPY ["Web/Web.csproj", "Web/"]

# download and install latest credential provider. Not required after https://github.com/dotnet/dotnet-docker/issues/878
**RUN wget -O - https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh  | bash**

# Environment variable to enable seesion token cache. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
**ENV DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0**

# Environment variable for adding endpoint credentials. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
# Add "FEED_URL" AND "PAT" using --build-arg in docker build step. "endpointCredentials" field is an array, you can add multiple endpoint configurations.
# Make sure that you *do not* hard code the "PAT" here. That is a sensitive information and must not be checked in.
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS {\"endpointCredentials\": [{\"endpoint\":\"${FEED_URL}\", \"username\":\"ArtifactsDocker\", \"password\":\"${PAT}\"}]}

COPY nuget.config ./
RUN dotnet restore "Web/Web.csproj" --configfile nuget.config -nowarn:msb3202,nu1503 --verbosity diag
COPY . .
WORKDIR "/src/Web"
RUN dotnet build "eb.csproj" -c Release -o /app

FROM build AS publish
RUN dotnet publish "Web.csproj" -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Web.dll"]

Thank you krub its very good

from artifacts-credprovider.

For me the Solution works on sdk 3.1 but not on sdk 5.0
Issue

from artifacts-credprovider.

@Gladskih I'm having the same issue as you. Works on SDK 3.1, but once we tried to upgrade, we got this same issue:

GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (SPNEGO cannot find mechanisms to negotiate)

And yes we've been using the above solution for months. MSFT we need you to support Linux here a little better! This is a brutal solution to the problem. And the kicker for us here is that 3.1 is NOT LTS.

from artifacts-credprovider.

@Gladskih I'm having the same issue as you. Works on SDK 3.1, but once we tried to upgrade, we got this same issue:

GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (SPNEGO cannot find mechanisms to negotiate)

And yes we've been using the above solution for months. MSFT we need you to support Linux here a little better! This is a brutal solution to the problem. And the kicker for us here is that 3.1 is NOT LTS.

@irperez , I found the solution for this. I do not use the recommended above where you use installcredprovider.sh. I have configured my nuget.config to make this works and also I has the self signed cert issue. to fix the cert issue I did as above

COPY ca-bundle.crt ../usr/local/share/ca-certificates/
RUN update-ca-certificates

THEN to fix the GSSAPI issue I found this link with how to solve it : https://developercommunity.visualstudio.com/t/azure-artifacts-nuget-feed-gives-error-during-rest/711941

In the nuget.config I added the

<packageSourceCredentials>
    <AzureDevops>
      <add key="Username" value="xxxxxx" />
      <add key="ClearTextPassword" value="xxxxx" />
      <add key="ValidAuthenticationTypes" value="basic" />  <--This is important to not to try to use NTLM apparently
    </AzureDevops>  
  </packageSourceCredentials>

from artifacts-credprovider.

I tried everything under the sun and nothing helped. Until... I found this setting under the Build Definition > Options > Build Job. Set it to Project Collection instead of Current project, and finally the PAT from the agent worked inside my docker build.

from artifacts-credprovider.