hcsshim: timeout waiting for notification extra info about hcsshim HOT 65 OPEN

This is my number one issue with Docker for Windows containers.

Is there a timeline for a solution?

from hcsshim.

I have the same problem.

FROM microsoft/windowsservercore:1709

RUN mkdir C:\Build

PS C:\Users\anton\Desktop\DockerTest> docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM microsoft/windowsservercore:1709
 ---> be1324f21832
Step 2/2 : RUN mkdir C:\Build
 ---> Running in b568056319be
container b568056319bef54ed28031228d38362153da8775b715b7a3ab11af2e772b5da6 encountered an error during CreateContainer: hcsshim: timeout waiting for notification extra info: {"SystemType":"Container","Name":"b568056319bef54ed28031228d38362153da8775b715b7a3ab11af2e772b5da6","Owner":"docker","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"D:\\Docker2\\windowsfilter\\b568056319bef54ed28031228d38362153da8775b715b7a3ab11af2e772b5da6","Layers":[{"ID":"8a1cd827-e3f3-52e5-a2a2-18edd87a32c3","Path":"D:\\Docker2\\windowsfilter\\a4c1853a70a019beeb62b7184361f855be533ce1c933dd49158420bc05b17fa2"},{"ID":"0509ff50-38ef-5064-9ef8-93099eb63817","Path":"D:\\Docker2\\windowsfilter\\d532172bb554c431d4f4b6dfca2474e96ec1f563f0b3b26c30d210f0ef080a36"}],"HostName":"b568056319be","HvPartition":true,"EndpointList":["707ba3e2-0e0f-479f-845b-137a4c0efac1"],"HvRuntime":{"ImagePath":"D:\\Docker2\\windowsfilter\\a4c1853a70a019beeb62b7184361f855be533ce1c933dd49158420bc05b17fa2\\UtilityVM"},"AllowUnqualifiedDNSQuery":true}
PS C:\Users\anton\Desktop\DockerTest>

@darrenstahlmsft
I sent a trace to you.

from hcsshim.

@dermeister0 @ClintSulis I had a chance to look at both your traces today, they are both the same issue, and we understand the problem. I don't have any timelines or workarounds at this time, but know we are working to get our fix out via Windows Servicing.

from hcsshim.

So I had this issue now for a couple of days and couldn't really pin point it down.

What did the trick for me was:

• Uninstalling docker
• Removing all docker folders
• Removing the windows features Containers and Hyper-v
• Restart
• Reinstall docker
• Let docker handle the Windows features
• Restart

And that's it...

from hcsshim.

I ran into the same issue : hcsshim: timeout waiting for notification extra info
I solved it by increasing the RAM usable for a build with the -m option, like so:

docker build -m 4GB -t <tag> .

Replace 4GB with the amount of RAM you need. By default, it's limited to 1GB.

Source: https://sarafian.github.io/sdl/knowledge-center/2017/03/11/docker-windows-containers-out-of-memory-windows-10.html

from hcsshim.

I found a workaround that works for me.
I run a docker ps -a to see if I have an exited copy of my container sitting around.
Then I run a docker rm containerid to remove that exited container and then my docker compose works again from visual studio.

Also, I found that my dockerfile was not updated to use the correct base and build images when I changed my project from using .net core 2.0 to 2.1. I created a new project which targeted .net core 2.1 and copied the FROM lines to my original project and things started to work again.

from hcsshim.

HI @halbherz Thanks. It worked for me. I want to install a .exe file to nanoserver-insider container Is that possible? if possible how

from hcsshim.

Many reproductions in this thread. I'm the latest (see my mention linked above at microsoft/Windows-Containers#153). The last comment from anyone associated with Microsoft was from @lowenna, who no longer works there (and maybe not anywhere, judging by "mostly retired"). This thread was created in 2017, and the last Microsoft comment was March 5, 2019, over two years ago. Most of the threads I run into regarding Windows containers have no Microsoft participation. I can't get anyone's attention on Twitter or Stack Overflow. Are Windows containers not a priority for Microsoft? Am I wasting my time trying to figure this stuff out? What does it take to start a conversation with a Microsoft engineer?

from hcsshim.

I have figured out a solution to my particular problem, which was trying to run a Windows container from a Windows virtual machine. @scottresnik, it might help you too. Good luck to everyone else.

from hcsshim.

who no longer works there (and maybe not anywhere, judging by "mostly retired")

You're right, I don't 😄

from hcsshim.

Hey there,

I just installed Docker and running it for the first time and I'm getting this error with windows containers

I attempted @halbherz suggestion and was unable to get it working after doing that. Not sure what else to do at this point.

Running Win 10 Enterprise
version 1709
os build 16299.98

Running latest vs 2017, .net core 2.1 with docker and brand new project (letting VS do all the docker file stuff).

Any suggestions would be welcome.

from hcsshim.

Seems @AdamRehn

In my testing under Win10, it seems that altering the memory limit for containers between subsequent Docker runs (e.g using -m 6GB for one run and then -m 8GB for the next) can reduce the frequency with which this error occurs. Since Win10 always uses Hyper-V isolation, I'm guessing that changing the memory limit forces Docker to provision a new Hyper-V VM, preventing it from re-using an existing VM that may have fallen victim to whatever condition triggers the timeout.

and @Kuinox

Hi,
I alsot get this error, but mostly when the system is out of ram and is swapping.

are on to something here.

I noticed that my issue seems to be related to memory problems as well. My machine was at 5.3GB/8GB and closing applications and dropping to 3GB/8GB didn't help until I swapped -m 4GB to -m 2GB. When it broke again, it didn't fix until I went to -m 3GB. I noticed my FCNS (Finished Creating, Never Started) container issues seems to be related to the vmmem processes popping up with a weird GUID user. If the vmmem process indicated 'suspended' or 0kb used, it was a failure. I also noticed on one occassion that even with -m 2GB, the vmmem was using over 2,500,000Kb. I noticed that no new VMs appeared in the Hyper-V manager when the container was running so there's no way (that I know of) to monitor the progress otherwise. There's a strong correlation to hard faults and this error as well. Even with only 5.3GB/8GB used, I was getting lots of hard faults and the build process was failing for the mssql-server-windows-developer dockerfile. Swapping around the -m parameter seems to stir something and suddenly I was moving again. My machine with 256GB RAM (but no internet) never had this particular error.

Hoping @darstahl or @jhowardmsft can provide some update.

from hcsshim.

As mentioned previously I'm running the mssql-server-windows-developer dockerfile on Win 10 1809 with hyper-v.

Had a little more detail at microsoft/mssql-docker#420

Repro steps:

1. High RAM usage (committed is higher than total physical, e.g. 8.7GB committed, 8GB physical). Lots of hard faults indicated in resource manager.
2. Slow HDD may be part of the equation to increase delays with vm swap
3. docker build -m 4GB -t mssql-repro-test .

It's not 100% consistent but I definitely got it on several different machines many times. Once I started reducing my RAM usage and fiddling with the -m parameter I finally got the image to build consistently. 2GB seemed like a sweet spot. 1GB wasn't enough for the container to install CU13 when I tried to modify it. 4GB seemed to just make the RAM issue worse.

Sometimes I got a generic timeout, other times it called out hcsshim. Every time I got a "created" container that doesn't respond to 'stop', 'kill', or 'rm'. Had to restart docker desktop before I could 'rm' it.

Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        6247962
 Built:             Sun Feb 10 04:12:31 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.24)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 04:28:48 2019
  OS/Arch:          windows/amd64
  Experimental:     false

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 28
Server Version: 18.09.2
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows 10 Pro Version 1809 (OS Build 17763.316)
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 7.955GiB
Name: REDACTED
ID: OTZO:EQUA:G54M:CFBN:O5V6:DYPK:3POL:HTUQ:C2RH:55AS:MVMC:JJY2
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 26
 System Time: 2019-03-06T11:42:56.2623186-05:00
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

from hcsshim.

This timeout is caused by the HcsCreateComputeSystem (Windows API) call being done async and hanging for some reason. Since the hang is in the platform, not Go, we'll need to collect an HCS trace to debug this.

EDIT: See below for updated trace instructions

from hcsshim.

Got it. Forwarding to the internal team to take a look.

from hcsshim.

Hi @darrenstahlmsft!
Do you have any updates on the issue?
I've downloaded the WPA to analyze my log.
I'm trying to make any sense of it.

from hcsshim.

No updates yet. So far this only seems to be affecting Hyper-V isolation containers on RS3 (1709) Windows builds. If anyone is seeing this on other hosts or Windows Server containers, let me know. I'll keep this thread updated.

from hcsshim.

Hey!
After a recent Windows update, our docker hosting machine (Win 10) has updated, I am also getting this error.

Are there any updates?

from hcsshim.

Hey @darrenstahlmsft ,

I am running docker on Windows 10 professional. I also have the similar issue when i am trying to install my node dependencies using "npm install".

Hope you find the cause of this issue soon.

from hcsshim.

I'm having the same issue on Win10 Pro with Windows server images.
I am also unable to run Linux images because:
Docker hv-sock proxy (vsudd) is not reachable
at Docker.Backend.ContainerEngine.Linux.ConnectToVsud(TaskCompletionSource`1 vmId) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Linux.cs:line 293
at Docker.Backend.ContainerEngine.Linux.DoStart(Settings settings, String daemonOptions) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Linux.cs:line 260
at Docker.Backend.ContainerEngine.Linux.Start(Settings settings, String daemonOptions) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Linux.cs:line 130
at Docker.Core.Pipe.NamedPipeServer.<>c__DisplayClass9_0.b__0(Object[] parameters) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 47
at Docker.Core.Pipe.NamedPipeServer.RunAction(String action, Object[] parameters) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 145

not sure if that's related. However Docker on windows is now unusable

from hcsshim.

Guys, small update.

For me this happens if I execute the following sequence:

docker-compose -f docker-compose.yml -f docker-compose.staging.yml -p MyServer down --rmi local 
docker-compose -f docker-compose.yml -f docker-compose.staging.yml -p MyServer build --no-cache
docker-compose -f docker-compose.yml -f docker-compose.staging.yml -p MyServer up -d --force-recreate

if i skip the build --no-cache, docker builds and starts up all fine.

maybe this will be of help. This is not a 100% fix, but might work every nth time.

still would like to know the actual problem and the fix :)

from hcsshim.

I'm not using docker compose, just trying to run windows containers on Windows 10. I was getting this error earlier, I think when I was trying to allocate too much memory. But last night Windows installed some updates and now I can't run my container at all. Just trying to do the basic docker run for microsoft/mssql-server-windows-developer.

I'm pretty sure my system was fully up-to-date prior to yesterday's updates, and the only things that changed were the Tuesday security patches, but I'm not sure I can verify that yet. This is all in a VM and I do have a backup from a few days ago, so I might be able to try that.

Anything else I could do to provide diagnostic information?

from hcsshim.

Any updates on this issue or how to work around it?

from hcsshim.

We're tracking a few internal issues that all result in this same error message. In order to better understand the different causes, I'm uploading a new trace profile (HcsTraceProfile2.wprp) that will trace more details than the above version.

Wpr.exe -start HcsTraceProfile2.wprp!Hcs -filemode

<repro the issue here>

Wpr.exe -stop HcsTraceTimeout.etl "Some description"

Also get the event logs with the following:

wevtutil.exe epl Microsoft-Windows-Hyper-V-Worker-Analytic worker-analytic.evtx
wevtutil.exe epl Microsoft-Windows-Hyper-V-Worker-Admin worker-admin.evtx

Note that the resulting HcsTraceTimeout.etl may contain personal information, such as path names accessed on the host, current running executables, etc, so I suggest sending it to me directly at [email protected].

If you can send all the above files, then we can better make sure that all the underlying issues are understood.

from hcsshim.

Any updates on how to work around it?

from hcsshim.

I moved everything to a new visual studio solution and it worked.

from hcsshim.

Any updates? I just installed docker the other day and I'm having this problem with all windows containers. Linux containers work fine. Are you still looking for those files? I should have the time to generate them this week.

from hcsshim.

There are multiple issues that result in this same error message. I'd still be happy to take the above files so I can make sure we're addressing all the underlying issues that might cause this.

from hcsshim.

I tried this again today and did not experience the issue. Maybe the computer just needed a restart?

from hcsshim.

@darrenstahlmsft

I'm uploading a new trace profile (HcsTraceProfile2.wprp)

I sent an updated trace to you.

from hcsshim.

@darrenstahlmsft

I sent a trace to you as well. Hope it helps!

from hcsshim.

from hcsshim.

Does anybody have a fix for this? It has been blocking me for 3 days now. I have been de-installing windows updates to try to get back to a state where I don't encounter this problem.

from hcsshim.

Any update on this? Just ran into this exact error today on a developer's machine

from hcsshim.

Same error here. Win 10 latest docker host with several windows server core images

from hcsshim.

Please, some kind of update would be greatly appreciated.

from hcsshim.

In my testing under Win10, it seems that altering the memory limit for containers between subsequent Docker runs (e.g using -m 6GB for one run and then -m 8GB for the next) can reduce the frequency with which this error occurs. Since Win10 always uses Hyper-V isolation, I'm guessing that changing the memory limit forces Docker to provision a new Hyper-V VM, preventing it from re-using an existing VM that may have fallen victim to whatever condition triggers the timeout.

from hcsshim.

Hi,
I alsot get this error, but mostly when the system is out of ram and is swapping.

from hcsshim.

I am running into the same issue from time to time by building the Dockerfile hosted here: https://github.com/mback2k/docker-windows-buildbot-msys2/blob/master/Dockerfile

from hcsshim.

Can confirm that this error still occurs as of the Windows 10 April 2018 update (Version 1803) with Docker CE Edge Version 18.05.0-ce-rc1-win63 (17439).

Though when I encounter the error, it is slightly different from what others have mentioned in this issue. I receive it during image creation with Docker timing out while waiting for the temporary container to shutdown.

The command 'cmd /S /C powershell c:/content/install.ps1' returned a non-zero code: 4294967295: failed to shutdown container: container 429047b1fdd4adcd8099081ecf0a2cee26fcb819121f2c90710b4fe2a2a58b63 encountered an error during WaitTimeout: hcsshim: timeout waiting for notification

Where install.ps1 simply runs a WiX-based installer bundle:

Start-Process -FilePath "c:/content/installer.exe" -ArgumentList "-q" -PassThru | Wait-Process

This was based on a suggestion made in Moby Issue #30395. The installer completes successfully with Exit code: 0x0, restarting: No, but the container itself hangs for approximately 5 minutes afterwards before the timeout.

The Dockerfile is:

FROM microsoft/iis:windowsservercore-ltsc2016

RUN mkdir C:\content

ADD content/ /content

RUN powershell c:/content/install.ps1

from hcsshim.

#Hi @darstahl I have the same issue when I run docker run from the Windows Insider Program build 17711. I am not able to do docker run for the microsoft/nanoserver-insder image. The following error I got.

docker run --isolation=hyperv -it microsoft/nanoserver-insider cmd

C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: container 16cf36301b02952d8a5befdacf025d996dc93985666483879aa8fef15218c8c9 encountered an error during CreateContainer: hcsshim: timeout waiting for notification extra info: {"SystemType":"Container","Name":"16cf36301b02952d8a5befdacf025d996dc93985666483879aa8fef15218c8c9","Owner":"docker","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\ProgramData\Docker\windowsfilter\16cf36301b02952d8a5befdacf025d996dc93985666483879aa8fef15218c8c9","Layers":[{"ID":"0be3898b-4e57-5b03-9fbe-6de1fec37724","Path":"C:\ProgramData\Docker\windowsfilter\449f1bb3e4f81fa2a6800de27c37066520954c92da260dd14cffd1171a75076e"}],"HostName":"","HvPartition":true,"EndpointList":["],"HvRuntime":{"ImagePath":"C:\ProgramData\Docker\windowsfilter\449f1bb3e4f81fa2a6800de27c37066520954c92da260dd14cffd1171a75076e\UtilityVM"},"AllowUnqualifiedDNSQuery":true}.

from hcsshim.

Hi there, i'm also getting same error as "hcsshim: timeout waiting for notification extra info" still unable to figure out the cause of problem and fix. Please comment if you know the way around to fix, thanks.

from hcsshim.

This issue seems to be temporary and it worked for me when i closed all of the running application on my machine which released few GB of RAM.
So, i can conclude it as "This error is coming due to memory issue", Thanks.

from hcsshim.

I've got a 16GB machine, I have nothing else running so have about 10GB ram free and I still get this problem. It's one painful error after another with Windows containers on Windows 10

I'm on Windows 10 1803 on my machine

from hcsshim.

This issue prevented one of my projects from progressing this week. I managed to fix it by reinstalling Docker for Windows.

Next are some details about it:

Environment specs

• Docker Version: 18.06.1-ce
• Storage Driver: windowsfilter (windows) lcow (linux)
• Kernel Version: 10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804)
• Operating System: Windows 10 Enterprise Version 1803 (OS Build 17134.285)
• Total Memory: 31.95GiB
• Daemon configuration file:

{
  "registry-mirrors": [],
  "insecure-registries": [],
  "debug": true,
  "experimental": true,
  "graph": "H:/ProgramData/Docker",
  "storage-opts": [
    "size=20G"
  ]
}

Notice that my docker storage folder is not in "C:". It is in an alternate folder.

Fixing Process

The PowerShell script provided here was used to uninstall Docker for Windows. Please read to the end before trying to use the script.

This is the script:

$ErrorActionPreference = "SilentlyContinue"

kill -force -processname 'Docker for Windows', com.docker.db, vpnkit, com.docker.proxy, com.docker.9pdb, moby-diag-dl, dockerd

try {
	./MobyLinux.ps1 -Destroy
} Catch {}

$service = Get-WmiObject -Class Win32_Service -Filter "Name='com.docker.service'"
if ($service) { $service.StopService() }
if ($service) { $service.Delete() }
Start-Sleep -s 5
Remove-Item -Recurse -Force "~/AppData/Local/Docker"
Remove-Item -Recurse -Force "~/AppData/Roaming/Docker"
if (Test-Path "C:\ProgramData\Docker") { takeown.exe /F "C:\ProgramData\Docker" /R /A /D Y }
if (Test-Path "C:\ProgramData\Docker") { icacls "C:\ProgramData\Docker\" /T /C /grant Administrators:F }
Remove-Item -Recurse -Force "C:\ProgramData\Docker"
Remove-Item -Recurse -Force "C:\Program Files\Docker"
Remove-Item -Recurse -Force "C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Docker"
Remove-Item -Force "C:\Users\Public\Desktop\Docker for Windows.lnk"
Get-ChildItem HKLM:\software\microsoft\windows\currentversion\uninstall | % {Get-ItemProperty $_.PSPath}  | ? { $_.DisplayName -eq "Docker" } | Remove-Item -Recurse -Force
Get-ChildItem HKLM:\software\classes\installer\products | % {Get-ItemProperty $_.pspath} | ? { $_.ProductName -eq "Docker" } | Remove-Item -Recurse -Force
Get-Item 'HKLM:\software\Docker Inc.' | Remove-Item -Recurse -Force
Get-ItemProperty HKCU:\software\microsoft\windows\currentversion\Run -name "Docker for Windows" | Remove-Item -Recurse -Force
#Get-ItemProperty HKCU:\software\microsoft\windows\currentversion\UFH\SHC | ForEach-Object {Get-ItemProperty $_.PSPath} | Where-Object { $_.ToString().Contains("Docker for Windows.exe") } | Remove-Item -Recurse -Force $_.PSPath
#Get-ItemProperty HKCU:\software\microsoft\windows\currentversion\UFH\SHC | Where-Object { $(Get-ItemPropertyValue $_) -Contains "Docker" }

The PowerShell script "MobyLinux.ps1" was also downloaded from here and executed by the previous one.

The script removed the files from:

• "C:\ProgramData\Docker"
• "C:\Program Files\Docker"
• "C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Docker"

However, the script did not properly uninstall Docker for Windows. It removed most of its files, leaving some files and key entries in the Windows Registry preventing the Docker installation to work afterwards.

After manually removing the damaged "Docker Engine" service and some entries from the Windows Registry, the "Docker for Windows Installer.exe" installation program was able to re-install docker.

Once Docker was installed the original Daemon configuration (displayed above) was restored and, without re-starting the machine, Docker started to work normally using the original images (sored in H:).

I'm unsure what exactly fixed the issue. It may be something the above script did or simply the fact of re-installing Docker for Windows.

I hope it helps.

from hcsshim.

Uninstalling 18.0.6 Stable and reinstalling Docker for Windows 2.0.0. beta1 Edge resolved the issue for me (I suspect reinstalling 18.0.6 would have worked as well). Not sure what caused the failure in the first place, but a Windows update had occurred between the time I had used Docker last and the time it broke.

from hcsshim.

Suddenly, this issue come again to me today. This is the second time for me. First time happened a few months ago, and resolved just by reinstaling Docker.
I created a PowerShell script to reproduce:

foreach($i in @(1..10)) {
  docker run --rm --memory=16g webkitdev/msbuild cmd /c echo pass $i
}

Here is the result.

pass 1
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: CreateComputeSystem 8958a4b18e2c257f0e8df7ae4ee4efa16178ded20436ec31e3332d1b923bd1d7: hcsshim: timeout waiting for notification
(extra info: {"SystemType":"Container", (...) ,"AllowUnqualifiedDNSQuery":true}).
pass 3
pass 4
pass 5
pass 6
pass 7
pass 8
pass 9
pass 10

PS C:\work\docker> docker version
Client: Docker Engine - Community
 Version:           18.09.0-ce-beta1
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        78a6bdb
 Built:             Thu Sep  6 22:42:13 2018
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0-ce-beta1
  API version:      1.39 (minimum version 1.24)
  Go version:       go1.10.3
  Git commit:       78a6bdb
  Built:            Thu Sep  6 22:57:45 2018
  OS/Arch:          windows/amd64
  Experimental:     true

from hcsshim.

Docker updated, threw an error during the update and appeared to uninstall itself. Reinstalled and now I get this message :(

I'm on Windows 10 (1709)
I also uninstalled and reinstalled Docker 2.0.0 and get the same error.

Client: Docker Engine - Community
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:47:51 2018
OS/Arch: windows/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.24)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:56:41 2018
OS/Arch: windows/amd64
Experimental: false

from hcsshim.

      Docker updated, threw an error during the update and appeared to uninstall itself.  Reinstalled and now I get this message :(

Client: Docker Engine - Community
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:47:51 2018
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.24)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:56:41 2018
OS/Arch: windows/amd64
Experimental: false

Same here. Unable to work after update. Changing memory limit is not helping

from hcsshim.

@jhowardmsft - Can you follow up here? It doesn't look like Darren ever got to the bottom of this.

from hcsshim.

I am getting this exact error: docker: Error response from daemon: container ABC encountered an error during CreateContainer: hcsshim: timeout waiting for notification.

I have found a rather inconvenient workaround. (⚠️WARNING: This will delete all of your images and containers!⚠️)

After trying to pull/run the image and it fails, stop all containers and run docker system prune -a. This will delete all of your images AND cached layers. Then, pull the image again. Make sure that none of the layers say layer already exists. Now, when running the image, it succeeds.

Unfortunately, this means that you must wipe your layer cache every time. In my instance, the image is very large with many layers, so this add HOURS to time for running the image. But, it will work consistently.

EDIT: Actually, this workaround does not work consistently, sorry for the misinformation.

from hcsshim.

Windows Server 2016 (RS1) or Windows Server 2019 (RS5)? We don't believe this error still occurs on RS5 builds and latest docker versions.

from hcsshim.

Windows 10 v1803

from hcsshim.

from hcsshim.

@ducttapecoder-vt

Can you provide output of docker version and docker info? And detailed repro steps/files needed to repro? And an indication if you're hitting this using Hyper-V containers or process isolated? 100% repro or intermittent? We might need ETL traces from Windows to root cause based on above.

from hcsshim.

Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        6247962
 Built:             Sun Feb 10 04:12:31 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.24)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 04:28:48 2019
  OS/Arch:          windows/amd64
  Experimental:     false

 Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.09.2
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804)
Operating System: Windows 10 Enterprise Version 1803 (OS Build 17134.590)
OSType: windows
Architecture: x86_64
CPUs: 12
Total Memory: 15.92GiB
Name: US509A4C4495E8
ID: 4JZP:RUDL:BNMW:GW7Z:EONZ:ASRI:GAGK:EW6Q:BEQ2:QBDE:FHLU:AQUT
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 26
 System Time: 2019-03-05T17:20:41.8230139-08:00
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Unfortunately, the images I am working with are proprietary and I cannot share them. But, they are based on servercore:1709_KB4480978 (10.0.16299.904).

This is on Windows 10, so I believe that means it must be hyper-v.

I can't 100% repro immediately after docker system prune -a, but after running the same image a number of times in a row, it will fail, and then continues to fail consistently after that.

from hcsshim.

@cowlinator Ah, this is 1803. See my previous comment. I'm really looking for a repro where this repros on Windows version 1809/RS5/Windows Server 2019.

from hcsshim.

I can reproduce failed to shutdown container, and subsequent terminate also failed [error=container xxx encountered an error during WaitTimeout: hcsshim: timeout waiting for notification module=libcontainerd container=xxx namespace=moby] in Windows Server 2019 with UBR 437 and 503 every time.

UBR 316 and 379 works well. We didn't see this issue with the same setup.

More detail info can be found via rancher/rancher#20440

Update:

UBR 529 fixed it! It works for us.

May 21, 2019—KB4497934 (OS Build OS 17763.529) https://support.microsoft.com/en-us/help/4497934

from hcsshim.

I am having the same issue, I have tried every suggestion in the forum and I am still unable to build the docker containers.

Has there been a definitive resolution for this?

from hcsshim.

I'm having this issue only on the newest Patch
mcr.microsoft.com/windows/servercore:1607-KB4512517 Docker image.
The image before that,
mcr.microsoft.com/windows/servercore:1607-KB4507460 is working.

docker run -it mcr.microsoft.com/windows/servercore:1607-KB4512517 cmd
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: hcsshim::CreateComputeSystem 1f978a1262f1c08848abb15602dfa829e739b5b365e23b39274b1866a164ff64: hcsshim: timeout waiting for notification
(extra info: {"SystemType":"Container","Name":"1f978a1262f1c08848abb15602dfa829e739b5b365e23b39274b1866a164ff64","Owner":"docker","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\Docker\\windowsfilter\\1f978a1262f1c08848abb15602dfa829e739b5b365e23b39274b1866a164ff64","Layers":[{"ID":"a9ab901e-0019-5e3d-a674-4047745fe107","Path":"C:\\ProgramData\\Docker\\windowsfilter\\3128f32ed0a0b7ced2503b3c32b8ff5bde17571a6787415d8e0475c234cb3968"},{"ID":"085dbfb9-4864-5b7d-9c85-30a0cb291186","Path":"C:\\ProgramData\\Docker\\windowsfilter\\bd6c5e246547defee9be0f35dba4f6cb79e198acc1ede93230ac380cf7212865"}],"HostName":"1f978a1262f1","HvPartition":true,"EndpointList":["E0BE75BE-65D6-4088-8761-E8440F384A46"],"HvRuntime":{"ImagePath":"C:\\ProgramData\\Docker\\windowsfilter\\3128f32ed0a0b7ced2503b3c32b8ff5bde17571a6787415d8e0475c234cb3968\\UtilityVM"},"AllowUnqualifiedDNSQuery":true}).

docker info

Client:
 Debug Mode: false

Server:
 Containers: 5
  Running: 0
  Paused: 0
  Stopped: 5
 Images: 59
 Server Version: 19.03.1
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics l2bridge l2tunnel nat null overlay transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
 Operating System: Windows 10 Pro Version 1809 (OS Build 17763.678)
 OSType: windows
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.88GiB
 Name: computername
 ID: anonym:OPWF
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: true
  File Descriptors: -1
  Goroutines: 28
  System Time: 2019-08-29T13:10:14.0679381+02:00
  EventsListeners: 1
 HTTP Proxy: http://someproxy.com:80
 HTTPS Proxy: http://someproxy.com:80
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

docker version

Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:17:08 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:25:07 2019
  OS/Arch:          windows/amd64
  Experimental:     false

from hcsshim.

Hi, we have same issue. I tried to analyze the hang, unfortunately, most of the analysis is guesswork. Perhaps something can be found useful from it.

docker version

Client:
 Version:           18.09.3
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        142dfcedca
 Built:             02/28/2019 06:33:17
 OS/Arch:           windows/amd64
 Experimental:      false
 
Server:
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.24)
  Go version:       go1.10.8
  Git commit:       142dfcedca
  Built:            02/28/2019 06:31:15
  OS/Arch:          windows/amd64
  Experimental:     false

Also i`ve collected extra info using slightly modified script from https://docs.microsoft.com/en-us/virtualization/windowscontainers/troubleshooting

Checking for common problems
Container Host OS Product Name: Windows Server 2019 Standard
Container Host OS Build Label: 17763.1.amd64fre.rs5_release.180914-1434
Showing output from: docker info
Containers: 24
 Running: 1
 Paused: 0
 Stopped: 23
Images: 4
Server Version: 18.09.3
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Standard Version 1809 (OS Build 17763.720)
OSType: windows
Architecture: x86_64
CPUs: 30
Total Memory: 48GiB
Name: *************
ID: NKUD:RO2K:6G34:ZMJB:HJAQ:PQWX:LINX:GB6H:A4EY:37FU:DLZG:KOU7
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): false
HTTP Proxy: *********************
No Proxy: 127.0.0.1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
 
 
Showing output from: docker version
Client:
 Version:           18.09.3
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        142dfcedca
 Built:             02/28/2019 06:33:17
 OS/Arch:           windows/amd64
 Experimental:      false
 
Server:
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.24)
  Go version:       go1.10.8
  Git commit:       142dfcedca
  Built:            02/28/2019 06:31:15
  OS/Arch:          windows/amd64
  Experimental:     false
 
Showing output from: docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
d9cd97f2f5bc        nat                 nat                 local
2937c9348a8a        none                null                local
 
Getting Warnings & errors in the Windows event logs from the last 24 hours
 
 
   ProviderName: docker
 
TimeCreated                     Id LevelDisplayName Message
-----------                     -- ---------------- -------
...
03.09.2019 14:44:53              1 Warning          driver error disconnecting container D97D86A5_container : The virtual machine or container was forcefully exited.
03.09.2019 14:44:53             11 Warning          failed to shutdown container [namespace=moby container=ec4dd407ba164e780fce8ea830d7f0bdd104b3bb92d61d618c39f3f7c5c2b78b process=init module=libcontainerd error=container ec4dd407ba164e780fce8ea830d7f0bdd104b3bb92d61...
03.09.2019 14:44:53             11 Error            failed to shutdown container, and subsequent terminate also failed [module=libcontainerd namespace=moby error=container ec4dd407ba164e780fce8ea830d7f0bdd104b3bb92d61d618c39f3f7c5c2b78b encountered an error during Wa...
03.09.2019 14:39:10              1 Warning          CreateCompleteSystem 4b18b01dfb3dd2fbfde34fdcaa59bd3234ac576d94eae965dac5f741cc377c5d: {"SystemType":"Container","Name":"4b18b01dfb3dd2fbfde34fdcaa59bd3234ac576d94eae965dac5f741cc377c5d","Owner":"docker","VolumePath...

I`ve noticed first warning:

03.09.2019 14:39:10 1 Warning CreateCompleteSystem 4b18b01dfb3dd2fbfde34fdcaa59bd3234ac576d94eae965dac5f741cc377c5d ... Did not complete within 4m0s. This may indicate a platform issue. If it appears to be making no forward progress, obtain the stacks and see is there is a syscall stuck in the platform API for a significant length of time.

After that i`ve got full memory dump of dockerd service and i saw following stack:

00 : ntdll!NtAlpcSendWaitReceivePort+0x14
01 : rpcrt4!LRPC_BASE_CCALL::SendReceive+0x12f
02 : rpcrt4!NdrpClientCall3+0x786
03 : rpcrt4!NdrClientCall3+0xf1
04 : vmcompute!ComputeService::Client::InvokeRpcFunction...
05 : vmcompute!ComputeService::Client::InvokeRpcFunctionWithRetry...
06 : vmcompute!HcsRpcServer::CreateSystem+0xa8
07 : vmcompute!HcsClient::ClientComputeSystem::Create+0x107
08 : vmcompute!HcsClient::CreateComputeSystem+0x137
09 : vmcompute!HcsCreateComputeSystem+0x100
0a : dockerd!cgo_topofstack+0xfbe
0b : 0x000000c0`44bcda70
0c : 0x000000c0`44449800

This thread waits for some ALPC request. I`ve noticed name of function
vmcompute!HcsCreateComputeSystem and i supposed that there may be RPC to vmcompute process.

Here is most interesting part of stack of vmcompute process:

00  ntdll!NtAlpcSendWaitReceivePort+0x14
...
11  vmcompute!ComputeService::Net::HnsAttachEndpoint+0x246
12  vmcompute!ComputeService::Management::Details::AttachNetworkEndpoints+0xed
13  vmcompute!ComputeService::Management::WindowsContainerOrchestrator::Construct+0x366
14  vmcompute!ComputeService::Management::ComputeSystemManager::CreateComputeSystem+0x5d5
15  vmcompute!HcsRpc_CreateSystem+0x34a
...

I saw that there was a call to another process, i think that this was HNS (Host Network Service) - see name of function: vmcompute!ComputeService::Net::HnsAttachEndpoint

I found interesting stack in HNS process:

00 ntdll!NtDeviceIoControlFile+0x14
01 KERNELBASE!DeviceIoControl+0x67
02 kernel32!DeviceIoControlImplementation+0x80
03 vmsif!LibIoctlPrivSendIoctl+0x142
04 vmsif!LibIoctlDeviceIoControl+0x1e6
05 vmsif!VmsIfNicDisconnect+0xc3
06 NetMgmtIF!NetMgmtDisconnectSwitchPort+0x73d
07 hostnetsvc!SwitchHelper::DisconnectNicPort+0xe1
08 hostnetsvc!SwitchHelper::DisconnectNicPort+0x5f
09 hostnetsvc!HNS::Service::Resource::HostPortResource::HardDisconnect+0xff
0a hostnetsvc!HNS::Service::Network::Endpoint::Disconnect+0x16a
0b hostnetsvc!HNS::Service::Network::Endpoint::Attach+0x31e
0c hostnetsvc!HNS::Service::Network::BaseNetwork::AttachEndpoint+0xd0
0d hostnetsvc!HNS::Service::Core::NetworkEntityManager::AttachEndpoint+0xff
0e hostnetsvc!HNS::Service::Request::EndpointRequest::AttachDetach+0x12d
0f hostnetsvc!HNS::Service::Request::EndpointRequest::Post+0x1dc
10 hostnetsvc!HNS::Service::Request::RequestManager::Process+0x15e
11 hostnetsvc!HNS::Service::Request::RequestManager::HandleRequest+0x116
12 hostnetsvc!HNS::Service::Server::ComHNSApi::Request+0x261

As you can see HNS tries to create new network device (according to function name). During this construction HNS sends IOCTL to some device (ntdll!NtDeviceIoControlFile). Unfortunately i do not have full memory kernel dump so my analysis is incomplete.

May be my analysis will help someone for further investigation of this issue (CC: @jhowardmsft )

from hcsshim.

Started receiving this error trying to run Windows images only (Linux ones run fine) after upgrading to 20H2 (19042.450). Have tried restarting and the problem still persists. Happy to provide traces if they're still useful, as I'm only experimenting with Docker currently so a) have nothing important/sensitive in there and b) don't mind if it stays broken for a few days

from hcsshim.

I am having this issue as well. @MatthewSteeples comment about 20H2 let me to try rolling back to 1909. However, that did not solve the problem. What I discovered through some trial an error was that the issue only manifests itself when running Windows 10 as a virtual machine and attempting to use Windows containers. Linux containers work fine, but I can only run Windows containers on a physical Windows 10 machine.

from hcsshim.

I hit the same issue on my setup when I run Windows containers, and I run the Windows VM on ESXi Server.
My Windows VM is configured with version as the following:

WindowsBuildLabEx                                       : 17763.1.amd64fre.rs5_
                                                          release.180914-1434
WindowsCurrentVersion                                   : 6.3
WindowsEditionId                                        : ServerStandard
WindowsInstallationType                                 : Server
WindowsInstallDateFromRegistry                          : 1/2/2020 11:04:05 AM
WindowsProductId                                        : 00429-70000-00000-AA1
                                                          52
WindowsProductName                                      : Windows Server 2019
                                                          Standard
WindowsRegisteredOrganization                           :
WindowsRegisteredOwner                                  : Windows User
WindowsSystemRoot                                       : C:\Windows
WindowsVersion                                          : 1809

The Windows hns service is stuck when attaching the HNSEndpoint to the containers, and I dump the stack of hns service as

from hcsshim.

@thejohnfreeman @scottresnik did either of you find a solution to running the windows containers from a Windows VM on a windows host? @scottresnik the solution you posted throws an incompatible OS error regardless of what version I pull

from hcsshim.