Currently, Windows containers are not supported on MiniKube. This repo describes the steps for a prototype of running Windows containers on MiniKube.
Important: This is not a supported project by MiniKube. If the maintainers of MiniKube decide to support Windows containers, there's additional development work needed.
The goal of this project is to implement a Windows node to a MiniKube cluster, with the intent to try out Windows containers. What the steps in this project will do, is to use the existing Kubernetes mechanism to add a Windows node to your MiniKube cluster, configure the Flannel CNI so the Windows and Linux can co-exist and communicate with each other, to then allow you to deploy Windows containers to the Windows nodes. After completion, your MiniKube API won't be aware of the new Windows node, but the underlying Kubernetes infrastructure will. As a result, you can use kubectl, but not the minikube cli.
To get started you'll need:
- Windows 11 or 10 host with Hyper-V installed.
- Check out the documentation on how to install Hyper-V on Windows.
- An External Switch must be configured
- Check out the documentation on how to create a new Hyper-V Switch.
- MiniKube for Windows must be installed
- Check out the documentation on how to install MiniKube on Windows
- An Evaluation install media (ISO file) for Windows Server 2022
- You can download an evaluation media for Windows Server 2022 here.
Open a new elevated PowerShell session. Let's start by creating a new MiniKube cluster:
$SwitchName = Read-Host -Prompt "Please provide the name of the External Virtual Switch to be used"
minikube start --driver=hyperv --hyperv-virtual-switch=$SwitchName --nodes=2 --cni=flannel --container-runtime=containerdImportant: The above will deploy a 2 node cluster. You can change to a single node by removing '--nodes=2'. If deploy a single node, the command below needs to be run on just that node.
Next, we need to prepare the Linux nodes for Windows-specific Flannel CNI configuration:
minikube ssh
sudo sysctl net.bridge.bridge-nf-call-iptables=1
exit
minikube ssh -n minikube-m02
sudo sysctl net.bridge.bridge-nf-call-iptables=1
exitNext, we will update the Flannel CNI for Windows. For this project, I have configured the kube-flannel.yaml file with the appropriate settings:
wget -Uri https://raw.githubusercontent.com/vrapolinario/MinikubeWindowsContainers/main/kube-flannel.yaml -OutFile .\kube-flannel.yaml
kubectl apply -f kube-flannel.yamlNow we need to make sure the flannel daemon set is restarted to reflect the new Windows-specific configuration.
kubectl get ds -A
kubectl rollout restart ds kube-flannel-ds -n kube-flannel
kubectl get pods -AMake sure the pod has terminated and the new one is running. All pods should show a running state at this point. The output of the command should show:
PS C:\minikube\v2> kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-6mjwl 1/1 Running 0 56s
kube-flannel kube-flannel-ds-9q9rf 1/1 Running 0 89s
kube-system coredns-5d78c9869d-46hpn 1/1 Running 0 149m
kube-system etcd-minikube 1/1 Running 0 149m
kube-system kube-apiserver-minikube 1/1 Running 0 149m
kube-system kube-controller-manager-minikube 1/1 Running 0 149m
kube-system kube-proxy-2gs29 1/1 Running 0 148m
kube-system kube-proxy-bkmhw 1/1 Running 0 149m
kube-system kube-scheduler-minikube 1/1 Running 0 149m
kube-system storage-provisioner 1/1 Running 0 149mFor Linux nodes, you don't need to manually deploy the VM and OS, because MiniKube takes care of creating a new VM, downloading the OS image, and deploying the image to the VM. For Windows this is not supported, so we will perform these steps manually.
Let's start by creating a new VM:
$VMName = 'minikube-m03'
$SwitchName = Read-Host -Prompt "Please provide the name of the External Virtual Switch to be used (This should be the same Switch as the MiniKube VMs"
$ISOFile = Read-Host -Prompt "Please provide the full path for the Windows Server 2022 install media (ISO file)"
New-VM -Name $VMName -Generation 1 -MemoryStartupBytes 6000MB -Path ${env:homepath}\.minikube\machines\ -NewVHDPath ${env:homepath}\.minikube\machines\$VMName\VHD.vhdx -NewVHDSizeBytes 127000MB -SwitchName $SwitchName
Set-VM -Name $VMName -ProcessorCount 2 -AutomaticCheckpointsEnabled $false
Set-VMProcessor -VMName $VMName -ExposeVirtualizationExtensions $true
Set-VMDvdDrive -VMName $VMName -Path $ISOFile
Start-VM -Name $VMNameYou will need to install Windows Server 2022 on the VM. Follow the installation wizard to Install a Server Core instance:
After the installation is complete, you need to set up a new local admin password:
Once you're logged in, type 15 to exit the SCOnfig screen.
Important: The commands in this section need to be run inside the Windows node.
With the VM created and OS installed, we can now move to configuring the Windows node for MiniKube. Let's start by changing the instance name and installing the Continers feature:
Set-SConfig -AutoLaunch $false
Rename-Computer -NewName minikube-m03
Install-WindowsFeature -Name containers
Restart-Computer -ForceImportant: You can use a different name for this Windows instance. I'm keeping "minikube-m03" for consistency with the other nodes.
Now let's download ContainerD:
$Version="1.7.3"
curl.exe -L https://github.com/containerd/containerd/releases/download/v$Version/containerd-$Version-windows-amd64.tar.gz -o containerd-windows-amd64.tar.gz
tar.exe xvf .\containerd-windows-amd64.tar.gzAfter downloading, let's configure ContainerD for Windows:
Copy-Item -Path ".\bin\" -Destination "$Env:ProgramFiles\containerd" -Recurse -Force
cd $Env:ProgramFiles\containerd\
.\containerd.exe config default | Out-File config.toml -Encoding ascii
notepad config.tomlThe command above will copy the ContainerD binaries to the Program Files folder, and create a new config file. After that, Notepad will open so you can change some settings. You need to change the following parameters:
| Setting | Old value | New Value |
|---|---|---|
| bin_dir | "C:\\Program Files\\containerd\\cni\\bin" | "c:\\opt\\cni\\bin" |
| conf_dir | "C:\\Program Files\\containerd\\cni\\conf" | "c:\\etc\\cni\\net.d\\" |
Save the file and close Notepad. Now let's create the folders above and register (and start) the ContainerD service:
mkdir c:\opt\cni\bin
mkdir c:\etc\cni\net.d
.\containerd.exe --register-service
Start-Service containerd
get-service containerdThe output of the command above should show the ContainerD service as Running. Now, let's install NSSM:
mkdir c:\k
$arch = "win64"
curl.exe -L https://k8stestinfrabinaries.blob.core.windows.net/nssm-mirror/nssm-2.24.zip -o nssm.zip
tar.exe C c:\k\ -xvf .\nssm.zip --strip-components 2 */$arch/*.exeNow, let's install kubelet:
$KubernetesVersion="v1.27.3"
curl.exe -L https://dl.k8s.io/$KubernetesVersion/bin/windows/amd64/kubelet.exe -o c:\k\kubelet.exe
@"
# Start file needed to support kubeadm extra args
`$FileContent = Get-Content -Path "/var/lib/kubelet/kubeadm-flags.env"
`$kubeAdmArgs = `$FileContent.TrimStart(`'KUBELET_KUBEADM_ARGS=`').Trim(`'"`')
`$args = "--cert-dir=`$env:SYSTEMDRIVE/var/lib/kubelet/pki",
"--config=`$env:SYSTEMDRIVE/var/lib/kubelet/config.yaml",
"--bootstrap-kubeconfig=`$env:SYSTEMDRIVE/etc/kubernetes/bootstrap-kubelet.conf",
"--kubeconfig=`$env:SYSTEMDRIVE/etc/kubernetes/kubelet.conf",
"--hostname-override=$(hostname)",
"--enable-debugging-handlers",
"--cgroups-per-qos=false",
"--enforce-node-allocatable=``"``"",
"--resolv-conf=``"``""
`$kubeletCommandLine = "c:\k\kubelet.exe " + (`$args -join " ") + " `$kubeAdmArgs"
Invoke-Expression `$kubeletCommandLine
"@ | Set-Content -Path c:\k\Start-kubelet.ps1
c:\k\nssm.exe install kubelet Powershell -ExecutionPolicy Bypass -NoProfile c:\k\Start-kubelet.ps1
c:\k\nssm.exe set Kubelet AppStdout C:\k\kubelet.log
c:\k\nssm.exe set Kubelet AppStderr C:\k\kubelet.err.logThe command above will download and install kubelet for Windows. However, notice the kubelet service will not start until a further stage on which we join the Windows node to the MiniKube Kubernetes cluster using kubeadm. Now we need to open the appropriate ports on Windows Firewall:
New-NetFirewallRule -Name kubelet -DisplayName 'kubelet' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250The next step is a hack for the Windows node to be able to find the master Linux node via its FQDN:
notepad C:\Windows\System32\drivers\etc\hosts
# Add the IP address and control-plane.minikube.internal to the Hosts fileYou will need to append the IP address of the master node to the Hosts file on the Windows node.
You can find the IP address of the master Linux node by running the following command on your machine:
minikube node list
Your hosts file should look similar to this:
Save the file and close Notepad. Now, let's download kubeadm:
curl.exe -L https://dl.k8s.io/$KubernetesVersion/bin/windows/amd64/kubeadm.exe -o c:\k\kubeadm.exe
cd c:\kAt this point, your Windows node is essentially ready to join the MiniKube Kubernetes cluster. However, you will need the join command from the Linux nodes to proceed.
To get the join command for kubeadm, run the following on your machine:
minikube ssh
cd /var/lib/minikube/binaries/v1.27.3/
sudo ./kubeadm token create --print-join-commandCopy the output for the join command. You will need to append --cri-socket "npipe:////./pipe/containerd-containerd" to the command. On the Windows node, run the command provided on the previous step with the above setting for ContainerD. Your command will look like this:
.\kubeadm.exe join control-plane.minikube.internal:8443 --token c3k6qb.zn7kjtq7l74m0zcz --discovery-token-ca-cert-hash sha256:8ceaf34a09e0e5cf4f82d523d3c9068e78d390b221d606dc6f1224400b164f6c --cri-socket "npipe:////./pipe/containerd-containerd"Important: An error at this stage is expected. MiniKube expects to find some files in a different folder than kubelet, so we need to fix that before the command succeed. Do not stop the command until it finishes.
After getting the error from kubelet, run the following to fix it:
mkdir c:\var\lib\minikube\certs
Copy-Item C:\etc\kubernetes\pki\ca.crt -Destination C:\var\lib\Minikube\Certs
Remove-Item C:\etc\kubernetes\pki\ca.crtNow run the kubeadm join command again. This time, the command should succeed.
To ensure your Windows node has properly joined the cluster, you can run the following on your machine:
kubectl get nodes -o wideThe output of the command should show the Windows node along with the Linux nodes.
PS C:\Users\viniap> kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready control-plane 14s v1.27.3 10.137.188.75 <none> Buildroot 2021.02.12 5.10.57 containerd://1.7.2
minikube-m02 Ready <none> 14s v1.27.3 10.137.188.117 <none> Buildroot 2021.02.12 5.10.57 containerd://1.7.2
minikube-m03 NotReady <none> 14s v1.27.3 10.137.190.85 <none> Windows Server 2022 Datacenter Evaluation 10.0.20348.587 containerd://1.7.3You will notice the status of the Windows node it "NotReady". This is because the networking for this node is not properly configured yet.
Now that the Windows node has joined the cluster, we can configure the networking settings as any Kubernetes cluster, by using kubectl. To get started, let's apply the Flannel Overlay configuration by leveraging the official tools from SIG-Windows:
wget https://raw.githubusercontent.com/vrapolinario/MinikubeWindowsContainers/main/flannel-overlay.yaml -OutFile .\flannel-overlay.yaml
kubectl apply -f .\flannel-overlay.yamlNow we need to configure kube-proxy. For the purpose of this project, I have configured the YAML file:
wget https://raw.githubusercontent.com/vrapolinario/MinikubeWindowsContainers/main/kube-proxy.yaml -OutFile .\kube-proxy.yaml
kubectl apply -f .\kube-proxy.yamlThe commands above will result in two new pods being created under the kube-system namespace. You can check the progress of the deployment by running:
PS C:\minikube\v2> kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-6mjwl 1/1 Running 0 3h7m
kube-flannel kube-flannel-ds-9q9rf 1/1 Running 0 3h8m
kube-flannel kube-flannel-ds-windows-amd64-psr5z 1/1 Running 0 4m42s
kube-system coredns-5d78c9869d-46hpn 1/1 Running 0 5h36m
kube-system etcd-minikube 1/1 Running 0 5h36m
kube-system kube-apiserver-minikube 1/1 Running 0 5h36m
kube-system kube-controller-manager-minikube 1/1 Running 0 5h36m
kube-system kube-proxy-2gs29 1/1 Running 0 5h35m
kube-system kube-proxy-bkmhw 1/1 Running 0 5h36m
kube-system kube-proxy-windows-xbsk9 1/1 Running 1 (3m4s ago) 3m10s
kube-system kube-scheduler-minikube 1/1 Running 0 5h36m
kube-system storage-provisioner 1/1 Running 0 5h36mYou can now check the status of your Windows node by running the kubectl get nodes -o wide command again as above. The output now should show the Windows node as Ready:
PS C:\minikube\v2> kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready control-plane 5h42m v1.27.3 10.137.188.75 <none> Buildroot 2021.02.12 5.10.57 containerd://1.7.2
minikube-m02 Ready <none> 5h41m v1.27.3 10.137.188.93 <none> Buildroot 2021.02.12 5.10.57 containerd://1.7.2
minikube-m03 Ready <none> 25m v1.27.3 10.137.188.106 <none> Windows Server 2022 Datacenter Evaluation 10.0.20348.587 containerd://1.7.3Congrats! Now your MiniKube Kubernetes cluster is ready receive a Windows container application.
The example here simply brings up a new container with exposed ports. The intent is to simply prove that a Windows container is running.
To get a Server Core container running on your environment with LogMonitor as the entry process to monitor logs emitted to STDOUT, you can run:
wget https://raw.githubusercontent.com/vrapolinario/MinikubeWindowsContainers/main/iis-log-monitor.yaml -OutFile .\iis-log-monitor.yaml
kubectl apply -f .\iis-log-monitor.yamlIt will take a while for the image to download. You can check the status of the container by running:
kubectl get pods Now the IIS pod should appear in the list of pods
PS C:\minikube\v2> kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default iis-logmonitor-5cfbfbf855-mxjm2 1/1 Running 0 19m
kube-flannel kube-flannel-ds-6mjwl 1/1 Running 0 3h7m
kube-flannel kube-flannel-ds-9q9rf 1/1 Running 0 3h8m
kube-flannel kube-flannel-ds-windows-amd64-psr5z 1/1 Running 0 4m42s
kube-system coredns-5d78c9869d-46hpn 1/1 Running 0 5h36m
kube-system etcd-minikube 1/1 Running 0 5h36m
kube-system kube-apiserver-minikube 1/1 Running 0 5h36m
kube-system kube-controller-manager-minikube 1/1 Running 0 5h36m
kube-system kube-proxy-2gs29 1/1 Running 0 5h35m
kube-system kube-proxy-bkmhw 1/1 Running 0 5h36m
kube-system kube-proxy-windows-xbsk9 1/1 Running 1 (3m4s ago) 3m10s
kube-system kube-scheduler-minikube 1/1 Running 0 5h36m
kube-system storage-provisioner 1/1 Running 0 5h36mOnce the container is in a "Running" state, you can interact with it:
kubectl exec <pod-name> -- powershell dirThe above will execute the "dir" command inside the container and return the output.
PS C:\minikube\v2> kubectl exec iis-logmonitor-5cfbfbf855-mxjm2 -- powershell dir
Directory: C:\LogMonitor
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 9/13/2023 9:45 AM 738304 LogMonitor.exe
-a---- 9/18/2023 5:50 PM 1442 LogMonitorConfig.jsonWe can get the logs streamed to STDOUT using the command below
kubectl logs <pod-name>
Use the command below which maps an external ip to the service
minikube tunnel 
Open a second powershell window and get the EXTERNAL-IP that we will use to access the default IIS page hosted in iis-logmonitor app
PS C:\minikube\v2> kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
iis-logmonitor LoadBalancer 10.107.41.1 10.107.41.1 80:31379/TCP 12h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 18hIn the above output the EXTERNAL-IP is 10.107.41.1. To get to the IIS Windows Server page access this link on your browser http://10.107.41.1:80 and you should see the webpage below
- The issue with networking has been solved. You can now fully use Windows nodes and Windows containers with MiniKube.
- While Windows containers will run fine in this prototype, there's a known networking issue. MiniKube exposes NodePort or LoadBalancer in a different way than regular Kubernetes. Since this prototype (as of this date) does not inform MiniKube of the new node, MiniKube is unable to expose ports (and consequently the service) for the Windows node. There's currently an issue open and an ask for help to get this working on GitHub.
- You might see a "CrashLoopBackError" for the kube-proxy pod after applying the final configurations before it shows the status "Running". Since this pod uses host process containers to configure the host, waiting a few retries should work and the pod should show "Running" after a few attempts.
- If you try to run a Windows container and it fails, you might need to update the Windows node. Open the VM and log in. Type "sconfig" on PowerShell and select option #6 to start Windows update.
To clean up your environment, you can run the following:
#Delete MiniKube cluster
minikube delete --all
#Remove VM
$VMName = 'minikube-m03'
Stop-VM -Name $VMName -TurnOff
Remove-VM -Name $VMName -Force
Remove-Item -Path ${env:homepath}\.minikube\machines\$VMName -Force -Recurse
#Remove External Virtual Switch
Remove-VMSwitch $SwitchName
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent