justinas / nixos-ha-kubernetes Goto Github PK
View Code? Open in Web Editor NEWToy highly-available Kubernetes cluster on NixOS
License: MIT License
Toy highly-available Kubernetes cluster on NixOS
License: MIT License
Flannel 0.15.1 contains a bug fix for the MAC address assignment bug.
nixos-ha-kubernetes/modules/worker/flannel.nix
Lines 44 to 56 in 6442aaa
However, Nixpkgs is still on 0.13.0.
Have a simple ping
or curl
call between pods (running in separate nodes) in check.sh. Network-MultiTool may prove useful.
It would be amazing if it was possible to use this with macvlans, so that I could treat this cluster as a basically real one within my network.
The CI could at least run ci-lint
script inside the nix-shell. There seems to be no way to run virtual machines on GH Actions, so no end-to-end testing can be done.
following the guide the setup always fails for me with ter apply
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
libvirt_volume.nixos_boot: Creating...
╷
│ Error: can't find storage pool 'default'
│
│ with libvirt_volume.nixos_boot,
│ on main.tf line 50, in resource "libvirt_volume" "nixos_boot":
│ 50: resource "libvirt_volume" "nixos_boot" {
│
╵
I made sure that I am in the correct group:
$ groups
users wheel networkmanager libvirtd docker
and that libvirtd is running:
$ systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/etc/systemd/system/libvirtd.service; enabled; preset: enabled)
Drop-In: /nix/store/y07n6kr0sk83gqji3lqzykzsrk94f319-system-units/libvirtd.service.d
└─overrides.conf
Active: active (running) since Thu 2023-03-23 15:12:51 CET; 14s ago
TriggeredBy: ● libvirtd-ro.socket
● libvirtd.socket
● libvirtd-admin.socket
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 5239 (.libvirtd-wrapp)
IP: 0B in, 0B out
IO: 0B read, 0B written
Tasks: 19 (limit: 32768)
Memory: 5.2M
CPU: 150ms
CGroup: /system.slice/libvirtd.service
└─5239 /nix/store/ziiig351fxfw1rizgnfpbkllxx2xlzvk-libvirt-8.10.0/sbin/libvirtd --config /nix/store/rd9dai9irpxsz9bw98fz47m1wb9zl31g-libvirtd.conf --timeout 120
Mär 23 15:12:51 gestalt systemd[1]: Starting Virtualization daemon...
Mär 23 15:12:51 gestalt systemd[1]: Started Virtualization daemon.
Hello!
Thank you for this great job!
I'm trying to create environment to learn k8s with your project.
After all steps ./chech.sh give this:
❯ ./check.sh
https://10.240.0.201:2379, 46916c050e984dc0, 3.5.9, 364 kB, false, false, 2, 370, 370,
https://10.240.0.216:2379, 96221fa7145e7558, 3.5.9, 360 kB, true, false, 2, 370, 370,
https://10.240.0.120:2379, db650873350377d7, 3.5.9, 360 kB, false, false, 2, 370, 370,
Kubernetes control plane is running at https://10.240.0.10
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
uid=0(root) gid=0(root) groups=0(root),10(wheel)
Server: 10.240.0.157
Address: 10.240.0.157:53
** server can't find kubernetes.cluster.local: NXDOMAIN
Name: kubernetes.default.svc.cluster.local
Address: 10.32.0.1
** server can't find kubernetes.svc.cluster.local: NXDOMAIN
** server can't find kubernetes.cluster.local: NXDOMAIN
** server can't find kubernetes.svc.cluster.local: NXDOMAIN
pod default/busybox terminated (Error)
Success.
So the next steps I tryed was:
nixos-ha-kubernetes on master [!?] via 💠 default via impure (nix-shell)
❯ k top pod
error: Metrics API not available
nixos-ha-kubernetes on master [!?] via 💠 default via impure (nix-shell)
❯ k top node
error: Metrics API not available
What do I need to do to get the right result?
Best wishes,
Sergei
Currently, we run one CoreDNS service per worker
node (good), but the containers' resolv.conf
points only to the host node itself as the name server (bad):
If Kubelet is alive, but CoreDNS is dead on the node, containers running on that node will fail to resolve names.
And I also tried to add dashboard with this commands:
nixos-ha-kubernetes on master [!?] via 💠 default via impure (nix-shell)
❯ helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/ --kubeconfig ./certs/generated/kubernetes/admin.kubeconfig
nixos-ha-kubernetes on master [!?] via 💠 default via impure (nix-shell)
❯ helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard --kubeconfig ./certs/generated/kubernetes/admin.kubeconfig
Release "kubernetes-dashboard" does not exist. Installing it now.
NAME: kubernetes-dashboard
LAST DEPLOYED: Thu Jun 6 05:47:44 2024
NAMESPACE: kubernetes-dashboard
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
*************************************************************************************************
*** PLEASE BE PATIENT: Kubernetes Dashboard may need a few minutes to get up and become ready ***
*************************************************************************************************
Congratulations! You have just installed Kubernetes Dashboard in your cluster.
To access Dashboard run:
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443
NOTE: In case port-forward command does not work, make sure that kong service name is correct.
Check the services in Kubernetes Dashboard namespace using:
kubectl -n kubernetes-dashboard get svc
Dashboard will be available at:
https://localhost:8443
And I get this issue:
nixos-ha-kubernetes on master [!?] via 💠 default via impure (nix-shell)
❯ k get pods -o wide -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kubernetes-dashboard-api-868878b978-z4l5w 0/1 CrashLoopBackOff 11 (3m11s ago) 34m 10.200.38.3 worker2 <none> <none>
kubernetes-dashboard-auth-66564b9c9c-xpwxr 1/1 Running 0 34m 10.200.38.5 worker2 <none> <none>
kubernetes-dashboard-kong-76dff7b666-g64jb 1/1 Running 0 34m 10.200.38.4 worker2 <none> <none>
kubernetes-dashboard-metrics-scraper-555758b9bf-f5xpf 1/1 Running 0 34m 10.200.5.3 worker1 <none> <none>
kubernetes-dashboard-web-846f5f49b-2gtgt 1/1 Running 0 34m 10.200.5.4 worker1 <none> <none>
nixos-ha-kubernetes on master [!?] via 💠 default via impure (nix-shell)
❯ k logs kubernetes-dashboard-api-868878b978-z4l5w -n kubernetes-dashboard
I0606 03:19:28.381701 1 main.go:40] "Starting Kubernetes Dashboard API" version="1.7.0"
I0606 03:19:28.381772 1 init.go:47] Using in-cluster config
E0606 03:19:28.381817 1 config.go:529] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: error creating pool from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: data does not contain any valid RSA or ECDSA certificates
F0606 03:19:28.384162 1 main.go:159] Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get "https://10.32.0.1:443/version": tls: failed to verify certificate: x509: certificate signed by unknown authority
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ
Can you please help me?
Best wishes,
Sergei
Normal Killing 6s (x2 over 79s) kubelet Stopping container tshoot
Normal SandboxChanged 5s (x2 over 79s) kubelet Pod sandbox changed, it will be killed and re-created.
The issue appeared in 97ac448
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.