kubernetes-retired / frakti Goto Github PK
View Code? Open in Web Editor NEWThe hypervisor-based container runtime for Kubernetes.
License: Apache License 2.0
The hypervisor-based container runtime for Kubernetes.
License: Apache License 2.0
For now we will list all pods (containers) managed by hyperd, thus user created pod will also be parsed by frakti and report failed to parse container name
.
We need to consider add a filter to it. (e.g. special prefix?)
Need to support CNI plugin in Kubernetes + Frakti. The implementation requires finding a way to generate network configuration from CNI and pass these information to hyperd in expected syntax.
A very similar approach is CNI plugin support in CRI-O, please check these for details:
cri-o/cri-o#237
https://github.com/kubernetes-incubator/cri-o#setup-cni-networking
TODOs:
make install
When creating a pod with host network, it failed to create symbolic link for logs (warning).
pod config
apiVersion: v1
kind: Pod
metadata:
name: pod
labels:
name: pod
spec:
hostNetwork: true
dnsPolicy: ClusterFirst
containers:
- name: nginx
image: nginx
Kubelet events:
df21b37c7a7f3aff14ba2034b76d4896406cf501cc59f80b74b49e032934
36m 36m 1 kubelet, instance-2 spec.containers{nginx} Warning Failed Failed to start container with id a741df21b37c7a7f3aff14ba2034b76d4896406cf501cc59f80b74b49e032934 with error: rpc error: code = 2 desc = failed to create symbolic link "/var/log/pods/318a363b-0f73-11e7-9e6c-42010a8c0003/nginx_0.log" to the container log file "" for container "a741df21b37c7a7f3aff14ba2034b76d4896406cf501cc59f80b74b49e032934": symlink /var/log/pods/318a363b-0f73-11e7-9e6c-42010a8c0003/nginx_0.log: no such file or directory
36m 36m 1 kubelet, instance-2 Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "nginx" with rpc error: code = 2 desc = failed to create symbolic link "/var/log/pods/318a363b-0f73-11e7-9e6c-42010a8c0003/nginx_0.log" to the container log file "" for container "a741df21b37c7a7f3aff14ba2034b76d4896406cf501cc59f80b74b49e032934": symlink /var/log/pods/318a363b-0f73-11e7-9e6c-42010a8c0003/nginx_0.log: no such file or directory: "Start Container Failed"
This is a issue for Google Summer of Code 2017, view docs here.
Frakti is a well-designed CRI implementation for hypervisor-based runtimes, it would be not so hard to have Unikernels runtime into its picture.
A build-in unikshim
(or any other name) for frakti need to be developed to manage Unikernels workloads.
Implementation Tips:
kubelet -> frakti -> hypervisor manager (libvirt or QEMU) -> Unikernels machines
Goals
one-pod-one-container
model is required.Nice To Have
unikshim
.Non Goals
emptyDir
and hostPath
.Is this normal behavior. Under kubernetes without hyperd, I noticed that there is no attempt to spin up DNS pods. However, when I issue a kubeadm reset to start over there is an attempt by hyperd to start a kube-dns pod:
hyperd[1009]: E0524 10:19:59.988202 1009 persist.go:113] Pod[k8s_POD.0_kube-dns-3913472980-2jz1j_kube-system_af1d4552-3f1e-11e7-aeb4-0021ccd48ab5_10cd9672] failed to load sandbox info: leveldb: not found
Have not noticed this behavior under a non-hyperd kubernetes cluster.
Perhaps this is just a remnant but I did perform a kubeadm reset and then decided to reboot before peforming a new kubeadm init. In checking the status of hyperd after the reboot (prior to peforming a new kubeadm init) the above was the result.
Also when a kubeadm reset is done, all files in /etc/cni/net.d are removed. This should be noted in the deploy.md just as a warning or note
Thanks
DB
I really like the idea of being able to schedule pods into lightweight vm's. There are times though when having access to the underlying node is needed. So its rather awkward to only have one at a time. It would be awesome if the docker backend and hyper backend could co'exist on the host at the same time, and pick one or the other based on pod annotation. You could then even possibly use kubernetes itself, with a daemonset that's docker based, to enable and manage hyper support on a kubernetes cluster. Looking at hyper a bit, it looks like it could containerize nicely.
We should improve unit tests coverage.
CRI has changed its API to proto3. We should upgrade our vendor.
Many CNI Plugins don't properly support DEL (See kubernetes/kubernetes#43488 for more details).
We should upgrade to CNI v0.5.1 after kubernetes/kubernetes#43546 is merged.
In fresh new environment:
E0314 02:26:12.316677 11713 hyper_checkpoint.go:119] Failed to list checkpoint store: open /var/lib/frakti/sandbox: no such file or directory
@YaoZengzeng Is this directory expected to automatically created?
The current frakti requires CNI plugins set before starting. However, with latest kubeadm and kubernetes, network plugins may be initialized dynamically.
Related upstream issue: kubernetes/kubernetes#43474.
There are many redundant flags because of introducing dockershim. We should clean them up.
--storage-driver-buffer-duration duration Writes in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction (default 1m0s)
--storage-driver-db string database name (default "cadvisor")
--storage-driver-host string database host:port (default "localhost:8086")
--storage-driver-password string database password (default "root")
--storage-driver-secure use secure connection with database
--storage-driver-table string table name (default "stats")
--storage-driver-user string database username (default "root")
Encountered when verifying PR #62 by running kubectl run --image=nginx nginx
:
I0118 11:38:52.943145 23702 manager.go:177] CreateContainer with request pod_sandbox_id:"k8s_POD.0_nginx-2932712833-gc82w_default_78e43fe8-dd72-11e6-b3f3-fa163ea7515e_a76f41fe" config:<metadata:<name:"nginx" attempt:0 > image:<image:"" > working_dir:"" envs:<key:"KUBERNETES_SERVICE_HOST" value:"10.0.0.1" > envs:<key:"KUBERNETES_SERVICE_PORT" value:"443" >....
E0118 11:38:52.943961 23702 container.go:45] Create container nginx in pod k8s_POD.0_nginx-2932712833-gc82w_default_78e43fe8-dd72-11e6-b3f3-fa163ea7515e_a76f41fe failed: rpc error: code = 2 desc = No command specified
The problem is image:<image:"" >
, which expected to be imageRef of nginx image.
cc/ @resouer @YaoZengzeng
The design doc of pod level resource has been updated with more details, and there's something different from our original design:
according to https://github.com/kubernetes/community/blob/master/contributors/design-proposals/pod-resource-management.md#burstable-qos, there's chance that a BU pod does not have memory limit, so it's mem limit will be set to node's available resource. But frakti will keep using this value (which is obvious too big), it would be wrong.
Need to figure out this use case and set default value instead (may need to import qos pkg).
Refer kubernetes/kubernetes#29579.
This could be done by using kubelet streaming liberay.
Currently, sandbox checkpoint is only removed after the pod is deleted successfully. However, if the hyper pod has already been removed outside, the checkpoint will be left over and kubelet will always fail on stopping sandbox.
We should ensure checkpoint and network plugin are cleaned up properly once frakti has detected the sandbox has already been non-exist.
(Are the deployment instructions even up to date? starting to see some steps left out)
Using Centos 7.3 with usual repo updates
Follow the deployment steps per : https://github.com/kubernetes/frakti/blob/master/docs/deploy.md
Using 3 VMs (1 master, 2 minons) or that's the plan
By the way the repos are updated with current 1.6.2, the "note" below the yum install references an older 1.6.0-beta 4
On master run kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version stable (latest gives 1.7 alpha)
kubeadm does all of the preflight checks and as expected everything passes. Note there should be a step prior to running this that enables and starts kubelet service, otherwise the warning will pop up.
result... hung at:
[apiclient] Created API client, waiting for the control plane to become ready
No docker images are pulled, no docker containers appear to be running or created
First time I have had issues with kubeadm under a normal docker based install over attempting a fresh Frakti one. These are all fresh VMs so nothing legacy on them other than Centos 7.3. FOllowed deployment instructions exactly.
DB
Also curious as to why this has two Hypervisor entries: refer to Hypervisor=libvert and Hypervisor=qemu, is this not redundant and would the config not read the second entry?
echo -e "Hypervisor=libvirt\n
Kernel=/var/lib/hyper/kernel\n
Initrd=/var/lib/hyper/hyper-initrd.img\n
Hypervisor=qemu\n
StorageDriver=overlay\n
gRPCHost=127.0.0.1:22318" > /etc/hyper/config
Another thing to note is kubelet failed:
-- Unit kubelet.service has begun starting up.
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.355299 2726 feature_gate.go:144] feature gates: map[DynamicVolumeProvisioning:true TaintBasedE
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.359676 2726 server.go:232] Starting Kubelet configuration sync loop
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.359702 2726 server.go:407] failed to init dynamic Kubelet configuration sync: cloud provider w
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.375962 2726 manager.go:143] cAdvisor running in container: "/"
May 01 18:14:33 kube01 kubelet[2726]: W0501 18:14:33.476000 2726 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.550699 2726 fs.go:117] Filesystem partitions: map[/dev/vda1:{mountpoint:/ major:253 minor:1 fs
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.552778 2726 manager.go:198] Machine: {NumCores:2 CpuFrequency:2399998 MemoryCapacity:397522534
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.588629 2726 manager.go:204] Version: {KernelVersion:3.10.0-514.16.1.el7.x86_64 ContainerOsVers
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.589372 2726 server.go:509] --cgroups-per-qos enabled, but --cgroup-root was not specified. de
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.591154 2726 container_manager_linux.go:245] container manager verified user specified cgroup-r
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.591188 2726 container_manager_linux.go:250] Creating Container Manager object based on Node Co
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.591400 2726 kubelet.go:255] Adding manifest file: /etc/kubernetes/manifests
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.591442 2726 kubelet.go:265] Watching apiserver
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.597608 2726 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.597697 2726 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.597777 2726 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1
May 01 18:14:33 kube01 kubelet[2726]: W0501 18:14:33.598688 2726 kubelet_network.go:63] Hairpin mode set to "promiscuous-bridge" but container runt
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.598721 2726 kubelet.go:494] Hairpin mode set to "none"
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.598922 2726 remote_runtime.go:41] Connecting to runtime service /var/run/frakti.sock
May 01 18:14:33 kube01 kubelet[2726]: 2017/05/01 18:14:33 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "t
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.599459 2726 remote_runtime.go:63] Version from runtime service failed: rpc error: code = 14 de
May 01 18:14:33 kube01 kubelet[2726]: 2017/05/01 18:14:33 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "t
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.600929 2726 kuberuntime_manager.go:154] Get runtime version failed: rpc error: code = 14 desc
May 01 18:14:33 kube01 kubelet[2726]: error: failed to run Kubelet: failed to create kubelet: rpc error: code = 14 desc = grpc: the connection is unav
May 01 18:14:33 kube01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
May 01 18:14:33 kube01 systemd[1]: Unit kubelet.service entered failed state.
May 01 18:14:33 kube01 systemd[1]: kubelet.service failed.
We could switch most tests to critest, while only keep some frakti-specific tests.
# crictl exec $cid ps aux
PID USER TIME COMMAND
1 root 0:00 /init
3 root 0:00 top
4 root 0:00 ps aux
execing command in container failed: command terminated with exit code 0
After running the allinone.sh, kubeadm init completes as expect. However, in checking service status, frakti shows this:
● frakti.service - Hypervisor-based container runtime for Kubernetes
Loaded: loaded (/usr/lib/systemd/system/frakti.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-05-09 18:12:23 UTC; 3min 56s ago
Docs: https://github.com/kubernetes/frakti
Main PID: 3199 (frakti)
CGroup: /system.slice/frakti.service
└─3199 /usr/bin/frakti --v=3 --log-dir=/var/log/frakti --logtostderr=false --cgroup-driver=systemd --listen=/var/run/frakti.sock --strea...
May 09 18:12:23 kube01 systemd[1]: Started Hypervisor-based container runtime for Kubernetes.
May 09 18:12:23 kube01 systemd[1]: Starting Hypervisor-based container runtime for Kubernetes...
May 09 18:12:23 kube01 frakti[3199]: E0509 18:12:23.563497 3199 ocicni.go:110] error updating cni config: No networks found in /etc/cni/net.d
May 09 18:12:23 kube01 frakti[3199]: E0509 18:12:23.571272 3199 ocicni.go:110] error updating cni config: No networks found in /etc/cni/net.d
May 09 18:12:24 kube01 systemd[1]: [/usr/lib/systemd/system/frakti.service:8] Unknown lvalue 'TasksMax' in section 'Service'
May 09 18:12:38 kube01 systemd[1]: [/usr/lib/systemd/system/frakti.service:8] Unknown lvalue 'TasksMax' in section 'Service'
[root@kube01 ~]# ls /etc/cni/net.d
10-mynet.conf 99-loopback.conf
To fix the nil pointer here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/security_context.go#L129
cc @Crazykev Watch out for staging
part
Readonly rootfs is not supported because hyperd. We should fix it after hyperhq/hyperd#638.
The test is disabled at #119 because of hyperhq/hyperd#604.
We should enable it again after hyperhq/hyperd#604 is fixed.
I'm thinking about spinning this back up across physical machines, about 5 HP Proliants to test. Where are things with supporting Calico or Flannel CNI?
Thanks
DB
Now frakti cni integration only support bridge plugin. I think it is necessary to support non bridge plugins like macvlan, calico etc. @feiskyer
Kubernetes service is not accessible after pod created, it is related with a hyperd bug introduced recently hyperhq/hyperd#468.
When I use cri-tools to test frakti(docker runtime), I find stop container will time out.
• Failure [60.490 seconds]
[k8s.io] Container
/home/xlgao/gocode/src/github.com/kubernetes-incubator/cri-tools/pkg/framework/framework.go:72
runtime should support basic operations on container
/home/xlgao/gocode/src/github.com/kubernetes-incubator/cri-tools/pkg/validate/container.go:111
runtime should support stopping container [Conformance] [It]
/home/xlgao/gocode/src/github.com/kubernetes-incubator/cri-tools/pkg/validate/container.go:89
Mar 31 17:04:17.269: stop container "0924ab698482d90591ff93552a44144340836760c04929370e834bd41abcef7d" timeout.
/home/xlgao/gocode/src/github.com/kubernetes-incubator/cri-tools/pkg/validate/container.go:215
Does the version need updating in the code. I cloned the master branch, compiled, and result of frakti --version is:
I0605 11:24:34.272388 11749 frakti.go:71] frakti version: 0.1
Should this be 0.2? just wondering as I want to make sure I have the latest code with the ability to specify cpu and mem.
Thanks
DB
CNI doesn't do port-mapping today. We could support port mapping via iptables.
Hypernetes doesn't support port forwarding. We should decide whether/how we could support it in frakti.
Result:
KUBELET
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Mon 2017-06-05 12:52:23 CDT; 1s ago
Docs: http://kubernetes.io/docs/
Process: 7419 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CGROUP_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
Main PID: 7419 (code=exited, status=1/FAILURE)
Jun 05 12:52:23 master01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Jun 05 12:52:23 master01 systemd[1]: Unit kubelet.service entered failed state.
Jun 05 12:52:23 master01 systemd[1]: kubelet.service failed.
HYPERD
● hyperd.service - hyperd
Loaded: loaded (/usr/lib/systemd/system/hyperd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2017-06-05 12:50:52 CDT; 4min 46s ago
Docs: http://docs.hypercontainer.io
Main PID: 7193 (hyperd)
CGroup: /system.slice/hyperd.service
└─7193 /usr/bin/hyperd --log_dir=/var/log/hyper
Jun 05 12:50:52 master01 systemd[1]: Started hyperd.
Jun 05 12:50:52 master01 systemd[1]: Starting hyperd...
Jun 05 12:50:53 master01 hyperd[7193]: time="2017-06-05T12:50:53-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jun 05 12:50:53 master01 hyperd[7193]: time="2017-06-05T12:50:53-05:00" level=info msg="Firewalld running: false"
Jun 05 12:50:53 master01 hyperd[7193]: time="2017-06-05T12:50:53-05:00" level=info msg="Loading containers: start."
Jun 05 12:50:53 master01 hyperd[7193]: ...
Jun 05 12:50:53 master01 hyperd[7193]: time="2017-06-05T12:50:53-05:00" level=info msg="Loading containers: done."
Jun 05 12:50:53 master01 hyperd[7193]: E0605 12:50:53.215808 7193 persist.go:113] Pod[k8s_POD.0_kube-dns-2612786695-kqqhv_kube-system_4664f72a-409d-11e7-9579-0021ccd48ab5_9acb0442] failed to load sandbox info: leveldb: not found
FRAKTI
● frakti.service - Hypervisor-based container runtime for Kubernetes
Loaded: loaded (/usr/lib/systemd/system/frakti.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2017-06-05 12:50:31 CDT; 5min ago
Docs: https://github.com/kubernetes/frakti
Main PID: 7126 (frakti)
CGroup: /system.slice/frakti.service
└─7126 /usr/bin/frakti --v=3 --log-dir=/var/log/frakti --logtostderr=false --cgroup-driver=systemd --listen=/var/run/frakti.sock --streaming-server-addr=master01 --hyper-endpoint=127.0.0.1:22318
Jun 05 12:50:31 master01 systemd[1]: Started Hypervisor-based container runtime for Kubernetes.
Jun 05 12:50:31 master01 systemd[1]: Starting Hypervisor-based container runtime for Kubernetes...
Jun 05 12:50:52 master01 frakti[7126]: 2017/06/05 12:50:52 transport: http2Client.notifyError got notified that the client transport was broken EOF.
Jun 05 12:50:52 master01 frakti[7126]: 2017/06/05 12:50:52 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 127.0.0.1:22318: getsockopt: connection refused"; Reconnecting to {127.0.0.1:22318 }
Jun 05 12:52:22 master01 frakti[7126]: 2017/06/05 12:52:22 transport: http2Server.HandleStreams failed to read frame: read unix /var/run/frakti.sock->@: read: connection reset by peer
Jun 05 12:55:24 master01 frakti[7126]: 2017/06/05 12:55:24 transport: http2Server.HandleStreams failed to read frame: read unix /var/run/frakti.sock->@: read: connection reset by peer
Jun 05 12:55:45 master01 frakti[7126]: 2017/06/05 12:55:45 transport: http2Server.HandleStreams failed to read frame: read unix /var/run/frakti.sock->@: read: connection reset by peer
Another thing I've noticed is that that does not survive reboots all that well.
Thoughts where I should look?
DB
I0315 20:12:03.360743 12416 manager.go:107] RunPodSandbox with request &RunPodSandboxRequest{Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:busybox,Uid:c57bea76-08a6-11e7-a282-4ccc6a36dc71,Namespace:default,Attempt:0,},Hostname:busybox,LogDirectory:/var/log/pods/c57bea76-08a6-11e7-a282-4ccc6a36dc71,DnsConfig:&DNSConfig{Servers:[8.8.8.8],Searches:[],Options:[ndots:5],},PortMappings:[&PortMapping{Protocol:TCP,ContainerPort:8080,HostPort:0,HostIp:,}],Labels:map[string]string{io.kubernetes.pod.name: busybox,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: c57bea76-08a6-11e7-a282-4ccc6a36dc71,name: busybox,},Annotations:map[string]string{kubernetes.io/config.seen: 2017-03-14T19:10:02.897218293+08:00,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/besteffort/podc57bea76-08a6-11e7-a282-4ccc6a36dc71,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{HostNetwork:false,HostPid:false,HostIpc:false,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,},},},}
I0315 20:12:03.363198 12416 network.go:66] Get interface information in container ns, skip non-veth device lo
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x48b8d2]
goroutine 31 [running]:
panic(0x15fb640, 0xc4200140b0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
k8s.io/frakti/pkg/hyper.networkInfoFromNs(0x2320f00, 0xc420478370, 0x37)
/home/crazykev/go-project/src/k8s.io/frakti/pkg/hyper/network.go:46 +0xa2
k8s.io/frakti/pkg/hyper.(*Runtime).RunPodSandbox(0xc4203a5b40, 0xc4203b22a0, 0x1d, 0xc42053da40, 0x1, 0x1)
/home/crazykev/go-project/src/k8s.io/frakti/pkg/hyper/sandbox.go:62 +0x49b
k8s.io/frakti/pkg/manager.(*FraktiManager).RunPodSandbox(0xc4203a5c40, 0x7f4f39ec7000, 0xc4202bacc0, 0xc42008c260, 0x0, 0x1d8, 0x1d8)
/home/crazykev/go-project/src/k8s.io/frakti/pkg/manager/manager.go:109 +0x123
k8s.io/frakti/vendor/k8s.io/kubernetes/pkg/kubelet/api/v1alpha1/runtime._RuntimeService_RunPodSandbox_Handler(0x176a820, 0xc4203a5c40, 0x7f4f39ec7000, 0xc4202bacc0, 0xc42022d810, 0x0, 0x0, 0x0, 0x523f10, 0xc4204a0070)
/home/crazykev/go-project/src/k8s.io/frakti/vendor/k8s.io/kubernetes/pkg/kubelet/api/v1alpha1/runtime/api.pb.go:2300 +0x27d
k8s.io/frakti/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc420418f00, 0x2322ee0, 0xc420490a20, 0xc4200bbe00, 0xc4203f74d0, 0x22fc818, 0xc4202bac90, 0x0, 0x0)
/home/crazykev/go-project/src/k8s.io/frakti/vendor/google.golang.org/grpc/server.go:608 +0xc50
k8s.io/frakti/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420418f00, 0x2322ee0, 0xc420490a20, 0xc4200bbe00, 0xc4202bac90)
/home/crazykev/go-project/src/k8s.io/frakti/vendor/google.golang.org/grpc/server.go:766 +0x6b0
k8s.io/frakti/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4204bbab0, 0xc420418f00, 0x2322ee0, 0xc420490a20, 0xc4200bbe00)
/home/crazykev/go-project/src/k8s.io/frakti/vendor/google.golang.org/grpc/server.go:419 +0xab
created by k8s.io/frakti/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/home/crazykev/go-project/src/k8s.io/frakti/vendor/google.golang.org/grpc/server.go:420 +0xa3
On a separate system I've installed Kubernetes with kubeadm 1.6.2 & 1.6.4. On the system with standard Kubernetes (non-frakti) Rook.io installs and runs. The rook operator succeeds and runs. On the system with Kubernetes based on Frakti I noticed that the rook operator fails yielding CrashLoopBackOff and no logs generated. Is there some behavior that prevents this from working. I know that on each node, ceph-common is to be installed. Is this an issue where the hypervisor is isolated and does not have access to such modules?
Thanks
DB
Currently filter does not work properly, we are temporarily list all images and skip filter:
https://github.com/kubernetes/frakti/pull/67/files#diff-f7ec4be94a67a866942db2ebd145d763R278
This should be fixed as soon as hyperd being updated.
Since we now have docker as alternative runtime. We should consider support systemd cgroups in frakti because in many distribution (e.g. CentOS), dockerd use systemd as default.
Or, we should let user know that only cgroupfs mode of dockerd is supported at least.
Why is there a taint of the master in the allinone.sh script. I'd recommend leaving it out and letting the end user decide. By default the master should not have pods scheduled unless it's explicitly called out
Just a suggestion. I can see this if it is just a 1 node install so makes sense to test things. Maybe the option to specify.
Would you mind if I work with this script and extend it with options? By the way great work on frakti
For the allinone with Centos 7 a line:
echo 1 >/proc/sys/net/bridge/bridge-nf-call-iptables
Will need to be added for nodes to join. And probably for the master as well. Additionally I have found that some Centos 7 distros for some reason are not disabling selinux so /etc/selinux/config set SELINUX=disabled
DB
Currently, only nameserver is set in /etc/resolv.conf
. We should also set dns search domain and options.
Expected /etc/resolv.conf
example:
search default.svc.cluster.local svc.cluster.local cluster.local home
nameserver 10.96.0.10
options ndots:5
Current implement StopContainer()
only cancel request when timeout, this forcibly kill container should guaranteed by underlying runtime, will fix this after hyperhq/hyperd#559 merged
Node e2e test (conformance) results of latest frakti on master branch + hyperd 0.8.0.1 + Kubernetes v160-beta.4.
The test command is:
make test-e2e-node PARALLELISM=2 TEST_ARGS='--kubelet-flags="--container-runtime=remote --container-runtime-endpoint=/var/run/frakti.sock --feature-gates=AllAlpha=true,Accelerators=false"' FOCUS="\[Conformance\]"
The node e2e result is:
Summarizing 9 Failures:
[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0644,tmpfs) [Conformance] [Volume]
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197
[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0777,default) [Conformance] [Volume]
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197
[Fail] [k8s.io] Projected [It] should project all components that make up the projection API [Conformance] [Volume] [Projection]
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197
[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0644,default) [Conformance] [Volume]
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197
[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0666,default) [Conformance] [Volume]
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197
[Fail] [k8s.io] KubeletManagedEtcHosts [It] should test kubelet managed /etc/hosts file [Conformance]
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/exec_util.go:107
[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0777,tmpfs) [Conformance] [Volume]
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197
[Fail] [k8s.io] Kubelet when scheduling a read only busybox container [It] it should not write to root filesystem [Conformance]
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:151
[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0666,tmpfs) [Conformance] [Volume]
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197
Ran 121 of 211 Specs in 1987.027 seconds
FAIL! -- 112 Passed | 9 Failed | 0 Pending | 90 Skipped
In the 9 failed issues:
6 issues related mound volume with specified mode, which seems buggy in hyperd.
1 issue is know issue of hyperd hyperhq/hyperd#564
2 issues still have no clear root cause found
cc @feiskyer
When enabled frakti with kubelet, kubectl attach
failed with errors:
# cluster/kubectl.sh get pods
NAME READY STATUS RESTARTS AGE
busybox-983667326-wlbt7 1/1 Running 0 21m
# cluster/kubectl.sh attach busybox-983667326-wlbt7
If you don't see a command prompt, try pressing enter.
error: unable to upgrade connection: pod busybox-983667326-wlbt7_default not found
Kubelet is panic while doing attach:
I1206 08:36:42.017475 31088 server.go:741] POST /attach/default/busybox-983667326-wlbt7/busybox?error=1&output=1: (3.672091ms) 500
goroutine 1433 [running]:
k8s.io/kubernetes/pkg/httplog.(*respLogger).recordStatus(0xc42017cc40, 0x1f4)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xbb
k8s.io/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc42017cc40, 0x1f4)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x35
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteHeader(0xc420bb8f00, 0x1f4)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:200 +0x41
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteErrorString(0xc420bb8f00, 0x1f4, 0xc4212ae180, 0x2d, 0xc421956180, 0x1f)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:180 +0x46
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteError(0xc420bb8f00, 0x1f4, 0x5b36f60, 0xc421326e10, 0x0, 0xc420bdc07d)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:165 +0x78
k8s.io/kubernetes/pkg/kubelet/server.(*Server).getAttach(0xc4200a35e0, 0xc4201fac00, 0xc420bb8f00)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:611 +0x1fb
k8s.io/kubernetes/pkg/kubelet/server.(*Server).(k8s.io/kubernetes/pkg/kubelet/server.getAttach)-fm(0xc4201fac00, 0xc420bb8f00)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:323 +0x3e
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch.func6(0xc4201fac00, 0xc420bb8f00)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:267 +0x3e
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc4201faea0, 0xc4201fac00, 0xc420bb8f00)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:21 +0x95
k8s.io/kubernetes/pkg/kubelet/server.(*Server).InstallAuthFilter.func1(0xc4201fac00, 0xc420bb8f00, 0xc4201faea0)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:249 +0x151
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc4201faea0, 0xc4201fac00, 0xc420bb8f00)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:19 +0x68
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc420f78000, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:269 +0x996
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm(0x5b5b920, 0xc42017cc40, 0xc421779d10)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48
net/http.HandlerFunc.ServeHTTP(0xc4204ce610, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
/usr/local/go/src/net/http/server.go:1726 +0x44
net/http.(*ServeMux).ServeHTTP(0xc420e69c50, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
/usr/local/go/src/net/http/server.go:2022 +0x7f
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP(0xc420f78000, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:287 +0x4d
k8s.io/kubernetes/pkg/kubelet/server.(*Server).ServeHTTP(0xc4200a3590, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:740 +0x10e
net/http.serverHandler.ServeHTTP(0xc4203ad600, 0x5b619a0, 0xc42121ed00, 0xc421779d10)
/usr/local/go/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc420924300, 0x5b646a0, 0xc4216de1c0)
/usr/local/go/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2293 +0x44d
logging error output: "pod busybox-983667326-wlbt7_default not found"
Refer kubernetes/kubernetes#31546 and https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/pod-resource-management.md.
Notes: scripts to enable pod-level cgroups in local cluster:
export KUBERNETES_PROVIDER=local
# enable cri
export CONTAINER_RUNTIME=remote
export EXPERIMENTAL_CRI=true
export CONTAINER_RUNTIME_ENDPOINT=/var/run/frakti.sock
# also enable pod-level cgroups
export EXPERIMENTAL_CGROUPS_PER_QOS=true
export CGROUP_ROOT=/
hack/local-up-cluster.sh
We should update dockershim vendor to support hostPid.
Refer kubernetes/kubernetes#44097, which already part kubernetes v1.6.2.
The expected container log path is PodSandboxConfig.LogDirectory/ContainerConfig.LogPath
// Path relative to PodSandboxConfig.LogDirectory for container to store
// the log (STDOUT and STDERR) on the host.
// E.g.,
// PodSandboxConfig.LogDirectory = `/var/log/pods/<podUID>/`
// ContainerConfig.LogPath = `containerName_Instance#.log`
//
// WARNING: Log management and how kubelet should interface with the
// container logs are under active discussion in
// https://issues.k8s.io/24677. There *may* be future change of direction
// for logging as the discussion carries on.
optional string log_path = 11;
cc/ @gnawux
Performed a fresh install as before, clean Centos 7.3 VM , 1 master, 2 nodes
Everything looked good. Launched heapster, influxdb, grafana, dashboard
ran kubectl proxy as usual , default 127.0.0.1:8001
Accessing the dashboard:
http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/
yields - Error: 'dial tcp 10.244.1.3:9090: getsockopt: no route to host' Trying to reach: 'http://10.244.1.3:9090/'
Accessing grafana:
http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/
yields - http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/
I will leave this cluster up for troubleshooting for the time being
DB
When enabled frakti with kubelet, kubectl exec
failed with errors:
cluster/kubectl.sh exec busybox-983667326-wlbt7 ip addr
Error from server: error dialing backend: dial tcp 172.16.1.134:22521: getsockopt: connection refused
Test ENV:
kubernetes 1.6 + frakti + hyperd
Issue:
kubectl attach
does not return anything
Expected:
kubectl attach
should return process output in the container
This is because of lacking TTYResize in hyper api.
This is problem caused by hyperd, related with hyperhq/hyperd#613.
This document tracks the process of the container runtime interface integration with Hyper.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.