kubernetes-retired / frakti Goto Github PK

Is this normal behavior. Under kubernetes without hyperd, I noticed that there is no attempt to spin up DNS pods. However, when I issue a kubeadm reset to start over there is an attempt by hyperd to start a kube-dns pod:

hyperd[1009]: E0524 10:19:59.988202 1009 persist.go:113] Pod[k8s_POD.0_kube-dns-3913472980-2jz1j_kube-system_af1d4552-3f1e-11e7-aeb4-0021ccd48ab5_10cd9672] failed to load sandbox info: leveldb: not found

Have not noticed this behavior under a non-hyperd kubernetes cluster.

Perhaps this is just a remnant but I did perform a kubeadm reset and then decided to reboot before peforming a new kubeadm init. In checking the status of hyperd after the reboot (prior to peforming a new kubeadm init) the above was the result.

Also when a kubeadm reset is done, all files in /etc/cni/net.d are removed. This should be noted in the deploy.md just as a warning or note

Thanks
DB

The test is disabled at #119 because of hyperhq/hyperd#604.

We should enable it again after hyperhq/hyperd#604 is fixed.

This is a issue for Google Summer of Code 2017, view docs here.

Frakti is a well-designed CRI implementation for hypervisor-based runtimes, it would be not so hard to have Unikernels runtime into its picture.

A build-in unikshim(or any other name) for frakti need to be developed to manage Unikernels workloads.

Implementation Tips:

1. The workflow will be like:

kubelet -> frakti -> hypervisor manager (libvirt or QEMU) -> Unikernels machines

2. frakti has already implemented Kubernetes resource model, CNI network, native volumes, so most of upper-layer concepts from Kubernetes can be easily fit to Unikernels container.

Goals

1. A PoC that can manage Unikernels workload lifecycle with Kubernetes.
2. All pod level validation tests should be passed by using cri-tools
3. Only one-pod-one-container model is required.
4. Only consider using CNI bridge mode network.

Nice To Have

1. Implementation of container level API in unikshim.

Non Goals

1. Supporting all kinds of CNI plugins other than bridge mode
2. Supporting all kinds of volume plugins other than emptyDir and hostPath.
3. Any other features not in the scope of CRI implementation (not covered by CRI validation tests in cri-tools)

To fix the nil pointer here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/security_context.go#L129

cc @Crazykev Watch out for staging part

Since we now have docker as alternative runtime. We should consider support systemd cgroups in frakti because in many distribution (e.g. CentOS), dockerd use systemd as default.

Or, we should let user know that only cgroupfs mode of dockerd is supported at least.

Readonly rootfs is not supported because hyperd. We should fix it after hyperhq/hyperd#638.

CRI has changed its API to proto3. We should upgrade our vendor.

Now frakti cni integration only support bridge plugin. I think it is necessary to support non bridge plugins like macvlan, calico etc. @feiskyer

Performed a fresh install as before, clean Centos 7.3 VM , 1 master, 2 nodes

Everything looked good. Launched heapster, influxdb, grafana, dashboard

ran kubectl proxy as usual , default 127.0.0.1:8001

Accessing the dashboard:
http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/
yields - Error: 'dial tcp 10.244.1.3:9090: getsockopt: no route to host' Trying to reach: 'http://10.244.1.3:9090/'

Accessing grafana:
http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/
yields - http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/

I will leave this cluster up for troubleshooting for the time being

DB

When creating a pod with host network, it failed to create symbolic link for logs (warning).

pod config

apiVersion: v1
kind: Pod
metadata:
  name: pod
  labels:
    name: pod
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirst
  containers:
  - name: nginx
    image: nginx

Kubelet events:

df21b37c7a7f3aff14ba2034b76d4896406cf501cc59f80b74b49e032934
  36m		36m		1	kubelet, instance-2	spec.containers{nginx}	Warning		Failed		Failed to start container with id a741df21b37c7a7f3aff14ba2034b76d4896406cf501cc59f80b74b49e032934 with error: rpc error: code = 2 desc = failed to create symbolic link "/var/log/pods/318a363b-0f73-11e7-9e6c-42010a8c0003/nginx_0.log" to the container log file "" for container "a741df21b37c7a7f3aff14ba2034b76d4896406cf501cc59f80b74b49e032934": symlink  /var/log/pods/318a363b-0f73-11e7-9e6c-42010a8c0003/nginx_0.log: no such file or directory
  36m		36m		1	kubelet, instance-2			Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "nginx" with rpc error: code = 2 desc = failed to create symbolic link "/var/log/pods/318a363b-0f73-11e7-9e6c-42010a8c0003/nginx_0.log" to the container log file "" for container "a741df21b37c7a7f3aff14ba2034b76d4896406cf501cc59f80b74b49e032934": symlink  /var/log/pods/318a363b-0f73-11e7-9e6c-42010a8c0003/nginx_0.log: no such file or directory: "Start Container Failed"

Refer kubernetes/kubernetes#31546 and https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/pod-resource-management.md.

Notes: scripts to enable pod-level cgroups in local cluster:

export KUBERNETES_PROVIDER=local
# enable cri
export CONTAINER_RUNTIME=remote
export EXPERIMENTAL_CRI=true
export CONTAINER_RUNTIME_ENDPOINT=/var/run/frakti.sock
# also enable pod-level cgroups
export EXPERIMENTAL_CGROUPS_PER_QOS=true
export CGROUP_ROOT=/

hack/local-up-cluster.sh

CNI doesn't do port-mapping today. We could support port mapping via iptables.

When I use cri-tools to test frakti(docker runtime), I find stop container will time out.

• Failure [60.490 seconds]
[k8s.io] Container
/home/xlgao/gocode/src/github.com/kubernetes-incubator/cri-tools/pkg/framework/framework.go:72
  runtime should support basic operations on container
  /home/xlgao/gocode/src/github.com/kubernetes-incubator/cri-tools/pkg/validate/container.go:111
    runtime should support stopping container [Conformance] [It]
    /home/xlgao/gocode/src/github.com/kubernetes-incubator/cri-tools/pkg/validate/container.go:89

    Mar 31 17:04:17.269: stop container "0924ab698482d90591ff93552a44144340836760c04929370e834bd41abcef7d" timeout.
    

    /home/xlgao/gocode/src/github.com/kubernetes-incubator/cri-tools/pkg/validate/container.go:215

1. Compiled current code base for frakti to pick up changes for default cpu/mem
2. Copied frakti binary to 1 master, 5 minon nodes
3. Restart all nodes

Result:
KUBELET
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Mon 2017-06-05 12:52:23 CDT; 1s ago
Docs: http://kubernetes.io/docs/
Process: 7419 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CGROUP_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
Main PID: 7419 (code=exited, status=1/FAILURE)

Jun 05 12:52:23 master01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Jun 05 12:52:23 master01 systemd[1]: Unit kubelet.service entered failed state.
Jun 05 12:52:23 master01 systemd[1]: kubelet.service failed.

HYPERD
● hyperd.service - hyperd
Loaded: loaded (/usr/lib/systemd/system/hyperd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2017-06-05 12:50:52 CDT; 4min 46s ago
Docs: http://docs.hypercontainer.io
Main PID: 7193 (hyperd)
CGroup: /system.slice/hyperd.service
└─7193 /usr/bin/hyperd --log_dir=/var/log/hyper

Jun 05 12:50:52 master01 systemd[1]: Started hyperd.
Jun 05 12:50:52 master01 systemd[1]: Starting hyperd...
Jun 05 12:50:53 master01 hyperd[7193]: time="2017-06-05T12:50:53-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jun 05 12:50:53 master01 hyperd[7193]: time="2017-06-05T12:50:53-05:00" level=info msg="Firewalld running: false"
Jun 05 12:50:53 master01 hyperd[7193]: time="2017-06-05T12:50:53-05:00" level=info msg="Loading containers: start."
Jun 05 12:50:53 master01 hyperd[7193]: ...
Jun 05 12:50:53 master01 hyperd[7193]: time="2017-06-05T12:50:53-05:00" level=info msg="Loading containers: done."
Jun 05 12:50:53 master01 hyperd[7193]: E0605 12:50:53.215808 7193 persist.go:113] Pod[k8s_POD.0_kube-dns-2612786695-kqqhv_kube-system_4664f72a-409d-11e7-9579-0021ccd48ab5_9acb0442] failed to load sandbox info: leveldb: not found

FRAKTI
● frakti.service - Hypervisor-based container runtime for Kubernetes
Loaded: loaded (/usr/lib/systemd/system/frakti.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2017-06-05 12:50:31 CDT; 5min ago
Docs: https://github.com/kubernetes/frakti
Main PID: 7126 (frakti)
CGroup: /system.slice/frakti.service
└─7126 /usr/bin/frakti --v=3 --log-dir=/var/log/frakti --logtostderr=false --cgroup-driver=systemd --listen=/var/run/frakti.sock --streaming-server-addr=master01 --hyper-endpoint=127.0.0.1:22318

Jun 05 12:50:31 master01 systemd[1]: Started Hypervisor-based container runtime for Kubernetes.
Jun 05 12:50:31 master01 systemd[1]: Starting Hypervisor-based container runtime for Kubernetes...
Jun 05 12:50:52 master01 frakti[7126]: 2017/06/05 12:50:52 transport: http2Client.notifyError got notified that the client transport was broken EOF.
Jun 05 12:50:52 master01 frakti[7126]: 2017/06/05 12:50:52 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 127.0.0.1:22318: getsockopt: connection refused"; Reconnecting to {127.0.0.1:22318 }
Jun 05 12:52:22 master01 frakti[7126]: 2017/06/05 12:52:22 transport: http2Server.HandleStreams failed to read frame: read unix /var/run/frakti.sock->@: read: connection reset by peer
Jun 05 12:55:24 master01 frakti[7126]: 2017/06/05 12:55:24 transport: http2Server.HandleStreams failed to read frame: read unix /var/run/frakti.sock->@: read: connection reset by peer
Jun 05 12:55:45 master01 frakti[7126]: 2017/06/05 12:55:45 transport: http2Server.HandleStreams failed to read frame: read unix /var/run/frakti.sock->@: read: connection reset by peer

Another thing I've noticed is that that does not survive reboots all that well.

Thoughts where I should look?
DB

The expected container log path is PodSandboxConfig.LogDirectory/ContainerConfig.LogPath

    // Path relative to PodSandboxConfig.LogDirectory for container to store
    // the log (STDOUT and STDERR) on the host.
    // E.g.,
    //     PodSandboxConfig.LogDirectory = `/var/log/pods/<podUID>/`
    //     ContainerConfig.LogPath = `containerName_Instance#.log`
    //
    // WARNING: Log management and how kubelet should interface with the
    // container logs are under active discussion in
    // https://issues.k8s.io/24677. There *may* be future change of direction
    // for logging as the discussion carries on.
    optional string log_path = 11;

cc/ @gnawux

The design doc of pod level resource has been updated with more details, and there's something different from our original design:

according to https://github.com/kubernetes/community/blob/master/contributors/design-proposals/pod-resource-management.md#burstable-qos, there's chance that a BU pod does not have memory limit, so it's mem limit will be set to node's available resource. But frakti will keep using this value (which is obvious too big), it would be wrong.

Need to figure out this use case and set default value instead (may need to import qos pkg).

Currently, sandbox checkpoint is only removed after the pod is deleted successfully. However, if the hyper pod has already been removed outside, the checkpoint will be left over and kubelet will always fail on stopping sandbox.

We should ensure checkpoint and network plugin are cleaned up properly once frakti has detected the sandbox has already been non-exist.

In fresh new environment:

E0314 02:26:12.316677   11713 hyper_checkpoint.go:119] Failed to list checkpoint store: open /var/lib/frakti/sandbox: no such file or directory

@YaoZengzeng Is this directory expected to automatically created?

Test ENV:
kubernetes 1.6 + frakti + hyperd

Issue:
kubectl attach does not return anything

Expected:
kubectl attach should return process output in the container

I'm thinking about spinning this back up across physical machines, about 5 HP Proliants to test. Where are things with supporting Calico or Flannel CNI?

Thanks
DB

The current frakti requires CNI plugins set before starting. However, with latest kubeadm and kubernetes, network plugins may be initialized dynamically.

Related upstream issue: kubernetes/kubernetes#43474.

When enabled frakti with kubelet, kubectl exec failed with errors:

cluster/kubectl.sh exec busybox-983667326-wlbt7 ip addr
Error from server: error dialing backend: dial tcp 172.16.1.134:22521: getsockopt: connection refused

When enabled frakti with kubelet, kubectl attach failed with errors:

# cluster/kubectl.sh get pods
NAME                      READY     STATUS    RESTARTS   AGE
busybox-983667326-wlbt7   1/1       Running   0          21m
# cluster/kubectl.sh attach busybox-983667326-wlbt7
If you don't see a command prompt, try pressing enter.
error: unable to upgrade connection: pod busybox-983667326-wlbt7_default not found

Kubelet is panic while doing attach:

I1206 08:36:42.017475   31088 server.go:741] POST /attach/default/busybox-983667326-wlbt7/busybox?error=1&output=1: (3.672091ms) 500
goroutine 1433 [running]:
k8s.io/kubernetes/pkg/httplog.(*respLogger).recordStatus(0xc42017cc40, 0x1f4)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xbb
k8s.io/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc42017cc40, 0x1f4)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x35
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteHeader(0xc420bb8f00, 0x1f4)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:200 +0x41
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteErrorString(0xc420bb8f00, 0x1f4, 0xc4212ae180, 0x2d, 0xc421956180, 0x1f)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:180 +0x46
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteError(0xc420bb8f00, 0x1f4, 0x5b36f60, 0xc421326e10, 0x0, 0xc420bdc07d)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:165 +0x78
k8s.io/kubernetes/pkg/kubelet/server.(*Server).getAttach(0xc4200a35e0, 0xc4201fac00, 0xc420bb8f00)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:611 +0x1fb
k8s.io/kubernetes/pkg/kubelet/server.(*Server).(k8s.io/kubernetes/pkg/kubelet/server.getAttach)-fm(0xc4201fac00, 0xc420bb8f00)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:323 +0x3e
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch.func6(0xc4201fac00, 0xc420bb8f00)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:267 +0x3e
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc4201faea0, 0xc4201fac00, 0xc420bb8f00)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:21 +0x95
k8s.io/kubernetes/pkg/kubelet/server.(*Server).InstallAuthFilter.func1(0xc4201fac00, 0xc420bb8f00, 0xc4201faea0)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:249 +0x151
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc4201faea0, 0xc4201fac00, 0xc420bb8f00)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:19 +0x68
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc420f78000, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:269 +0x996
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm(0x5b5b920, 0xc42017cc40, 0xc421779d10)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48
net/http.HandlerFunc.ServeHTTP(0xc4204ce610, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
	/usr/local/go/src/net/http/server.go:1726 +0x44
net/http.(*ServeMux).ServeHTTP(0xc420e69c50, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
	/usr/local/go/src/net/http/server.go:2022 +0x7f
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP(0xc420f78000, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:287 +0x4d
k8s.io/kubernetes/pkg/kubelet/server.(*Server).ServeHTTP(0xc4200a3590, 0x5b5b920, 0xc42017cc40, 0xc421779d10)
	/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:740 +0x10e
net/http.serverHandler.ServeHTTP(0xc4203ad600, 0x5b619a0, 0xc42121ed00, 0xc421779d10)
	/usr/local/go/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc420924300, 0x5b646a0, 0xc4216de1c0)
	/usr/local/go/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2293 +0x44d

logging error output: "pod busybox-983667326-wlbt7_default not found"

Need to support CNI plugin in Kubernetes + Frakti. The implementation requires finding a way to generate network configuration from CNI and pass these information to hyperd in expected syntax.

A very similar approach is CNI plugin support in CRI-O, please check these for details:

cri-o/cri-o#237
https://github.com/kubernetes-incubator/cri-o#setup-cni-networking

TODOs:

• Basic cni network plugin support #62
• Checkpointing network info
• CNI configure examples
• CNI e2e tests
• deployment, e.g. by make install

Refer kubernetes/kubernetes#29579.

• execsync
• exec
• attach
• port-forward

This could be done by using kubelet streaming liberay.

Node e2e test (conformance) results of latest frakti on master branch + hyperd 0.8.0.1 + Kubernetes v160-beta.4.

The test command is:

make test-e2e-node PARALLELISM=2 TEST_ARGS='--kubelet-flags="--container-runtime=remote --container-runtime-endpoint=/var/run/frakti.sock --feature-gates=AllAlpha=true,Accelerators=false"' FOCUS="\[Conformance\]"

The node e2e result is:

Summarizing 9 Failures:

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0644,tmpfs) [Conformance] [Volume] 
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0777,default) [Conformance] [Volume] 
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197

[Fail] [k8s.io] Projected [It] should project all components that make up the projection API [Conformance] [Volume] [Projection] 
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0644,default) [Conformance] [Volume] 
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0666,default) [Conformance] [Volume] 
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197

[Fail] [k8s.io] KubeletManagedEtcHosts [It] should test kubelet managed /etc/hosts file [Conformance] 
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/exec_util.go:107

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0777,tmpfs) [Conformance] [Volume] 
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197

[Fail] [k8s.io] Kubelet when scheduling a read only busybox container [It] it should not write to root filesystem [Conformance] 
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:151

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0666,tmpfs) [Conformance] [Volume] 
/root/go-project/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197

Ran 121 of 211 Specs in 1987.027 seconds
FAIL! -- 112 Passed | 9 Failed | 0 Pending | 90 Skipped 

In the 9 failed issues:
6 issues related mound volume with specified mode, which seems buggy in hyperd.
1 issue is know issue of hyperd hyperhq/hyperd#564
2 issues still have no clear root cause found

cc @feiskyer

Hypernetes doesn't support port forwarding. We should decide whether/how we could support it in frakti.

Currently, only nameserver is set in /etc/resolv.conf. We should also set dns search domain and options.

Expected /etc/resolv.conf example:

search default.svc.cluster.local svc.cluster.local cluster.local home
nameserver 10.96.0.10
options ndots:5

Currently filter does not work properly, we are temporarily list all images and skip filter:
https://github.com/kubernetes/frakti/pull/67/files#diff-f7ec4be94a67a866942db2ebd145d763R278

This should be fixed as soon as hyperd being updated.

(Are the deployment instructions even up to date? starting to see some steps left out)

Using Centos 7.3 with usual repo updates

Follow the deployment steps per : https://github.com/kubernetes/frakti/blob/master/docs/deploy.md

Using 3 VMs (1 master, 2 minons) or that's the plan

By the way the repos are updated with current 1.6.2, the "note" below the yum install references an older 1.6.0-beta 4

On master run kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version stable (latest gives 1.7 alpha)

kubeadm does all of the preflight checks and as expected everything passes. Note there should be a step prior to running this that enables and starts kubelet service, otherwise the warning will pop up.

result... hung at:
[apiclient] Created API client, waiting for the control plane to become ready

No docker images are pulled, no docker containers appear to be running or created

First time I have had issues with kubeadm under a normal docker based install over attempting a fresh Frakti one. These are all fresh VMs so nothing legacy on them other than Centos 7.3. FOllowed deployment instructions exactly.

DB

Also curious as to why this has two Hypervisor entries: refer to Hypervisor=libvert and Hypervisor=qemu, is this not redundant and would the config not read the second entry?

echo -e "Hypervisor=libvirt\n
Kernel=/var/lib/hyper/kernel\n
Initrd=/var/lib/hyper/hyper-initrd.img\n
Hypervisor=qemu\n
StorageDriver=overlay\n
gRPCHost=127.0.0.1:22318" > /etc/hyper/config

Another thing to note is kubelet failed:

-- Unit kubelet.service has begun starting up.
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.355299 2726 feature_gate.go:144] feature gates: map[DynamicVolumeProvisioning:true TaintBasedE
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.359676 2726 server.go:232] Starting Kubelet configuration sync loop
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.359702 2726 server.go:407] failed to init dynamic Kubelet configuration sync: cloud provider w
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.375962 2726 manager.go:143] cAdvisor running in container: "/"
May 01 18:14:33 kube01 kubelet[2726]: W0501 18:14:33.476000 2726 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.550699 2726 fs.go:117] Filesystem partitions: map[/dev/vda1:{mountpoint:/ major:253 minor:1 fs
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.552778 2726 manager.go:198] Machine: {NumCores:2 CpuFrequency:2399998 MemoryCapacity:397522534
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.588629 2726 manager.go:204] Version: {KernelVersion:3.10.0-514.16.1.el7.x86_64 ContainerOsVers
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.589372 2726 server.go:509] --cgroups-per-qos enabled, but --cgroup-root was not specified. de
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.591154 2726 container_manager_linux.go:245] container manager verified user specified cgroup-r
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.591188 2726 container_manager_linux.go:250] Creating Container Manager object based on Node Co
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.591400 2726 kubelet.go:255] Adding manifest file: /etc/kubernetes/manifests
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.591442 2726 kubelet.go:265] Watching apiserver
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.597608 2726 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.597697 2726 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.597777 2726 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1
May 01 18:14:33 kube01 kubelet[2726]: W0501 18:14:33.598688 2726 kubelet_network.go:63] Hairpin mode set to "promiscuous-bridge" but container runt
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.598721 2726 kubelet.go:494] Hairpin mode set to "none"
May 01 18:14:33 kube01 kubelet[2726]: I0501 18:14:33.598922 2726 remote_runtime.go:41] Connecting to runtime service /var/run/frakti.sock
May 01 18:14:33 kube01 kubelet[2726]: 2017/05/01 18:14:33 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "t
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.599459 2726 remote_runtime.go:63] Version from runtime service failed: rpc error: code = 14 de
May 01 18:14:33 kube01 kubelet[2726]: 2017/05/01 18:14:33 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "t
May 01 18:14:33 kube01 kubelet[2726]: E0501 18:14:33.600929 2726 kuberuntime_manager.go:154] Get runtime version failed: rpc error: code = 14 desc
May 01 18:14:33 kube01 kubelet[2726]: error: failed to run Kubelet: failed to create kubelet: rpc error: code = 14 desc = grpc: the connection is unav
May 01 18:14:33 kube01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
May 01 18:14:33 kube01 systemd[1]: Unit kubelet.service entered failed state.
May 01 18:14:33 kube01 systemd[1]: kubelet.service failed.

Current implement StopContainer() only cancel request when timeout, this forcibly kill container should guaranteed by underlying runtime, will fix this after hyperhq/hyperd#559 merged

This is problem caused by hyperd, related with hyperhq/hyperd#613.

We could switch most tests to critest, while only keep some frakti-specific tests.

This document tracks the process of the container runtime interface integration with Hyper.

Frakti core features

• Manage pods/containers/images according to kubelet new runtime API
  • pod sandboxes
    • create pod sandbox
    • list pod sandbox
    • delete pod sandbox #11
    • get pod sandbox’s status
  • containers
    • create container
    • start container
    • stop container
    • delete container
    • list containers
    • get container status
    • exec in container
    • port forwarding
    • container logging
  • images
    • list images
    • pull image
    • delete image
    • get image status
• Hack scripts for easily setup a local enviroment
  • hack/local-up-cluster.sh
  • Vagrant or minikube
• testing
  • Unit tests
  • End-to-end tests
  • Kubernetes node e2e tests
• Cluster deployment
  • Cluster deploy frakti for kubernetes clusters
• Documentation
  • Admin and user guide
  • Frakti specific features which are different from Docker

Requirements for Hyper

• Container-level APIs: StartContainer/StopContainer hyperhq/hyperd#445

References

• Container runtime interface planning
• Network plugin for client/server container runtimes
• Client/server container runtime proposal

This is because of lacking TTYResize in hyper api.

Does the version need updating in the code. I cloned the master branch, compiled, and result of frakti --version is:

I0605 11:24:34.272388 11749 frakti.go:71] frakti version: 0.1

Should this be 0.2? just wondering as I want to make sure I have the latest code with the ability to specify cpu and mem.

Thanks
DB

I really like the idea of being able to schedule pods into lightweight vm's. There are times though when having access to the underlying node is needed. So its rather awkward to only have one at a time. It would be awesome if the docker backend and hyper backend could co'exist on the host at the same time, and pick one or the other based on pod annotation. You could then even possibly use kubernetes itself, with a daemonset that's docker based, to enable and manage hyper support on a kubernetes cluster. Looking at hyper a bit, it looks like it could containerize nicely.

We should update dockershim vendor to support hostPid.

Refer kubernetes/kubernetes#44097, which already part kubernetes v1.6.2.

There are many redundant flags because of introducing dockershim. We should clean them up.

      --storage-driver-buffer-duration duration   Writes in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction (default 1m0s)
      --storage-driver-db string                  database name (default "cadvisor")
      --storage-driver-host string                database host:port (default "localhost:8086")
      --storage-driver-password string            database password (default "root")
      --storage-driver-secure                     use secure connection with database
      --storage-driver-table string               table name (default "stats")
      --storage-driver-user string                database username (default "root")

On a separate system I've installed Kubernetes with kubeadm 1.6.2 & 1.6.4. On the system with standard Kubernetes (non-frakti) Rook.io installs and runs. The rook operator succeeds and runs. On the system with Kubernetes based on Frakti I noticed that the rook operator fails yielding CrashLoopBackOff and no logs generated. Is there some behavior that prevents this from working. I know that on each node, ceph-common is to be installed. Is this an issue where the hypervisor is isolated and does not have access to such modules?

Thanks
DB

Kubernetes service is not accessible after pod created, it is related with a hyperd bug introduced recently hyperhq/hyperd#468.

Encountered when verifying PR #62 by running kubectl run --image=nginx nginx:

I0118 11:38:52.943145   23702 manager.go:177] CreateContainer with request pod_sandbox_id:"k8s_POD.0_nginx-2932712833-gc82w_default_78e43fe8-dd72-11e6-b3f3-fa163ea7515e_a76f41fe" config:<metadata:<name:"nginx" attempt:0 > image:<image:"" > working_dir:"" envs:<key:"KUBERNETES_SERVICE_HOST" value:"10.0.0.1" > envs:<key:"KUBERNETES_SERVICE_PORT" value:"443" >....
E0118 11:38:52.943961   23702 container.go:45] Create container nginx in pod k8s_POD.0_nginx-2932712833-gc82w_default_78e43fe8-dd72-11e6-b3f3-fa163ea7515e_a76f41fe failed: rpc error: code = 2 desc = No command specified

The problem is image:<image:"" >, which expected to be imageRef of nginx image.

cc/ @resouer @YaoZengzeng

We should improve unit tests coverage.

Why is there a taint of the master in the allinone.sh script. I'd recommend leaving it out and letting the end user decide. By default the master should not have pods scheduled unless it's explicitly called out

Just a suggestion. I can see this if it is just a 1 node install so makes sense to test things. Maybe the option to specify.

Would you mind if I work with this script and extend it with options? By the way great work on frakti

For the allinone with Centos 7 a line:

echo 1 >/proc/sys/net/bridge/bridge-nf-call-iptables

Will need to be added for nodes to join. And probably for the master as well. Additionally I have found that some Centos 7 distros for some reason are not disabling selinux so /etc/selinux/config set SELINUX=disabled

DB

Many CNI Plugins don't properly support DEL (See kubernetes/kubernetes#43488 for more details).

We should upgrade to CNI v0.5.1 after kubernetes/kubernetes#43546 is merged.

For now we will list all pods (containers) managed by hyperd, thus user created pod will also be parsed by frakti and report failed to parse container name.

We need to consider add a filter to it. (e.g. special prefix?)

After running the allinone.sh, kubeadm init completes as expect. However, in checking service status, frakti shows this:

● frakti.service - Hypervisor-based container runtime for Kubernetes
Loaded: loaded (/usr/lib/systemd/system/frakti.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-05-09 18:12:23 UTC; 3min 56s ago
Docs: https://github.com/kubernetes/frakti
Main PID: 3199 (frakti)
CGroup: /system.slice/frakti.service
└─3199 /usr/bin/frakti --v=3 --log-dir=/var/log/frakti --logtostderr=false --cgroup-driver=systemd --listen=/var/run/frakti.sock --strea...

May 09 18:12:23 kube01 systemd[1]: Started Hypervisor-based container runtime for Kubernetes.
May 09 18:12:23 kube01 systemd[1]: Starting Hypervisor-based container runtime for Kubernetes...
May 09 18:12:23 kube01 frakti[3199]: E0509 18:12:23.563497 3199 ocicni.go:110] error updating cni config: No networks found in /etc/cni/net.d
May 09 18:12:23 kube01 frakti[3199]: E0509 18:12:23.571272 3199 ocicni.go:110] error updating cni config: No networks found in /etc/cni/net.d
May 09 18:12:24 kube01 systemd[1]: [/usr/lib/systemd/system/frakti.service:8] Unknown lvalue 'TasksMax' in section 'Service'
May 09 18:12:38 kube01 systemd[1]: [/usr/lib/systemd/system/frakti.service:8] Unknown lvalue 'TasksMax' in section 'Service'
[root@kube01 ~]# ls /etc/cni/net.d
10-mynet.conf 99-loopback.conf