Comments (3)
@luisdavim did you check the deployment yaml files i linked below.
According to https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md, vendor drivers should be installed in the volume plugin path on every node, for Azure specific drivers, it seems like they can be installed in a DeamonSet form with using hostPath
mount points. e.g.:
- https://github.com/Azure/kubernetes-volume-drivers/tree/master/flexvolume/blobfuse/deployment
- https://github.com/Azure/kubernetes-volume-drivers/tree/master/flexvolume/dysk/deployment
- https://github.com/Azure/kubernetes-volume-drivers/tree/master/csi/dysk/deployment
With one caution, apparently these deployment files are designed for AKS or acs-engine/aks-engine which both uses --volume-plugin-dir=/etc/kubernetes/volumeplugins
for kubelets. But the default value of the --volume-plugin-dir
parameter is /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
. Seems like deployment yaml files are mostly for AKS/aks-engine and may need some tuning when using with Gardener.
Also i'm not sure if we can use hostPath's in our shoots by default, last time i checked PSP's they werre not prevented but thats something to consider.
from gardener-extension-provider-azure.
I've tried the DaemonSet
approach but it doesn't seem to work:
$ cat blobfuse-flexvol-installer-1.9.yaml
apiVersion: v1
kind: Namespace
metadata:
name: flex
labels:
type: system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: blobfuse-flexvol-installer
namespace: flex
labels:
k8s-app: blobfuse
spec:
selector:
matchLabels:
name: blobfuse
template:
metadata:
labels:
name: blobfuse
spec:
containers:
- name: blobfuse-flexvol-installer
image: mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume
imagePullPolicy: Always
volumeMounts:
- name: volplugins
mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
- name: varlog
mountPath: /var/log/
volumes:
- name: varlog
hostPath:
path: /var/log/
- name: volplugins
hostPath:
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
$ kctl get ds -n flex
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
blobfuse-flexvol-installer 4 4 0 4 0 <none> 73s
$ kctl describe po blobfuse-flexvol-installer-4ctr7 -n flex
Name: blobfuse-flexvol-installer-4ctr7
Namespace: flex
Priority: 0
PriorityClassName: <none>
Node: shoot--studio--studio-cpu-worker-66bd458847-vtd6t/10.250.0.9
Start Time: Thu, 03 Jan 2019 09:32:38 +0000
Labels: controller-revision-hash=5d9655ccf7
name=blobfuse
pod-template-generation=1
Annotations: cni.projectcalico.org/podIP: 100.96.3.125/32
kubernetes.io/psp: gardener.privileged
Status: Running
IP: 100.96.3.125
Controlled By: DaemonSet/blobfuse-flexvol-installer
Containers:
blobfuse-flexvol-installer:
Container ID: docker://cd394c1535752172061037fb296d51ba10a7b394b1e50db3d30cd6bfcb472566
Image: mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume
Image ID: docker-pullable://mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:f9822fd37987419f28c6fa26e3e9838bdd5f46bf8acf3a23cf471aa672607005
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: error while creating mount source path '/usr/libexec/kubernetes/kubelet-plugins/volume/exec': mkdir /usr/libexec/kubernetes: read-only file system
Exit Code: 128
Started: Thu, 03 Jan 2019 09:34:00 +0000
Finished: Thu, 03 Jan 2019 09:34:00 +0000
Ready: False
Restart Count: 4
Environment: <none>
Mounts:
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/ from volplugins (rw)
/var/log/ from varlog (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ctq79 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
varlog:
Type: HostPath (bare host directory volume)
Path: /var/log/
HostPathType:
volplugins:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
HostPathType:
default-token-ctq79:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ctq79
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m5s default-scheduler Successfully assigned flex/blobfuse-flexvol-installer-4ctr7 to shoot--studio--studio-cpu-worker-66bd458847-vtd6t
Normal Pulling 103s (x5 over 3m4s) kubelet, shoot--studio--studio-cpu-worker-66bd458847-vtd6t pulling image "mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume"
Normal Pulled 103s (x5 over 3m2s) kubelet, shoot--studio--studio-cpu-worker-66bd458847-vtd6t Successfully pulled image "mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume"
Normal Created 102s (x5 over 3m2s) kubelet, shoot--studio--studio-cpu-worker-66bd458847-vtd6t Created container
Warning Failed 102s (x5 over 3m2s) kubelet, shoot--studio--studio-cpu-worker-66bd458847-vtd6t Error: failed to start container "blobfuse-flexvol-installer": Error response from daemon: error while creating mount source path '/usr/libexec/kubernetes/kubelet-plugins/volume/exec': mkdir /usr/libexec/kubernetes: read-only file system
Warning BackOff 89s (x5 over 2m34s) kubelet, shoot--studio--studio-cpu-worker-66bd458847-vtd6t Back-off restarting failed container
from gardener-extension-provider-azure.
I don't think we will do that soon @luisdavim, maybe we can address this one after extensibility? However, if you have capacity, feel free to file a PR if that might be too long for you.
from gardener-extension-provider-azure.
Related Issues (20)
- Expose NAT gateway IP in the shoot cluster HOT 4
- guestbook test is failing on Azure for multi-zone clusters HOT 1
- Deletion fails due to empty string vnet reference
- Seed deletion fails due to missing backupbucket secret HOT 1
- Support PremiumV2_LRS storage HOT 2
- Improve error code matching HOT 1
- Add Infrastructure integration test for invalid credentials HOT 1
- Error code not added HOT 2
- Intermittent Azure API fault results in zombie NatGateway and persistent shoot creation failure HOT 2
- Add native support for generation 1 and 2 OS images and machine types
- Enhance error code
- Improve error classification for `PublicIPAddressInUse` HOT 1
- Change StorageAccount for backupbucket from LRS to ZRS HOT 1
- Enable Serial Console Support in Worker Node HOT 3
- Integrate vSMP MemeoryOne in Azure HOT 6
- Cannot delete `BackupEntries` which use the secret from a `core.BackupBucket.status.generatedSecretRef` reference HOT 2
- [CPM] Restoration of cluster fails if it's `Infrastructure` resource on the source `Seed` was annotated with `migration.azure.provider.extensions.gardener.cloud/zone` HOT 1
- Map context timeout/deadline exceeded code during VM Deletion to gardener ERR_INFRA_DEPENDENCIES HOT 1
- Parametrize and Expose Azure disks IOPs and Throughput
- Can't use azure dns provider with non-azure infrastructure provider HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gardener-extension-provider-azure.