Code Monkey home page Code Monkey logo

budgetkey-k8s's People

Contributors

akariv avatar orihoch avatar

Stargazers

avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

omertalmi5 wsheffel

budgetkey-k8s's Issues

cluster migration plan

  • fonts - copy and expose as themes
  • datapackages - copy from server gz
  • travis - use deploy

[incident] pipelines pod stuck in ContainerCreating due to persistent disk unable to mount

  • kubectl get pods
    • pipelines pod stuck in ContainerCreating status
  • kubectl describe pod pipelines-79dd8668bb-dmnc4_budgetkey
    • timeout expired waiting for volumes to attach/mount for pod "budgetkey"/"pipelines-79dd8668bb-dmnc4". list of unattached/unmounted volumes=[data]
  • gcloud compute disks describe budgetkey-pipelines-data-3
    • the persistent disk is still attached to the node
    • users: - https://www.googleapis.com/compute/v1/projects/hasadna-general/zones/europe-west1-b/instances/gke-hasadna-cluster-default-pool-17373db3-dlpj
  • gcloud compute ssh gke-hasadna-cluster-default-pool-17373db3-dlpj
    • sudo journalctl -n 500 -f
    • error should appear periodically - as kubernetes retries everything
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]: I0627 10:19:42.835137    1516 reconciler.go:186] operationExecutor.UnmountVolume started for volume "data" (UniqueName: "kubernetes.io/gce-pd/budgetkey-pipelines-data-3") pod "459bd8f9-79e9-11e8
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]: E0627 10:19:42.858324    1516 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/gce-pd/budgetkey-pipelines-data-3\" (\"459bd8f9-79e9-11e8-8769-42010a840006\")" failed. No retries pe
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]: Unmounting arguments: /var/lib/kubelet/pods/459bd8f9-79e9-11e8-8769-42010a840006/volumes/kubernetes.io~gce-pd/data
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]: Output: umount: /var/lib/kubelet/pods/459bd8f9-79e9-11e8-8769-42010a840006/volumes/kubernetes.io~gce-pd/data: target is busy
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]:         (In some cases useful info about processes that
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]:          use the device is found by lsof(8) or fuser(1).)
  • check the pod name - cat /var/lib/kubelet/pods/459bd8f9-79e9-11e8-8769-42010a840006/etc-hosts
  • can inspect other directories in the pod dir as well
  • run lsof to determine which process is keeping it -
  • toolbox - starts a privileged docker container for debugging (due to container optimized os limits)
    • apt install lsof
    • lsof -w | grep 459bd8f9-79e9-11e8-8769-42010a840006
    • kill / investigate the process
    • kubernetes keeps retrying, so wait ~1 minute and it should work

[incident] elasticsearch release fails - "has no deployed releases"

  • helm ls --all
    • budgetkey-elasticsearch-budgetkey release is in PENDING_UPGRADE state
  • delete the release and recreate
    • helm delete --purge budgetkey-elasticsearch-budgetkey
    • ./helm_upgrade_external_chart.sh elasticsearch
  • alternatively - you can try to inspect the configmap directly
    • kubectl get cm -n kube-system | grep budgetkey-elasticsearch-budgetkey
    • can try to manually modify the configmap - to force a successful status

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.