Comments (15)
camilb
commented on June 2, 2024
Hi @grifonas can you check if the PVC are created and bound?
kubectl get pvc -n monitoring
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
grafana-persistent-storage Bound pvc-9625a608-ecea-11e7-8f01-0ede025574a6 5Gi RWO ssd 3d
prometheus-k8s-db-prometheus-k8s-0 Bound pvc-23786da3-e00f-11e7-86d8-0e2d7d9cf658 40Gi RWO ssd 19d
prometheus-k8s-db-prometheus-k8s-1 Bound pvc-237a8987-e00f-11e7-86d8-0e2d7d9cf658 40Gi RWO ssd 19dfrom prometheus-kubernetes.
grifonas
commented on June 2, 2024
Hi @camilb,
Yes. The PVC does get created. And its status is "Bound":
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-k8s-db-prometheus-k8s-0 Bound pvc-a9692c9c-ef0b-11e7-a246-1207993b6e7c 40Gi RWO ssd 18h
from prometheus-kubernetes.
camilb
commented on June 2, 2024
Can you please delete the pod and check if it solves the issue?
kubectl delete pod prometheus-k8s-0 -n monitoring
from prometheus-kubernetes.
grifonas
commented on June 2, 2024
Yep. Tried that as well. Ends up stuck in the same state with the same error.
I also tried deleting the whole thing (with your teardown script) apart from the volume itself (leaving the PVC).
I then tried deleting the PVC but leaving the AWS volume.
And then deleting everything including the PVC and the AWS volume...
No luck =(
from prometheus-kubernetes.
camilb
commented on June 2, 2024
How many availability zones do you have?
from prometheus-kubernetes.
grifonas
commented on June 2, 2024
from prometheus-kubernetes.
grifonas
commented on June 2, 2024
People are complaining about it here kubernetes/kubernetes#28962 as well.
from prometheus-kubernetes.
camilb
commented on June 2, 2024
Can you contact me on kubernetes slack team? I have the user camil
from prometheus-kubernetes.
grifonas
commented on June 2, 2024
One sec.
from prometheus-kubernetes.
camilb
commented on June 2, 2024
@grifonas Just finished creating a new cluster using kops and didn't find any issue.
Here is my test configuration:
export NAME=kops.example.com
export KOPS_STATE_STORE=s3://some-bucketkops create cluster \
--kubernetes-version=1.8.4 \
--zones=us-east-1a,us-east-1b,us-east-1c \
--master-zones=us-east-1a,us-east-1b,us-east-1c \
--node-count=3 \
--node-size=t2.medium \
--master-size=t2.medium \
${NAME}kops update cluster kops.example.com --yes$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-172-20-115-107.ec2.internal Ready master 7m v1.8.4 52.90.170.49 Debian GNU/Linux 8 (jessie) 4.4.102-k8s docker://1.13.1
ip-172-20-35-115.ec2.internal Ready master 8m v1.8.4 54.198.26.129 Debian GNU/Linux 8 (jessie) 4.4.102-k8s docker://1.13.1
ip-172-20-44-87.ec2.internal Ready node 6m v1.8.4 34.238.163.158 Debian GNU/Linux 8 (jessie) 4.4.102-k8s docker://1.13.1
ip-172-20-75-20.ec2.internal Ready node 6m v1.8.4 54.236.36.128 Debian GNU/Linux 8 (jessie) 4.4.102-k8s docker://1.13.1
ip-172-20-84-198.ec2.internal Ready master 8m v1.8.4 34.205.93.187 Debian GNU/Linux 8 (jessie) 4.4.102-k8s docker://1.13.1
ip-172-20-98-36.ec2.internal Ready node 6m v1.8.4 54.196.140.27 Debian GNU/Linux 8 (jessie) 4.4.102-k8s docker://1.13.1$ kubectl get pods -n monitoring -o wide
NAME READY STATUS RESTARTS AGE IP NODE
alertmanager-main-0 2/2 Running 0 3m 100.96.3.6 ip-172-20-98-36.ec2.internal
alertmanager-main-1 2/2 Running 0 3m 100.96.5.4 ip-172-20-44-87.ec2.internal
alertmanager-main-2 2/2 Running 0 3m 100.96.4.5 ip-172-20-75-20.ec2.internal
grafana-58cb75bbfc-c29pc 2/2 Running 0 3m 100.96.3.7 ip-172-20-98-36.ec2.internal
kube-state-metrics-5fb4849844-87bvb 2/2 Running 0 4m 100.96.4.3 ip-172-20-75-20.ec2.internal
node-exporter-7f2xx 1/1 Running 0 4m 172.20.84.198 ip-172-20-84-198.ec2.internal
node-exporter-fkcpq 1/1 Running 0 4m 172.20.98.36 ip-172-20-98-36.ec2.internal
node-exporter-flp6x 1/1 Running 0 4m 172.20.35.115 ip-172-20-35-115.ec2.internal
node-exporter-h6sb9 1/1 Running 0 4m 172.20.115.107 ip-172-20-115-107.ec2.internal
node-exporter-n64qd 1/1 Running 0 4m 172.20.44.87 ip-172-20-44-87.ec2.internal
node-exporter-wcgv9 1/1 Running 0 4m 172.20.75.20 ip-172-20-75-20.ec2.internal
prometheus-k8s-0 2/2 Running 0 3m 100.96.5.5 ip-172-20-44-87.ec2.internal
prometheus-k8s-1 2/2 Running 0 3m 100.96.4.6 ip-172-20-75-20.ec2.internal
prometheus-operator-68589bfbfd-vc5dm 1/1 Running 0 4m 100.96.3.3 ip-172-20-98-36.ec2.internalfrom prometheus-kubernetes.
grifonas
commented on June 2, 2024
Very odd indeed. Will continue trying to figure it out.
Oh, by the way, my cluster is in an existing VPC:
kops create cluster \
--name=$KOPS_CLUSTER_NAME \
--state=s3://$CONFIG_BUCKET \
--vpc=$VPC_ID \
--zones us-east-1a,us-east-1b \
--api-loadbalancer-type public \
--associate-public-ip=false \
--authorization RBAC \
--cloud aws \
--dns public \
--dns-zone=$HOSTED_ZONE_ID \
--image ami-aa2ea6d0 \
--kubernetes-version 1.8.4 \
--master-size t2.large \
--master-volume-size 60 \
--master-zones us-east-1a \
--networking weave \
--node-count 2 \
--node-size m5.xlarge \
--node-volume-size 60 \
--ssh-public-key mykey.pub \
--topology private \
--cloud-labels kubernetes.io/cluster/$KOPS_CLUSTER_NAME=ownedThanks!
from prometheus-kubernetes.
camilb
commented on June 2, 2024
We had a similar issue in kube-aws kubernetes-retired/kube-aws#1053
New AWS instance types: P3, C5, M5, H1. Please note that NVME volumes are not supported on the default jessie image, so masters will not boot on M5 and C5 instance types unless a stretch image is chosen (change stretch to jessie in the image name). Also note that kubernetes will not support mounting persistent volumes on NVME instances until Kubernetes 1.9.`
from prometheus-kubernetes.
grifonas
commented on June 2, 2024
I think you might be onto something here! Will give it a go m4 a go. Thanks.
from prometheus-kubernetes.
grifonas
commented on June 2, 2024
@camilb
You were absolutely right. The instance type was indeed the issue.
Thanks again!
from prometheus-kubernetes.
commented on June 2, 2024
I am also facing the same issue with one avaliability zone.
from prometheus-kubernetes.
Related Issues (20)
- Grafana service error
- error validating data: [apiVersion not set, kind not set]; HOT 5
- Feature Request: thanos support
- Target Labels are "dropped" HOT 1
- Stuck in Waiting for Operator to register custom resource definitions HOT 3
- Pods/Container metrics not show up in prometheus HOT 1
- AKS - Waiting for Operator to register custom resource definitions.... HOT 1
- Unable to mount volumes for pod
- I want to rest external_labels
- Kubernetes 1.11 update removed heapster HOT 2
- Kubelet Connection Refused HOT 1
- Custom: deploy prometheus-k8s error: error converting YAML to JSON: yaml: line 21 HOT 3
- Updating prometheus configuration
- Custom,minikube, Deploying Prometheus,prometheus-k8s.yaml: error converting YAML to JSON: yaml: line 22: mapping values are not allowed in this context HOT 2
- Where are the grafana dashboards? HOT 1
- Missing pod/container metrics from prometheus,no data point in grafana HOT 2
- Is this repo under the Apache licence? HOT 1
- pod cpu usage graph is wrong.
- Deploy script getting stuck at Waiting for Operator to register custom resource definitions HOT 1
- line 1921: did not find expected key HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prometheus-kubernetes.