Comments (14)
camilb
commented on June 2, 2024
Can you post the logs from the nginx ingress controller?
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
What is the name of ingress controler? I can't find it. Maybe it did not get created?
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
I ran the script again, i get this
Waiting 10 seconds for the Ingress Controller to become available.
No resources found.
Configure "/etc/hosts" or create DNS records for these hosts:
Cleaning modified files
Done
Seems it did not get created.
from prometheus-kubernetes.
camilb
commented on June 2, 2024
Ok, then it was the same issue I fixed in #24. Please erase the nginx-ingress namespace, pull the changes in the repository and run the script again.
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
I did this. Things changed a bit. Now I can do from the master links grafana.hostname.of.the.cluser and get to the prompt to enter password in ncurses environment for the grafana. But does not accept password, just removes username and password and let me type again.
When I try to access from outside, from my laptop using Ingress loadbalancer link after typing kubectl describe svc -n nginx-ingress it still gives me error above.
cluster created with kops version of kubernetes 1.7.5
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
I just tried with SSH tunelling as well. It still gives me default backend - 404
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
Not sure why links gives password prompt then
from prometheus-kubernetes.
camilb
commented on June 2, 2024
Are you planning to use the links without a password? By default it is set to use a basic auth. You can change this in the ingress, if you want.
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
No, password is needed, this will also be installed on production cluster also. Currently trying on dev. I used links only to see if it is accessing. Not sure why links prompts for password while ssh tunneling gives same error message as accessing over ingress load balancer
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
kubectl describe ing --all-namespaces
Name: prometheus
Namespace: monitoring
Address: a6fc456dadsadadadad835dadsa0332a2-4621221227.us-east-1.elb.amazonaws.com
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
prometheus.dev-kube.host.io
/ prometheus:9090 (<none>)
alertmanager.dev-kube.host.io
/ alertmanager:9093 (<none>)
grafana.dev-kube.host.io
/ grafana:3000 (<none>)
Annotations:
auth-secret: basic-auth
auth-type: basic
auth-realm: Authentication Required
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
24m 24m 1 ingress-controller Normal CREATE Ingress monitoring/prometheus
23m 23m 1 ingress-controller Normal UPDATE Ingress monitoring/prometheus
Hostanmes and urls sanitized. Seems that ingress is not connected with prometheus nor alertmanaer nor grafana. How can this be fixed?
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
Here are logs from ingress you asked the other day, but at the time there was no ingress. Now here it is:
kubectl logs -n monitoring prometheus-0
level=info ts=2017-09-23T14:21:37.455866215Z caller=main.go:214 msg="Starting prometheus" version="(version=2.0.0-beta.5, branch=HEAD, revision=64c7932e851f3276391fc3e216cd30f447d4924e)"
level=info ts=2017-09-23T14:21:37.45594732Z caller=main.go:215 build_context="(go=go1.8.3, user=root@df40cc0bd79b, date=20170921-16:23:04)"
level=info ts=2017-09-23T14:21:37.455974516Z caller=main.go:216 host_details="(Linux 4.4.65-k8s #1 SMP Tue May 2 15:48:24 UTC 2017 x86_64 prometheus-0 (none))"
level=info ts=2017-09-23T14:21:37.459668251Z caller=web.go:367 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2017-09-23T14:21:37.459670894Z caller=main.go:308 msg="Starting TSDB"
level=info ts=2017-09-23T14:21:37.460924936Z caller=targetmanager.go:68 component="target manager" msg="Starting target manager..."
level=info ts=2017-09-23T14:21:37.467237555Z caller=main.go:320 msg="TSDB started"
level=info ts=2017-09-23T14:21:37.467262329Z caller=main.go:387 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2017-09-23T14:21:37.468620451Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.469264926Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.471695147Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.472187768Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.473134388Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.474981697Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.478900826Z caller=main.go:364 msg="Server is ready to receive requests."
ts=2017-09-23T17:00:00.789738113Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506175200000 maxt=1506182400000
ts=2017-09-23T17:00:01.339959677Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.289187ms
ts=2017-09-23T17:00:01.340030358Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=1.302µs
ts=2017-09-23T19:00:00.799108843Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506182400000 maxt=1506189600000
ts=2017-09-23T19:00:01.319870037Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.328385ms
ts=2017-09-23T19:00:01.319923278Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=1.112µs
ts=2017-09-23T21:00:00.816460695Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506189600000 maxt=1506196800000
ts=2017-09-23T21:00:01.339474813Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.184751ms
ts=2017-09-23T21:00:01.571722822Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=232.192677ms
ts=2017-09-23T21:00:01.640337482Z caller=compact.go:359 component=tsdb msg="compact blocks" count=2 mint=1506175200000 maxt=1506189600000
ts=2017-09-23T23:00:00.798122586Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506196800000 maxt=1506204000000
ts=2017-09-23T23:00:01.315683703Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.394554ms
ts=2017-09-23T23:00:01.418545214Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=102.7232ms
ts=2017-09-24T01:00:00.788365919Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506204000000 maxt=1506211200000
ts=2017-09-24T01:00:01.299229535Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.687927ms
ts=2017-09-24T01:00:01.405483798Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=106.197858ms
ts=2017-09-24T01:00:01.476536425Z caller=compact.go:359 component=tsdb msg="compact blocks" count=3 mint=1506189600000 maxt=1506211200000
ts=2017-09-24T03:00:00.80940169Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506211200000 maxt=1506218400000
ts=2017-09-24T03:00:01.398754424Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.573839ms
ts=2017-09-24T03:00:01.686396571Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=287.583164ms
ts=2017-09-24T03:00:01.765048482Z caller=compact.go:359 component=tsdb msg="compact blocks" count=2 mint=1506175200000 maxt=1506211200000
ts=2017-09-24T05:00:00.795917393Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506218400000 maxt=1506225600000
ts=2017-09-24T05:00:01.339708957Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.540354ms
ts=2017-09-24T05:00:01.444667574Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=104.899596ms
ts=2017-09-24T07:00:00.817403025Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506225600000 maxt=1506232800000
ts=2017-09-24T07:00:01.436544263Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.755068ms
ts=2017-09-24T07:00:01.739252265Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=302.64951ms
ts=2017-09-24T07:00:01.805270867Z caller=compact.go:359 component=tsdb msg="compact blocks" count=3 mint=1506211200000 maxt=1506232800000
ts=2017-09-24T09:00:00.817406718Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506232800000 maxt=1506240000000
ts=2017-09-24T09:00:01.399094954Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.106398ms
ts=2017-09-24T09:00:01.503549072Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=104.395695ms
ts=2017-09-24T11:00:00.791152687Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506240000000 maxt=1506247200000
ts=2017-09-24T11:00:01.317133151Z caller=head.go:273 component=tsdb msg="head GC completed" duration=19.06363ms
ts=2017-09-24T11:00:01.421245059Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=104.056625ms
ts=2017-09-24T13:00:00.793268102Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506247200000 maxt=1506254400000
ts=2017-09-24T13:00:01.360833676Z caller=head.go:273 component=tsdb msg="head GC completed" duration=30.003962ms
ts=2017-09-24T13:00:01.919111942Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=558.220151ms
ts=2017-09-24T13:00:01.985767239Z caller=compact.go:359 component=tsdb msg="compact blocks" count=3 mint=1506232800000 maxt=1506254400000
ts=2017-09-24T15:00:00.814216845Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506254400000 maxt=1506261600000
ts=2017-09-24T15:00:01.366858312Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.596172ms
ts=2017-09-24T15:00:01.480299747Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=113.381002ms
ts=2017-09-24T17:00:00.802356794Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506261600000 maxt=1506268800000
ts=2017-09-24T17:00:01.405654684Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.258604ms
ts=2017-09-24T17:00:01.685660072Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=279.94733ms
[centos@ip-10-1-0-140 ingress]$ kubectl logs -n monitoring prometheus-0
level=info ts=2017-09-23T14:21:37.455866215Z caller=main.go:214 msg="Starting prometheus" version="(version=2.0.0-beta.5, branch=HEAD, revision=64c7932e851f3276391fc3e216cd30f447d4924e)"
level=info ts=2017-09-23T14:21:37.45594732Z caller=main.go:215 build_context="(go=go1.8.3, user=root@df40cc0bd79b, date=20170921-16:23:04)"
level=info ts=2017-09-23T14:21:37.455974516Z caller=main.go:216 host_details="(Linux 4.4.65-k8s #1 SMP Tue May 2 15:48:24 UTC 2017 x86_64 prometheus-0 (none))"
level=info ts=2017-09-23T14:21:37.459668251Z caller=web.go:367 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2017-09-23T14:21:37.459670894Z caller=main.go:308 msg="Starting TSDB"
level=info ts=2017-09-23T14:21:37.460924936Z caller=targetmanager.go:68 component="target manager" msg="Starting target manager..."
level=info ts=2017-09-23T14:21:37.467237555Z caller=main.go:320 msg="TSDB started"
level=info ts=2017-09-23T14:21:37.467262329Z caller=main.go:387 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2017-09-23T14:21:37.468620451Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.469264926Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.471695147Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.472187768Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.473134388Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.474981697Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-09-23T14:21:37.478900826Z caller=main.go:364 msg="Server is ready to receive requests."
ts=2017-09-23T17:00:00.789738113Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506175200000 maxt=1506182400000
ts=2017-09-23T17:00:01.339959677Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.289187ms
ts=2017-09-23T17:00:01.340030358Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=1.302µs
ts=2017-09-23T19:00:00.799108843Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506182400000 maxt=1506189600000
ts=2017-09-23T19:00:01.319870037Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.328385ms
ts=2017-09-23T19:00:01.319923278Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=1.112µs
ts=2017-09-23T21:00:00.816460695Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506189600000 maxt=1506196800000
ts=2017-09-23T21:00:01.339474813Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.184751ms
ts=2017-09-23T21:00:01.571722822Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=232.192677ms
ts=2017-09-23T21:00:01.640337482Z caller=compact.go:359 component=tsdb msg="compact blocks" count=2 mint=1506175200000 maxt=1506189600000
ts=2017-09-23T23:00:00.798122586Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506196800000 maxt=1506204000000
ts=2017-09-23T23:00:01.315683703Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.394554ms
ts=2017-09-23T23:00:01.418545214Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=102.7232ms
ts=2017-09-24T01:00:00.788365919Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506204000000 maxt=1506211200000
ts=2017-09-24T01:00:01.299229535Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.687927ms
ts=2017-09-24T01:00:01.405483798Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=106.197858ms
ts=2017-09-24T01:00:01.476536425Z caller=compact.go:359 component=tsdb msg="compact blocks" count=3 mint=1506189600000 maxt=1506211200000
ts=2017-09-24T03:00:00.80940169Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506211200000 maxt=1506218400000
ts=2017-09-24T03:00:01.398754424Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.573839ms
ts=2017-09-24T03:00:01.686396571Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=287.583164ms
ts=2017-09-24T03:00:01.765048482Z caller=compact.go:359 component=tsdb msg="compact blocks" count=2 mint=1506175200000 maxt=1506211200000
ts=2017-09-24T05:00:00.795917393Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506218400000 maxt=1506225600000
ts=2017-09-24T05:00:01.339708957Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.540354ms
ts=2017-09-24T05:00:01.444667574Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=104.899596ms
ts=2017-09-24T07:00:00.817403025Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506225600000 maxt=1506232800000
ts=2017-09-24T07:00:01.436544263Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.755068ms
ts=2017-09-24T07:00:01.739252265Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=302.64951ms
ts=2017-09-24T07:00:01.805270867Z caller=compact.go:359 component=tsdb msg="compact blocks" count=3 mint=1506211200000 maxt=1506232800000
ts=2017-09-24T09:00:00.817406718Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506232800000 maxt=1506240000000
ts=2017-09-24T09:00:01.399094954Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.106398ms
ts=2017-09-24T09:00:01.503549072Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=104.395695ms
ts=2017-09-24T11:00:00.791152687Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506240000000 maxt=1506247200000
ts=2017-09-24T11:00:01.317133151Z caller=head.go:273 component=tsdb msg="head GC completed" duration=19.06363ms
ts=2017-09-24T11:00:01.421245059Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=104.056625ms
ts=2017-09-24T13:00:00.793268102Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506247200000 maxt=1506254400000
ts=2017-09-24T13:00:01.360833676Z caller=head.go:273 component=tsdb msg="head GC completed" duration=30.003962ms
ts=2017-09-24T13:00:01.919111942Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=558.220151ms
ts=2017-09-24T13:00:01.985767239Z caller=compact.go:359 component=tsdb msg="compact blocks" count=3 mint=1506232800000 maxt=1506254400000
ts=2017-09-24T15:00:00.814216845Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506254400000 maxt=1506261600000
ts=2017-09-24T15:00:01.366858312Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.596172ms
ts=2017-09-24T15:00:01.480299747Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=113.381002ms
ts=2017-09-24T17:00:00.802356794Z caller=compact.go:359 component=tsdb msg="compact blocks" count=1 mint=1506261600000 maxt=1506268800000
ts=2017-09-24T17:00:01.405654684Z caller=head.go:273 component=tsdb msg="head GC completed" duration=17.258604ms
ts=2017-09-24T17:00:01.685660072Z caller=head.go:282 component=tsdb msg="WAL truncation completed" duration=279.94733ms
from prometheus-kubernetes.
camilb
commented on June 2, 2024
These are the logs from prometheus, the ingress logs are in nginx-ingress namespace.
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
This is still making problems. I have contacted few of my coworkers who use Macs, they get the same message like me. Seems it is working only on your Mac. Strangely, today stoped even giving me that message from yesterday.
from prometheus-kubernetes.
mikimtm
commented on June 2, 2024
Seems Ingress controler is broken in some way.
from prometheus-kubernetes.
Related Issues (20)
- Grafana service error
- error validating data: [apiVersion not set, kind not set]; HOT 5
- Feature Request: thanos support
- Target Labels are "dropped" HOT 1
- Stuck in Waiting for Operator to register custom resource definitions HOT 3
- Pods/Container metrics not show up in prometheus HOT 1
- AKS - Waiting for Operator to register custom resource definitions.... HOT 1
- Unable to mount volumes for pod
- I want to rest external_labels
- Kubernetes 1.11 update removed heapster HOT 2
- Kubelet Connection Refused HOT 1
- Custom: deploy prometheus-k8s error: error converting YAML to JSON: yaml: line 21 HOT 3
- Updating prometheus configuration
- Custom,minikube, Deploying Prometheus,prometheus-k8s.yaml: error converting YAML to JSON: yaml: line 22: mapping values are not allowed in this context HOT 2
- Where are the grafana dashboards? HOT 1
- Missing pod/container metrics from prometheus,no data point in grafana HOT 2
- Is this repo under the Apache licence? HOT 1
- pod cpu usage graph is wrong.
- Deploy script getting stuck at Waiting for Operator to register custom resource definitions HOT 1
- line 1921: did not find expected key HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prometheus-kubernetes.