Empty reply from server about caddy-prometheus HOT 13 CLOSED

[ Quoting [email protected] in "[miekg/caddy-prometheus] Empty repl..." ]

I am running this using docker image zzrot/alpine-caddy
This is my Caddyfile

https://dashboard.mydomain.com {
 prometheus
 tls /certs/dashboard.mydomain.com.crt /certs/dashboard.mydomain.com.key
 proxy / node1:4000 node2:4000 {
   proxy_header Host {host}
   proxy_header X-Real-IP {remote}
   proxy_header X-Forwarded-Proto {scheme}
   websocket
   except /css /fonts /js /img
 }
 root /node-app/public
 log stdout
 errors stderr
}

Every thing works as expected except for prometheus

when curl http://localhost:9180/metrics I get curl: (52) Empty reply from server Also if I replace the localhost with the actual IP of the host (I have the port 9180 exposed) I get connection refused. Am I missing anything?

where in the caddy stack is prometheus configured? Still should not give 'empty
reply from server' though

/Miek

Miek Gieben

from caddy-prometheus.

I am not sure which order its in. Looking at Dockerfile for ZZROTDesign/alpine-caddy, Its downloading from https://caddyserver.com/download/build?os=linux&arch=amd64&features=cors,git,hugo,ipfilter,jsonp,jwt,mailout,prometheus,realip,search,upload

And the build order I see from https://github.com/caddyserver/buildsrv/blob/master/features/registry.go
is prometheus is listed as last

from caddy-prometheus.

[ Quoting [email protected] in "Re: [miekg/caddy-prometheus] Empty ..." ]

I am not sure which order its in. Looking at Dockerfile for ZZROTDesign/alpine-caddy, Its downloading from https://caddyserver.com/download/build?os=linux&arch=amd64&features=cors,git,hugo,ipfilter,jsonp,jwt,mailout,prometheus,realip,search,upload

And the build order I see from https://github.com/caddyserver/buildsrv/blob/master/features/registry.go
is prometheus is listed as last

interesting... If you can repeat this behavior with prometheus at index 0 (this
is what I use for my setup) then I can take a look.

/Miek

Miek Gieben

from caddy-prometheus.

Shouldn't this be a PR to caddyserver/buildsrv to make prometheus to be at index 0? Since docker image is using the buildsrv to build the caddy

from caddy-prometheus.

Still same issue after having only prometheus add on using build url
https://caddyserver.com/download/build?os=linux&arch=amd64&features=prometheus
This should have placed prometheus at index 0

from caddy-prometheus.

I ended up building my own caddy docker image and added prometheus at index 0
This is my Dockerfile

FROM golang:1.6-alpine

RUN apk --no-cache add git

RUN go get github.com/mholt/caddy \
    && go get github.com/caddyserver/caddyext \
    && go get github.com/miekg/caddy-prometheus \
    && caddyext install prometheus:github.com/miekg/caddy-prometheus \
    && caddyext move prometheus 0 \
    && cd /usr/bin \
    && caddyext build

COPY ./Caddyfile /etc/Caddyfile
CMD ["customCaddy", "--conf", "/etc/Caddyfile"]

this is my Caddyfile

localhost:8888 {
  prometheus
}

This works when I do local build and run it
But when I run this through docker I get empty reply from server

from caddy-prometheus.

[ Quoting [email protected] in "Re: [miekg/caddy-prometheus] Empty ..." ]

I ended up building my own caddy docker image and added prometheus at index 0
This is my Dockerfile

FROM golang:1.6-alpine

RUN apk --no-cache add git

RUN go get github.com/mholt/caddy \
   && go get github.com/caddyserver/caddyext \
   && go get github.com/miekg/caddy-prometheus \
   && caddyext install prometheus:github.com/miekg/caddy-prometheus \
   && caddyext move prometheus 0 \
   && cd /usr/bin \
   && caddyext build

COPY ./Caddyfile /etc/Caddyfile
CMD ["customCaddy", "--conf", "/etc/Caddyfile"]

this is my Caddyfile

localhost:8888 {
 prometheus
}

This works when I do local build and run it
But when I run this through docker I get empty reply from server

Ah! Thanks for the debugging, that is a nice data point.

What happens when you run with docker --net host? Still empty reply?
shouldn't you EXPOSE some extra ports in that docker file for the metrics?

/Miek

Miek Gieben

from caddy-prometheus.

Did some more testing

What happens when you run with docker --net host? Still empty reply?

When I run it with docker run -p="8888:8888" -p="9180:9180" -d --net=host test-caddy I can access the metrics locally using http://localhost:9180/metrics. But when I try to access from outside using the IP address (prometheus container is running in another host) I get connection refused. Also I am trying to run caddy as reverse proxy using docker-compose and I need to use dns setting which is not compatible with network_mode: host (same as --net).

shouldn't you EXPOSE some extra ports in that docker file for the metrics?

I exposed then using -p flags in docker run

from caddy-prometheus.

I am not a Go developer but Shouldn't addr be just the port? I see from the doc log.Fatal(http.ListenAndServe(":8080", nil))

from caddy-prometheus.

[ Quoting [email protected] in "Re: [miekg/caddy-prometheus] Empty ..." ]

Did some more testing

What happens when you run with docker --net host? Still empty reply?

When I run it with docker run -p="8888:8888" -p="9180:9180" -d --net=host test-caddy I can access the metrics locally using http://localhost:9180/metrics. But when I try to access from outside using the IP address (prometheus container is running in another host) I get connection refused. Also I am trying to run caddy as reverse proxy using docker-compose and I need to use dns setting which is not compatible with network_mode: host (same as --net).

So how is this not working as intended? You listen on localhost on the
docker-networking-foo can't forward (or whatever it does) the port from the
outside.

/Miek

Miek Gieben

from caddy-prometheus.

You can just use:
prometheus :8080 which should work for you

from caddy-prometheus.

Yea thanks. That works. It would be nice to add a section in Readme to do that if anyone is running caddy and this under docker

Thanks

from caddy-prometheus.

[ Quoting [email protected] in "Re: [miekg/caddy-prometheus] Empty ..." ]

Yea thanks. That works. It would be nice to add a section in Readme to do that if anyone is running caddy and this under docker

Feel free to propose some text. But I agree the README is a bit terse at the
moment.

/Miek

Miek Gieben

from caddy-prometheus.