aschzero / hera Goto Github PK

I've noticed when using hera with multiple projects on the same host, the moment I try to run two instances of Hera they seem to fight with eachother.
Errors about "cant find container with id " show up in both hera instance's logs when run together, but when run individually there are no errors.

Steps to reproduce

1. Create projectA/docker-compose.yml:

version: "3"
                                                                                                                               
services:
  hera:
    image: aschzero/hera:latest
    container_name: projectA_hera
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./data/hera:/certs
    links:
      - web
    networks:
      - projectA
                                                                                                                               
  web:
    image: nginx:latest
    expose:
      - 8080
    volumes:
      - ./etc/nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./html:/var/www/html
    networks:
      - projectA
    labels:
      hera.hostname: projectA.example.com
      hera.port: 8080
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080"]

networks:
  projectA:

2. And then create projectB/docker-compose.yml with almost the same content, just a few lines changed:

version: "3"
                                                                                                                               
services:
  hera:
    image: aschzero/hera:latest
    container_name: projectB_hera
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./data/hera:/certs
    links:
      - web
    networks:
      - projectB
                                                                                                                               
  web:
    image: nginx:latest
    expose:
      - 8080
    volumes:
      - ./etc/nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./html:/var/www/html
    networks:
      - projectB
    labels:
      hera.hostname: projectB.example.com
      hera.port: 8080
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080"]

networks:
  projectB:

3. Start both projects cd projectA; docker-compose up -d; cd ../projectB; docker-compose up -d

4. Errors like this appear in both separate Hera instance's docker logs:

projectA_hera  | [INFO] Unable to connect, retrying... (4/5)
projectA_hera  | [INFO] Unable to connect, retrying... (5/5)
projectA_hera  | [ERROR] Unable to connect to e8f8bfe62cb8
projectA_hera  | [INFO] Container found, connecting to db3985d686ee...
projectA_hera  | [INFO] Unable to connect, retrying... (1/5)
projectA_hera  | [INFO] Unable to connect, retrying... (2/5)
projectA_hera  | [INFO] Unable to connect, retrying... (3/5)
projectA_hera  | [INFO] Unable to connect, retrying... (4/5)
projectA_hera  | [INFO] Unable to connect, retrying... (5/5)
projectA_hera  | [ERROR] Unable to connect to db3985d686ee
projectA_hera  | [INFO] Container found, connecting to 2754df6f4c5b...
projectA_hera  | [INFO] Unable to connect, retrying... (1/5)
projectA_hera  | [INFO] Unable to connect, retrying... (2/5)

Looks like this image only works for amd64 / x86_64. Would be great to have a multi-arch docker image (as explained here).

Looks like the Dockerfile needs a few adjustments too in the URLs and paths.

I have replicas for some containers which should be internally load-balanced by docker swarm, and allow for rolling updates. For production, I currently have a separate cloudflared container running for each hostname, pointing to the service names (rather than IP address). I'm not sure if cloudflared caches the IP address.

When using hera instead of the cloudflared containers, I'm seeing an issue where upon restarting the hera container, only 2-3 of the 5 expected tunnel config files are being created.

I check the logs and see e.g.:

[INFO] Hera v0.2.5 has started
[INFO] Found certificate: mydomain.co.nz.pem
[INFO] Container found, connecting to 430c25080893...
[INFO] Registering tunnel sub1.mydomain.co.nz
[INFO] Container found, connecting to d805ef7e8d27...
[INFO] Registering tunnel sub2.mydomain.co.nz
[INFO] Container found, connecting to 2f91eca07de6...
[INFO] Container found, connecting to 09e5ccb1465f...
[INFO] Registering tunnel sub3.mydomain.co.nz
[INFO] Container found, connecting to 33ebdf48b35d...
[INFO] Restarting tunnel sub1.mydomain.co.nz
...

And a bunch of other logs from s6.

So it looks like hera is seeing a second container from the same service and deciding to restart the tunnel. Because there's always only one log entry about restarting a tunnel, and its always the last from hera, I'm figuring it also causes hera to stop creating the tunnels.

Quickly looking through the code, I'm also thinking that having services and replicas might lead to more unexpected behaviour, such as removing or restarting tunnels during rolling updates, and pointing to a singular container.

Also, should the service check take into account the health status of the containers?

What are your thoughts on supporting services with replicas?

Got an alert that my 48GB server was running out of ram, when it normally has ~25GB free. I found this little 😈 feasting itself on my spare RAM.

Unfortunately when I tried to strace it, htop froze, and I had to Ctrl+c to exit, which also killed cloudflared.

You're welcome to close it as "unable to reproduce" if you want, I can always comment back with more details if it happens again,

this would eliminate the requirement for this label and thus make it optional.

When hera detects a container and tries to spin up the container it seems to fail.
I changed the domain name to mydomain.com, that's not the actual one.

[INFO] Hera v0.2.5 has started
[INFO] Found certificate: mydomain.com.pem
[INFO] Container found, connecting to 769e01d96270...
[INFO] Registering tunnel heratest.mydomain.com
[INFO] Hera is listening
time="2021-05-04T10:59:46Z" level=info msg="Build info: {GoOS:linux GoVersion:go1.11.5 GoArch:amd64}"
time="2021-05-04T10:59:46Z" level=info msg="Version 2019.4.1"
time="2021-05-04T10:59:46Z" level=info msg=Flags config=/var/run/s6/services/heratest.mydomain.com/config.yml hostname=heratest.mydomain.com logfile=/var/log/hera/heratest.mydomain.com.log no-autoupdate=true origincert=/certs/mydomain.com.pem proxy-dns-upstream="https://1.1.1.1/dns-query, https://1.0.0.1/dns-query" url="172.23.0.3:9000"
time="2021-05-04T10:59:46Z" level=info msg="Starting metrics server" addr="127.0.0.1:34401"
time="2021-05-04T10:59:46Z" level=info msg="Proxying tunnel requests to http://172.23.0.3:9000"
time="2021-05-04T10:59:46Z" level=error msg="Unable to dial edge" error="Handshake with edge error: x509: certificate has expired or is not yet valid"
time="2021-05-04T10:59:46Z" level=info msg="Retrying in 1s seconds"

I tried with both the latest CLI and cloudflare/cloudflared:2020.7.0 and that worked just fine. Any suggestions?

Cloudflare Load Balancer is supported in cloudflared when using the --lp-pool parameter (see https://developers.cloudflare.com/argo-tunnel/reference/load-balancing/). Ex:

$ cloudflared tunnel --hostname my.example.com --url http://localhost:8080 --lb-pool my-pool-name-goes-here

Please add support to Hera for --lb-pool parameter.

When using this addon (which is very very useful) for Hassio (home assistant docker based system) I'm having an issue with a manual restart on Home Assistant itself which does restart the container. When it happens I get the following in hera:

[INFO] Stopping tunnel name.test.com
time="2020-01-16T03:04:55Z" level=info msg="Initiating graceful shutdown..."
[INFO] Container found, connecting to a806d1313c78...
[INFO] Restarting tunnel name.test.com
time="2020-01-16T03:05:25Z" level=info msg=Quitting...
time="2020-01-16T03:05:25Z" level=info msg="Metrics server stopped"

Is there a way for the tunnel to be opened when a container restarts?

Hera is working great for my setup, but found that I cannot use it for any internal services that use https. Whenever I try, I get this message trying to access my service:

400 Bad Request

The plain HTTP request was sent to HTTPS port.

Reviewing the logs in Portainer, I see that Hera is exposing the service as http://:, regardless of if its https or not.

Can we get two things added?

1. Add the ability to specify if its https which would force the url= parameter to be https://:
2. Add the ability to –no-tls-verify parameter so that self-signed certs will work as well.

Could probably add these into the labels for each container.

Thanks.

I'm looking at using this project in my homelab stack but am unsure if it's still actively maintained. It seems that there are issues that haven't had responses and open pull requests that also haven't had activity for a while.

Just wondering if this project is still being actively maintained or if the author, @aschzero is coming back to it at some point.

this would eliminate the requirement for this label and thus make it optional.

I'd like to create a second tunnel to my open port because CloudFlare Argo was down twice in 60 days. I see that once I run Hera with my Docker app, the docker port is not accessible from localhost.

Is there any way I can use both Hera/CF and another tunnel?

seems that if the domain (in my case something like home.net.za) results in the above error.

hera               | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
hera               | [s6-init] ensuring user provided files have correct perms...exited 0.
hera               | [fix-attrs.d] applying ownership & permissions fixes...
hera               | [fix-attrs.d] 01-log-permissions: applying...
hera               | [fix-attrs.d] 01-log-permissions: exited 0.
hera               | [fix-attrs.d] done.
hera               | [cont-init.d] executing container initialization scripts...
hera               | [cont-init.d] 01-setup-logs: executing...
hera               | [cont-init.d] 01-setup-logs: exited 0.
hera               | [cont-init.d] 02-symlink-certs: executing...
hera               | [cont-init.d] 02-symlink-certs: exited 0.
hera               | [cont-init.d] done.
hera               | [services.d] starting services
hera               | [INFO] Hera v0.2.4 has started
hera               | [INFO] Found certificate: domain.net.za.pem
hera               | [services.d] done.
hera               | [INFO] Hera container found, connecting to e6d474a.......
hera               | [ERROR] Unable to parse hostname https://pihole.domain.net.za: tld not found
hera               | [INFO] Hera is listening

my docker compose: (${DOMAIN} is basicaly just my domain as in domain.net.za)

pihole:
      container_name: pihole
      image: pihole/pihole
      ports:
         - 53:53
         - 53:53/udp
         - 67:67/udp
         - 8081:80
      environment:
         TZ: ${TZ}
         WEBPASSWORD: stars
         ServerIP: 192.168.0.10
         PROXY_LOCATION: pihole
         VIRTUAL_HOST: pihole.${DOMAIN}
         VIRTUAL_PORT: 80
      volumes:
         - /storage.config/pihole:/etc/pihole/
         - /storage.config/dnsmasq:/etc/dnsmasq.d/
         - /storage.config/05-lan.conf:/etc/dnsmasq.d/05-lan.conf
      dns:
         - 127.0.0.1
         - 1.1.1.1
         - 8.8.8.8
      cap_add:
         - NET_ADMIN
      restart: unless-stopped
      labels:
         - hera.hostname=pihole.${DOMAIN}
         - hera.port=80

   hera:
      image: aschzero/hera
      container_name: hera
      volumes:
         - /var/run/docker.sock:/var/run/docker.sock
         - /storage.config/hera/certs:/certs
         - /storage.config/hera/logs:/var/log/hera

also +1 to the idea of being able to configure it directly. like set up hostdomain in the block and then use labels for subdomains or folders?

There are a few use cases for allowing a single container to serve multiple hostnames via either the same port, or different ports.

E.g.

www.mydomain.co.nz        -> mycontainer:8080
www.mydomain.com.au       -> mycontainer:8080
admin.mydomain.co.nz      -> mycontainer:8081
api.mydomain.co.nz        -> camel:8081
privateapi.mydomain.co.nz -> camel:8082
...

The current labelling system does not appear to allow for this kind of flexibility. I am wondering if this might be either changed or extended.

In some cases, this could be worked around by creating duplicate services for each hostname. However, it may not always be practical if numerous hostnames are required.

In my case, it would not be desirable to multiply the number of running containers, because I need to dedicate quite a bit of memory to each instance, some of which may receive relatively little traffic.

Right now hera looks for hera.hostname and hera.port to configure the tunnels. I was wondering if it's possible to customize the lefthand side to be something other than hera so that we can namespace them a bit more, e.g. argo_zone1.hostname and argo_zone1.port.

The other question is whether it's possible to add a hera.upstream label so that we can use argo's SSH tunnelling feature and https endpoints e.g. hera.upstream: "https://localhost:32400" or hera.upstream: "ssh://localhost".

Another issue that could be fixed with additional namespacing options is the ability to have multiple exposed services from one container, e.g. hera.<servicename>.hostname/port.

Full example docker-compose.yml:

version: '3'

services:
    hera:
        image: aschzero/hera:latest
        container_name: hera_zone1
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - ./data/hera:/certs
        environment:
            - HERA_LABEL_PREFIX=zone1_argo   # specify the left-hand side of the label (defaults to "hera", the current behavior)
            - HERA_LABEL_SUBLABELS=true      # allow hera.<anything here>.port in order to support multiple tunnels per container
        networks:
            - zone1

    nginx:
        image: nginx:alpine
        volumes:
            - ./html/:/var/www/html
            - ./etc/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
        networks:
            - zone1
        expose:
            - 8080
            - 9090
        labels:
            zone1_argo.client_dashboard.hostname: dashboard.example.com
            zone1_argo.client_dashboard.port: 8080
            zone1_argo.admin_dashboard.hostname: admin.example.com
            zone1_argo.admin_dashboard.port: 9090
            zone1_argo.ssh.hostanme: ssh.example.com
            zone1_argo.ssh.upstream: "ssh://nginx:22"   # <- ability to support upstream in url form instead of just port

networks:
    zone1:

Hi,

is this project still maintained? After starting your image, i'll just get the warning

ERR Initiating shutdown error="you are using Cloudflare Tunnel in a (legacy) way that is no longer supported.Migrate to Named Tunnels to continue exposing your private origin through Cloudflare's Network.You can follow this guide: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/do-more-with-tunnels/migrate-legacy-tunnels as an example.Alternatively, you can go to https://dash.teams.cloudflare.com/ -> Access -> Tunnels -> Create and rely on the User Interface instead."

Looks like cloudflare changed from these legacy tunnels to named tunnels. Will this project adapt this to be useable again?

Thanks!