Description
I don't know if it's a bug or a feature. I have a service api that have multiple listening ports. Let's say one for public and one for admin/private. I want to publish each port to a different serviceDomain.
[I'm using manual configuration (with curl call).]
Each configuration works well, but I didn't manage to make them work together. The last one always override the previous configuration for the same serviceName. I suspect a name matching is done before inserting a new service but it does not take port in consideration.
I would like serviceA:8080 -> public.test.com and serviceA:8090 -> admin.test.com

Steps to reproduce the issue:

1. Create multi port container
2. Using manual config map each port to a different domainName

Describe the results you received:
Last config override the previous one for a same serviceName

Describe the results you expected:
Each port should be accessible on the requested domaineName

Additional information you deem important (e.g. issue happens only occasionally):

Additional environment details (AWS, VirtualBox, physical, etc.):
VirtualBox

BTW thanks a lot for this great product.

Hi,
I try to build a custom fe and be template but I don't get it working. May you give a brief example how to use the golang templating in order to render the services and use the userDef attributes.

Any help is highly appreciated. Thanks in advance.

Cheers,
Volker

When I want to use HAProxys native syslog logging facility how should I configure it in docker-flow-proxy. Use templates? Frontend and Global extra blocks? I understand that from syslog perspective I either have to use remote IP or map /dev/log from host into container.

Description
Question: We have a requirement to setup a TCP socket microservice. We currently use docker flow proxy in our swarm for all of our http, https, and wss traffic. Works great! Fantastic project!

I've been looking into using reqMode=tcp. That in itself works great, but wasn't quite sure how or if it's even possible to use it for SSL termination and not just SSL Passthru.

Additional environment details (AWS, VirtualBox, physical, etc.):
Docker 18.03-ce using AWS and Docker4AWS cloudformation template with 5 managers and 19 workers.

I think I could add this myself with custom template support but maybe this is something that most of the users would need they just don't know it.

HAProxy has health check support, and in case of containerized environment thats something we really want. We do not want to sent trafic to instances that are still starting up or shutting down.

More information from here: https://www.serverlab.ca/tutorials/linux/network-services/how-to-configure-haproxy-health-checks/

Hi,

First of all thanks for dfp it's very nice and the documentation is well done so it's actually very usable :)

I've been settings things up in a dev. environment using a couple of VMs and so far it's looking great. However I just ran into an issue and I'm wondering if i'm missing something or it's a limitation.

So i've set up a stack deployment of the proxy + swarm listener as shown in the documentation and that's working perfectly fine.

Now I'm trying to setup a service with credentials but I'm running into issues with docker stack prefixing the secrets with the stack's name while dnf expects dfp_users_<secret_name>.

So as a reference I'm setting up a basic docker registry with authentication like this:

version: '3.7'

services:
  registry:
    image: registry:2
    networks:
      - proxy
    volumes:
      - type: volume
        source: registrydata
        target: /var/lib/registry
        volume:
          nocopy: true
    deploy:
      placement:
        constraints: [node.labels.registry == true]
      labels:
        - com.df.notify=true
        - com.df.serviceDomain=mydomain
        - com.df.port=5000
        - com.df.usersPassEncrypted=true
        - com.df.usersSecret=credentials

volumes:
  registrydata:

secrets:
  dfp_users_credentials:
    file: ./credentials.txt

networks:
  proxy:
    external: true

This spawns everything as it should but the secret created is registry_dfp_users_credentials

$ docker secret ls
ID                          NAME                             DRIVER              CREATED             UPDATED
kmqx8y2cd8r9xsq1vo71tf16i   registry_dfp_users_credentials                       17 seconds ago      17 seconds ago

As a workaround I could always make the secret external and simply define it manually but I was wondering if there was a better way of doing it.

Thanks

Description

Trying to configure a service to allow for proxying multiple https ports from outside the swarm.

Our proxy configuration which is pretty much out of the box
dfp-config.yaml.txt

My latest shot in trying to configure multiple ports and their associated srcPort.
wso-config.yaml.txt

The two container called api-manager uses the following ports for different purposes:

• http port 8280 for API-REST stuff.
• https port 8243 for API-REST stuff.
• https port 9443 for GUI stuff.

Obviously we could go for a port 80 -> 8280 solution, but there is still a need for mapping two different https port to the same container. The motive behind not using 80/443 is that the above ports are the default ports used in WSO2 products, and are known in our organization. We try not to confuse our poor developers by switching ports on them 😉

I am not aware of there being different paths for the different ports, but I'll check up on it as soon as I can get a hold of one of our architects on the solution.

**Create new cluster on AWS on master two ubuntu 16 deploy with stack and manually the proxy ( as explain in this page http://proxy.dockerflow.com/swarm-mode-auto/ the service is not starting and stack on start only if i delete the network it started **

Steps to reproduce the issue:

1. create new cluster with one master two node on ubuntu 16
2. create netwrk proxy
3. try to deploy th eproxy

Describe the results you received:

2018/06/13 11:33:19 Starting HAProxy
2018/06/13 11:38:25 Starting HAProxy
2018/06/13 11:26:15 Starting HAProxy
2018/06/13 11:20:46 Starting HAProxy
2018/06/13 11:30:10 Starting HAProxy
Describe the results you expected:
need to run...

Additional information you deem important (e.g. issue happens only occasionally):

Additional environment details (AWS, VirtualBox, physical, etc.):

Description

I have DFP deployed as Docker Swarm stack 2 replicas across 2 different nodes. In front of DFP there is a loadbalancer. At the moment LB points to just one node in order to reduce the debugging surface. For the sake of the test I have also passed LB and executed requests directly against DFP and found out it does not have impact.

Steps to reproduce the issue:

1. Deploy DFP ... wait
2. Deploy application stack on application node 1
3. Wait, to give cluster the chance to not have excuses
4. Repetitively execute requests against DFP
5. Deploy another version of stack on application node 2
6. Wait, to give cluster the chance to not have excuses
7. Repetitively execute requests against DFP

Describe the results you received:

Usually when first app stack is deployed DFP does not return 503. But almost always it does randomly when second stack is deployed.

I have also checked /cfg/haproxy.cfg on both proxy nodes but they are the same. I have also tried DO_NOT_RESOLVE_ADDR=true but without effect.

Once I decrease the number of replicas to 1, everything is stable.

In the proxy log on the node where the request reaches I get something like this:

2018/06/09 15:43:11 HAPRoxy: 10.255.0.9:43070 [09/Jun/2018:15:43:11.898] services~ services/<NOSRV> -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET /console/ HTTP/1.1"

... although the service (the container is up & listening, the entry is in haproxy.cfg) is up.

Describe the results you expected:

I expect no 503 for replicas > 1 as well.

Additional information you deem important (e.g. issue happens only occasionally):

Additional environment details (AWS, VirtualBox, physical, etc.):

DFP tested: 18.05.26-51 and previously also 17.09.22-18
Docker engine: 17.09.0-ce

http://proxy.dockerflow.com/swarm-mode-auto/

docker service create --name swarm-listener
--network proxy
--mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock"
-e DF_NOTIFY_CREATE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/reconfigure
-e DF_NOTIFY_REMOVE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/remove
--constraint 'node.role==manager'
docker-flow/docker-flow-swarm-listener

"docker-flow/docker-flow-swarm-listener" should be
"dockerflow/docker-flow-swarm-listener" I believe

Same with the docker-flow-proxy command afterwards.

Description

How can we specify resolvers to the haproxy? Not the docker resolver.

Just to understand our use case:
We have a few dynamic hostnames that periodically changes their IP, if we don't set a few resolvers with the "hold" option haproxy does not handle traffic to the correct IP.

I've found the DO_NOT_RESOLVE_ADDR but I think it's not what we are looking for.

Additional information you deem important (e.g. issue happens only occasionally):

Just to let you know our use case, is something like this, and for this example I used the public internet dns servers we know. But, we use a private dns cluster there.

resolvers dns
	nameserver dns-0  4.4.2.1:53
	nameserver dns-1  8.8.8.8:53
	nameserver dns-2  4.4.2.2:53
	nameserver dns-3  8.8.4.4:53
	hold valid 300s

backend infra-ddns
	mode http
        timeout connect 60s
	timeout server  60s
	server ddns1 ddns.domain.net:9000 resolvers dns check
	server ddns2 ddns2.domain.net:9000 resolvers dns check 

Description

When using DFP to redirect http requests to https, the client accessing the url quits reporting too many redirects.

Steps to reproduce the issue:

1. Create proxy network
2. Create cert secret
3. Deploy proxy-stack.yml and go-demo.yml
4. curl -L https://test.mydomain.com/demo/hello (you may need to change domain)

Describe the results you received:

curl: (47) Maximum (50) redirects followed

Describe the results you expected:

Hello, World!

Additional information you deem important (e.g. issue happens only occasionally):

Previous versions of DFP did not have this problem (in particular, vfarcic/docker-flow-proxy:17.10.22-47).

Additional environment details (AWS, VirtualBox, physical, etc.):

DFP logs when the go-demo service is deployed. Here is a trace

Tried on AWS and Virtualbox environment.

Description

The issue #1 introduced the RESOLVERS enviroment but it rely on the CHECK_RESOLVERS enviroment and right now we can't overwrite it on a per service basis.

Describe the results you expected:

Be able to set CHECK_RESOLVERS per service

I'm having some issues with Jenkins and it is currently unavailable. I'll do my best to fix it soon.

I have a backend which listens on two ports, 8000 and 8008. On port 8000 the service talks "http", on 8008 and it speaks an arbitrary protocol, on top of that it will only accept traffic when a given SNI header is provided.

As such, to route my traffic I need to use a mixtures of request modes, namely http and sni. This works well using the index suffix you can provide to reqMode. However, one cannot currently do the same for pathType meaning that the config will either be using a) path_beg (the default) for both or b) whatever I put in com.df.pathType.

Can we please allow for pathType to also support an index suffix?

The outcome I would like to see is that I can render the following configuration:

frontend services
    bind *:80
    bind *:443 ssl crt-list /cfg/crt-list.txt alpn h2,http/1.1
    mode **http**
    option  forwardfor 
    ...
    acl url_xxx8000_1 **path_beg** /path
    acl domain_xxx8000_1 hdr_beg(host) -i myhost.mydomain
   
frontend service_443
    bind *:443
    mode **tcp**
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    acl sni_xxx8008_2 **req_ssl_sni -m reg -i** MyMagicSNIHeader
    ....

Every time I switch a service off by scaling it to 0 replicates, none of the other services are available, because haproxy cannot find the hostname. I'd love to see docker-flow-proxy skip all services which have 0 replicates, to keep the haproxy configuration valid.

If this is achievable with a custom template, that would be fine too, but I'd need a hint as to how the template should look like.

Description
I'm experiencing random 503 errors and I was wondering if two things could be linked ?!
Could it be some latency induced by the reloading of the config ?
I have 3 instances running and all of them spit out logs by the second, reloading and validating the configuration

Describe the results you received:
proxy_proxy.3.cex3u74s4xfh@mgr3 | 2018/06/29 06:32:52 Reloading the proxy
proxy_proxy.3.cex3u74s4xfh@mgr3 | 2018/06/29 06:32:52 Validating configuration
proxy_proxy.3.cex3u74s4xfh@mgr3 | Configuration file is valid
proxy_proxy.3.cex3u74s4xfh@mgr3 | 2018/06/29 06:32:52 Proxy config was reloaded
proxy_proxy.3.cex3u74s4xfh@mgr3 | 2018/06/29 06:32:57 Reloading the proxy
proxy_proxy.3.cex3u74s4xfh@mgr3 | 2018/06/29 06:32:57 Validating configuration
proxy_proxy.3.cex3u74s4xfh@mgr3 | Configuration file is valid
proxy_proxy.3.cex3u74s4xfh@mgr3 | 2018/06/29 06:32:58 Proxy config was reloaded
proxy_proxy.3.cex3u74s4xfh@mgr3 | 2018/06/29 06:33:03 Reloading the proxy
proxy_proxy.3.cex3u74s4xfh@mgr3 | 2018/06/29 06:33:03 Validating configuration
proxy_proxy.3.cex3u74s4xfh@mgr3 | Configuration file is valid
proxy_proxy.3.cex3u74s4xfh@mgr3 | 2018/06/29 06:33:03 Proxy config was reloaded
proxy_proxy.1.i54r1xgen9nd@node3 | 2018/06/29 06:33:04 Reloading the proxy
proxy_proxy.1.i54r1xgen9nd@node3 | 2018/06/29 06:33:04 Validating configuration
proxy_proxy.1.i54r1xgen9nd@node3 | Configuration file is valid
proxy_proxy.1.i54r1xgen9nd@node3 | 2018/06/29 06:33:04 Proxy config was reloaded
proxy_proxy.2.osq32xxyjhij@node1 | 2018/06/29 06:33:07 Reloading the proxy
proxy_proxy.2.osq32xxyjhij@node1 | 2018/06/29 06:33:07 Validating configuration
proxy_proxy.2.osq32xxyjhij@node1 | Configuration file is valid
proxy_proxy.2.osq32xxyjhij@node1 | 2018/06/29 06:33:07 Proxy config was reloaded

Additional information
DFP version :latest
Docker version 18.03.1-ce

DFP Swarm config
version: "3.2"
networks:
proxy:
external: true
services:
proxy:
image: dockerflow/docker-flow-proxy:${DFPTAG:-latest}
ports:
- 80:80
- 443:443
networks:
- proxy
environment:
- LISTENER_ADDRESS=swarm-listener
- MODE=swarm
- CONNECTION_MODE=${CONNECTION_MODE:-http-keep-alive}
deploy:
replicas: 3
swarm-listener:
image: dockerflow/docker-flow-swarm-listener:${DFPSLTAG:-latest}
networks:
- proxy
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- DF_NOTIFY_CREATE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/reconfigure
- DF_NOTIFY_REMOVE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/remove
deploy:
placement:
constraints: [node.role == manager]

Services configuration
-- extract:

deploy:
  labels:
    - com.df.notify=true
    - com.df.distribute=true
    - com.df.serviceDomain=host.domain.com
    - com.df.port=8069
    - com.df.reqMode=http

By default docker-flow-proxy isn't listening only to ipv4 could we have an option to listen to ipv6 also?

root@f22d1f4a9ec5:/# netstat -anp | grep :80
tcp        0      0 172.x.x.x:44546        123.123.123.123:80        TIME_WAIT   -               
tcp        0      0 172.x.x.x:34684        123.123.123.123:80        TIME_WAIT   -               
tcp        0      0 172.x.x.x:37834        123.123.123.123:80         TIME_WAIT   - 

This feature has been previously requested, then implemented and since regressed.

Original feature request: vfarcic#351
Implementation: vfarcic#374
Caused a problem: #4
Regression: #5

To sum up the original feature request, there is a need to perform a redirect unless the X-Forwarded-Proto: https header is present. Apparently, solving it by applying inverse logic based on the current query parameter redirectWhenHttpProto is not sufficient, ref. #4 .

So I'm suggesting a new service query parameter redirectUnlessHttpsProto to fix exactly our problem, avoiding making any bad surprises for anyone else. I'd be happy to supply a PR if there's general agreement that this is the way to go.

The main issue is that the repository source code is not searchable. To quote GitHub: Searching in Forks:

Forks are only indexed for code search when they have more stars than the parent repository. You will not be able to search the code in a fork that has less stars than its parent.

As it stands, this repo will be without code search for a long time.

More reading about how's and why's: Transferring a repository owned by your personal account

Description

Would it be possible to update haproxy 1.8.3 to newer version?

When using IE in win10 with https (haproxy is ssl termination), it uses http/2 by default. As result some pages don't load because it takes to long and there is timeout. In chrome and firefox this works.
I tried to building custom image with haproxy 1.8.4 and it resolved the issue. So would it be possible to
update version in Dockerfile?

I'd like to run multiple dfp instances with one dfsl and let the service decide which proxy to configure with a service label.

Currently dfsl can have multiple urls specified in DF_NOTIFY_CREATE_SERVICE_URL, also the proxy can have a custom name PROXY_INSTANCE_NAME already, the missing thing is a service label, something like com.df.proxyInstanceName would be great if we wanted some services to register in only one of the dfp instances.

I am using the new com.df.discoveryType: DNS as described in #13

Unfortunately it doesn't work when the service is set to mode: global. I have to use replicated mode and set a specific number of replicas to get it to work.

If I try it with mode: global I get the Following error:

2018-05-24T10:54:15.821057657Z proxy_proxy.0.kvxmdwqiqg57@swarm-manager-01    | 2018/05/24 10:54:15 Validating configuration
2018-05-24T10:54:15.828949479Z proxy_proxy.0.kvxmdwqiqg57@swarm-manager-01    | [ALERT] 143/105415 (265) : parsing [/cfg/haproxy.cfg:87] : Wrong server-template number or range arg '0'.
2018-05-24T10:54:15.836071732Z proxy_proxy.0.kvxmdwqiqg57@swarm-manager-01    | [ALERT] 143/105415 (265) : Error(s) found in configuration file : /cfg/haproxy.cfg
2018-05-24T10:54:15.836131332Z proxy_proxy.0.kvxmdwqiqg57@swarm-manager-01    | [ALERT] 143/105415 (265) : Fatal errors found in configuration.
2018-05-24T10:54:15.850056632Z proxy_proxy.0.kvxmdwqiqg57@swarm-manager-01    | Exit Status: 1

The generated Backend looks like this:

backend app_web-be80_0
     mode http
     http-request redirect scheme https if !{ ssl_fc }
     http-request add-header X-Forwarded-Proto https if { ssl_fc }
     server-template app_web 0 app_web:80 check

This is probably intended. Not sure if there is an easy way for docker flow proxy to get the number of replicas running for a global service. Just wanted to make sure that this is not a bug.

It would be great if docker-flow-proxy allows canary deployments.

Let's say we deploy two versions of the same service, mapped with the same docker-flow-proxy configuration. It would be great if the HAProxy is configured to redirect traffic to both versions of the application. Currently, as far as I've been able to test, that's not the case.

With a docker-compose file like this one:

version: '3.5'

services:
  canary:
    build:
      context: .
      dockerfile: Dockerfile
    image: canary:0.1.0
    deploy:
      labels:
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.servicePath=/canary
        - com.df.port=8080
      replicas: 2
      update_config:
        parallelism: 1
        failure_action: rollback
    networks:
      - proxy
      - service
  canaryV2:
      build:
        context: .
        dockerfile: Dockerfile
      image: canary:0.2.0
      deploy:
        labels:
          - com.df.notify=true
          - com.df.distribute=true
          - com.df.servicePath=/canary
          - com.df.port=8080
        replicas: 2
        update_config:
          parallelism: 1
          failure_action: rollback
      networks:
        - proxy
        - service
networks:
  proxy:
    external: true
  service:
    external: true

The only version that gets traffic is Version 1. First approach would be to redirect traffic in a round-robin way. Most advanced scenarios would be to use request headers to identify what traffic goes to which version of the service.

Let me know if I can help with this feature.

This is a feature request.
Would it be possible to add multiple listener addresses to DFP ? DFP could then get list of backends from multiple DFSL in different swarms.

Currently it's possible to run multiple DFSL in different swarms and have them update DFP and work correctly, until DFP restarts or scales. Adding multiple listener addresses to DFP could be a way around this.

I think this would also need an option added to DFSL to include a prefix to the service name to avoid collisions where two stacks have the same name in different swarms.

Thoughts ?

We are experiencing SSL connection errors from client on first connections to our site which are docker services sitting behind the DFP proxy docker service. They occasionally get the error when connecting then future page loads seem to work ok. The errors occur in Edge, Chrome & OpenSSL client when testing.

The site is a swarm service setup behind the DFP. We are experiencing the same problem with a certificate issued by our internal CA and also when we use the lets encrypt companion DFP service with a letsencrypt certificate and experiencing the same problem in different environments.

Our upstream services consist of an Angular app and dotnet core back end services which we are experiencing the errors when accessing the site and the back end services directly when debugging the problem.

Examples of the errors:

Chrome:

OpenSSL debug via git bash:
mason@workstation MINGW64 ~
$ openssl s_client -debug -connect mysite.com:443
CONNECTED(00000074)
write to 0x2954bf0 [0x2955330] (308 bytes => 308 (0x134))
0000 - 16 03 01 01 2f 01 00 01-2b 03 03 03 96 2e 3d 79 ..../...+.....=y
0010 - 30 4d 37 90 09 7d e4 dc-1e d1 0a a7 59 f7 ad f0 0M7..}......Y...
0020 - 8c 52 d3 8a f9 f6 f8 ad-b3 5b 84 00 00 ac c0 30 .R.......[.....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-00 07 c0 11 c0 07 c0 0c .<./...A........
00c0 - c0 02 00 05 00 04 c0 12-c0 08 00 16 00 13 00 10 ................
00d0 - 00 0d c0 0d c0 03 00 0a-00 ff 02 01 00 00 55 00 ..............U.
00e0 - 0b 00 04 03 00 01 02 00-0a 00 1c 00 1a 00 17 00 ................
00f0 - 19 00 1c 00 1b 00 18 00-1a 00 16 00 0e 00 0d 00 ................
0100 - 0b 00 0c 00 09 00 0a 00-23 00 00 00 0d 00 20 00 ........#..... .
0110 - 1e 06 01 06 02 06 03 05-01 05 02 05 03 04 01 04 ................
0120 - 02 04 03 03 01 03 02 03-03 02 01 02 02 02 03 00 ................
0130 - 0f 00 01 01 ....
read from 0x2954bf0 [0x295a890] (7 bytes => 7 (0x7))
0000 - 48 54 54 50 2f 31 2e HTTP/1.

no peer certificate available

No client certificate CA names sent

SSL handshake has read 7 bytes and written 308 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1526296307
Timeout : 300 (sec)
Verify return code: 0 (ok)

10680:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:

mason@workstation MINGW64 ~

==> If I run this again a second time, it works successfully. (Ignore the 404 error from the proxy as it's routing the request based on hostname)

mason@workstation MINGW64 ~
$ openssl s_client -debug -connect mysite.com:443
CONNECTED(000001B4)
write to 0x2a24250 [0x2a25330] (308 bytes => 308 (0x134))
0000 - 16 03 01 01 2f 01 00 01-2b 03 03 d1 02 ef 8d 97 ..../...+.......
0010 - f0 65 7e 7e 8e d0 31 a8-db 8d f9 6f f7 79 bf d1 .e~~..1....o.y..
0020 - 1a d7 4b 0e 39 35 4a 82-46 0d 32 00 00 ac c0 30 ..K.95J.F.2....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-00 07 c0 11 c0 07 c0 0c .<./...A........
00c0 - c0 02 00 05 00 04 c0 12-c0 08 00 16 00 13 00 10 ................
00d0 - 00 0d c0 0d c0 03 00 0a-00 ff 02 01 00 00 55 00 ..............U.
00e0 - 0b 00 04 03 00 01 02 00-0a 00 1c 00 1a 00 17 00 ................
00f0 - 19 00 1c 00 1b 00 18 00-1a 00 16 00 0e 00 0d 00 ................
0100 - 0b 00 0c 00 09 00 0a 00-23 00 00 00 0d 00 20 00 ........#..... .
0110 - 1e 06 01 06 02 06 03 05-01 05 02 05 03 04 01 04 ................
0120 - 02 04 03 03 01 03 02 03-03 02 01 02 02 02 03 00 ................
0130 - 0f 00 01 01 ....
read from 0x2a24250 [0x2a2a890] (7 bytes => 7 (0x7))
0000 - 16 03 03 00 42 02 ....B.
0007 - <SPACES/NULS>
read from 0x2a24250 [0x2a2a89a] (64 bytes => 64 (0x40))
0000 - 00 3e 03 03 cc 11 71 78-6b 42 e8 e7 c9 c0 db 79 .>....qxkB.....y
0010 - e3 da 4e 7e c6 52 3d 5a-f3 3c cd b2 f8 c0 90 73 ..N~.R=Z.<.....s
0020 - f6 3d fe 15 00 c0 30 00-00 16 ff 01 00 01 00 00 .=....0.........
0030 - 0b 00 04 03 00 01 02 00-23 00 00 00 0f 00 01 01 ........#.......
read from 0x2a24250 [0x2a2a893] (5 bytes => 5 (0x5))
0000 - 16 03 03 0b c9 .....
read from 0x2a24250 [0x2a2a898] (3017 bytes => 1084 (0x43C))
0000 - 0b 00 0b c5 00 0b c2 00-07 26 30 82 07 22 30 82 .........&0.."0.
0010 - 06 0a a0 03 02 01 02 02-12 04 4f 75 81 ac a1 3e ..........Ou...>
0020 - 1b ec 44 e2 e8 73 a1 c3-02 41 3a 30 0d 06 09 2a ..D..s...A:0...*
0030 - 86 48 86 f7 0d 01 01 0b-05 00 30 4a 31 0b 30 09 .H........0J1.0.
0040 - 06 03 55 04 06 13 02 55-53 31 16 30 14 06 03 55 ..U....US1.0...U
0050 - 04 0a 13 0d 4c 65 74 27-73 20 45 6e 63 72 79 70 ....Let's Encryp
0060 - 74 31 23 30 21 06 03 55-04 03 13 1a 4c 65 74 27 t1#0!..U....Let'
0070 - 73 20 45 6e 63 72 79 70-74 20 41 75 74 68 6f 72 s Encrypt Author
0080 - 69 74 79 20 58 33 30 1e-17 0d 31 38 30 35 31 31 ity X30...180511
0090 - 30 30 30 31 33 33 5a 17-0d 31 38 30 38 30 39 30 000133Z..1808090
00a0 - 30 30 31 33 33 5a 30 27-31 25 30 23 06 03 55 04 00133Z0'1%0#..U.
00b0 - 03 13 1c 70 72 69 76 78-2d 69 64 61 61 73 2d 64 ...mysite.com...
00c0 - 65 6d 6f 2e 65 64 65 6c-6b 65 79 2e 6e 65 74 30 ................
00d0 - 82 02 22 30 0d 06 09 2a-86 48 86 f7 0d 01 01 01 .."0....H......
00e0 - 05 00 03 82 02 0f 00 30-82 02 0a 02 82 02 01 00 .......0........
00f0 - 99 a0 79 34 89 b1 b7 2b-e1 af a5 51 f5 b0 c7 79 ..y4...+...Q...y
0100 - a7 15 10 7b e8 6a 7b 3e-85 37 48 a9 b0 a5 e8 7f ...{.j{>.7H.....
0110 - 47 46 05 1d 4d ac 59 96-e1 ba e2 68 03 8a 83 21 GF..M.Y....h...!
0120 - 2a 8d a3 25 bf c4 54 c8-bf 8f 3b 23 ff 95 20 e6 ..%..T...;#.. .
0130 - 1b 16 b9 40 25 a9 67 21-88 5b ce 80 79 6b 37 22 ...@%.g!.[..yk7"
0140 - 22 0b 50 ce 8d 7c 18 e0-a7 8a 1a dc 2f a0 6f 08 ".P..|....../.o.
0150 - 47 49 2b 58 2f c2 38 60-c1 59 69 97 4c 87 eb 99 GI+X/.8.Yi.L... 0160 - 51 fa b5 aa 50 8f d5 0c-bc 74 78 8a c1 be a1 38 Q...P....tx....8 0170 - fb 23 9b 88 72 ec 8b 32-fe 3a 1b e1 39 1c 06 a2 .#..r..2.:..9... 0180 - 9c ea 47 6a a3 6d 9d 2c-3c 11 86 e4 6a 5e 94 7b ..Gj.m.,<...j^.{ 0190 - bf 8d 13 8b 71 13 02 27-8c fb a6 f6 79 e5 26 1f ....q..'....y.&. 01a0 - 65 57 9f 5f 9f 10 b0 4f-65 dc 5c 05 5e b8 99 35 eW._...Oe.\.^..5 01b0 - 8c 9d 06 09 99 eb ab 07-28 ea 28 b4 d7 8a 67 53 ........(.(...gS 01c0 - b5 62 ce 16 1d f9 11 f0-3c 48 c3 e8 69 32 d4 b0 .b......<H..i2.. 01d0 - b1 af aa 9f 1e b6 61 0a-9a 30 3f f9 ba 34 64 37 ......a..0?..4d7 01e0 - ef 5d 74 bb 01 67 86 a7-23 db 68 f1 25 08 3f 57 .]t..g..#.h.%.?W 01f0 - f7 e0 f8 de 87 53 a9 64-0a 45 14 d5 da 85 8a 45 .....S.d.E.....E 0200 - 7c 7b 6c 8d 15 3d 8b b4-0e b4 9d ad 9c 43 f1 e0 |{l..=.......C.. 0210 - c9 ba ea d5 0b a4 c6 02-2a 70 c1 57 8a 59 ba 14 ........*p.W.Y.. 0220 - 14 16 31 d9 23 d8 39 f0-03 34 43 69 d4 10 47 64 ..1.#.9..4Ci..Gd 0230 - 42 26 23 de ce 74 3f 57-2a d3 36 a7 6f 98 d8 55 B&#..t?W*.6.o..U 0240 - 2b 65 f5 9a 26 18 6d 54-96 95 c6 2b 5d 58 e2 47 +e..&.mT...+]X.G 0250 - 5e a3 f8 ff 1b a5 d5 e4-24 bf 55 05 5d 35 a4 4f ^.......$.U.]5.O 0260 - fd 8d f5 be 3a 75 8b 94-31 c2 41 84 1c f5 b9 32 ....:u..1.A....2 0270 - 42 6e 09 e6 71 c3 76 8b-dc 34 91 08 37 95 28 22 Bn..q.v..4..7.(" 0280 - fc 12 1f c9 33 8d 4e da-4c d8 13 ea 78 79 9d af ....3.N.L...xy.. 0290 - 09 0b 06 f0 76 8c b1 f9-b2 ad d9 e7 dc b5 ef 33 ....v..........3 02a0 - 53 3a 67 23 8e 0a 3e b7-cf b5 e1 36 77 38 ed 0b S:g#..>....6w8.. 02b0 - ea 4c 17 db 02 f8 94 0e-7f 77 8d 69 ea 2f 34 8b .L.......w.i./4. 02c0 - 55 f9 ed b1 41 21 67 9e-e1 91 dc a6 d8 36 f1 c0 U...A!g......6.. 02d0 - e1 92 3b 10 c3 14 50 3b-10 f5 81 67 f3 da 13 00 ..;...P;...g.... 02e0 - 18 8c 6f 10 40 1a e1 8c-df d4 d1 8c 68 6b bf 87 [email protected].. 02f0 - 02 03 01 00 01 a3 82 03-23 30 82 03 1f 30 0e 06 ........#0...0.. 0300 - 03 55 1d 0f 01 01 ff 04-04 03 02 05 a0 30 1d 06 .U...........0.. 0310 - 03 55 1d 25 04 16 30 14-06 08 2b 06 01 05 05 07 .U.%..0...+..... 0320 - 03 01 06 08 2b 06 01 05-05 07 03 02 30 0c 06 03 ....+.......0... 0330 - 55 1d 13 01 01 ff 04 02-30 00 30 1d 06 03 55 1d U.......0.0...U. 0340 - 0e 04 16 04 14 2e 1f 0d-06 2e 75 3a 2f b0 1b ab ..........u:/... 0350 - 36 61 52 31 96 a1 9c cf-51 30 1f 06 03 55 1d 23 6aR1....Q0...U.# 0360 - 04 18 30 16 80 14 a8 4a-6a 63 04 7d dd ba e6 d1 ..0....Jjc.}.... 0370 - 39 b7 a6 45 65 ef f3 a8-ec a1 30 6f 06 08 2b 06 9..Ee.....0o..+. 0380 - 01 05 05 07 01 01 04 63-30 61 30 2e 06 08 2b 06 .......c0a0...+. 0390 - 01 05 05 07 30 01 86 22-68 74 74 70 3a 2f 2f 6f ....0.."http://o 03a0 - 63 73 70 2e 69 6e 74 2d-78 33 2e 6c 65 74 73 65 csp.int-x3.letse 03b0 - 6e 63 72 79 70 74 2e 6f-72 67 30 2f 06 08 2b 06 ncrypt.org0/..+. 03c0 - 01 05 05 07 30 02 86 23-68 74 74 70 3a 2f 2f 63 ....0..#http://c 03d0 - 65 72 74 2e 69 6e 74 2d-78 33 2e 6c 65 74 73 65 ert.int-x3.letse 03e0 - 6e 63 72 79 70 74 2e 6f-72 67 2f 30 27 06 03 55 ncrypt.org/0'..U 03f0 - 1d 11 04 20 30 1e 82 1c-70 72 69 76 78 2d 69 64 ... 0........... 0400 - 61 61 73 2d 64 65 6d 6f-2e 65 64 65 6c 6b 65 79 ................ 0410 - 2e 6e 65 74 30 81 fe 06-03 55 1d 20 04 81 f6 30 ...t0....U. ...0 0420 - 81 f3 30 08 06 06 67 81-0c 01 02 01 30 81 e6 06 ..0...g.....0... 0430 - 0b 2b 06 01 04 01 82 df-13 01 01 01 .+.......... read from 0x2a24250 [0x2a2acd4] (1933 bytes => 1933 (0x78D)) 0000 - 30 81 d6 30 26 06 08 2b-06 01 05 05 07 02 01 16 0..0&..+........ 0010 - 1a 68 74 74 70 3a 2f 2f-63 70 73 2e 6c 65 74 73 .http://cps.lets 0020 - 65 6e 63 72 79 70 74 2e-6f 72 67 30 81 ab 06 08 encrypt.org0.... 0030 - 2b 06 01 05 05 07 02 02-30 81 9e 0c 81 9b 54 68 +.......0.....Th 0040 - 69 73 20 43 65 72 74 69-66 69 63 61 74 65 20 6d is Certificate m 0050 - 61 79 20 6f 6e 6c 79 20-62 65 20 72 65 6c 69 65 ay only be relie 0060 - 64 20 75 70 6f 6e 20 62-79 20 52 65 6c 79 69 6e d upon by Relyin 0070 - 67 20 50 61 72 74 69 65-73 20 61 6e 64 20 6f 6e g Parties and on 0080 - 6c 79 20 69 6e 20 61 63-63 6f 72 64 61 6e 63 65 ly in accordance 0090 - 20 77 69 74 68 20 74 68-65 20 43 65 72 74 69 66 with the Certif 00a0 - 69 63 61 74 65 20 50 6f-6c 69 63 79 20 66 6f 75 icate Policy fou 00b0 - 6e 64 20 61 74 20 68 74-74 70 73 3a 2f 2f 6c 65 nd at https://le 00c0 - 74 73 65 6e 63 72 79 70-74 2e 6f 72 67 2f 72 65 tsencrypt.org/re 00d0 - 70 6f 73 69 74 6f 72 79-2f 30 82 01 03 06 0a 2b pository/0.....+ 00e0 - 06 01 04 01 d6 79 02 04-02 04 81 f4 04 81 f1 00 .....y.......... 00f0 - ef 00 76 00 db 74 af ee-cb 29 ec b1 fe ca 3e 71 ..v..t...)....>q 0100 - 6d 2c e5 b9 aa bb 36 f7-84 71 83 c7 5d 9d 4f 37 m,....6..q..].O7 0110 - b6 1f bf 64 00 00 01 63-4c b7 a3 a1 00 00 04 03 ...d...cL....... 0120 - 00 47 30 45 02 20 0b 73-8e 1b 52 ab 04 49 1b 2f .G0E. .s..R..I./ 0130 - 43 80 60 49 6e 41 b3 8e-3a e9 69 f5 f4 c1 72 62 C.InA..:.i...rb
0140 - f5 8e be 96 b8 46 02 21-00 97 2b f9 3f 3a 47 cc .....F.!..+.?:G.
0150 - bf ad 02 ad 7c 27 77 68-22 e9 b6 e3 5e 3d f8 ca ....|'wh"...^=..
0160 - 66 65 aa 78 fc 5e ed 0e-aa 00 75 00 29 3c 51 96 fe.x.^....u.)<Q.
0170 - 54 c8 39 65 ba aa 50 fc-58 07 d4 b7 6f bf 58 7a T.9e..P.X...o.Xz
0180 - 29 72 dc a4 c3 0c f4 e5-45 47 f4 78 00 00 01 63 )r......EG.x...c
0190 - 4c b7 a3 84 00 00 04 03-00 46 30 44 02 20 64 83 L........F0D. d.
01a0 - 01 e7 bc be 11 e0 33 88-eb 16 5a 8a 01 64 20 92 ......3...Z..d .
01b0 - a2 8d ad 89 b2 05 1b cb-b3 a5 d3 f1 fd d1 02 20 ...............
01c0 - 13 b9 12 38 94 6a 8b cc-ce f3 74 d1 52 82 af d5 ...8.j....t.R...
01d0 - 88 fd fe df b9 96 a7 f7-44 89 d2 c7 dc 5e 9d d4 ........D....^..
01e0 - 30 0d 06 09 2a 86 48 86-f7 0d 01 01 0b 05 00 03 0....H.........
01f0 - 82 01 01 00 67 51 71 f2-cc 12 82 04 f6 72 24 7e ....gQq......r$~
0200 - 97 35 85 a3 60 ff 1c 95-1d ac d7 95 0f 23 7f 81 .5..........#.. 0210 - d9 9f bf 6c 14 02 9c 94-45 45 8a 19 54 91 c1 5e ...l....EE..T..^ 0220 - 1a 0e fa 97 e6 19 05 3b-be 3a 19 38 38 84 87 42 .......;.:.88..B 0230 - cc 75 ff fb 50 33 3a c9-ac fa 5e 7f 97 92 b5 4d .u..P3:...^....M 0240 - 39 31 04 69 dd 12 78 6e-66 74 16 88 c9 cc b9 5d 91.i..xnft.....] 0250 - f1 13 6f d5 bc dc 9e ad-17 b7 24 53 6b c2 b0 01 ..o.......$Sk... 0260 - e4 8d e9 a5 d0 15 a4 32-9d 13 dc e8 71 68 6c 72 .......2....qhlr 0270 - 5e a4 f5 85 73 e4 2d 27-7d f5 91 7d d5 1a 86 f0 ^...s.-'}..}.... 0280 - 24 bb 8e 4a ca 08 b8 f4-d9 33 ed 2a 8a 38 69 50 $..J.....3.*.8iP 0290 - 77 07 88 e4 07 37 1f 83-5a 05 a6 f6 14 66 5a 52 w....7..Z....fZR 02a0 - c1 7f 4f 3b a0 3b 85 22-b8 00 77 75 00 fb 3a f2 ..O;.;."..wu..:. 02b0 - f5 62 c6 54 d5 cc 09 5c-ca 09 25 ca 9f bc 31 d0 .b.T...\..%...1. 02c0 - e5 2d d7 a9 eb 96 4f 94-00 cf 29 fe 59 47 8a f3 .-....O...).YG.. 02d0 - af f7 ea 1c d7 58 fb 2e-e5 e7 e7 10 17 8c 0f 07 .....X.......... 02e0 - 4b b2 a1 9f 89 32 26 4b-0b 01 3c a2 ff 23 bd 0c K....2&K..<..#.. 02f0 - 69 5a b1 cf 00 04 96 30-82 04 92 30 82 03 7a a0 iZ.....0...0..z. 0300 - 03 02 01 02 02 10 0a 01-41 42 00 00 01 53 85 73 ........AB...S.s 0310 - 6a 0b 85 ec a7 08 30 0d-06 09 2a 86 48 86 f7 0d j.....0...*.H... 0320 - 01 01 0b 05 00 30 3f 31-24 30 22 06 03 55 04 0a .....0?1$0"..U.. 0330 - 13 1b 44 69 67 69 74 61-6c 20 53 69 67 6e 61 74 ..Digital Signat 0340 - 75 72 65 20 54 72 75 73-74 20 43 6f 2e 31 17 30 ure Trust Co.1.0 0350 - 15 06 03 55 04 03 13 0e-44 53 54 20 52 6f 6f 74 ...U....DST Root 0360 - 20 43 41 20 58 33 30 1e-17 0d 31 36 30 33 31 37 CA X30...160317 0370 - 31 36 34 30 34 36 5a 17-0d 32 31 30 33 31 37 31 164046Z..2103171 0380 - 36 34 30 34 36 5a 30 4a-31 0b 30 09 06 03 55 04 64046Z0J1.0...U. 0390 - 06 13 02 55 53 31 16 30-14 06 03 55 04 0a 13 0d ...US1.0...U.... 03a0 - 4c 65 74 27 73 20 45 6e-63 72 79 70 74 31 23 30 Let's Encrypt1#0 03b0 - 21 06 03 55 04 03 13 1a-4c 65 74 27 73 20 45 6e !..U....Let's En 03c0 - 63 72 79 70 74 20 41 75-74 68 6f 72 69 74 79 20 crypt Authority 03d0 - 58 33 30 82 01 22 30 0d-06 09 2a 86 48 86 f7 0d X30.."0...*.H... 03e0 - 01 01 01 05 00 03 82 01-0f 00 30 82 01 0a 02 82 ..........0..... 03f0 - 01 01 00 9c d3 0c f0 5a-e5 2e 47 b7 72 5d 37 83 .......Z..G.r]7. 0400 - b3 68 63 30 ea d7 35 26-19 25 e1 bd be 35 f1 70 .hc0..5&.%...5.p 0410 - 92 2f b7 b8 4b 41 05 ab-a9 9e 35 08 58 ec b1 2a ./..KA....5.X..* 0420 - c4 68 87 0b a3 e3 75 e4-e6 f3 a7 62 71 ba 79 81 .h....u....bq.y. 0430 - 60 1f d7 91 9a 9f f3 d0-78 67 71 c8 69 0e 95 91 .......xgq.i...
0440 - cf fe e6 99 e9 60 3c 48-cc 7e ca 4d 77 12 24 9d .....<H.~.Mw.$. 0450 - 47 1b 5a eb b9 ec 1e 37-00 1c 9c ac 7b a7 05 ea G.Z....7....{... 0460 - ce 4a eb bd 41 e5 36 98-b9 cb fd 6d 3c 96 68 df .J..A.6....m<.h. 0470 - 23 2a 42 90 0c 86 74 67-c8 7f a5 9a b8 52 61 14 #*B...tg.....Ra. 0480 - 13 3f 65 e9 82 87 cb db-fa 0e 56 f6 86 89 f3 85 .?e.......V..... 0490 - 3f 97 86 af b0 dc 1a ef-6b 0d 95 16 7d c4 2b a0 ?.......k...}.+. 04a0 - 65 b2 99 04 36 75 80 6b-ac 4a f3 1b 90 49 78 2f e...6u.k.J...Ix/ 04b0 - a2 96 4f 2a 20 25 29 04-c6 74 c0 d0 31 cd 8f 31 ..O* %)..t..1..1 04c0 - 38 95 16 ba a8 33 b8 43-f1 b1 1f c3 30 7f a2 79 8....3.C....0..y 04d0 - 31 13 3d 2d 36 f8 e3 fc-f2 33 6a b9 39 31 c5 af 1.=-6....3j.91.. 04e0 - c4 8d 0d 1d 64 16 33 aa-fa 84 29 b6 d4 0b c0 d8 ....d.3...)..... 04f0 - 7d c3 93 02 03 01 00 01-a3 82 01 7d 30 82 01 79 }..........}0..y 0500 - 30 12 06 03 55 1d 13 01-01 ff 04 08 30 06 01 01 0...U.......0... 0510 - ff 02 01 00 30 0e 06 03-55 1d 0f 01 01 ff 04 04 ....0...U....... 0520 - 03 02 01 86 30 7f 06 08-2b 06 01 05 05 07 01 01 ....0...+....... 0530 - 04 73 30 71 30 32 06 08-2b 06 01 05 05 07 30 01 .s0q02..+.....0. 0540 - 86 26 68 74 74 70 3a 2f-2f 69 73 72 67 2e 74 72 .&http://isrg.tr 0550 - 75 73 74 69 64 2e 6f 63-73 70 2e 69 64 65 6e 74 ustid.ocsp.ident 0560 - 72 75 73 74 2e 63 6f 6d-30 3b 06 08 2b 06 01 05 rust.com0;..+... 0570 - 05 07 30 02 86 2f 68 74-74 70 3a 2f 2f 61 70 70 ..0../http://app 0580 - 73 2e 69 64 65 6e 74 72-75 73 74 2e 63 6f 6d 2f s.identrust.com/ 0590 - 72 6f 6f 74 73 2f 64 73-74 72 6f 6f 74 63 61 78 roots/dstrootcax 05a0 - 33 2e 70 37 63 30 1f 06-03 55 1d 23 04 18 30 16 3.p7c0...U.#..0. 05b0 - 80 14 c4 a7 b1 a4 7b 2c-71 fa db e1 4b 90 75 ff ......{,q...K.u. 05c0 - c4 15 60 85 89 10 30 54-06 03 55 1d 20 04 4d 30 .....0T..U. .M0
05d0 - 4b 30 08 06 06 67 81 0c-01 02 01 30 3f 06 0b 2b K0...g.....0?..+
05e0 - 06 01 04 01 82 df 13 01-01 01 30 30 30 2e 06 08 ..........000...
05f0 - 2b 06 01 05 05 07 02 01-16 22 68 74 74 70 3a 2f +........"http:/
0600 - 2f 63 70 73 2e 72 6f 6f-74 2d 78 31 2e 6c 65 74 /cps.root-x1.let
0610 - 73 65 6e 63 72 79 70 74-2e 6f 72 67 30 3c 06 03 sencrypt.org0<..
0620 - 55 1d 1f 04 35 30 33 30-31 a0 2f a0 2d 86 2b 68 U...50301./.-.+h
0630 - 74 74 70 3a 2f 2f 63 72-6c 2e 69 64 65 6e 74 72 ttp://crl.identr
0640 - 75 73 74 2e 63 6f 6d 2f-44 53 54 52 4f 4f 54 43 ust.com/DSTROOTC
0650 - 41 58 33 43 52 4c 2e 63-72 6c 30 1d 06 03 55 1d AX3CRL.crl0...U.
0660 - 0e 04 16 04 14 a8 4a 6a-63 04 7d dd ba e6 d1 39 ......Jjc.}....9
0670 - b7 a6 45 65 ef f3 a8 ec-a1 30 0d 06 09 2a 86 48 ..Ee.....0....H
0680 - 86 f7 0d 01 01 0b 05 00-03 82 01 01 00 dd 33 d7 ..............3.
0690 - 11 f3 63 58 38 dd 18 15-fb 09 55 be 76 56 b9 70 ..cX8.....U.vV.p
06a0 - 48 a5 69 47 27 7b c2 24-08 92 f1 5a 1f 4a 12 29 H.iG'{.$...Z.J.)
06b0 - 37 24 74 51 1c 62 68 b8-cd 95 70 67 e5 f7 a4 bc 7$tQ.bh...pg....
06c0 - 4e 28 51 cd 9b e8 ae 87-9d ea d8 ba 5a a1 01 9a N(Q.........Z...
06d0 - dc f0 dd 6a 1d 6a d8 3e-57 23 9e a6 1e 04 62 9a ...j.j.>W#....b.
06e0 - ff d7 05 ca b7 1f 3f c0-0a 48 bc 94 b0 b6 65 62 ......?..H....eb
06f0 - e0 c1 54 e5 a3 2a ad 20-c4 e9 e6 bb dc c8 f6 b5 ..T..*. ........
0700 - c3 32 a3 98 cc 77 a8 e6-79 65 07 2b cb 28 fe 3a .2...w..ye.+.(.:
0710 - 16 52 81 ce 52 0c 2e 5f-83 e8 d5 06 33 fb 77 6c .R..R......3.wl
0720 - ce 40 ea 32 9e 1f 92 5c-41 c1 74 6c 5b 5d 0a 5f [email protected]...\A.tl[].
0730 - 33 cc 4d 9f ac 38 f0 2f-7b 2c 62 9d d9 a3 91 6f 3.M..8./{,b....o
0740 - 25 1b 2f 90 b1 19 46 3d-f6 7e 1b a6 7a 87 b9 a3 %./...F=.~..z...
0750 - 7a 6d 18 fa 25 a5 91 87-15 e0 f2 16 2f 58 b0 06 zm..%......./X..
0760 - 2f 2c 68 26 c6 4b 98 cd-da 9f 0c f9 7f 90 ed 43 /,h&.K.........C
0770 - 4a 12 44 4e 6f 73 7a 28-ea a4 aa 6e 7b 4c 7d 87 J.DNosz(...n{L}.
0780 - dd e0 c9 02 44 a7 87 af-c3 34 5b b4 42 ....D....4[.B
read from 0x2a24250 [0x2a2a893] (5 bytes => 5 (0x5))
0000 - 16 03 03 02 4d ....M
read from 0x2a24250 [0x2a2a898] (589 bytes => 589 (0x24D))
0000 - 0c 00 02 49 03 00 17 41-04 3b 87 a9 91 e3 2f 06 ...I...A.;..../.
0010 - 1a 2e 35 28 b4 53 2f 7a-37 a1 5a f1 2c 8c 50 85 ..5(.S/z7.Z.,.P.
0020 - c5 c5 a2 9a 44 de 68 53-63 98 cd 80 a5 ee 4e 97 ....D.hSc.....N.
0030 - 42 23 1b 6d e8 5c 17 5e-04 87 8a f2 24 59 b3 66 B#.m..^....$Y.f
0040 - 8c 0b 29 22 aa 48 eb 3e-af 06 01 02 00 26 fa 92 ..)".H.>.....&..
0050 - 5e 8b 2f 05 cb 32 48 14-fd 39 39 60 3b 32 55 8d ^./..2H..99;2U. 0060 - b8 ba c5 6b 90 aa 56 58-56 f7 fa fb 71 9d 77 03 ...k..VXV...q.w. 0070 - a7 91 fc ef d1 c6 2b 7d-b0 82 be 63 f2 93 bb c9 ......+}...c.... 0080 - 34 c3 4b 9f 4c 83 71 92-d2 f2 55 af 42 a5 b7 30 4.K.L.q...U.B..0 0090 - 77 bf 61 1b d4 d1 bf 6a-55 64 30 4b 79 55 39 47 w.a....jUd0KyU9G 00a0 - bf f6 b2 b2 8d 88 1f d0-3c 9b 02 6c 70 f2 30 0a ........<..lp.0. 00b0 - 68 ed 6a 89 b8 9e 40 a7-6a 72 5e f8 f1 85 6d e6 [email protected]^...m. 00c0 - 6b 63 4f a0 cf 31 d3 1c-f7 42 7c 44 59 07 fc 6e kcO..1...B|DY..n 00d0 - 9a e2 f1 e0 4f 01 97 ff-36 ac bf 11 01 93 6d a2 ....O...6.....m. 00e0 - 5c 5a d8 ca a6 38 19 da-35 5d 1e ad ab ea 92 9c \Z...8..5]...... 00f0 - 2f ea 06 4b 3f ee 17 08-69 40 eb 9e 17 6b 2e c4 /[email protected].. 0100 - 71 ee 15 0b a4 25 5e 78-de 4c aa 46 b1 a3 7c 3e q....%^x.L.F..|> 0110 - 81 98 95 e1 15 cb 0f b6-5c af 8f 7c 46 94 a7 55 ........\..|F..U 0120 - f5 31 97 28 ba 1d e2 e0-ed af 08 26 fd 8e bb 4a .1.(.......&...J 0130 - 89 eb 66 bb fc 88 50 26-a3 96 fb 74 ea 2c e0 62 ..f...P&...t.,.b 0140 - f5 e6 b0 09 94 49 5c e1-35 df db e5 b8 0e c4 3a .....I\.5......: 0150 - 3d dc 2c 51 a8 87 89 b8-a8 80 0c 4d e9 ad 9c 97 =.,Q.......M.... 0160 - 88 49 cd 8c 8d 34 85 e1-28 ad 53 f8 26 7f 1f 0b .I...4..(.S.&... 0170 - 04 14 7d 23 b3 12 78 08-4f 19 ad 9d fc 5d 0e c3 ..}#..x.O....].. 0180 - da 78 70 6c ae 3a 8c 59-e5 bf 3b d6 d6 c6 6c b4 .xpl.:.Y..;...l. 0190 - 5a 04 e0 24 b0 72 00 08-23 80 7d 5d 15 ea 74 6f Z..$.r..#.}]..to 01a0 - 7a 75 87 fa c5 05 3d 5d-48 5e f2 db ad 82 de 75 zu....=]H^.....u 01b0 - de 3d 20 f2 7e 64 b0 84-03 a6 98 f8 b1 b1 53 34 .= .~d........S4 01c0 - 68 1a 66 eb cd 87 ca 5f-ab e7 a3 17 75 6a 1c ba h.f...._....uj.. 01d0 - b6 94 78 c4 87 eb 19 76-25 60 ad 61 e3 e3 cd 54 ..x....v%.a...T
01e0 - d7 0d 6d f6 78 41 04 0d-87 3e e6 5e 4d 19 75 bf ..m.xA...>.^M.u.
01f0 - 04 7d fe 07 b8 a2 4f 02-4a c7 7c bf c4 d7 81 0d .}....O.J.|.....
0200 - cd be 46 27 8d 15 98 3a-01 4f ac ba 92 48 90 45 ..F'...:.O...H.E
0210 - b4 fd 17 1e 42 f3 82 7a-0c 4b f6 6c e0 99 79 e1 ....B..z.K.l..y.
0220 - 6b 52 5b 88 58 5b 64 3e-19 43 ce ae 05 fc be 4a kR[.X[d>.C.....J
0230 - 48 66 d2 c4 f6 ca 4e 38-1b 60 9d d9 b0 c4 f2 7e Hf....N8......~ 0240 - 1f 29 1c 9e 82 74 bf 2b-6d 15 fb 38 62 .)...t.+m..8b read from 0x2a24250 [0x2a2a893] (5 bytes => 5 (0x5)) 0000 - 16 03 03 00 04 ..... read from 0x2a24250 [0x2a2a898] (4 bytes => 4 (0x4)) 0000 - 0e . 0004 - <SPACES/NULS> write to 0x2a24250 [0x2a342b0] (75 bytes => 75 (0x4B)) 0000 - 16 03 03 00 46 10 00 00-42 41 04 a9 96 c7 e8 95 ....F...BA...... 0010 - 95 6e c9 41 90 ad ab ae-eb be 92 66 78 83 c7 40 .n.A.......fx..@ 0020 - fc 84 18 01 08 d2 07 08-f0 7b 33 35 56 30 bb ab .........{35V0.. 0030 - 2f e0 af 4c d0 51 25 44-73 52 c4 bb e8 c8 82 8b /..L.Q%DsR...... 0040 - 8d 9e f7 2d cc 31 57 70-ef 3b 7c ...-.1Wp.;| write to 0x2a24250 [0x2a342b0] (6 bytes => 6 (0x6)) 0000 - 14 03 03 00 01 01 ...... write to 0x2a24250 [0x2a342b0] (45 bytes => 45 (0x2D)) 0000 - 16 03 03 00 28 fe 5b 72-ce c3 69 2d ee 3e 35 1d ....(.[r..i-.>5. 0010 - 42 66 97 5d d9 91 e8 12-86 53 d9 a6 70 09 37 6c Bf.].....S..p.7l 0020 - 4a 13 3f 79 57 cc 6e 9a-b3 e0 c5 98 da J.?yW.n...... read from 0x2a24250 [0x2a2a893] (5 bytes => 5 (0x5)) 0000 - 16 03 03 00 aa ..... read from 0x2a24250 [0x2a2a898] (170 bytes => 170 (0xAA)) 0000 - 04 00 00 a6 00 00 01 2c-00 a0 ad 53 29 ac de 7f .......,...S)... 0010 - 7d 32 82 19 4f 76 46 7c-08 77 f5 e6 db 88 66 70 }2..OvF|.w....fp 0020 - 68 d0 ac 12 75 1e 63 24-3e a6 67 4f e3 d5 51 8b h...u.c$>.gO..Q. 0030 - 29 c8 48 15 d2 30 d9 8d-98 8f 42 b8 42 53 b5 41 ).H..0....B.BS.A 0040 - 28 ae 41 66 84 b7 d8 fb-57 59 1d 78 c4 b9 7f 95 (.Af....WY.x.... 0050 - 37 31 7c d1 9a 23 6c fd-52 d2 52 ad e3 86 18 4a 71|..#l.R.R....J 0060 - 66 ed 3b 7e 7a 3d dc ab-91 00 f5 cb 9e c8 15 37 f.;~z=.........7 0070 - 24 c3 b5 9c c0 32 8f 67-63 29 1b 43 1d a0 99 6c $....2.gc).C...l 0080 - 3b a5 f5 7f 92 9b 03 71-dd 62 e3 ed fe 9f 0a b5 ;......q.b...... 0090 - b0 c0 6c 7f fc 6f 7f 70-50 3a 3e 8e 40 60 59 27 ..l..o.pP:>.@Y'
00a0 - 88 a3 71 3f 37 ca c7 7d-a8 7c ..q?7..}.|
read from 0x2a24250 [0x2a2a893] (5 bytes => 5 (0x5))
0000 - 14 03 03 00 01 .....
read from 0x2a24250 [0x2a2a898] (1 bytes => 1 (0x1))
0000 - 01 .
read from 0x2a24250 [0x2a2a893] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 28 ....(
read from 0x2a24250 [0x2a2a898] (40 bytes => 40 (0x28))
0000 - 9c 69 e5 e3 9f 89 0c b7-b7 17 67 0c e0 33 62 8d .i........g..3b.
0010 - 3e 5d 86 cd 25 69 5c 06-ba 67 22 c0 65 72 09 39 >]..%i..g".er.9
0020 - 18 ea fc c6 1d e5 e2 f0- ........

Certificate chain
0 s:/CN=mysite.com
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Server certificate
-----BEGIN CERTIFICATE-----
MIIHIjCCBgqgAwIBAgISBE91gayhPhvsROLoc6HDAkE6MA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODA1MTEwMDAxMzNaFw0x
ODA4MDkwMDAxMzNaMCcxJTAjBgNVBAMTHHByaXZ4LWlkYWFzLWRlbW8uZWRlbGtl
eS5uZXQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCZoHk0ibG3K+Gv
pVH1sMd5pxUQe+hqez6FN0ipsKXof0dGBR1NrFmW4briaAOKgyEqjaMlv8RUyL+P
OyP/lSDmGxa5QCWpZyGIW86AeWs3IiILUM6NfBjgp4oa3C+gbwhHSStYL8I4YMFZ
aZdMh+uZUfq1qlCP1Qy8dHiKwb6hOPsjm4hy7Isy/job4TkcBqKc6kdqo22dLDwR
huRqXpR7v40Ti3ETAieM+6b2eeUmH2VXn1+fELBPZdxcBV64mTWMnQYJmeurByjq
KLTXimdTtWLOFh35EfA8SMPoaTLUsLGvqp8etmEKmjA/+bo0ZDfvXXS7AWeGpyPb
aPElCD9X9+D43odTqWQKRRTV2oWKRXx7bI0VPYu0DrSdrZxD8eDJuurVC6TGAipw
wVeKWboUFBYx2SPYOfADNENp1BBHZEImI97OdD9XKtM2p2+Y2FUrZfWaJhhtVJaV
xitdWOJHXqP4/xul1eQkv1UFXTWkT/2N9b46dYuUMcJBhBz1uTJCbgnmccN2i9w0
kQg3lSgi/BIfyTONTtpM2BPqeHmdrwkLBvB2jLH5sq3Z59y17zNTOmcjjgo+t8+1
4TZ3OO0L6kwX2wL4lA5/d41p6i80i1X57bFBIWee4ZHcptg28cDhkjsQwxRQOxD1
gWfz2hMAGIxvEEAa4Yzf1NGMaGu/hwIDAQABo4IDIzCCAx8wDgYDVR0PAQH/BAQD
AgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAA
MB0GA1UdDgQWBBQuHw0GLnU6L7AbqzZhUjGWoZzPUTAfBgNVHSMEGDAWgBSoSmpj
BH3duubRObemRWXv86jsoTBvBggrBgEFBQcBAQRjMGEwLgYIKwYBBQUHMAGGImh0
dHA6Ly9vY3NwLmludC14My5sZXRzZW5jcnlwdC5vcmcwLwYIKwYBBQUHMAKGI2h0
dHA6Ly9jZXJ0LmludC14My5sZXRzZW5jcnlwdC5vcmcvMCcGA1UdEQQgMB6CHHBy
aXZ4LWlkYWFzLWRlbW8uZWRlbGtleS5uZXQwgf4GA1UdIASB9jCB8zAIBgZngQwB
AgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxl
dHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIwgZ4MgZtUaGlzIENlcnRpZmljYXRl
IG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5IFJlbHlpbmcgUGFydGllcyBhbmQg
b25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhlIENlcnRpZmljYXRlIFBvbGljeSBm
b3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0Lm9yZy9yZXBvc2l0b3J5LzCCAQMG
CisGAQQB1nkCBAIEgfQEgfEA7wB2ANt0r+7LKeyx/so+cW0s5bmquzb3hHGDx12d
Tze2H79kAAABY0y3o6EAAAQDAEcwRQIgC3OOG1KrBEkbL0OAYEluQbOOOulp9fTB
cmL1jr6WuEYCIQCXK/k/OkfMv60CrXwnd2gi6bbjXj34ymZlqnj8Xu0OqgB1ACk8
UZZUyDlluqpQ/FgH1Ldvv1h6KXLcpMMM9OVFR/R4AAABY0y3o4QAAAQDAEYwRAIg
ZIMB57y+EeAziOsWWooBZCCSoo2tibIFG8uzpdPx/dECIBO5EjiUaovMzvN00VKC
r9WI/f7fuZan90SJ0sfcXp3UMA0GCSqGSIb3DQEBCwUAA4IBAQBnUXHyzBKCBPZy
JH6XNYWjYP8clR2s15UPI3+B2Z+/bBQCnJRFRYoZVJHBXhoO+pfmGQU7vjoZODiE
h0LMdf/7UDM6yaz6Xn+XkrVNOTEEad0SeG5mdBaIycy5XfETb9W83J6tF7ckU2vC
sAHkjeml0BWkMp0T3OhxaGxyXqT1hXPkLSd99ZF91RqG8CS7jkrKCLj02TPtKoo4
aVB3B4jkBzcfg1oFpvYUZlpSwX9PO6A7hSK4AHd1APs68vVixlTVzAlcygklyp+8
MdDlLdep65ZPlADPKf5ZR4rzr/fqHNdY+y7l5+cQF4wPB0uyoZ+JMiZLCwE8ov8j
vQxpWrHP
-----END CERTIFICATE-----
subject=/CN=mysite.com
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3

No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 3922 bytes and written 434 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 1D90A5743671612511F38A56A01B668C1667BF4E54CEE063A8ADE37C2C7762C2
Session-ID-ctx:
Master-Key: B13CED937290AD677DCF03CD9F20C5BDDC28A7727084C1378C04DBC6D24FE21A721161114D158CD3BFA61209D8AF0D79
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - ad 53 29 ac de 7f 7d 32-82 19 4f 76 46 7c 08 77 .S)...}2..OvF|.w
0010 - f5 e6 db 88 66 70 68 d0-ac 12 75 1e 63 24 3e a6 ....fph...u.c$>.
0020 - 67 4f e3 d5 51 8b 29 c8-48 15 d2 30 d9 8d 98 8f gO..Q.).H..0....
0030 - 42 b8 42 53 b5 41 28 ae-41 66 84 b7 d8 fb 57 59 B.BS.A(.Af....WY
0040 - 1d 78 c4 b9 7f 95 37 31-7c d1 9a 23 6c fd 52 d2 .x....71|..#l.R.
0050 - 52 ad e3 86 18 4a 66 ed-3b 7e 7a 3d dc ab 91 00 R....Jf.;~z=....
0060 - f5 cb 9e c8 15 37 24 c3-b5 9c c0 32 8f 67 63 29 .....7$....2.gc)
0070 - 1b 43 1d a0 99 6c 3b a5-f5 7f 92 9b 03 71 dd 62 .C...l;......q.b
0080 - e3 ed fe 9f 0a b5 b0 c0-6c 7f fc 6f 7f 70 50 3a ........l..o.pP:
0090 - 3e 8e 40 60 59 27 88 a3-71 3f 37 ca c7 7d a8 7c >.@`Y'..q?7..}.|

Start Time: 1526296687
Timeout   : 300 (sec)
Verify return code: 0 (ok)

HTTP/1.0 408 Request Timeout
Cache-Control: no-cache
Connection: close
Content-Type: text/html

I'm attempting to start a dockerflow/docker-flow-proxy service on a swarm with one manager and two workers, but the command never completes and never reports any errors. So, it never returns to the command prompt. Instead, it repeatedly switches between, "new", "assigned", "ready", and "starting".

Note that although the screenshot shows the image vfarcic/docker-flow-proxy I'm having exactly the same problem with dockerflow/docker-flow-proxy.

This is from the book "The DevOps 2.1 Toolkit - Docker Swarm" by @vfarcic
Here's the command I used to start the service

docker service create --name proxy -p 80:80 -p 443:443 -p 8080:8080 --network proxy -e MODE=swarm dockerflow/docker-flow-proxy

If I Ctrl-C it shows

Operation continuing in background.
Use docker service ps p473obsbo9tgjom1fd61or5ap to check progress.

... and returns to the command prompt

If I use docker service ps --no-trunc p473obsbo9tgjom1fd61or5ap I get:

ID                          NAME                IMAGE                                                                                                         NODE                DESIRED STATE       CURRENT STATE               ERROR                                                          PORTS
dbzvr67uzg5fq9w64uh324dtn   proxy.1             dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Running             Starting 11 seconds ago
vzlttk2xdd4ynmr4lir4t38d5    \_ proxy.1         dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Shutdown            Failed 16 seconds ago       "task: non-zero exit (137): dockerexec: unhealthy container"
q5dlvb7xz04cwxv9hpr9w5f7l    \_ proxy.1         dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Shutdown            Failed 49 seconds ago       "task: non-zero exit (137): dockerexec: unhealthy container"
z86h0pcj21joryb3sj59mx2pq    \_ proxy.1         dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Shutdown            Failed about a minute ago   "task: non-zero exit (137): dockerexec: unhealthy container"
uj2axgfm1xxpdxrkp308m28ys    \_ proxy.1         dockerflow/docker-flow-proxy:latest@sha256:e4e4a684d703bec18385caed3b3bd0482cefdff73739ae21bd324f5422e17d93   node-1              Shutdown            Failed about a minute ago   "task: non-zero exit (137): dockerexec: unhealthy container"

So, no mention of node-2 or node-3 and I don't know what the error message means.

If I run docker service logs -f proxy the following repeats every minute or so:

proxy.1.s585dio0clan@node-1    | 2018/09/12 03:13:02 Starting HAProxy
proxy.1.s585dio0clan@node-1    | 2018/09/12 03:13:02 Getting certs from http://202.71.99.195:8080/v1/docker-flow-proxy/certs

I can only guess that it fails to get whatever certs it's looking for. There doesn't seem to be anything at that address, but an IP address lookup shows that it leads to my ISP. Where's that coming from?

Steps to reproduce the issue:

1. Create three nodes with docker-machine
2. Initialize the swarm with the first node as the manager
3. Join the swarm with the other two nodes.
4. Create a network using the overlay driver.
5. Create a service, using dockerflow/docker-flow-proxy; publishing mirrored ports 80, 443, and 8080; and setting the MODE=swarm environment variable.

Describe the results you received:

overall progress: 0 out of 1 tasks
1/1 starting

(never returns to command prompt)

Describe the results you expected:
Something like:

overall progress: 1 out of 1 tasks
verify: Service converged

Additional information you deem important (e.g. issue happens only occasionally):

Additional environment details (AWS, VirtualBox, physical, etc.):
I'm running this on a physical computer running Linux Mint 19 (based on Ubuntu 18.04). The driver / provider for docker-machine is virtualbox.

Steps to reproduce the issue:

Following the instructions found here.

I used certbot to generate a certificate for my subdomain using docker on the masternode:

sudo docker run --rm \
-p 81:81 -p 443:443 -p 80:80 \
-v /etc/letsencrypt:/etc/letsencrypt \
certbot/certbot certonly \
-d my-subdomain.mydomain.com \
--standalone \
-m [email protected] --agree-tos

Which resulted in:

./cert
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for my-subdomain.my-domain.com
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/my-subdomain.my-domain.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/my-subdomain.my-domain.com/privkey.pem
   Your cert will expire on 2018-10-09. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Then i added the pem to the df-proxy service:

docker secret create cert-my-subdomain.mydomain.com /etc/letsencrypt/live/my-subdomain.mydomain.com/fullchain.pem
docker service update --secret-add cert-my-subdomain.mydomain.com proxy_proxy

Describe the results you received:

https://my-subdomain.my-domain/demo/hello results in ->

Secure Connection Failed

An error occurred during a connection to menards-test.wts-ops.com. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG 

http://my-subdomain.my-domain/demo/hello results in ->

503 Service Unavailable
No server is available to handle this request. 

Describe the results you expected:

http redirects to https, and https resolves with "hello world" in browser.

Additional information you deem important (e.g. issue happens only occasionally):

Before adding the secret to the proxy service, hello world was resolving on http. It no longer does after adding secret. Odd.

Additional environment details (AWS, VirtualBox, physical, etc.):

Docker Swarm. Azure. Deployed proxy and hello demo using "swarm mode with docker stack tutorial

Hi @vfarcic & (@thomasjpfan)

In response of our Skype session, I am submitting this issue, possibly related to #6

Please consider the following compose file, which creates two services mqtt_emqtt-master and mqtt_emqtt-worker. Each service deploys 1 container.

################################################################################
# MQTT Stack
################################################################################
#$ docker stack deploy mqtt --compose-file docker-compose-mqtt.yml
################################################################################
version: "3.6"

services:
  emqtt-master:
    image: raymondmm/emqtt
    hostname: emqtt-master
    environment:
      - "EMQ_NAME=emq"
      - "EMQ_HOST=master.mq.tt"
      - "EMQ_NODE__COOKIE=ef16498f66804df1cc6172f6996d5492"
      - "EMQ_WAIT_TIME=60"
    networks:
      indonesia-net:
        aliases:
          - master.mq.tt
      proxy_indonesia-net:
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/TZ:ro
    deploy:
      placement:
        constraints: [node.role == manager]
      labels:
        - com.df.notify=true

        # EMQTT dashboard
        - com.df.serviceDomain.1=emqtt.indonesia
        - com.df.port.1=18083
        - com.df.reqMode.1=http

        # EMQTT tcp connection
        - com.df.port.2=1883
        - com.df.srcPort.2=1883
        - com.df.reqMode.2=tcp

  emqtt-worker:
    image: raymondmm/emqtt
    hostname: emqtt-worker
    environment:
      - "[email protected]"
      - "EMQ_NODE__COOKIE=ef16498f66804df1cc6172f6996d5492"
      - "EMQ_WAIT_TIME=60"
    depends_on:
     - emqtt-master
    networks:
      - indonesia-net
      - proxy_indonesia-net
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/TZ:ro
    deploy:
      placement:
        constraints: [node.role == worker]
      labels:
        - com.df.notify=true
        
        # EMQTT tcp connection
        - com.df.port.1=1883
        - com.df.srcPort.1=1883
        - com.df.reqMode.1=tcp

networks:
  indonesia-net:
    external: false
  proxy_indonesia-net:
    external: true

The haproxy.cfg looks like this:

global
    pidfile /var/run/haproxy.pid
    tune.ssl.default-dh-param 2048

    # disable sslv3, prefer modern ciphers
    ssl-default-bind-options no-sslv3
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

    ssl-default-server-options no-sslv3
    ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

resolvers docker
    nameserver dns 127.0.0.11:53

defaults
    mode    http
    balance roundrobin

    option  dontlognull
    option  dontlog-normal
    option  http-keep-alive
    option  forwardfor
    option  redispatch

    errorfile 400 /errorfiles/400.http
    errorfile 403 /errorfiles/403.http
    errorfile 405 /errorfiles/405.http
    errorfile 408 /errorfiles/408.http
    errorfile 429 /errorfiles/429.http
    errorfile 500 /errorfiles/500.http
    errorfile 502 /errorfiles/502.http
    errorfile 503 /errorfiles/503.http
    errorfile 504 /errorfiles/504.http

    maxconn 5000
    timeout connect 5s
    timeout client  20s
    timeout server  20s
    timeout queue   30s
    timeout tunnel  3600s
    timeout http-request 5s
    timeout http-keep-alive 15s

    stats enable
    stats refresh 30s
    stats realm Strictly\ Private
    stats uri /admin?stats
    stats auth my-user:my-pass
frontend services
    bind *:80
    bind *:443
    mode http

    acl url_mqtt_emqtt-master18083_1 path_beg /
    acl domain_mqtt_emqtt-master18083_1 hdr_beg(host) -i emqtt.indonesia
    use_backend mqtt_emqtt-master-be18083_1 if url_mqtt_emqtt-master18083_1 domain_mqtt_emqtt-master18083_1

frontend tcpFE_1883
    bind *:1883
    mode tcp
    default_backend mqtt_emqtt-master-be1883_2
    default_backend mqtt_emqtt-worker-be1883_1

backend mqtt_emqtt-master-be18083_1
    mode http
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    server mqtt_emqtt-master mqtt_emqtt-master:18083

backend mqtt_emqtt-master-be1883_2
    mode tcp
    server mqtt_emqtt-master mqtt_emqtt-master:1883

backend mqtt_emqtt-worker-be1883_1
    mode tcp
    server mqtt_emqtt-worker mqtt_emqtt-worker:1883

Unfortunately as I understood dfp currently doesn't support load balancing between two (or more) services.

In this case all incoming mqtt tcp connections are getting connection with the worker and none with the master. When the worker is shutdown (replica set to zero), all the incoming mqtt tcp connections fail and not redirected to the master.

What I would like is that dfp load balances the incoming mqtt tcp connection over the two (or more) containers.

Find below an haproxy.cfg example which worked for me before i used dfp:

global
  ulimit-n 99999
  maxconn 99999
  maxpipes 99999
  tune.maxaccept 500
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  chroot /var/lib/haproxy
  user haproxy
  group haproxy

defaults
  log global
  mode http
  option dontlognull
  timeout connect 5000ms
  timeout client 50000ms
  timeout server 50000ms
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

listen stats :80
  stats enable
  stats uri / # must be present to see the logs
  stats auth admin:admin

listen mqtt
  bind *:1883
  bind *:8883 ssl crt /certs/lelylan-mqtt.pem
  mode tcp
  #Use this to avoid the connection loss when client subscribed for a topic and its idle for sometime
  option clitcpka # For TCP keep-alive
  timeout client 3h #By default TCP keep-alive interval is 2hours in OS kernal, 'cat /proc/sys/net/ipv4/tcp_keepalive_time'
  timeout server 3h #By default TCP keep-alive interval is 2hours in OS kernal
  option tcplog
  balance leastconn
  server mosca_1 178.62.122.204:1883 check
  server mosca_2 178.62.104.172:1883 check

Hope you guys have time to look into this :-)

Hi, is it possible to specify multiple service domains for different ports ?

E.g. traefik allow me to do something like this:

labels:
- "traefik.gitlab.frontend.rule=Host:gitlab.example.com"
- traefik.gitlab.port=80
- "traefik.registry.frontend.rule=Host:registry.example.com"
- traefik.registry.port=5001

Hi When there are multiple proxy in the cluster
The session settings is different on each proxy, can do a IP sort first when templating the config?

PROXY NODE 1

    server service_golang_0 10.129.0.130:8000 check cookie service_golang_0
    server service_golang_1 10.129.0.138:8000 check cookie service_golang_1
    server service_golang_2 10.129.0.140:8000 check cookie service_golang_2
    server service_golang_3 10.129.0.116:8000 check cookie service_golang_3
    server service_golang_4 10.129.0.100:8000 check cookie service_golang_4
    server service_golang_5 10.129.0.253:8000 check cookie service_golang_5

PROXY NODE 2

    server service_golang_0 10.129.0.100:8000 check cookie service_golang_0
    server service_golang_1 10.129.0.253:8000 check cookie service_golang_1
    server service_golang_2 10.129.0.130:8000 check cookie service_golang_2
    server service_golang_3 10.129.0.138:8000 check cookie service_golang_3
    server service_golang_4 10.129.0.140:8000 check cookie service_golang_4
    server service_golang_5 10.129.0.116:8000 check cookie service_golang_5

Can you please add more details on how to use custom error pages. Preferably the solution is not to create a custom image. I would like the error pages to be stored on the file system outside of docker for easy maintenance (actually a mounted share to be used across the swarm). Can I use a bind mount to /errorpages? That did not work for me, I guess because image does not expose it.

It would be great if you could implement it in the project :)

I checked some solution and dnsrr mode is working with haproxy "server-template" future very well.
Here is my custom backend template:

{{range $sd := .ServiceDest}}
    backend {{$.AclName}}-be{{.Port}}_{{.Index}}
    mode http
    balance roundrobin
    server-template {{$.ServiceName}} 1-10 {{$.ServiceName}}:{{.Port}}  check resolvers docker
{{end}}

I hardcoded servers(1-10) because I didn't find any template var with number of replicas.

I am trying to launch a MSSQL database in a container using DFP as proxy. MSSQL database uses port 1433 for connectivity. If I create a service on my local machine like this:

docker service create --name mssql -p 1433:1433 --env="SA_PASSWORD=Qwerty!234" --env="ACCEPT_EULA=true" microsoft/mssql-server-linux:latest

I can now connect to the MSSQL database:

sqlcmd -S localhost,1433 -U sa

with password Qwerty!234

When I try to use DFP to achieve the same thing I run into problems.
I create the DFP setup:

docker network create --driver overlay proxy
docker service create --name swarm-listener --network proxy --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" -e DF_NOTIFY_CREATE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/reconfigure -e DF_NOTIFY_REMOVE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/remove --constraint "node.role==manager" vfarcic/docker-flow-swarm-listener
docker service create --name proxy -p 80:80 -p 443:443 --network proxy -e LISTENER_ADDRESS=swarm-listener -e DEBUG=true vfarcic/docker-flow-proxy

I then add the 1433 port to the proxy configuration:

docker service update --publish-add 1433:1433 proxy

And finally I create the MSSQL database:

docker service create --name mssql --network proxy --label com.df.notify=true --label com.df.distribute=true --label com.df.serviceDomain=mssql.domain.local --label com.df.port=1433 --label com.df.srcPort=1433 --label com.df.reqMode=tcp --env="SA_PASSWORD=Qwerty!234" --env="ACCEPT_EULA=true" microsoft/mssql-server-linux

Now I cannot connect to the database. I get logon timeouts:

sqlcmd -S mssql.domain.local,1433 -U sa

mssql.domain.local I have setup in my local Windows hosts file to point to 127.0.0.1

First looking at the haproxy configuration ( /cfg/haproxy.conf ) by sh'ing into the proxy container I see:

frontend tcpFE_1433
    bind *:1433
    mode tcp
    option tcplog
    log global
    acl domain_mssql1433_0 hdr_beg(host) -i mssql.domain.local
    use_backend mssql-be1433_0 if domain_mssql1433_0

But looking at the debug logs from the DFP proxy container I see:

2018/09/19 07:28:40 HAPRoxy: 10.255.0.2:34136 [19/Sep/2018:07:28:40.791] tcpFE_1433 tcpFE_1433/<NOSRV> -1/-1/0 0 SC 1/1/0/0/0 0/0
Indicating that the request doesn't get forwarded to the backend haproxy configuration.

I have a web server service running 2 instances of a web application.
When i do a rolling update of my service or scale it up or down the proxy is not getting notified causing the proxy to not be able service requests for that service.

docker-compose.txt

HA Proxy supports optional ssl configuration and sni filters per certificate so you can for example have client required verification on certain SSL hosts only. The documentation is at https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.1-crt-list

A sample crt-list.txt file I have created to test this looks like:

/certs/nginx-test-2.example.com.pem [ca-file /certs/tmp/ca.crt verify required]
/certs/nginx-test-1.example.com.pem

As expected, test-1 can be accessed freely by anyone; test-2 requires valid client certificates to be present for the service to be hit.

At the moment I see the SSL certificate list is created by adding any files from /certs. Given the above configuration can get quite complex, I propose the addition of the option DF_CRT_LIST_PATH to DFP. When set, DFP will no longer try and create the crt-list.txt file and will instead rely on outside sources to correctly create that file.

This is just one possible solution. I'd love the ability to achieve per certificate sslbind options and if anyone has a better or more convenient solution I'd be happy to go with that.

I'm aware there are a few other places that SSL certificates are read/used in DFP but I confess I am not fully aware of the feature set those instances provide. The above proposal does not really consider how to handle certificate addition or removal from the crt-list. I'd assume that whatever system creates the crt-list would also need to trigger a reload ?

What are peoples thoughts ? Does this seem like a feature we could viably add to DFP?

When using dockerflow/docker-flow-proxy:18.05.08-45-linux-arm docker pulls the wrong image from docker hub. Error shown at container level is: "no suitable node ... ", which indicates wrong architecture.

I'll investigate what's wrong here and fix it as soon as possible with a PR. Meanwhile just use latest or no tag at all to pull the right image.

Description
I am using DFP and DFSL together, if the DFP and DFSL container is on the same docker node and the node needs to be taking down for maintenance, the DFP container will general startup before the DFSL, causing 503 to be sent until the DFSL is available.

Steps to reproduce the issue:

1. Establish a 2 or 3 node docker swarm cluster
2. Set the replica for DFP service to match the number of nodes in the cluster
3. Set the node that hosts the DFSL and one of the DFP to drain, while generating http requests

Describe the results you received:
When the DFP moves to another node, it will generate 503's until the DFSL is online

Describe the results you expected:
If the DFP service is configured to use a DFSL, do not start the DFP until the container can gather the correct haproxy cfg from the DFSL.

Additional information you deem important (e.g. issue happens only occasionally):

Additional environment details (AWS, VirtualBox, physical, etc.):

It is a common pattern to reset service by scaleing it to 0 and then scaling pack to original count. Unfortunately this will take haproxy offline with the following errors:
2018/09/06 19:36:31 Validating configuration
[ALERT] 248/193631 (31) : parsing [/cfg/haproxy.cfg:96] : 'server app_server' : could not resolve address 'app_server'.
[ALERT] 248/193631 (31) : Failed to initialize server(s) addr.
Exit Status: 1
2018/09/06 19:36:36 Config validation failed. Will try again...
2018/09/06 19:36:37 Validating configuration
[ALERT] 248/193637 (42) : parsing [/cfg/haproxy.cfg:96] : 'server app_server' : could not resolve address 'app_server'.
[ALERT] 248/193637 (42) : Failed to initialize server(s) addr.
Exit Status: 1
Unfortunately the Portainer is behind the docker-flow-proxy as well, so I can't scale it back now unless I log to host via ssh.

Because the project has a new location, I'm transferring the issue here.

Hi,

Let's say that we have three services and each one is listening on port 8888 but it is started with a different configuration (e.g. a different mounting volume).

Is there a way to configure the proxy in order to obtain this behavior:
when we are trying to access http://localhost/, the proxy will redirect the request to any service that is alive?

Thank you!

Description

I need to pass SSL to different nginx proxys in docker swarm. Im trying to set it up but having problems.

Steps to reproduce the issue:

1. Deploy DFP
2. Deploy different nginx https proxys with different certificates with sni configuration in docker-compose.
3. Deploy a service behind nginx proxy
4. curl that service

Describe the results you received:
Some of the requests go through some get Docker Flow Proxy: 503 Service Unavailable.

Proxy log:
2018/07/04 11:17:02 HAPRoxy: 172.28.22.175:61976 [04/Jul/2018:11:17:01.573] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 7/1/469 3806 -- 1/1/0/0/0 0/0
2018/07/04 11:17:03 HAPRoxy: 172.28.22.175:61977 [04/Jul/2018:11:17:03.044] services~ services/ -1/-1/-1/-1/1 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
2018/07/04 11:17:04 HAPRoxy: 172.28.22.175:61978 [04/Jul/2018:11:17:04.214] services~ services/ -1/-1/-1/-1/1 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
2018/07/04 11:17:05 HAPRoxy: 172.28.22.175:61979 [04/Jul/2018:11:17:05.258] services~ services/ -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
2018/07/04 11:17:06 HAPRoxy: 172.28.22.175:61980 [04/Jul/2018:11:17:06.273] services~ services/ -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
2018/07/04 11:17:07 HAPRoxy: 172.28.22.175:61981 [04/Jul/2018:11:17:07.251] services~ services/ -1/-1/-1/-1/1 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
2018/07/04 11:17:08 HAPRoxy: 172.28.22.175:61982 [04/Jul/2018:11:17:08.244] services~ services/ -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
2018/07/04 11:17:09 HAPRoxy: 172.28.22.175:61983 [04/Jul/2018:11:17:08.903] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 7/1/431 3807 -- 1/1/0/0/0 0/0
2018/07/04 11:17:10 HAPRoxy: 172.28.22.175:61984 [04/Jul/2018:11:17:09.911] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 8/1/426 3806 -- 1/1/0/0/0 0/0
2018/07/04 11:17:11 HAPRoxy: 172.28.22.175:61985 [04/Jul/2018:11:17:10.901] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 7/0/423 3806 -- 1/1/0/0/0 0/0
2018/07/04 11:17:12 HAPRoxy: 172.28.22.175:61986 [04/Jul/2018:11:17:11.871] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 7/1/429 3806 -- 1/1/0/0/0 0/0
2018/07/04 11:17:13 HAPRoxy: 172.28.22.175:61987 [04/Jul/2018:11:17:13.123] services~ services/ -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" (edited)

Requests work 100% when done from inside DPF container.

Additional information you deem important (e.g. issue happens only occasionally):
Nginx labels:
com.df.notify: 'true'
com.df.pathType: "req_ssl_sni -i -m end"
com.df.servicePath: ".test1.company.xyz,.test1.companyassets.xyz"
com.df.srcPort: 443
com.df.reqMode: sni
com.df.port: 1443

Proxy config:
frontend service_443
bind *:443
mode tcp
option tcplog
log global
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
acl sni_test1_nginx-consul-proxy1443-1 req_ssl_sni -i -m end .test1.company.xyz req_ssl_sni -i -m end .test1.companyassets.xyz
acl srcPort_test1_nginx-consul-proxy443_0 dst_port 443
use_backend test1_nginx-consul-proxy-be1443_0 if sni_test1_nginx-consul-proxy1443-1 srcPort_test1_nginx-consul-proxy443_0

backend test1_nginx-consul-proxy-be1443_0
mode tcp
server test1_nginx-consul-proxy test1_nginx-consul-proxy:1443

Certificates are mounted to the proxy at /certs
Wildcard letsencypt certificates to *.test1.company.xyz and *.test1.companyassets.xyz domains.

DFP version 18.07.03-68

I have a service mapped to some DFP servicePath, say /x. This service also internally serves all its endpoints under /x. It returns links to its own endpoints, which it is able to do thanks to HTTP Host and X-Forwarded-Proto headers.

A problem arises if I want to expose the same service on different service paths. (real life example, under a different domain it's supposed to have a different service path). It's possible to get the proxy routing right using reqPathSearchReplace:

- com.df.serviceDomain.1=x.com
- com.df.servicePath.1=/x
- com.df.serviceDomain.2=y.com
- com.df.servicePath.2=/y
- com.df.reqPathSearchReplace=/y,/x

If I call the service through y.com/y, its generated internal links end up being y.com/x/foo, because it has no idea about the path/search/replace that's going on in haproxy.

There's a conventional HTTP header for dealing with this situation, named X-Forwarded-Prefix, I think of it as a "variable service path". At least https://github.com/spring-projects/spring-framework and a number of other web application frameworks support this header out of the box.

So, for reqPathSearchReplace this will not be able to fly, because that rule can replace any part of the path and a prefix cannot be computed statically for haproxy config.

A proposed way to be able to configure such a setup would be to use e.g. servicePathMapping or similar:

- com.df.serviceDomain.1=x.com
- com.df.servicePath.1=/x
- com.df.serviceDomain.2=y.com
- com.df.servicePathMapping.2=/y,/x

in which case the service would still be serving internally under /x but an incoming X-Forwarded-Prefix would kind-of overwrite that. (Request /y/foo goes to /x/foo with X-Forwarded-Prefix: /y)

or even simpler:

- com.df.serviceDomain.1=x.com
- com.df.servicePathMapping.1=/x,/
- com.df.serviceDomain.2=y.com
- com.df.servicePathMapping.2=/y,/

in which case the service would just be serving everything under root /.

Description

After a reboot of some of the worker nodes DFP starts returning 503 for all services. Even when the servicies are up and running again. DFP keeps on returning 503 until it's restarted

Steps to reproduce the issue:

1. Initialize a swarm with 3 managers and 3 workers
2. Deploy 15-20 services
3. Deploy DFP and DFSL. Everything works properly
4. Reboot all 3 workers at once
5. Wait for all services to come back.

Describe the results you received:

DFP keeps returning 503 even though all services are up and running

Describe the results you expected:

DFP starts routing traffic to services as soon as they get restarted

Additional information you deem important (e.g. issue happens only occasionally):

We need to restart (docker service update --force) both DFP and DFSL so they start working again.
When the proxy is not responding if we sh into the container and restart just the haproxy, it starts working
The HAProxy configuration looks ok although all requests return 503 (the basic black 503 message not the fancy 503 that the proxy returns when a service is not available)

Additional environment details (AWS, VirtualBox, physical, etc.):

We've been managed to reproduce this issue both in AWS and our own servers

I'm using VirtualBox with 3 Linux/Ubuntu 17.10.1 VMs. 1 manager, 2 workers. I read your responses in the other issues about it being something with the overlay network, but I'm at a loss at what to do or how to troubleshoot that. I just started experimenting with docker yesterday.

docker service logs dfp

user123@user123-VirtualBox:~/projects/myapp/razor2$ sudo docker service logs dfp
dfp.1.qvocubh1l9aa@user123-VirtualBox | 2018/04/24 18:17:53 Starting HAProxy
dfp.1.qvocubh1l9aa@user123-VirtualBox | 2018/04/24 18:17:54 Getting certs from http://198.105.254.24:8080/v1/docker-flow-proxy/certs
dfp.1.m77rmf7oucjk@user123-VirtualBox | 2018/04/24 18:18:17 Starting HAProxy
dfp.1.m77rmf7oucjk@user123-VirtualBox | 2018/04/24 18:18:17 Getting certs from http://198.105.254.24:8080/v1/docker-flow-proxy/certs

Commands run:

docker network create --driver overlay proxy

docker service create --name swarm-listener --network proxy --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" -e DF_NOTIFY_CREATE_SERVICE_URL=http://dfp:8080/v1/docker-flow-proxy/reconfigure -e DF_NOTIFY_REMOVE_SERVICE_URL=http://dfp:8080/v1/docker-flow-proxy/remove --constraint 'node.role==manager' dockerflow/docker-flow-swarm-listener

docker service create --name dfp -p 80:80 -p 443:443 --network proxy -e SERVICE_NAME=dfp -e LISTENER_ADDRESS=swarm-listener dockerflow/docker-flow-proxy

Recreating issue vfarcic#416

Docker Flow Proxy fails to provision config from time to time (often only on one or more instances), removing DFP from the stack and re-provisioning 'solves' the problem until it reappears again.

Description

Following http://proxy.dockerflow.com/swarm-mode-stack/ process to setup docker-flow-proxy stack. The stack is up and running and listening and registering new services. But except for the test service mentioned in doc all others services I deploy get registered but on trying the servicePath URL I get a 404 response for

Steps to reproduce the issue:

1. For no auth based service - Try deploying a basic consul cluster using deploy stack. The service works using the endpoint exposed. Labels used are

labels:
        - com.df.notify=true
        - com.df.servicePath=/consul
        - com.df.port=8500

2. For auth based service - I am deploying swarmpit. Labels used are

labels:
        - com.df.notify=true
        - com.df.servicePath=/swarmpit
        - com.df.users=admin:admin
        - com.df.port=8080

Assuming com.df.users help me provide the users for login to mapped service.

Describe the results you received:

• No auth based service logs on enabling debug on docker-flow-proxy are below:

df_proxy.0.aofchyt7vl1i@host1    | 2018/07/19 07:59:26 HAPRoxy: 10.0.86.152:46180 [19/Jul/2018:07:59:26.800] services sd_server-be8500_0/sd_server 0/0/1/1/2 404 82 - - ---- 2/2/0/1/0 0/0 "GET /consul HTTP/1.1"
df_proxy.0.aofchyt7vl1i@host1    | 2018/07/19 07:59:31 HAPRoxy: 10.0.86.152:46182 [19/Jul/2018:07:59:26.586] services services/<NOSRV> -1/-1/-1/-1/5000 408 666 - - cR-- 2/2/0/0/0 0/0 "<BADREQ>"

• Auth based service logs on enabling debug on docker-flow-proxy are below:

df_proxy.0.aofchyt7vl1i@host1    | 2018/07/19 08:01:04 HAPRoxy: 10.0.86.152:46258 [19/Jul/2018:08:01:04.302] services sp_app-be8080_0/<NOSRV> -1/-1/-1/-1/0 401 255 - - PR-- 1/1/0/0/3 0/0 "GET /swarmpit HTTP/1.1"
df_proxy.0.aofchyt7vl1i@host1    | 2018/07/19 08:01:10 HAPRoxy: 10.0.86.152:46260 [19/Jul/2018:08:01:10.902] services sp_app-be8080_0/sp_app 0/0/0/20/20 401 307 - - ---- 1/1/0/1/0 0/0 "GET /swarmpit HTTP/1.1"
df_proxy.0.aofchyt7vl1i@host1    | 2018/07/19 08:01:11 HAPRoxy: 10.0.86.152:46260 [19/Jul/2018:08:01:11.159] services services/<NOSRV> -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET /favicon.ico HTTP/1.1"

Describe the results you expected:
The urls should show me the services.

Additional information you deem important (e.g. issue happens only occasionally):
This is always happening, however I am able to access service using host exposed ports.

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical VMs.

If I create a service and later remove it docker flow proxy gets stuck into a reconfiguration loop.

This might be related to vfarcic#453 although it seems that this is not caused by the rapid creation and removal of services but just at any time when I remove a docker flow proxy labeled service with docker service rm service_name.

Docker Flog Proxy Logs:

proxy_proxy.1.lcn2oedr07ly@Docker-Manager    | 2018/04/19 12:24:08 Validating configuration
proxy_proxy.1.lcn2oedr07ly@Docker-Manager    | [ALERT] 108/122408 (208) : parsing [/cfg/haproxy.cfg:106] : 'server app_u0oy5r45p775' : could not resolve address 'app_u0oy5r45p775'.
proxy_proxy.1.lcn2oedr07ly@Docker-Manager    | [ALERT] 108/122408 (208) : Failed to initialize server(s) addr.
proxy_proxy.1.lcn2oedr07ly@Docker-Manager    | Exit Status: 1
proxy_proxy.1.lcn2oedr07ly@Docker-Manager    | 2018/04/19 12:24:08 Config validation failed. Will try again...

Docker Flow Swarm Listener Logs:

proxy_swarm-listener.1.r1hsv1vbaayp@Docker-Manager    | 2018/04/19 12:23:22 Sending service removed notification to http://proxy:8080/v1/docker-flow-proxy/remove?distribute=true&serviceName=app_u0oy5r45p775
proxy_swarm-listener.1.r1hsv1vbaayp@Docker-Manager    | 2018/04/19 12:23:22 Canceling service remove notification to http://proxy:8080/v1/docker-flow-proxy/remove?distribute=true&serviceName=app_u0oy5r45p775

Using dockerflow/docker-flow-proxy:18.04.18-37 and dockerflow/docker-flow-swarm-listener:18.04.15-5

Is anyone else getting weird behaviour, where if docker swarm (mode) performs a rollback automatically, then the old (rolled back to) version is not registered in proxy?

Thanks.

Typing it to help others and maybe get this updated in the doc or improved in the code.

I followed https://proxy.dockerflow.com/swarm-mode-auto/ and all worked great.
Then I tried something of my own and ran into 503 problems quickly.
Note: my proxy network is called proxy-network instead of just proxy to make it easier to comprehend for me. I have 2 nodes using latest dfp image.

docker service create --name proxy -p 80:80 -p 443:443 --network proxy-network --replicas 2
-e LISTENER_ADDRESS=swarm-listener
dockerflow/docker-flow-proxy

docker service create --name groovy-jetty -t --mount type=volume,source=webServer-rw,destination=/mnt/webServer
--network go-demo --network proxy-network --replicas 2 --label com.df.notify=true
--label com.df.port.1=9080
--label com.df.srcPort.1=80
--label com.df.port.2=9443
--label com.df.srcPort.2=443
--label com.df.distribute=true
--label com.df.reqMode=tcp
--label com.df.checkTcp=true
--label com.df.connectionMoode=http-tunnel
my-images/docker-groovy-with-jetty:0.1

With this setup I was getting intermittent 503 errors (definitely not production qualiy). So I started poking around. And looked at HAProxy config.
Note: the similar 503 errors were happening with http mode too when I used just 80 port (lost the details) but I needed tcp so that server can present its own cert (this is a temporary setup).

docker ps
Gave me the proxy container id

docker exec -ti 7b79959ae9c6 /bin/ash
/$ less /cfg/haproxy.cfg

Turned out there were 3 frontend entries:

frontend services
bind *:80
bind *:443
mode http
option forwardfor

frontend tcpFE_443
bind *:443
mode tcp
default_backend groovy-jetty-be9443_2

frontend tcpFE_80
bind *:80
mode tcp
default_backend groovy-jetty-be9080_1

Doing netstat -atnp confirmed my suspicion:

tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 5708/haproxy
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 5708/haproxy
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 5708/haproxy
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 5708/haproxy

Why OS allows that I don't know (I suspect its bc it is the same pid) but with this setup services listener is getting ~50% of the connections (depending which of the listeners grabs the next connection) and it has no backend hence intermittent 503 errors.

The workaround is to change default ports for proxy service (they aren't published so there is no harm):

-e DEFAULT_PORTS=81,444 \

Confirmed with below command showing zero 503 errors.

httperf --server=serverIP --port=80 --uri=/ --num-conns=300 --num-calls=10|grep -E "test-duration|5xx"

I believe there should be some logic added to prevent this kind of config created by dfp but I will leave this to the discretion of the maintainer as I'm just a noob :)
Hope this helps someone!!

Feature request
A way to specify what certificate Docker Flow whould use when there is no SNI field present in the TLS handshake.

Context
Hi, I have a setup where one server can be reached reached from root of the domain and one via a subdomain. It all seems to work, untill I try to reach reach the main domain from a WinXP client using IE8. IE on XP does not have support for SNI, which caused HAProxy to default to the first certificate it has, which is based on their alphabetical order in the certs directory. I do not have control over how the certificates are named, since I use docker-flow-letsencrypt which names them after the domain that they belong to.

/ # ls certs
portainer.domain.nl.combined.pem  domain.nl.combined.pem

Current workaround
Luckily my main domain name starts with a b and the first subdomain starts with a p, this issue only occurs on my dev server on a different domain. On the dev server its possible to change the subdomain from subdomain.domain.nl to zsubdomain.domain.nl