failed to determine master about stampede HOT 8 CLOSED

startup.sh only works if etcd is publicly available or available from containers.

from stampede.

@wmark what does your cloud config look like? It does assume etcd is listening on the IP on docker0.

from stampede.

Thank you for looking into this. It basically boils down to having etcd locked down using iptables to prevent malicious containers from messing with the cluster.

To solve this issue you just need to expose etcd ports (from the host) to the container.

Excerpt from the iptables rules for your interest:

  units:
    - name: iptables.service
      enable: true
    - name: iptables-restore.service
      command: start
    - name: ip6tables.service
      enable: true
    - name: ip6tables-restore.service
      command: start

write_files:
  - path: /var/lib/iptables/rules-save
    permissions: 0600
    owner: root:root
    content: |
      *filter
      :INPUT ACCEPT [0:0]
      :FORWARD DROP [0:0]
      :OUTPUT ACCEPT [0:0]
      :cluster - [0:0]
      :local-ap - [0:0]

      # NTP and etcd are restricted to servers in the cluster
      -A INPUT -p tcp -m multiport --dports 123,4001,7001 -j cluster
      -A INPUT -p udp -m multiport --dports 123,4001,7001 -j cluster

      -A cluster -s $private_ipv4/16 -j ACCEPT
      -A cluster -j local-ap

      # add private bridges (br0, virbr0, … — NOT docker0…) here
      -A local-ap ! -i lo -p tcp -j REJECT --reject-with tcp-reset
      -A local-ap ! -i lo -p udp -j REJECT --reject-with icmp-port-unreachable

      COMMIT
      # end
  - path: /var/lib/ip6tables/rules-save
    permissions: 0600
    owner: root:root
    content: |
      *filter
      :INPUT ACCEPT [0:0]
      :FORWARD DROP [0:0]
      :OUTPUT ACCEPT [0:0]
      :cluster - [0:0]
      :local-ap - [0:0]

      # NTP and etcd are restricted to servers in the cluster
      -A INPUT -p tcp -m multiport --dports 123,4001,7001 -j cluster
      -A INPUT -p udp -m multiport --dports 123,4001,7001 -j cluster

      -A cluster -s 2a03:f00d:f00d::/48 -j ACCEPT
      -A cluster -j local-ap

      # add private bridges (br0, virbr0, … — NOT docker0…) here
      -A local-ap ! -i lo -p tcp -j REJECT --reject-with tcp-reset
      -A local-ap ! -i lo -p udp -j REJECT --reject-with icmp6-port-unreachable

      COMMIT
      # end

from stampede.

I do consider one of the major hurdles to make stampede production ready is to address the security around etcd and fleet. Currently some things can be done to restrict access, but I think it's either too cumbersome to the user or not sufficient. You do bring up a good point. It's really not good for stampede to assume that etcd is available to the containers as that has security implications. I think I will change it such that etcd is only accessed from outside the containers in the host OS/namespace. @wmark Would that approach work better for you?

from stampede.

Darren, thanks a lot for taking the time to analyze the issue and your suggestion.

I believe that due to etcd-io/etcd#91 we can close the issue here.

from stampede.

@wmark I encourage you to check out rancher.io. Rancher.io is a continuation of work that started with stampede, but now we have a company and a large amount of resources dedicated to it.

from stampede.

Thanks Darren, I will definitely take a look.

Coming with some years of experience with Linux and Gentoo, I am currently working on a fork of CoreOS which incorporates all the missing items (central logging for example; HW monitoring and reporting, authneticated and encrypted-by-default networking between nodes) and fixing what's not done right yet (software versions, in-place updates using overlays, security settings). It's to run my encrypted email service, but with the goal in mind making it a Gentoo/ChromeOS distribution in the end.

from stampede.

Sounds really interesting, I'd love to see it.

from stampede.