saltstack-formulas / hostsfile-formula Goto Github PK

Your setup

Formula commit hash / release tag

02c5de72b32e5b14ced692ead558f74964c09841

Versions reports (master & minion)

Master is 3001.1, minion is 3000.3:

Minion

Salt Version:
           Salt: 3000.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.17 (default, Jul 20 2020, 15:37:01)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-118-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Master

Salt Version:
           Salt: 3001.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.7
         pygit2: Not Installed
         Python: 3.6.9 (default, Jul 17 2020, 12:50:27)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 17.1.2
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: ubuntu 18.04 Bionic Beaver
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-99-generic
         system: Linux
        version: Ubuntu 18.04 Bionic Beaver

Pillar / config used

Pillar config is minimal:

hostsfile:
  alias: ip
  minions: '*'

And the alias:

mine_functions:
  ip:
    - mine_function:  grains.get
    - path:to:grain

Bug details

Describe the bug

If the mine function specified under hostsfile:alias returns nothing for a given
minion, the state 1. always returns dirty, and 2. appends multiple
copies of some other host in the hosts file -- I think always the last
one present, but I'm not sure. So you get something like this, assuming
there are four hosts and host4 is the one with no valid return:

192.168.0.1 host1
192.168.0.2 host2
192.168.0.3 host3 host3 host3 host3 host3 host3 host3 (more)
# host4 never appears

Steps to reproduce the bug

Configuring the pillar as above and rigging the alias to return None for
minion should reproduce the problem.

Expected behaviour

Probably offending hosts should be ignored, or the state should return
an error, or something. In any case some check should be done that what
we're trying to add is really an IP address.

Attempts to fix the bug

I haven't poked around it yet.

Additional context

I am not sure if the underlying bug is in the formula or in salt itself.

I'm currently facing some issue using saltstack to enroll my landscape.
I would like that salt manages the /etc/hosts file on all my minions/servers.
Therefor I added the hostsfile-formular to my saltstack via a Git-remoteFS
but it seems to have no effect. No hosts file gets placed anywhere, Please also see:
What i'm doing wrong here?

my tree:

root@salt:/srv# tree
.
├── formulas
├── pillar
│   ├── hostsfile.sls
│   └── timezone.sls
├── README.md
└── salt
    ├── base.sh
    ├── docker
    │   └── init.sls
    ├── fail2ban
    │   ├── init.sls
    │   ├── jail.conf
    │   └── jail.local
    ├── glusterfs
    │   └── init.sls
    ├── iptables
    │   ├── init.sls
    │   ├── nodes
    │   └── template.j2
    ├── minion
    ├── scripts
    │   ├── add_minion.sh
    │   ├── remove_minion.sh
    │   └── salt_master_enroll.sh
    ├── ssh
    │   ├── authorized_keys
    │   ├── init.sls
    │   ├── motd
    │   ├── server.sls
    │   └── sshd_config
    ├── toolstack
    │   └── init.sls
    └── top.sls

/etc/salt/minion (same on all minions):

mine_functions:
  network.interfaces: []
  network.ip_addrs:
    - enp7s0
mine_interval: 2

/etc/salt/master (salt master only)

interface: 10.10.10.2


pillar_roots:
  base:
    - /srv/pillar


fileserver_backend:
  - git
  - roots

# Add your remote formular repos here:
gitfs_remotes:
  - https://github.com/saltstack-formulas/timezone-formula.git
  - https://github.com/saltstack-formulas/hostsfile-formula.git
  - https://github.com/salt-formulas/salt-formula-glusterfs.git

gitfs_provider: gitpython

/etc/salt/top.sls:

base:           # Apply SLS files from the directory root for the 'base' environment

  '*':
    - timezone  # Set the general timezone
    - hostsfile # Set the /etc/hosts file

  'salt':    	# For salt master node as there is only one
    - toolstack # Apply basic toolstack to node
...

Finally my hostsfile.sls pillar at /srv/pillar/hostsfile.sls:

hostsfile:
  domain: local
  hosts:
    salt: 10.10.10.2
    master1: 10.10.10.3
  only:
    127.0.0.1:
      - localhost
      - localhost.localdomain
    # Removes all entries for 127.0.1.1:
    127.0.1.1: []

Can smb explain to me why my hostsfile has no effect at all. What do I miss here?

Hi.

Would you be interested in adding formula to the saltstack-formula?
https://github.com/davidkarlsen/sysstat-formula

I found no email for the collection, so I just selected a repo to add an issue.

The following line seems no longer being used?

{%- set if = grains['maintain_hostsfile_interface'] %}

If it's still in use we should update readme, if it's not in use anymore maybe we can remove it. also setting a variable name to be if is really confusing as if can also be an if statement.

Link to code:
https://github.com/saltstack-formulas/hostsfile-formula/blob/master/hostsfile/init.sls#L15

Does this happen on RedHat also?

Doing something like this instead of unless: test ... does the trick. I'm sure we could make it compatible with RedHat too.

cmd.run:
  - name: hostname {{ grains['id'] }}
  - onchanges:
    - file: /etc/hostname

Cheers!

Your setup

Formula commit hash / release tag

3be1202 v0.8.0

Versions reports (master & minion)

2019.2.0

Pillar / config used

hostsfile:
  alias:
      main_ip
  hosts:
      None
  only:
      ----------
      127.0.0.1:
          - foo.tld
          - foo
            - localhost

main_ip is a mine_function we use and return an IP depending of the main interface serving a service.

Bug details

Describe the bug

When hosts is not completed the state is broken with a message:

    Rendering SLS 'hostsfile' failed: Jinja error: 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 392, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 22, in top-level template code
TypeError: 'NoneType' object is not iterable

; line 22

---

[...]
{%- set pillar_only = salt['pillar.get']('hostsfile:only', {}) %}
{%- set mine_hosts = salt['mine.get'](minions, minealias, tgt_type=minions_type) %}
{%- if mine_hosts is defined %}
{%-   do hosts.update(mine_hosts) %}
{%- endif %}
{%- do hosts.update(pillar_hosts) %}    <======================

I think this formula could be used to generate a /etc/hosts only using a mine query but no static host.

Steps to reproduce the bug

• install formula hostsfile
• create a pillar for your minion to configure hostsfile but let hostsfile:hosts empty
• run state.sls hostsfile

Expected behaviour

I expect the formula to be able to:

• run a mine query to populate /etc/hosts
• to insert static entries to /etc/hosts
• and any combination of mine query and static entries with none mandatory

Attempts to fix the bug

Working on it, hoping to find a bugfix soon.

Additional context

see http://docs.saltstack.com/en/latest/topics/mine/#mine-functions

To enable the Salt Mine the mine_functions option needs to be applied to a minion. This option can be applied via the minion's configuration file, or the minion's Pillar

So the 'must' part should be a bit better defined...
and maybe add a pillar.example file to show the user how he/she could set it up.

When I use keepalived , the interface have more then one ip address, if use network.ip_addrs we will get all ips , so virtual ips will be config to /etc/hosts, I change the mine function and sls , use network.interface_ip to get one ip address. I hope fqdn keep only one ip address.

If anyone have the same problems with me, you can refer to my practice, if most people don't want to virtual IP become the FQDN, we may consider to submit PR

Hi, could it be possible to add a newline at the end of hostsfile?
I know this could be "messy" to deal with when adding a new host, but it would be cleaner to read

Best regards

Is your feature request related to a problem?

If there's an existing minion that's replaced with another minion of the same name but different IP (and the original minion is renamed) the hostsfile formula will simply add an entry for the new node, and add the new name for the old node to the old node's entry. This means that the original name gets two entries in /etc/hosts. Visual example:
Original node (the "before" state):

10.0.0.2  prometheus.my.domain.com

< Rename prometheus.my.domain.com minion_id to "prometheus-old.my.domain.com" >
< Spin up new node and call it "prometheus.my.domain.com" >

Now when running hostsfile on any minions that already had the old entry, they grab both new and old node entries from the Mine, and their /etc/hosts looks like:

10.0.0.2  prometheus.my.domain.com prometheus-old.my.domain.com
10.0.0.3  prometheus.my.domain.com

Describe the solution you'd like

Add a "purge existing entries" behavior to the formula. Probably an if condition to add the clean flag to host.present in init.sls?

The intent would be to update minions' /etc/hosts to point prometheus.my.domain.com to the new minion and set the old entry to "prometheus-old.my.domain.com" only. Like so:

10.0.0.2  prometheus-old.my.domain.com
10.0.0.3  prometheus.my.domain.com

Describe alternatives you've considered

n/a

Additional context

n/a

Salt Cloud's Linode provider has strict rules on the label used for a VM's name:

[WARNING ] A Linode label may only contain ASCII letters or numbers, dashes, and underscores, must begin and end with letters or numbers, and be at least three characters in length.

Because dots are not allowed, a full domain name cannot be set as the minion ID. This is incompatible with the requirement of matching the minion ID to the FQDN.

I'll attempt to add an option to improve this.

Hello,

I'm a relatively new user of Salt and I've been working on a project that involves using this formula.
However, it would seem that the relatively recent domain part of the formula is not working properly in my environment:

OS: CentOS 7
Saltstack: 2016.11.8 with salt-ssh.

Indeed, it generates entries such as this one:

192.168.122.53     saltmaster saltmaster.<function caller at 0x7f4993c70aa0>

As you can see, it doesn't seem to be able to actually grab the domain and instead prints a reference to a function, which changes every run.

I've been trying to tinker a bit with the code and replacing salt.pillar.get with salt['pillar.get'] in https://github.com/saltstack-formulas/hostsfile-formula/blob/master/hostsfile/init.sls#L23 seems to do the trick.

If this workaround sounds reasonable I'll go ahead and open a PR with that modification but I am not familiar with best practices when it comes to making formulas so I'd rather get your input on that issue.

If no mined data exists for a host, it should be deleted from the hostsfiles to keep it clean

I've found that if I only use hostsfile.hostname state. It sets fqdn properly but not adding it to /etc/hosts, because of this on some OS(for example ubuntu) you will see warning in bash. Also some application need this

It seems that entries are only added when found, but not updated when the IP of a minion changes.

I have not read through the entire formula, so I am unsure if this is intended behavior, a general bug or maybe a bug specific to my setup.

I will provide details if needed.

Ability to group hosts to split projects, so targeted hosts will only receive certain host/ip information