neilmunday / salt-minion-inventory Goto Github PK

Add buttons to the minion table to allow filtering of minions, e.g. only show minions that are up etc.

Salt Master
OS version: Debian 10
SaltStack version: 3003

Salt Minions
OS version: Debian 10
SaltStack version: 3003

Issue description:
I have followed the installation instructions however when I run the command: salt '*' inventory.audit force=True all I receive is

'inventory.audit' is not available.

The salt master logs state:
[salt.utils.reactor:443 ][ERROR ][2335] Reactor 'inventory_present' failed to execute runner 'inventory.present': function not available

I have ensured that all the python modules are loaded and the database tables populated but im still having little success, in addressing this issue. Are you able to advise or point me in towards what action is required to address this.

From the salt grains add the serial number, manufacturer and product name to the minions table and display on the minions info page.

For each minion, record the users who are logged in.

For each minion, record information about the graphics card attached to the host.

The Salt server generates time stamps in UTC format and they are stored in MySQL as "DateTime". This removes the time zone information which results in misleading last seen and last audit values being displayed in the web interface.

Suggest storing the time stamps as Unix time stamps instead.

Allow users view package differences between two servers.

Add storing of CPU load, memory usage etc. to inventory.py module for the minions. Metrics captured should be displayed in the interface.

This should be implemented as a separate function in the inventory module so that it can be run more frequently than the audit checks.

Equally, a separate runner SLS will be required and an equivalent handler for the runner to update the database.

Issue description:

There is no any checks, when the minion properties does not changed only server_id field populated, and then exception raised:

salt/run/20201103071527850579/ret       {
    "_stamp": "2020-11-03T07:15:27.856485",
    "fun": "runner.inventory.audit",
    "fun_args": [
        "2020-11-03T07:15:27.842318",
        {
            "server_id": 1789550377
        },
        false
    ],
    "jid": "20201103071527850579",
    "return": "Exception occurred in runner inventory.audit: Traceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/salt/client/mixins.py\", line 397, in low\n    data[\"return\"] = func(*args, **kwargs)\n  File \"/var/lib/salt/runners/inventory.py\", line 149, in audit\n    vendorId = __getVendorId(db, cursor, properties[\"manufacturer\"])\nKeyError: 'manufacturer'\n",
    "success": false,
    "user": "Reactor"
}

I am investigating why Reactor proccess is leaking memory and suggested that this might be the reason

The inventory.audit reactor SLS fails to be parsed by the Salt master when using the 2019 version of Salt.

Error in Salt master logs:

2019-02-28 23:10:09,448 [salt.utils.reactor:106 ][ERROR   ][16095] Failed to render "/srv/reactor/inventory/audit.sls":
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/reactor.py", line 97, in render_reaction
    data=data)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 385, in render_template
    **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/template.py", line 101, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python2.7/site-packages/salt/renderers/yaml.py", line 57, in render
    raise SaltRenderError(err_type, line_num, exc.problem_mark.buffer)
SaltRenderError: found unexpected ':'

This appears due to the properties dictionary returned as part of the inventory.audit function.

Instead of invoking the inventory.audit function every few hours, implement "inventory.run" to execute every 10-15 minutes. This function should read the config files on the minion (see issue #21) to determine which functions need to be executed.

Proposed functions to run regularly:

• audit -> existing audit functionality minus logging user info
• user -> new function to report users currently logged into a minion
• metrics -> new function to report back metrics for the minion (e.g. CPU load, memory use, disk usage etc. see issue #16)

These functions may need to cache data on the minion which can then be used to decide whether to report back to the master or not. For example, if the users logged haven't changed then don't report back. This is similar to the existing audit cache.

Create a dashboard to summarise the state of all minions.

For tabs like the GPU tab that may have no content, disable them in this instance.

Add auto refresh to index.html.

The minion table in index.html should reported the kernel release, not the name of the kernel.

Initial testing was with Salt 2015 as supplied by CentOS 7.5. Further testing with Salt 2018 has revealed some issues with the handling of package versions data in the runner (inventory.py).

Add an about page to the web interface. Put link in top right?

Record which the name of the salt master associated which each minion.

Could also store info about the master in the database? This would require a scheduled task to run on the master.

Salt Master
salt-master 2019.2.1 (Fluorine)

OS version: CentOS 7.7
SaltSatck version:
salt-master 2019.2.1 (Fluorine)

Salt Minions
2019.2.1

OS version: Windows 2012R2
SaltStack version:

Issue description:
Biosreleasedate does not exist in windows grains, failure to populate database with minion return due to error.

The minion function caused an exception: Traceback (most recent call last):
  File "c:\salt\bin\lib\site-packages\salt\minion.py", line 1663, in _thread_return
    return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
  File "c:\salt\bin\lib\site-packages\salt\executors\direct_call.py", line 12, in execute
    return func(*args, **kwargs)
  File "c:\salt\var\cache\salt\minion\extmods\modules\inventory.py", line 66, in audit
    properties[p] = grains[p]
  File "c:\salt\bin\lib\site-packages\salt\utils\context.py", line 236, in __getitem__
    return self._dict()[key]
KeyError: 'biosreleasedate'

Correct bootstrap JS and CSS includes.

Salt Master

OS version: 22.04 Ubuntu Server
SaltSatck version: 3005

Salt Minions

OS version: Windows
SaltStack version: 3004

Issue description:
error in WEB Gui & inventory.audit missing

Update packages.php to use a JSON feed for the package data.

Add a "packages" tab to the minion info page and merge the existing packages page.

packages.php query does not filter by server ID which means all packages for all servers are returned.

To allow features for version 2.0 to be implemented, create a config file say /etc/salt-minion-inventory.conf and possibly a config directory, e.g. /etc/salt-minion-inventory.d to store settings to be used by the inventory module functions that are to execute on the minions.

The files should be staged to the minions by SaltStack before the module functions run.

Add the ability to delete minions via the web interface.

Hi,

We are trying your inventory runner, thanks for that,
we are facing issues.

database is setup and working,
module is functionnal,

$ salt -l debug minion1 inventory.audit force=True
[...]
[DEBUG   ] LazyLoaded nested.output
minion1:
    Success
[DEBUG   ] jid 20181221174623203872 found all minions set([u'minion1'])

We added a log line in __doQuery:

def __doQuery(cursor, query):
        """
        Helper function to execute a query on the given cursor object.
        """
        try:
                log.debug("query: %s" % query);       # <-------- here
                cursor.execute(query)
        except Exception as e:
                raise Exception("inventory.__query: %s\nquery was: %s" % (e, query))

All what we can see in logs are SELECTs (no INSERT, UPDATE...)

and salt logs show lines like that

2018-12-21 17:33:53,422 [salt.utils.event :739 ][DEBUG   ][29868] Sending event: tag = salt/run/20181221173353399703/ret; data = {u'fun_args': [u'2018-12-21T16:33:53.272762', OrderedDict([(u'server_id', 1564434732)]), False], u'jid': u'20181221173353399703', u'return': u'Exception occurred in runner inventory.audit: Traceback (most recent call last):\n  File "/usr/lib/python2.7/site-packages/salt/client/mixins.py", line 387, in _low\n    data[\'return\'] = self.functions[fun](*args, **kwargs)\n  File "/srv/salt/_runners/inventory.py", line 208, in audit\n    properties["id"],\nKeyError: \'id\'\n', u'success': False, u'_stamp': '2018-12-21T16:33:53.421964', u'user': u'Reactor', u'fun': u'runner.inventory.audit'}

Exception occurred in runner inventory.audit: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/client/mixins.py", line 387, in _low
    data[\'return\'] = self.functions[fun](*args, **kwargs)
  File "/srv/salt/_runners/inventory.py", line 208, in audit

Line 208 is an access to properties[], all accesses to properties[] give this error,
The error originally occurs in this line (we added our logs and debugs).

We are not at this point bearded enough to debug the properties argument in the audit python function.
Can you please provide us some help ?

Also, thanks for setting this repository up !

[WARNING ] /usr/lib/python2.7/site-packages/salt/payload.py:149: DeprecationWarning: encoding is deprecated, Use raw=False instead.
ret = msgpack.loads(msg, use_list=True, ext_hook=ext_type_decoder, encoding=encoding)

salt-syndic-node1:
salt-syndic-node1 properties have not changed

Add the boot time for a minion to the minion info page.

For each minion, record information about the physical disks attached.

Hi Neil,
i am trying to setup the inventory system for my salt master. have configured everthing as per your instructions. getting this error " root@saltsrv:/var/lib/salt/runners# salt labtest-ubuntu inventory.audit force=True
labtest-ubuntu:
'inventory.audit' is not available.
ERROR: Minions returned with non-zero exit code"

plz advise.

Update database schema to record the total for each type of GPU in a minion and display accordingly on the minion info page.

Is MariaDB supported out of the box, or some modifications is required for that?

Switching log_level to debug produces the following error:

May 26 10:28:57 gla-ccs salt-master[2473325]: [DEBUG ] Failed to import runners inventory:
May 26 10:28:57 gla-ccs salt-master[2473325]: Traceback (most recent call last):
May 26 10:28:57 gla-ccs salt-master[2473325]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 772, in _load_module
May 26 10:28:57 gla-ccs salt-master[2473325]: mod = self.run(spec.loader.load_module)
May 26 10:28:57 gla-ccs salt-master[2473325]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1232, in run
May 26 10:28:57 gla-ccs salt-master[2473325]: return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
May 26 10:28:57 gla-ccs salt-master[2473325]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1247, in _run_as
May 26 10:28:57 gla-ccs salt-master[2473325]: return _func_or_method(*args, **kwargs)
May 26 10:28:57 gla-ccs salt-master[2473325]: File "", line 548, in _check_name_wrapper
May 26 10:28:57 gla-ccs salt-master[2473325]: File "", line 1063, in load_module
May 26 10:28:57 gla-ccs salt-master[2473325]: File "", line 888, in load_module
May 26 10:28:57 gla-ccs salt-master[2473325]: File "", line 290, in _load_module_shim
May 26 10:28:57 gla-ccs salt-master[2473325]: File "", line 719, in _load
May 26 10:28:57 gla-ccs salt-master[2473325]: File "", line 688, in _load_unlocked
May 26 10:28:57 gla-ccs salt-master[2473325]: File "", line 883, in exec_module
May 26 10:28:57 gla-ccs salt-master[2473325]: File "", line 241, in _call_with_frames_removed
May 26 10:28:57 gla-ccs salt-master[2473325]: File "/var/lib/salt/runners/inventory.py", line 36, in
May 26 10:28:57 gla-ccs salt-master[2473325]: import MySQLdb
May 26 10:28:57 gla-ccs salt-master[2473325]: ModuleNotFoundError: No module named 'MySQLdb'

It can be solved by making sure the system has the MySQL dev tools on Ubuntu/Debian with:
sudo apt install libmysqlclient-dev

Then running
sudo salt-pip install configparser
sudo salt-pip install mysqlclient

Which installs the missing libraries into the OneDir python3.10