Munge service unable to start, incorrect permissions about munge HOT 12 CLOSED

What is the output from:
ls -ld /var/log/munge /var/log/munge/munged.log

What is the user id under which you are trying to start the daemon?

How are you starting the daemon (e.g., via the command-line, via systemd, via sysvinit)? What is the exact command-line you are using?

Based on the error message "/var/log/munge/munged.log" should be owned by UID 0, it appears that you're trying to run munged as root, and munged.log is not owned by root.

It's recommended that you run the daemon as a non-privileged user. Most distributions create a non-privileged munge account under which to run the daemon.

The --force command-line option changes certain errors (such as insecure file/directory permissions for the logfile) into warnings so the daemon will continue to run. This can be seen in the log messages you list at the end that begin with munged: Warning:.

from munge.

The output of the command is
drwx------. 1 munge munge 20 Sep 7 11:55 /var/log/munge -rw-r-----. 1 munge munge 2839 Sep 7 13:24 /var/log/munge/munged.log
I am attempting to start using the commandline systemctl
I am using systemctl start munge
A munge user is created however I am not sure how to use systemd as the munge user as it does not allow login.

from munge.

After the systemctl start munge command, what is the output of systemctl status -l munge?

from munge.

The output is

● munge.service - MUNGE authentication service
   Loaded: loaded (/etc/systemd/system/munge.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-09-07 15:58:53 EDT; 17s ago
     Docs: man:munged(8)
  Process: 7603 ExecStart=/usr/sbin/munged --syslog (code=exited, status=1/FAILURE)

Sep 07 15:58:53 dragonsdenN3 systemd[1]: Starting MUNGE authentication service...
Sep 07 15:58:53 dragonsdenN3 systemd[1]: munge.service: control process exited, code=exited status=1
Sep 07 15:58:53 dragonsdenN3 systemd[1]: Failed to start MUNGE authentication service.
Sep 07 15:58:53 dragonsdenN3 systemd[1]: Unit munge.service entered failed state.
Sep 07 15:58:53 dragonsdenN3 systemd[1]: munge.service failed.

from munge.

Do you see anything related to the munged failure in the output of journalctl -xe, or the file /var/log/messages?

What distribution are your running?

What are the contents of the file /etc/systemd/system/munge.service?

Is SELinux enabled? What is the output of sestatus?

from munge.

I am running CentOS 7.

Output of journalctl is

Sep 07 19:10:38 dragonsdenN3 systemd[1]: Starting MUNGE authentication service...
-- Subject: Unit munge.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit munge.service has begun starting up.
Sep 07 19:10:38 dragonsdenN3 munged[9700]: Failed to resolve host "dragonsdenN3"
Sep 07 19:10:38 dragonsdenN3 munged[9700]: Running on "dragonsdenN3" (0.0.0.0)
Sep 07 19:10:38 dragonsdenN3 munged[9700]: Ignoring PRNG seed "/var/lib/munge/munge.seed": Permission denied
Sep 07 19:10:38 dragonsdenN3 munged[9700]: PRNG seed dir is insecure: invalid ownership of "/var/lib/munge"
Sep 07 19:10:38 dragonsdenN3 munged[9698]: munged: Error: PRNG seed dir is insecure: invalid ownership of "/var/lib/munge"
Sep 07 19:10:38 dragonsdenN3 systemd[1]: munge.service: control process exited, code=exited status=1
Sep 07 19:10:38 dragonsdenN3 systemd[1]: Failed to start MUNGE authentication service.
-- Subject: Unit munge.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit munge.service has failed.
-- 
-- The result is failed.
Sep 07 19:10:38 dragonsdenN3 systemd[1]: Unit munge.service entered failed state.
Sep 07 19:10:38 dragonsdenN3 systemd[1]: munge.service failed.

Contents of systemd

[Unit]
Description=MUNGE authentication service
Documentation=man:munged(8)
After=network.target
After=syslog.target
After=time-sync.target

[Service]
Type=forking
ExecStart=/usr/sbin/munged         
PIDFile=/var/run/munge/munged.pid
User=munge
Group=munge
Restart=on-abort

[Install]
WantedBy=multi-user.target

from munge.

The fatal error is:
Error: PRNG seed dir is insecure: invalid ownership of "/var/lib/munge"

What is the output of ls -ld /var/lib/munge? That directory should be owned by the munge user and munge group.

from munge.

The output is
drwx------. 1 990 987 0 May 16 2014 /var/lib/munge

from munge.

That directory needs to be owned by the munge user and munge group:
chown -R munge:munge /var/lib/munge

from munge.

Running that still produces a failed start
Job for munge.service failed because the control process exited with error code. See "systemctl status munge.service" and "journalctl -xe" for details.
Output of journalctl is

30:02 dragonsdenN3 munged[20453]: munged: Error: Found pid 5795 bound to socket "/var/run/munge/munge.socket.2"
Sep 08 09:30:02 dragonsdenN3 systemd[1]: munge.service: control process exited, code=exited status=1
Sep 08 09:30:02 dragonsdenN3 systemd[1]: Failed to start MUNGE authentication service.
-- Subject: Unit munge.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit munge.service has failed.
-- 
-- The result is failed.
Sep 08 09:30:02 dragonsdenN3 systemd[1]: Unit munge.service entered failed state.
Sep 08 09:30:02 dragonsdenN3 systemd[1]: munge.service failed.

from munge.

The fatal error is the line with munged: Error::
munged: Error: Found pid 5795 bound to socket "/var/run/munge/munge.socket.2"

Another running process (pid 5795, presumably another instance of munged) is currently bound to the daemon's communication socket (/var/run/munge/munge.socket.2). Check that pid:
ps lp 5795

Since it appears that munged is already running, check its status:
systemctl status -l munge

If you want to restart the daemon:
systemctl restart munge

Once it's running: try to encode & decode a credential:
munge -n | unmunge

from munge.

systemctl restart munge did nothing, however after rebooting I was able to get munge to start properly by changing the permissions.

from munge.