gdraheim / docker-systemctl-replacement Goto Github PK

A little known fact is that systemd watches for an environment variable $SYSTEMD_UNIT_PATH to specify the locations where unit descriptor files are installed. Probably this has only some effect when the systemd daemon is run in user mode as you can change it for a PID-1 systemd daemon. In any case it would be cool when the systemctl.py script can watch for it instead of using other environment variable to set defaults in the code.

https://www.freedesktop.org/software/systemd/man/systemd.unit.html

To close the gap to normal docker-containers there is already an implementation of #28 to print service logs to the console log of PID 1. That helps to inspect the "system status" using "docker logs ".

Similarly, a user may want to check an exitcode of a docker container but so foar the init-replacement will always succeed. The problem is to define which exitcode should be removed the list of underlying services in a system.

The case of actually determining an early exit, when no-more-procs or no-more-active-services, has been already discussed in #27 ... however the exitcode may also be provided when doing a "systemctl --init default" from the command line.

The RemainAfterExit=yes option needs to be implemented. Without it a service may seem to be dead which is really running.

It was obverved specifically for Teamforge where the *.service file does not user type=forking where it should have:

[Service]
Type=simple
RemainAfterExit=yes

User=root
Group=root

ExecStart=/etc/init.d/collabnet start all
ExecStop=/etc/init.d/collabnet stop all

Support for the get/set-default commands is currently missing.

This is their intended function:

Unit File Commands:
[...]
  get-default                     Get the name of the default target
  set-default NAME                Set the default target

I currently needed the set-default command to not error-out during a post-install script execution, so I merely added an empty set_default_modules function to the Systemctl class, like this:

    def set_default_modules(self, *modules):
        return ""

and was able to proceed further.

When doing "systemctl status cron" then systemd may report

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

The systemctl.py replacement does create a path_journal_log as well, but it is never rotated. When using a container that runs for a long time then it might get to be a large file.

When daemon is not running (e.g. terminated, crashed, etc.) systemctl stop prints the following error:

kill: cannot find process ""

(My .service file contains ExecStop=/bin/kill ${MAINPID}).

The problem is that even though process is not there, systemctl does not remove PID file from /run on stop.

It should be possible to unmask an already unmasked unit:

root@653efcc50970:/etc/puppet# systemctl mask influxdb
root@653efcc50970:/etc/puppet# systemctl unmask influxdb
root@653efcc50970:/etc/puppet# systemctl unmask influxdb
ERROR:systemctl:target is not a symlink: /etc/systemd/system/influxdb.service

Last call should succeed.

Beginning in version 1.3.2236, systemctl start SERVICE no longer prints the output of the service as it starts. If this is deliberate, is there a way to enable the output?

https://www.freedesktop.org/software/systemd/man/systemctl.html

By default only units which are active, have pending jobs, or have failed are shown; this can be changed with option --all. If one or more PATTERNs are specified, only units matching one of them are shown. The units that are shown are additionally filtered by --type= and --state= if those options are specified.

The LOAD column shows the load state, one of loaded, not-found, bad-setting, error, masked. The ACTIVE columns shows the general unit state, one of active, reloading, inactive, failed, activating, deactivating. The SUB column shows the unit-type-specific detailed state of the unit, possible values vary by unit type. The list of possible LOAD, ACTIVE, and SUB states is not constant and new systemd releases may both add and remove values.

systemctl --state=help
command maybe be used to display the current set of possible values.

The --state filter is not supported up to v1.4.2505 of systemctl.py

Currently the special vars expansions (like %i) is only being done in the Exec* settings of the service descriptor (as of systemctl.py 1.4). It should be done however in other parts as well, especially PIDFile and EnvironmentFile as they are needed in template service files (i.e. an at-service).

EXAMPLE: ... /usr/lib/systemd/system/[email protected]

 [Unit]
 Description=OpenVPN tunneling daemon instance using /etc/openvpn/%I.conf
 After=network.target
 PartOf=openvpn.target

 [Service]
 Type=forking
 PrivateTmp=true
 PIDFile=/run/openvpn/%i.pid
 ExecStart=/usr/sbin/openvpn --daemon --suppress-timestamps \
            --writepid /run/openvpn/%i.pid --cd /etc/openvpn/ --config %i.conf
 ExecReload=/sbin/killproc -p /run/openvpn/%i.pid -HUP /usr/sbin/openvpn

Manpage of systemd.unit describes an optional drop-in directory for systemd unit files:

Along with a unit file foo.service, a "drop-in" directory foo.service.d/ may exist.

This files are ignored.

An example is Kubernetes, which installs files in drop-in directories.

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install kubelet kubeadm

installs the following files:

/lib/systemd/system/kubelet.service
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

The command

systemctl.py start kubelet

doesn't take 10-kubeadm.conf into account and kubelet service starts with wrong parameters.

systemctl stop myservice executes ExecStopPost handler unconditionally...

For example I have the following error printed to console when I attempt to stop not running service:

ERROR:systemctl:returncode 1
 /usr/bin/pkill -f ${DAEMON}
Traceback (most recent call last):
  File "/usr/local/bin/systemctl", line 2437, in <module>
    result = command_func(*modules)
  File "/usr/local/bin/systemctl", line 1193, in stop_of_units
    elif not self.stop_unit(unit):
  File "/usr/local/bin/systemctl", line 1201, in stop_unit
    return self.stop_unit_from(conf)
  File "/usr/local/bin/systemctl", line 1286, in stop_unit_from
    subprocess_wait(cmd, env, check=check)
  File "/usr/local/bin/systemctl", line 396, in subprocess_wait
    raise Exception("command failed")
Exception: command failed

In v0.9 of systemctl.py it is possible to start a number of units and they are reordered according to the After-clauses in the configs. This includes also hints like Wants.

In addition to that one would like to also start dependencies when they are marked with Requires-clauses. This would effectivly mean to expand the number of units to be started. The basic functionaliy is already there as can be seen in "systemctl.py list-dependencies"

Very cool project! - Maybe in future we could have replacements for hostnamectl and timedatectl as well :)

As seen in the bug report on #51 the systemd service reader behaves different about overrides directories.

A combination like this is possible in systemd but not in systemctl.py up to 1.4

/lib/systemd/system/kubelet.service
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

(in systemctl.py 1.4 only /lib/systemd/system/kubelet.service.d/10-kubeadm.conf would be parsed)

mask/unmask are missing

Context: struggling to get elasticsearch puppet module to work on a centos7 container on a Win10 host.

Please add support for File/Path/Mount conditions such as ConditionFileNotEmpty. Thanks.

This is for my comment on issue #45, and for others.

Here is an listing of popular services Dockerfile's that I created, and use daily, that will not work without docker-systemctl-replacement, and will not work using only docker-systemctl-replacement. An dual understanding, implementation needs to be applied.

freepbx
Issabel-PBX
Wine
pi-hole
fail2ban
BIND-9

All needed and containerstartup.service that ran an script that checked the services, it decided to use original or docker-systemctl-replacement, most of the time, i had to use both for proper operations.

The 1.0 version of systemctl.py will send the stop/kill signals only to the mainpid of a program. However in reality it should be sent to all processes in the same control group, so basically it includes the children.

This may or may not need to get some functionality about providing and querying a control group top function properly.

This systemctl replacement would be even more useful inside containers if it allows foreground start -- something like

systemctl start --foreground apache2

to make it suitable for Docker's ENTRYPOINT command.

Thank you.

Please expose ${MAINPID} to ExecStartPost handler so the following could work:

ExecStartPost=/bin/sh -c "touch /tmp/${MAINPID}"

It's valid to have multiple ExecStop commands. Currently units that use them fail to start with:

ERROR:systemctl: redis-server.service: there may be only one ExecStop statement (unless for 'oneshot' services).
                        Use ' ; ' for multiple commands or better use ExecStopPost

It's used eg. by Redis packages: official Debian, Ubuntu PPA (ppa:chris-lea/redis-server):

ExecStartPre=-/bin/run-parts --verbose /etc/redis/redis-server.pre-up.d
ExecStartPost=-/bin/run-parts --verbose /etc/redis/redis-server.post-up.d
ExecStop=-/bin/run-parts --verbose /etc/redis/redis-server.pre-down.d
ExecStop=/bin/kill -s TERM $MAINPID
ExecStopPost=-/bin/run-parts --verbose /etc/redis/redis-server.post-down.d

There is an obversation with some kubernetes (/rancher/openshift) setups where a docker-entrypoint.sh script is being used that pulls (!) settings from the runtime environment before doing the final exec $start command.
In a way that is related to the ExecStartPre statements in a *.service but with the idea that the pre-script may set (or override) some env-variables for the start-proc of the actual service jar. Of course these scripts do not have a meaningful outcome in a situation outside of a specific target environment, so they will probably not be encoded in the *.service script itself.
So it would be best to have a systemctl.py feature that supports additional Env-Scripts - which is a new feature over Env-Settings, Env-Files, and one should be able to extend an existing service outside of the *.service just like it is already common with *.wants and *.preset directories.

I discovered this issue while debugging why the PostgreSQL service would not start in my Docker container using systemctl.py.

The service definition specifies that the startup script should be run as user postgres who exists in the container and has the home directory /var/lib/pgsql.

The startup script (/usr/lib/postgresql-init) apparently expects to be run as user postgres and, at one point, does the following:

cd ~

A few lines later, it attempts to initialise the DB (if that hasn't been done before) sending stdout to a file named initlog, in the local working directory.

That all works fine, when the script is run by regular systemd, but systemctl.py apparently sets up the environment differently, so that ~ does not expand to the postgres user's home, but rather to /root.

This leads to the following errors being logged by the startup script:

/usr/lib/postgresql-init: line 42: cd: /root: Permission denied
Initializing PostgreSQL 9.6.9 at location /var/lib/pgsql/data
/usr/lib/postgresql-init: line 52: initlog: Permission denied
Initialisation failed. See //initlog .

… and lets the initialisation (and thus the service start) fail.

I was able to work around this issue, by putting a drop-in unit file into /usr/lib/systemd/system/postgresql.service.d that explicitly sets

[Service]
Environment="HOME=/var/lib/pgsql"

… but I really consider this a workaround, rather than a fix.

Ideally, when systemctl.py runs a shell script as some user, the tilde character ~ should expand to that users home directory (as specified in /etc/passwd) if used within the script. If only, because that seems to be the way systemd does it and there's at least one startup script out there that relies on this behaviour.

One feature of systemd is to allow a systemd.socket file to be created in parallel with a .service file.

In other words, you might have

foo.service
foo.socket

In the same directory.

These .socket files are apparently unsupported by systemctl.py.

Note also that they can have the same %t that is currently not supported in .service files, as reported in issue #29.

Service with EnvironmentFile=/path/do-not-exist starts anyway and does not fail to start (when it should).
Note that EnvironmentFile=-/path/do-not-exist should allow service start when file is not found.

in ubuntu the path of runuser is /sbin/runuser not sure if there is a better way of handling that other than a hardcoded path

Detecting whether systemd is the active init system may be difficult if systemctl replacement is in the PATH.

What would be helpful is to implement systemctl is-system-running as proxy to the original systemctl command so when is-system-running is given we find original systemctl executable and call it.

It looks like the '.include' directive is not supported by the systemctl script. For example, I'm running into an issue where "/etc/systemd/system/postgresql-9.6.service" has a line at the top that looks like this:

.include /usr/lib/systemd/system/postgresql-10.service

When I try to use systemctl.py to start the service, it reports:

ERROR:systemctl:Unit postgresql-9.6.service could not be found.

I know that the service file is valid because this command:

systemd-analyze verify postgresql-9.6.service

Gives me an exit code of 0.

There is a use case with Kubernetes and other cloud orchestration tools where some startup variables should be overridden. That may be additional Environment= or EnvrionmentFile= parts to be interpreted after the ones being specified in the dot-service file.

The way to add parts in the command line is not necessarily the way to go but it seems easy to implement.

Thanks for this project. It is very helpfull for me.

The issues is

command> systemctl.py start mysql
Failed to get D-Bus connection: Operation not permitted
ERROR:systemctl:returncode 1
/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION"
Traceback (most recent call last):
File "/usr/bin/dctl", line 1612, in
result = command_func(*modules)
File "/usr/bin/dctl", line 854, in restart_of_units
elif not self.restart_unit(unit):
File "/usr/bin/dctl", line 862, in restart_unit
return self.restart_unit_from(conf)
File "/usr/bin/dctl", line 883, in restart_unit_from
self.start_unit_from(conf)
File "/usr/bin/dctl", line 627, in start_unit_from
subprocess_wait(cmd, env, check=check)
File "/usr/bin/dctl", line 264, in subprocess_wait
raise Exception("command failed")
Exception: command failed

docker env

• centos 7.0
• mariadb

Installed on CentOS7 = postgresql-server-9.2.23-3.el7_4.x86_64

Running /usr/bin/postgresql-setup initdb

ERROR: failed to find PGDATA setting in postgresql.service

Reason: this section from "postgresql-setup":

# this parsing technique fails for PGDATA pathnames containing spaces,
# but there's not much I can do about it given systemctl's output format...
PGDATA=`systemctl show -p Environment "${SERVICE_NAME}.service" |
                sed 's/^Environment=//' | tr ' ' '\n' |
                sed -n 's/^PGDATA=//p' | tail -n 1`
if [ x"$PGDATA" = x ]; then
    echo "failed to find PGDATA setting in ${SERVICE_NAME}.service"
    exit 1
fi

When doing a "docker run" then "docker logs" will show the output of the process in the container. The systemctl replacement however will divert all output into some /var/log area and one has to use additional tools to get to the output (it would be journalctl for a real systemd).

When the systemctl docker replacement is running as the init process then it may be possible to keep the output channels of the started commands connected to the standard out/err of the script. That would allow access the results with "docker logs". May be one can dup() the channels to have both, but may be some forking services can not stay connected.

Somehow systemctl is unable to start systemd-resolved.service on Debian 9:

ERROR:systemctl:no such service 'systemd-resolved.service'

Workaround is to cp -v /lib/systemd/system/systemd-resolved.service /etc/systemd/system/ but that should not be necessary...

While systemctl.py will be dead in most circumstances, it is in fact a continuous process when it is the PID 1 in a docker container. At that place it may be useful to not only check for zombies regularly but also for dead process that a restart=on-abort rule in their service config. With the help of a pid file one can check for the condition regularly and run "start" again if it was found dead. That can help so people do not try to write watchdog wrappers for the services file which the systemd world has already mitigated.

It would be great if you could clarify license and improve license grant as well.

It is best to refer to the chosen license by its standard SPDX name as well as to include URL of the full canonical text of the license (as below).

You have to provide full text of the license with your software as well.

Also it would be best to add LICENSE file to the repository root to make it clear that license applies not only to main executable but also to tests, documentation files, etc.

Current statement inside systemctl.py is ambiguous:

(CC-BY, GPL, BSD)

"CC-BY" is ambiguous. See how to apply CC license instructions. Basically you have to use standard attribution text (so license can be detected automatically, etc) and include link to full text of the license. Please note that Creative Commons (recommends to) use a versioned license and it is always best to use the latest version.

"GPL" is ambiguous. Please follow instructions. You have to include full text of the license grant:

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.

"BSD" is extremely ambiguous and perhaps it is not the best license anyway. You probably want to use either BSD-2-Clause or MIT a.k.a. Expat license (or even Apache-2.0 license).

Thank you.

Service files support ""%t" unit specifier that stands for the runtime directory."

This was throwing me for a bit with a service that had a value similar to

%t/foo/socket

in it. I had to change it to explicit

/run/foo/socket

In order to get systemctl.py to properly start the service, with the socket being created. Of course, /run/foo had to be created and permissions set first.

As an alternative to #14 where one will automatically restart the services inside a container:

There are cloud services like a kubernetes service which will watch over the status of the containers. If any container has stopped/failed then it will restart the container whereas actually restarting the service inside the container.

In order to cooperate with external docker container managers the "--init" function of systemctl.py should not loop infinitely. Instead if no controlled service is in the "running" status anymore then the PID-1 systemctl.py should exit. This might be just done after some timeout (waiting for the startup of a service) but atleast it should always be done. Even when not using an external docker container manager the status of the container gets more visible if "docker ps" shows it as having exited when all services in the container have exited one way or the other.

This is a followup to "exit init-process when all services have died" #18 which has been implemented like that the docker container may stop by default when all processes have died (apart from PID 1).

In reality one may even want to have the exitcode from the container being pushed through to the docker container server - because many simple containers would use a docker entrypoint script with an 'exec' so that the returncode can be actually be found easily. Most java applications will have an exitcode of 143 because that's what it does when it receives a sigterm from a docker-stop.

The order of shutdowns and exitcodes is not directly definied for a "systemctl halt". The order of service shutdowns may even depend on the "After" clauses. So there must be a definition of "required services" to make it work as intended - for example when a container is run as "systemctl init ".

There may be other variants that would need to be explored?

$ sudo /usr/local/bin/systemctl restart httpd
httpd[183]: [:error] [pid 183] AH00000: sd_notifyf returned an error -4
Traceback (most recent call last):
  File "/usr/local/bin/systemctl", line 2927, in <module>
    result = command_func(*modules)
  File "/usr/local/bin/systemctl", line 1612, in restart_modules
    return self.restart_units(units) and found_all
  File "/usr/local/bin/systemctl", line 1617, in restart_units
    if not self.restart_unit(unit):
  File "/usr/local/bin/systemctl", line 1626, in restart_unit
    return self.restart_unit_from(conf)
  File "/usr/local/bin/systemctl", line 1649, in restart_unit_from
    self.start_unit_from(conf)
  File "/usr/local/bin/systemctl", line 1301, in start_unit_from
    raise Exception("could not start service")
Exception: could not start service

Spotted on CentOS-7.

FYI @colin-go.

# systemctl start rsyslog
ERROR:systemctl:the service is already running on PID 109

Command exits with return value 1 which is wrong (this is not an error condition).
Note that systemd returns 0 in such case (same as SysV).

It may be possible to allow an explicit start of *.target unit by simply expanding the argument into the list of *.service units.

Surely one would need to also check for active units that are not in the target. Atleast for usage in a container that should well be sufficient.

In rkt container runtime /var/run/systemd/notify socket inside containers is used to proxy notifications to the host.

Unfortunately systemctl replacement tries to use path to notify socket that is identical to the one owned/controlled by host's systemd.

Naturally there is an error when systemctl replacement attempts to use someone else's notify socket which it did not create:

# systemctl start php7.2-fpm.service
Traceback (most recent call last):
  File "/usr/local/bin/systemctl", line 3913, in <module>
    result = command_func(*modules)
  File "/usr/local/bin/systemctl", line 1538, in start_modules
    return self.start_units(units, init) and found_all
  File "/usr/local/bin/systemctl", line 1545, in start_units
    if not self.start_unit(unit):
  File "/usr/local/bin/systemctl", line 1564, in start_unit
    return self.start_unit_from(conf)
  File "/usr/local/bin/systemctl", line 1680, in start_unit_from
    notify = self.notify_socket_from(conf)
  File "/usr/local/bin/systemctl", line 1464, in notify_socket_from
    os.unlink(socketfile)
OSError: [Errno 16] Device or resource busy: '/var/run/systemd/notify'

Renaming _notify_socket_name to avoid clash fixes the problem.

unmask fails when I use it with services placed directly into /etc/systemd/system/, eg.

root@bd1:/etc/puppet# systemctl unmask kafka
ERROR:systemctl:target is not a symlink: /etc/systemd/system/kafka.service

In my case it fails with Puppet service resource, because when starting a service it unconditionally unmasks it.

It should do nothing and return successfully, like real systemd:

$ systemctl --version
systemd 229
$ ls -la /etc/systemd/system/kibana.service
-rw-r--r-- 1 root root 466 Jun 30 00:08 /etc/systemd/system/kibana.service
$ sudo systemctl unmask kibana
$ echo $?
0

There is a section in the systemd documentation that list LSB exit codes that should be followed. I know that quite some of the result values do not quite match but it would be better to follow style when having not checked what the systemd systemctl command returns.

https://www.freedesktop.org/software/systemd/man/systemd.exec.html

• 0 EXIT_SUCCESS Generic success code.
• 1 EXIT_FAILURE Generic failure or unspecified error.
• 2 EXIT_INVALIDARGUMENT Invalid or excess arguments.
• 3 EXIT_NOTIMPLEMENTED Unimplemented feature.
• 4 EXIT_NOPERMISSION The user has insufficient privileges.
• 5 EXIT_NOTINSTALLED The program is not installed.
• 6 EXIT_NOTCONFIGURED The program is not configured.
• 7 EXIT_NOTRUNNING The program is not running.

plus exit codes 200... 214

Some testcases on "ubuntu:16.04" did show an a different result than "ubuntu:18.04" and all the other distros.

I have tracked it down that python's os.waitpid() returns an exitcode==0 even when the
underlying process has actually failed with an exitcode<>0. It is unknown where that
bug comes from but it seems a bit serious to trash some very basic unix functionality.
Essentially a parent process does not get the correct exitcode from its own children.

P.S. after creating a testprogram it seems that os.waitpid is fine, but /bin/kill is broken

There is a phenomenon that current rules for Redhat Openshift require that a "USER" setting is to be present on on all images, so that implicitly all the processes in a container do not start as root. I don't know the reason for it, may be it is for the many containers running in --priviledged mode? As a matter of fact, there had been arguments with maintainers of Openshift containers that do not want to use systemctl.py.

As a consequence one should check a mode where systemctl.py should be to run as PID 1 without expecting write-access to the usual /var areas, but it should instead run as a specific --user process. That init-proc can still run (and only run) those services which have a matching User entry in the *.service descriptor. It might also be useful in other situations, doesn't it.

In one particular situation there is a collabnet.service that wants to run more "systemctl start" routines on other services. Upon a container start, both the collabnet.service and the other services are run by PID 1 as they are all enabled. As a result there may be two processes of the same service being run parallel to startup.

Now this situation has some obvious problems attached to it. Not even about possibly destroying data, but I noticed it first by having PIDFile knowing only the last process that has started, so a shutdown of the processes will not be complete when the container needs to shut down. A "systemctl halt" will leave some processes running. And that's a general problem with other circumenstances where a "start" call is issused in parallel from different sides.