racy use of shared file about console-login-helper-messages HOT 3 CLOSED

Could this possibly be related? I've see this failing together with systemd-udev-settle.service several times:

Red Hat Enterprise Linux CoreOS 43.81.202002032142.0
WARNING: Direct SSH access to machines is not recommended.

---
Last login: ...
[systemd]
Failed Units: 2
  console-login-helper-messages-issuegen.service
  systemd-udev-settle.service

See also https://bugzilla.redhat.com/show_bug.cgi?id=1739757

from console-login-helper-messages.

I've checked the logs, and it seems when this unit fails "udevadm settle" already failed a few seconds before. This unit fails after udev fails. So they are somehow related, I do not know if this is an additional failure or if this triggers the race on the temp file.

from console-login-helper-messages.

Thank you for reporting!

The problem looks like the race between the two invocations you mentioned - the part that writes to and deletes the .staged file is executed every time the udev rule invokes issuegen. It is possible that the udev invocation and console-login-helper-messages.service, which is run on boot, execute at the same time and one of the two deletes the file before the other executes the cat command to write the staged file to the generated file.

There is in fact no need to have the staged file - I shouldn't have added it in the first place in rfairley@d8ef47a. I have opened #36 to fix this, with more explanation there. This should fix the error causing console-login-helper-messages-issuegen.service to fail.

I've checked the logs, and it seems when this unit fails "udevadm settle" already failed a few seconds before. This unit fails after udev fails. So they are somehow related, I do not know if this is an additional failure or if this triggers the race on the temp file.

Could there be implicit ordering between systemd-udev-settle.service and console-login-helper-messages-issuegen.service that would make the likelihood of the race occurring higher if "udev settle" failed? E.g. console-login-helper-messages-issuegen.service waits for network-online.target. Thinking if the "udev settle" service failed for an unrelated reason after invoking /usr/lib/udev/rules.d/90-console-login-helper-messages-issuegen.rules, but terminated earlier than usual, console-login-helper-messages-issuegen.service could be started while the rules invocation is still executing. I can try to verify this by checking the unit ordering - however, the immediate issue should be fixed now by #36.

Given the timeline and the Generate /run/issue.d/console-login-helper-messages.issue message, it does seem to be only console-login-helper-messages-issuegen.service failing from a race triggered by the udev failure condition seen so far.

Will do another release once the fix is in and will build the RPM for RHCOS.

from console-login-helper-messages.