systemd tmpfiles unit intermittently fails on boot

[vance]

Approximately half the time on boot, the systemd unit which is responsible for creating directories under /run, fails. This can cause services which rely on those directories to fail. Typically mailman is the service affected, though not always.

$ systemctl --state=failed | cat
  UNIT                           LOAD   ACTIVE SUB    DESCRIPTION
* mailman.service                loaded failed failed GNU Mailing List Manager
* systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

I have since rebooted the machine so that it is running correctly. After that, no more details are available in the logs that I can find; perhaps more information can be obtained from systemd while the failure condition exists.

[vance]

The cause of Mailman failing seems to be when the /var/lock/mailman directory is not created.

$ journalctl -a -n 30 -u systemd-tmpfiles-setup.service | cat
-- Logs begin at Sun 2017-05-28 02:17:33 EDT, end at Sun 2017-05-28 02:22:37 EDT. --
May 28 02:17:33 linode.wplug.org systemd[1]: Starting Create Volatile Files and Directories...
May 28 02:17:33 linode.wplug.org systemd-tmpfiles[2250]: Failed to create directory or subvolume "/var/lock/mailman": No such file or directory
May 28 02:17:33 linode.wplug.org systemd-tmpfiles[2250]: Failed to create directory or subvolume "/var/lock/ppp": No such file or directory
May 28 02:17:33 linode.wplug.org systemd[1]: systemd-tmpfiles-setup.service: main process exited, code=exited, status=1/FAILURE
May 28 02:17:33 linode.wplug.org systemd[1]: Failed to start Create Volatile Files and Directories.
May 28 02:17:33 linode.wplug.org systemd[1]: Unit systemd-tmpfiles-setup.service entered failed state.
May 28 02:17:33 linode.wplug.org systemd[1]: systemd-tmpfiles-setup.service failed.
$ journalctl -a -n 30 -u mailman.service | cat
-- Logs begin at Sun 2017-05-28 02:17:33 EDT, end at Sun 2017-05-28 02:22:37 EDT. --
May 28 02:17:40 linode.wplug.org systemd[1]: Starting GNU Mailing List Manager...
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: Traceback (most recent call last):
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: File "/usr/lib/mailman/bin/mailmanctl", line 614, in <module>
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: main()
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: File "/usr/lib/mailman/bin/mailmanctl", line 449, in main
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: lock = acquire_lock(force)
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: File "/usr/lib/mailman/bin/mailmanctl", line 263, in acquire_lock
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: lock = acquire_lock_1(force)
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: File "/usr/lib/mailman/bin/mailmanctl", line 248, in acquire_lock_1
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: lock.lock(0.1)
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: File "/usr/lib/mailman/Mailman/LockFile.py", line 243, in lock
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: self.__write()
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: File "/usr/lib/mailman/Mailman/LockFile.py", line 422, in __write
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: fp = open(self.__tmpfname, 'w')
May 28 02:17:46 linode.wplug.org mailmanctl[3602]: IOError: [Errno 2] No such file or directory: '/var/lock/mailman/master-qrunner.linode.wplug.org.3602.1'
May 28 02:17:46 linode.wplug.org systemd[1]: mailman.service: control process exited, code=exited status=1
May 28 02:17:46 linode.wplug.org systemd[1]: Failed to start GNU Mailing List Manager.
May 28 02:17:46 linode.wplug.org systemd[1]: Unit mailman.service entered failed state.
May 28 02:17:46 linode.wplug.org systemd[1]: mailman.service failed.

[vance]

This appears to no longer occur. At least from the start of 2020 the server has not required multiple reboots until all services start properly. I don't know why; perhaps a bugfix from CentOS has resolved it. Marking as closed - a new ticket can be opened if it occurs again.