Managing services with systemd: systemctl and unit files

On almost every modern distribution the programs that work in the background, such as the web server, the SSH daemon or the printing service, are managed by systemd. The command you use to talk to systemd is systemctl. A dozen subcommands cover most everyday work, and understanding how units are organised saves you from editing files that the next update will overwrite.

Units: the basic concept

systemd thinks in terms of units. Each unit is described by a text file and has a type, shown by its extension:

• .service: a program to start and keep an eye on;
• .timer: a scheduled activation of another service;
• .mount: a mount point;
• .socket: a port or socket that starts the service on the first connection;
• .target: a group of units representing a state of the system.

When you type systemctl status ssh, the .service extension is implied.

Starting, stopping, restarting

The basic commands act immediately but do not survive a reboot:

sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx
sudo systemctl reload nginx

The difference between restart and reload matters on a production server. restart stops the process and starts it again, dropping connections in progress. reload asks the program to re-read its configuration without stopping, but it only works if the service supports it. For nginx, after changing the configuration, it is wise to check it first with nginx -t and then use reload.

Enabling at boot

Whether a service starts on its own at power-on is a separate matter from whether it is running right now. That is why there are separate commands:

sudo systemctl enable nginx
sudo systemctl disable nginx
sudo systemctl enable --now nginx

enable --now does both at once: it enables the service at boot and starts it immediately. There is also mask, which is more drastic: it links the unit to /dev/null and prevents it from being started in any way, even as a dependency of another unit. It is undone with unmask.

Reading the status

systemctl status name is the first command to run when something is wrong. It shows:

• Loaded: where the unit file is and whether it is enabled (enabled, disabled, masked);
• Active: the current state, for example active (running), inactive (dead) or failed, with the time of the last change;
• the main PID and the memory in use;
• the last lines of the service log.

For an overview, systemctl --failed lists only the failed units, and systemctl list-units --type=service shows every loaded service. The complete log lines are read with journalctl -u nginx.

Where unit files live

The golden rule: do not edit the files in /usr/lib. The next package update would replace them. To change a setting you use:

sudo systemctl edit nginx

The command creates an override file in /etc/systemd/system/nginx.service.d/ containing only the lines you want to change. To see the end result, the original file plus your changes, there is systemctl cat nginx.

Writing a simple service

Suppose you have a Python script that must always be running, for example a small bot watching a folder. A file /etc/systemd/system/watcher.service could look like this:

[Unit]
Description=Incoming folder watcher
After=network-online.target

[Service]
User=app
WorkingDirectory=/opt/watcher
ExecStart=/usr/bin/python3 /opt/watcher/main.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Each section has a precise role. [Unit] describes the service and its dependencies. [Service] says how to run it: as which user, from which folder, with which command, and what to do if it exits with an error. [Install] states which target to hook it into when you enable it.

After creating or changing a unit file you have to tell systemd:

sudo systemctl daemon-reload
sudo systemctl enable --now watcher

Two habits that, in my work, solve half the problems: always use absolute paths in ExecStart, because systemd does not use your shell's PATH, and run the service as a dedicated user rather than root.

Common mistakes

• Forgetting daemon-reload: systemd keeps using the previous version of the file and flags it with a warning in status.
• A service that starts and exits straight away: often the program puts itself in the background. In that case you need Type=forking, or better, run it in «foreground» mode if the program allows it.
• Restart loops: with Restart=always and a configuration error the service enters a loop; systemd stops it after a few rapid attempts. The real reason is in journalctl -u name.
• Missing environment variables: those set in .bashrc do not apply to services. They are declared with Environment= or EnvironmentFile=.

In short

• start, stop and restart act now; enable and disable decide what happens at boot.
• systemctl status and journalctl -u are the first tools for understanding a problem.
• Changes are made with systemctl edit, never in the files under /usr/lib.
• Every change to a unit file needs systemctl daemon-reload.
• A well-written service uses absolute paths, a dedicated user and a restart policy.