Skip to content

Central logfiles

Logfiles from all servers are streamed to the central log server using the systemd services systemd-journal-remote.service on the server side and systemd-journal-upload.service on the clients side.

After trying a number of central logfile solutions, this turned out to be the most suckless option.

You are welcome to configure a graphical / WebUI log management system on the log server - Blunix will be happy to help unless it includes elasticsearch.

How logs are streamed to the central log server

All logs are streamed in real time over the wireguard mesh to the central log server cus-util-prod-log-1 on port 19532 tcp. The transmission protocol is http.

The option to enable ssl encryption (httpS) for the connection is not used, as the systemd service does not run as root and hence has no access to the default letsencrypt certificate (/etc/letsencrypt/live/{{ inventory_hostname }}.{{ internal_private_domain }}) installed on all servers. The connection is however of course already encrypted due to using the wireguard mesh.

Viewing and live-streaming logs

Login to cus-util-prod-log-1 via ssh, for example with cake: cake ssh log

Logfiles of all hosts can be viewed and streamed live using the journalctl cli tool.

All logfiles are saved in the file /var/log/journal/remote/all.journal.

Man page of journalctl

man journalctl

Streaming all logs from all servers live as they come in

journalctl --file /var/log/journal/remote/all.journal -f

Filtering logfiles

Each log line is split up into components called "fields".

List avaialble output fields

journalctl --fields

Description of output fields

man systemd.journal-fields

View examples of output fields

journalctl --file /var/log/journal/remote/all.journal -n 10 -o verbose

View only specific fields

journalctl --file /var/log/journal/remote/all.journal -o cat --output-fields='_HOSTNAME,MESSAGE'

Grep production webservers nginx logs for specific strings

journalctl --file /var/log/journal/remote/all.journal -f _HOSTNAME=cus-www-prod-web-1 _HOSTNAME=cus-www-prod-web-2 _COMM=nginx --grep='404'

Matching specific fields

Fields can be given as arguments to journalctl.

View logs of a specific host

journalctl --file /var/log/journal/remote/all.journal -f _HOSTNAME=cus-util-prod-git-1

View logs of multiple hosts

journalctl --file /var/log/journal/remote/all.journal -f _HOSTNAME=cus-www-prod-web-1 _HOSTNAME=cus-www-prod-web-2

Filter by systemd service unit name

journalctl --file /var/log/journal/remote/all.journal -f _COMM=sshd
journalctl --file /var/log/journal/remote/all.journal -f _COMM=nginx

Viewing logs from a specific time period

systemd-journal-remote has a feature that can define the file size of /var/log/journal/remote/all.journal before it is "rotated". However this feature is not in debian 11 at the time of this writing.

This is documented here:

Because of this, it is rather annoying to find a journal file from a specific timeframe. Additionally, the naming scheme for the rotated files is not human readable (and I did not find documentation as to how the files are named).

Hence, we recommend to look the the filesystem timestamps to select the right file:

root@cus-util-prod-log-1 ~ # ls -tlah /var/log/journal/remote/
total 6,6G
-rw-r-----  1 systemd-journal-remote systemd-journal-remote  56M 22. Feb 18:13 all.journal
-rw-r-----  1 systemd-journal-remote systemd-journal-remote  88M 20. Feb 10:53 all@c5fd66026f8b4134a83d07b8174ee8ce-00000000005ec084-0005f52e9513970a.journal

When you have found the right file, you can select a timerange like so:

journalctl --file /var/log/journal/remote/all@c5fd66026f8b4134a83d07b8174ee8ce-00000000005ec084-0005f52e9513970a.journal --since="2023-20-02 08:00:00" --until="2020-20-02 08:30:00"

You can also use relative time periods (like "since one hour ago")

journalctl --file /var/log/journal/remote/all.journal --since=-1h
journalctl --file /var/log/journal/remote/all.journal --since=today

Viewing logfiles in production

journalctl is rather complex - we recommend to:

  • read the manual page carefully (man journalctl)
  • create bash scripts within cus-util-prod-log-1:/root/log-scripts/ for longer journalctl commands that are often used


There is currently no proper solution within systemd-journal-remote to generate alerts / emails / similar on specific log strings.

You can however setup a WebUI central log solution for that. Most all of them accept input from journalctl.


are documented and collected at

Please notify us if you find any additional problems.

Reasons we implemented it regardless of the problems

  • no elasticsarch
  • reliable
  • streams old logs if any component (server or client) is down temporarily
  • logs are signed on the server to detect tempering
  • journalctl search and filtering is quite powerful