Quick Links

Key Takeaways

  • Linux system logging changed with the introduction of systemd, which centralizes and manages system, boot, and kernel log files in a binary format.
  • The journalctl command is used to read and filter system log messages, allowing users to navigate and search through logs.
  • Users can customize the display format of journalctl output, such as using different output formats like short, json, verbose, or cat, to focus on specific log details.

Linux system logging changed with the introduction of systemd. Learn how to use the journalctl command to read and filter system log messages.

Centralized Logging

No stranger to controversy, the systemd system and service manager introduced a significant change in the way system logs are gathered. Logs used to be located at different places in the file system according to the service or daemon that was creating them. But they all had one thing in common. They were plain text files.

With systemd all the system, boot, and kernel log files are collected and managed by a central, dedicated logging solution. The format they are stored in is a binary one. One thing this facilitates is being able to extract the data in different formats, such as JSON, as we shall see. It can also make it easier to cross-reference related information that would have previously been recorded in separate log files. Because the data is now held in a single journal, the data from several sources of interest can be selected and displayed in a single interwoven list of entries.

journalctl is the tool used to work with the journal.

journalctl With No Frills

You can invoke journalctl with no command line parameters:

journalctl

journalctl in a terminal window

journalctl displays the entire journal, with the oldest entries at the top of the list. The list is displayed in less, allowing you to page and search using the usual navigation features of less. You can also use the Left Arrow and Right Arrow keys to scroll sideways to read wide log entries.

Output from journalctl in a terminal window

Pressing the End key will hop straight to the bottom of the list, and the newest log entries.

Output from journalctl in a terminal window showing the newest entries

Press Ctrl+C to exit.

Although journalctl can be called without using sudo, you will ensure you see all the detail within the log if you do use sudo.

sudo journalctl

sudo journalctl in a terminal window

If you need to, you can make journalctl send its output to the terminal window instead of to less, by using the --no-pager option.

sudo journalctl --no-pager

sudo journalctl --no-pager in a terminal window

The output scrolls quickly through the terminal window, and you are returned to the command prompt.

output from sudo journalctl --no-pager in a terminal window

To limit the number of lines that journalctl returns, use the -n (lines) option. Let's ask for ten lines of output:

sudo journalctl -n 10

sudo journalctl -n 10 in a terminal window

Following Journal Updates

To make journalctl display the newest entries as they arrive in the journal, use the -f (follow) option.

sudo journalctl -f

sudo journalctl -f in a terminal window
output from sudo journalctl -f in a terminal window

The newest entry has a timestamp of 07:09:07. As new activity takes place, the new entries are appended to the bottom of the display. Near real-time updates---cool!

output from sudo journalctl -f in a terminal window with new entries

At 07:09:59 an application called geek-app injected a log entry into the journal that said, "New Message from HTG."

Changing the Display Format

Because the journal is a binary file, the data in it needs to be translated or parsed into text before it can be displayed to you. With different parsers, different output formats can be created from the same binary source data. There are several different formats that journalctl can use.

The default output is the short format, which is very similar to the classic system log format. To explicitly request the short format, use the -o (output) option with the short modifier.

sudo journalctl -n 10 -o short-full

sudo journalctl -n 10 -o short in a terminal window

From left to right, the fields are:

  • The time the message was created, in local time.
  • The hostname.
  • The process name. This is the process that generated the message.
  • The log message.

To obtain a complete date and time stamp use the short-full modifier:

sudo journalctl -n 10 -o short-full

sudo journalctl -n 10 -o short-full in a terminal window

The date and time formats in this output are the format in which you need to provide dates and times when you are selecting log messages by period, as we shall see shortly.

To see all the metadata that accompanies each log message, use the verbose modifier.

sudo journalctl -n 10 -o verbose

sudo journalctl -n 10 -o verbose in a terminal window

There are many possible fields, but it is rare for all fields to be present in a message.

Output from sudo journalctl -n 10 -o verbose in a terminal window

One field worth discussing is the Priority field. In this example, it has a value of 6. The value represents the importance of the message:

  • 0: Emergency. The system is unusable.
  • 1: Alert. A condition has been flagged that should be corrected immediately.
  • 2: Critical. This covers crashes, coredumps, and significant failures in primary applications.
  • 3: Error. An error has been reported, but it is not considered severe.
  • 4: Warning. Brings a condition to your attention that, if ignored, may become an error.
  • 5: Notice. Used to report events that are unusual, but not errors.
  • 6: Information. Regular operational messages. These do not require action.
  • 7: Debug. Messages put into applications to make it easier for them to debug them.

If you want the output to be presented as properly formed JavaScript Object Notation (JSON) objects, use the json modifier:

sudo journalctl -n 10 -o json

sudo journalctl -n 10 -o json in a terminal window

Each message is properly wrapped as a well-formed JSON object, and displayed one message per line of output.

To have the JSON output pretty-printed, use the json-pretty modifier.

sudo journalctl -n 10 -o json-pretty

sudo journalctl -n 10 -o json-pretty in a terminal window

Each JSON object is split across multiple lines, with each name-value pair on a new line.

output from sudo journalctl -n 10 -o json-pretty in a terminal window

To only see the log entry messages, without time stamps or other metadata, use the cat modifier:

sudo journalctl -n 10 -o cat

sudo journalctl -n 10 -o cat in a terminal window

This display format can make it difficult to identify which process raised the log event, although some messages do contain a clue.

Output from sudo journalctl -n 10 -o cat in a terminal window

Selecting Log Messages By Time Period

To limit the output from journalctl to a time period you're interested in, use the -S (since) and -U (until) options.

To see the log entries since a particular time and date, use this command:

sudo journalctl -S "2020-91-12 07:00:00"

sudo journalctl -S "2020-91-12 07:00:00" in a terminal window

The display contains only messages that arrived after the date and time in the command.

Output from sudo journalctl -S "2020-91-12 07:00:00" in a terminal window

To define a time period you wish to report on, use both the -S (since) and -U (until) options together. This command looks at log messages from a 15 minute time period.:

sudo journalctl -S "2020-91-12 07:00:00" -U "2020-91-12 07:15:00"

sudo journalctl -S "2020-91-12 07:00:00" -U "2020-91-12 07:15:00" in a terminal window

This is a great combination use if you know something odd happened on your system, and roughly when it happened.

output from sudo journalctl -S "2020-91-12 07:00:00" -U "2020-91-12 07:15:00" in a terminal window

Using Relative Time Periods

You can use relative addressing when you select your time periods. That means you can say things like "show me all events from one day ago up until now." This is just what this command means. The "d" stands for "day", and the "-1" means one day in the past.

sudo journalctl -S -1d

sudo journalctl -S -1d in a terminal window

The log messages are listed from 00:00:00 yesterday, up until "now."

Output from sudo journalctl -S -1d in a terminal window

If you want to investigate something that happened in the recent past, you can specify a relative time period measured in hours. Here we're reviewing log messages from the last hour:

sudo journalctl -S -1h

sudo journalctl -S -1h in a terminal window

The messages from the last hour are displayed for you. You can also use "m" to set relative time periods measured in minutes, and "w" for weeks.

Output from sudo journalctl -S -1h in a terminal window

journalctl understands today, yesterday, and tomorrow. These modifiers provide a handy way to specify common time periods. To see all the events that happened yesterday, use this command:

sudo journalctl -S yesterday

sudo journalctl - S yesterday in a terminal window

All journal log events that happened yesterday, up to midnight 00:00:00, are retrieved and displayed for you.

Output from sudo journalctl - S yesterday in a terminal window

To see all the log messages received today so far, use this command:

sudo journalctl -S today

sudo journalctl -S today in a terminal window

Everything from 00:00:00 up until the time the command is issued, are displayed.

Output from sudo journalctl -S today in a terminal window

You're able to mix the different time period modifiers. To see everything from two days ago up until the start of today, use this command:

sudo journalctl -S -2d -U today

sudo journalctl -S -2d -U today in a terminal window

Everything since the day before yesterday up until today is retrieved and displayed.

Output from sudo journalctl -S -2d -U today in a terminal window

Selecting Log Messages By Data Fields

You can search for log messages that match a wide range of journal fields. These searches try to find matches in the metadata attached to each message. It is recommended that you refer to the list of fields and choose the ones that will be most useful to you.

Bear in mind, whether an application completes every field or not is entirely up to the authors of the application. You can't guarantee every field will be populated.

All of the journal field modifiers are used in the same way. We'll use a few in our examples below. To look for log messages from a specific application, use the _COMM (command) modifier. If you also use the -f (follow) option, journalctl will track new messages from this application as they arrive.

sudo journalctl -f _COMM=geek-app

sudo journalctl -f _COMM=geek-app in a terminal window

You can search for log entries using the process ID of the process that generated the log message. Use the ps command to find the process id of the daemon or application you're going to search for.

sudo journalctl _PID=751

sudo journalctl _PID=751 in a terminal window

On the machine used to research this article, the SSH daemon is process 751.

output from sudo journalctl _PID=751 in a terminal window

You can also search by user Id. This is the user ID fo the person who launched the application or command, or who owns the process.

sudo journalctl _UID=1000

sudo journalctl _UID=1000 in a terminal window

All messages associated with any other user ID's are filtered out. Only messages related to user 1000 are shown:

output from sudo journalctl _UID=1000 in a terminal window

Another way to search for log messages related to a specific application is to provide the path to the executable.

sudo journalctl /usr/bin/anacron

sudo journalctl /usr/bin/anacron in a terminal window

All of the anacron scheduler log messages are retrieved and displayed.

Output from sudo journalctl /usr/bin/anacron in a terminal window

To make searching easier, we can ask journalctl to list all the values it holds, for any of the journal fields.

To see the user ID's that journalctl has recorded log messages for, use the -F (fields) option, and pass the _UID field identifier.

journalctl -F _UID

journalctl -F _UID in a terminal window

Let's do that again and look at the group IDs (GID's):

journalctl -F _GID

journalctl -F _GID in a terminal window

You can do this with any of the journal field identifiers.

Listing Kernel Messages

There's a built-in way to isolate kernel messages quickly. You don't need to search and isolate them yourself. The -k (kernel) option removes all other messages and gives you an instant view of the kernel log entries.

sudo journalctl -k

sudo journalctl -k in a terminal window

The highlighting reflects the importance of the message, according to the values in the Priority field.

Output from sudo journalctl -k in a terminal window

Reviewing Boot Messages

If you've got an issue related to booting that you wish to investigate, journalctl has you covered. Perhaps you've added new hardware, and it isn't responding, or a previously working hardware component no longer works after your last system upgrade.

To see the log entries related to your last boot, use the -b (boot) option:

journalctl -b

journalctl -b in a terminal window

The log entries for the last boot are shown for you.

Output from journalctl -b in a terminal window

When we say "last boot," we mean the boot process that brought your computer to life for your current logged-in session. To see previous boots, you can use a number to tell journalctl which boot you're interested in. To see the third previous boot, use this command:

journalctl -b 3

journalctl -b 3 in a terminal window

Generally, if you've had a problem and had to reboot your machine, it is a previous boot sequence you're interested in. So this is a common command form.

Output from journalctl -b 3 in a terminal window

It is easy to get mixed up with the sequence of boots. To help, we can ask journalctl to list the boots that it has recorded in its journal, using the --list-boots option.

journalctl --list-boots

journalctl --list-boots in a terminal window

You can identify the boot you wish to see messages for from the date and time stamp, and then use the number in the left-hand column to obtain the log messages for that boot sequence. You can also pick the 32-bit boot identifier, and pass that to journalctl.

sudo journalctl -b 1f00248226ed4ab9a1abac86e0d540d7

sudo journalctl -b 1f00248226ed4ab9a1abac86e0d540d7 in a terminal window

The log messages from the boot sequence we requested are retrieved and displayed.

output from sudo journalctl -b 1f00248226ed4ab9a1abac86e0d540d7 in a terminal window

Managing Journal Hard Drive Space

Of course, the journal and all of its log messages are stored on your hard drive. That means they'll be taking up hard drive space. To see how much space has been taken by the journal, use the --disk-usage option.

journalctl --disk-usage

journalctl --disk-usage in a terminal window

With today's hard drives, 152 MB isn't much space at all, but for demonstration purposes, we'll still trim it back. There's two way we can do this. The first is to set a size limit that you want the journal reduced back to. It'll grow again, of course, but we can prune it now ready for that new growth.

We'll use the wonderfully titled --vacuum-size option, and pass in the size we'd like the journal reduced to. We'll ask for 100 MB. The way to think of this is we're asking journalctl to "throw away whatever you can, but don't go lower than 100 MB."

journalctl --vacuum-size=100M

journalctl --vacuum-size=100M in a terminal window

The other way to trim back the journal size is to use the --vacuum-time option. This option tells journalctl to discard messages that are older than the period you provide on the command line. You can use days, weeks, months, and years in the time period.

Let's weed out all messages older than one week:

journalctl --vacuum-time=1weeks

journalctl --vacuum-time=1weeks in a terminal window

Data vs. Information

Data isn't useful unless you can get at it and make use of it. Then it becomes useful information. The journalctl command is a flexible and sophisticated tool that allows you to get to the information of interest in a variety of ways.

You can use just about any snippet of information you have to home in on the log messages you need.

Linux Commands

Files

tar · pv · cat · tac · chmod · grep · diff · sed · ar · man · pushd · popd · fsck · testdisk · seq · fd · pandoc · cd · $PATH · awk · join · jq · fold · uniq · journalctl · tail · stat · ls · fstab · echo · less · chgrp · chown · rev · look · strings · type · rename · zip · unzip · mount · umount · install · fdisk · mkfs · rm · rmdir · rsync · df · gpg · vi · nano · mkdir · du · ln · patch · convert · rclone · shred · srm · scp · gzip · chattr · cut · find · umask · wc · tr

Processes

alias · screen · top · nice · renice · progress · strace · systemd · tmux · chsh · history · at · batch · free · which · dmesg · chfn · usermod · ps · chroot · xargs · tty · pinky · lsof · vmstat · timeout · wall · yes · kill · sleep · sudo · su · time · groupadd · usermod · groups · lshw · shutdown · reboot · halt · poweroff · passwd · lscpu · crontab · date · bg · fg · pidof · nohup · pmap

Networking

netstat · ping · traceroute · ip · ss · whois · fail2ban · bmon · dig · finger · nmap · ftp · curl · wget · who · whoami · w · iptables · ssh-keygen · ufw · arping · firewalld