Skip to main content

Journalctl Deep Dive: Debugging Linux with systemd Logs

·630 words·3 mins
linux systemd journalctl logging DevOps Cloud
Author
Emre Hayta - System Engineer
Table of Contents

Introduction
#

Modern Linux systems rely heavily on systemd, and with it comes a powerful logging system: journald. At the center of this ecosystem is journalctl, a command-line tool that allows sysadmins and DevOps engineers to efficiently query, filter, and analyze logs.

This article is a hands-on deep dive into journalctl, focusing on real-world debugging and operational use cases.


Understanding journald Basics
#

Unlike traditional text-based logs in /var/log, journald stores logs in a structured, indexed binary format. This enables:

  • Fast searching and filtering
  • Rich metadata (PID, UID, unit, boot ID, etc.)
  • Centralized logging for services, kernel, and user sessions

To view all logs:

journalctl

To follow logs in real time (like tail -f):

journalctl -f

Time-Based Log Filtering
#

Time filtering is one of the most common and powerful features.

Logs Since a Specific Time
#

journalctl --since "2026-01-23 09:00:00"

Logs Until a Specific Time
#

journalctl --until "2026-01-23 11:30:00"

Relative Time Filters
#

journalctl --since "1 hour ago"
journalctl --since "yesterday"
journalctl --since "2026-01-22"

This is extremely useful when debugging incidents within a known time window.


Filtering by Priority (Severity)
#

Systemd uses syslog-compatible priority levels:

LevelMeaning
0Emergency
1Alert
2Critical
3Error
4Warning
5Notice
6Info
7Debug

Show Errors Only
#

journalctl -p err

Show Warnings and Above
#

journalctl -p warning

Priority Within a Time Range
#

journalctl -p err --since "30 min ago"

Debugging Services and systemd Units
#

One of journalctl’s biggest advantages is native systemd unit integration.

Logs for a Specific Service
#

journalctl -u nginx.service

Follow a Service Log in Real Time
#

journalctl -u docker.service -f

Logs from Last Service Restart
#

journalctl -u ssh.service --since "5 min ago"

Combined Filters
#

journalctl -u postgresql.service -p err --since today

This is often enough to diagnose failed starts, crashes, or misconfigurations.


Kernel Logs with journalctl
#

journalctl also replaces dmesg in many scenarios.

View Kernel Logs
#

journalctl -k

Kernel Logs Since Boot
#

journalctl -k -b

Kernel Errors Only
#

journalctl -k -p err

This is particularly helpful when troubleshooting hardware issues, drivers, networking, or filesystems.


Working with Boot Sessions
#

Each system boot has a unique Boot ID. This is invaluable when diagnosing reboot-related issues.

List All Boots
#

journalctl --list-boots

Example output:

-2  9c1f2b2a6a2f4c0a9a7d...
-1  a3e91b1f3fdd4a2b8d8c...
 0  f7c21b8e99c245f1b62e...

Logs from Previous Boot
#

journalctl -b -1

Logs from a Specific Boot
#

journalctl -b f7c21b8e99c245f1b62e

Perfect for analyzing crashes, kernel panics, or failed startup services.


Advanced Output and Formatting
#

JSON Output (Great for Automation)
#

journalctl -o json

Pretty JSON
#

journalctl -o json-pretty

Short and Minimal Output
#

journalctl -o short-iso

Log Maintenance and Disk Usage
#

Journald manages log rotation automatically, but it’s important to keep an eye on disk usage.

Check Journal Disk Usage
#

journalctl --disk-usage

Vacuum Logs Older Than 7 Days
#

journalctl --vacuum-time=7d

Limit Journal Size to 1 GB
#

journalctl --vacuum-size=1G

For persistent logs across reboots, ensure:

/var/log/journal

exists and journald is configured accordingly.


Practical Debugging Workflow (Real World)
#

A typical production incident workflow might look like:

journalctl --since "10 min ago" -p err
journalctl -u app.service --since "10 min ago"
journalctl -k -p warning
journalctl -b -1

Fast, structured, and reliable — even on heavily loaded systems.


Final Thoughts
#

journalctl is far more than a log viewer — it’s a core debugging and observability tool for modern Linux systems. Mastering it will drastically reduce your mean time to resolution (MTTR) in production environments.

If you work with Linux in cloud, containerized, or bare-metal environments, journalctl should be second nature.


Need Help with Linux Troubleshooting?
#

If you need professional support with Linux troubleshooting, systemd debugging, or cloud operations,
check out https://techz.at — we help teams keep production systems stable, secure, and observable.