8.4 9 Lab Configure Logging On Linux

8.4.9 Lab: Configure Logging on Linux

In the world of Linux system administration, logging is your primary window into the machine's soul. It’s the continuous, silent record of everything happening—from kernel panics and failed login attempts to the successful start of a critical service. Mastering how to configure, manage, and interpret these logs transforms you from a reactive troubleshooter into a proactive guardian of your systems. This lab guides you through the essential steps to configure logging on a modern Linux system, focusing on the two dominant logging frameworks: the traditional syslog protocol (typically via rsyslog) and the modern systemd-journald. By the end, you will be able to direct log messages, set appropriate retention policies, and harness these tools for security auditing, debugging, and compliance.

Understanding the Linux Logging Ecosystem

Before diving into configuration, it’s crucial to understand the two main players. Historically, syslog (defined by RFC 5424) was the standard. The most common implementation is rsyslog, a high-performance, feature-rich syslog daemon. It receives messages from applications and the kernel, filters them based on facility (e.g., auth, cron, kern) and severity (e.g., debug, info, err, crit), and writes them to specified files (like /var/log/auth.log or /var/log/syslog).

The rise of systemd brought systemd-journald. This service captures logs from the kernel, early boot processes, and all systemd-managed services, storing them in a binary, indexed journal format. Its key advantages are structured logging (supporting key-value pairs), compression, and protection against log tampering. On most contemporary distributions (RHEL/CentOS 7+, Ubuntu 16.04+, Debian 8+), both daemons run in tandem. journald often acts as a central collector, while rsyslog can be configured to read from the journal and write to traditional text files for compatibility with legacy tools.

Core Configuration Files and Directories

Your primary configuration targets are:

• /etc/rsyslog.conf and /etc/rsyslog.d/: The main and drop-in configuration files for rsyslog.
• /etc/systemd/journald.conf: The configuration file for systemd-journald.
• /etc/logrotate.conf and /etc/logrotate.d/: Configuration for rotating and compressing old log files to prevent disk space exhaustion.
• Log Storage Locations:
  • rsyslog text files: /var/log/ (e.g., syslog, auth.log, kern.log, dpkg.log).
  • journald binary store: /var/log/journal/ (persistent) or /run/log/journal/ (volatile, if /var/log/journal/ doesn’t exist).

Step-by-Step Configuration Guide

1. Configure systemd-journald for Persistent Storage

By default, some distributions configure journald to store logs only in memory (/run/log/journal/), losing them on reboot. To enable persistent storage:

sudo nano /etc/systemd/journald.conf

Uncomment and set:

Storage=persistent

This directs journald to write logs to /var/log/journal/. You may also adjust SystemMaxUse= and SystemMaxFileSize= to control disk usage (e.g., SystemMaxUse=500M). Restart the service:

sudo systemctl restart systemd-journald

Verify with journalctl --disk-usage.

2. Configure rsyslog to Filter and Write Logs

The default rsyslog configuration is often sufficient for basic needs. To customize, edit /etc/rsyslog.conf or create a new file in /etc/rsyslog.d/ (e.g., 10-custom.conf). The syntax follows the rule: [facility].[severity] [action].

Example 1: Log all authentication messages (both successes and failures) to a dedicated, more secure file with restricted permissions.

auth,authpriv.*   /var/log/auth.log

Example 2: Log kernel messages of severity warning and higher to a separate file.

kern.warning      /var/log/kern.log

Example 3: Log all mail-related messages to a specific file.

mail.*            /var/log/mail.log

Example 4: Discard all debug and info messages from the cron facility to reduce noise, logging only warn and above.

cron.!debug,!info   /var/log/cron.log

After making changes, test the configuration syntax and restart rsyslog:

sudo rsyslogd -N1  # Syntax check
sudo systemctl restart rsyslog

3. Configure Log Rotation with logrotate

Log files grow indefinitely. logrotate manages this automatically, typically via a daily cron job. To customize rotation for a specific log (e.g., a high-volume application log at /var/log/myapp/app.log), create a file in /etc/logrotate.d/:

sudo nano /etc/logrotate.d/myapp

Add:

/var/log/myapp/app.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 myappuser myappgroup
    sharedscripts
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service > /dev/null 2>&1 || true
    endscript
}

This rotates the log daily, keeps 30 compressed archives, creates a new empty log with proper permissions after

…creates anew empty log with proper permissions after each rotation, ensuring that the application can continue writing without interruption. The postrotate stanza sends a SIGHUP to rsyslog, prompting it to reopen its file descriptors so that the freshly rotated file is used immediately.

4. Test and Validate the Rotation Policy

Before relying on the automated cron job, manually invoke logrotate to confirm that your configuration behaves as expected:

sudo logrotate -d /etc/logrotate.d/myapp   # dry‑run, shows what would happensudo logrotate -f /etc/logrotate.d/myapp   # force rotation now

Check the resulting files in /var/log/myapp/:

• The current app.log should be empty (or contain only new entries if the application was still running).
• A rotated file named app.log.1.gz (or similar, depending on the compress/delaycompress settings) should appear, containing the previous day's logs. Verify permissions:

ls -l /var/log/myapp/app.log*

You should see the mode 640 and ownership set to myappuser:myappgroup.

5. Forward Logs to a Centralized Server (Optional)

For environments with multiple hosts, forwarding logs to a central syslog collector simplifies correlation and long‑term archiving.

Using rsyslog as a sender Create a file /etc/rsyslog.d/30-forward.conf:

# Forward everything to log collector at logs.example.com:514 via UDP
*.* @logs.example.com:514
# For reliable TCP forwarding, use:
# *.* @@logs.example.com:514

Restart rsyslog:

sudo systemctl restart rsyslog

On the collector side, ensure the imudp (or imtcp) module is loaded and that the firewall permits traffic on port 514.

Using systemd-journald forwarding If you prefer to forward the journal directly, edit /etc/systemd/journald.conf:

ForwardToSyslog=yes
ForwardToWall=no

Then restart systemd-journald. The journal entries will appear in the local syslog socket and be forwarded by rsyslog as configured above.

6. Secure Log Files and Directories

• Restrict access: Ensure that only privileged users and the designated application accounts can read sensitive logs (e.g., /var/log/auth.log, /var/log/mail.log). Typical permissions are 640 with ownership root:adm or a custom group.
• Enable immutable flag for critical audit logs when tamper‑evidence is required:

  sudo chattr +i /var/log/audit/audit.log
  

  (Remember to remove the flag before rotating or manually editing.)
• Monitor for unauthorized changes using tools like auditd, osquery, or file‑integrity scanners (AIDE, Tripwire).

7. Routine Maintenance and Monitoring

1. Daily health check – Verify that logrotate ran without errors:

2. Disk usage alerts – Set up a simple systemd timer or cron job that runs journalctl --disk-usage and triggers an email if usage exceeds a threshold (e.g., 80 % of SystemMaxUse).
3. Log aggregation review – Periodically sample forwarded logs on the central server to confirm that timestamps, hostnames, and facility/severity fields are intact.
4. Rotate rotation configs – As applications evolve, revisit /etc/logrotate.d/ files to adjust rotate, size, or postrotate scripts.

Conclusion

By configuring systemd-journald for persistent storage, tailoring rsyslog filters to route messages to appropriate files, and leveraging logrotate for automated size‑ and time‑based rotation, you establish a robust, self‑maintaining logging foundation on any Linux system. Adding optional remote forwarding, securing file permissions, and instituting routine

...maintenance checks, you create a logging infrastructure that is not only reliable and secure but also adaptable to changing operational demands. This layered approach ensures that logs remain a trustworthy source of truth for troubleshooting, compliance, and security analysis, turning raw system events into actionable intelligence. Ultimately, the strength of your logging strategy lies in its consistency and the discipline of its upkeep—automating the routine while vigilantly monitoring the exceptional.