Blog

25 Linux Logs to Collect and Monitor

While “America runs on Dunkin”, IT increasingly runs on Linux. Between being open-source and highly customizable, everything from video games to enterprise servers can run on Linux. When cloud services took over the corporate IT environment, they brought Linux with them in the form of virtual servers and containers. Meanwhile, developers increasingly use Linux-based Docker to containerize applications and Kubernetes to manage the deployments.

 

As more of your development and IT environments rely on the open-source operating system, knowing which 25 Linux logs to collect and monitor can help you investigate performance issues and security incidents faster.

What are Linux Logs?

Linux logs record events, applications, and kernel information for activities within the Linux operating system. System administrators can use Linux logs for:

  • Troubleshooting: errors, warnings, or other messages to identify an issue’s root cause
  • Performance issues: system and application logs to identify issues like memory leaks
  • System health: system logs to identify patterns or trends before an issue arises
  • Compliance: documentation to prove systems work as intended
  • Security: monitoring and detecting suspicious activity or security issues, like failed logins

 

Stored in plain text format, Linux logs typically reside in the “/var/log” directory. While they have no specific naming convention to help when searching within the system, they often include the “.log” extension or contain “log” as part of the file name. Linux logs can be categorized as:

  • System logs: system operation, including boot messages, kernel messages, and hardware events
  • Service logs: service performance and monitoring, like network services or daemons
  • Application logs: application activity, like errors and warnings to identify or analyze performance
  • Event logs: information system activity, like logins or shutdowns, to detect and investigate security issues

 

How to Read Linux Logs

Linux logs are structured to contain the same standard information as event logs generated by other technologies:

  • Timestamp: date and time of event
  • Hostname: machine generating the log
  • Service or Application Name: service or application generating the log
  • Process ID (PID): process generating the log
  • Log Level: severity or importance, like informational (INFO), warning, or error
  • Message Body: event details

 

You can read logs directly through the Linux terminal. The command line interface is a native application that allows you to retrieve information from the different directories. Within the terminal you can use the following commands to change the information provided:

  • cat: displays the log file’s complete contents, including message body
  • less: displays one page of a file and supports searching the file, so you can more easily move between longer files and examine details
  • more: displays the file contents one screen at a time but limited searchability and navigation
  • tail: displays last ten entries of a file, often used for checking recent events
  • head: displays the beginning of the file to quickly scan initial entries
  • grep: search for patterns or keywords within a file to locate information when investigating or troubleshooting
  • awk: tool to extract information, filter using conditions, or perform operations
  • Sed: stream editor for processing and modifying log file text, like only printing those containing a specific keyword
  • dmesg: viewing system logs related to low-level hardware and kernel events, like driver problems
  • journalctl: displays logs from systemd daemon with the ability to filter by service, date, or other criteria

 

25 Linux Logs to Collect and Monitor

Since the sheer number of linux logs can become overwhelming, you may want to start by focusing on the following critical logs:

  1. /var/log/auth.log: documentation for failed and successful logins and authentication on Debian/Ubuntu
  2. /var/log/secure: documentation for failed and successful logins and authentication on RedHat/CentOS
  3. /var/log/boot.log: information about startup, shutdown, and boot, including initialization script
  4. /var/log/maillog: activities related to mail servers
  5. /var/log/kern: kernel logs and warning data for troubleshooting custom kernels
  6. /var/log/syslog: consolidated system-wide activity across different components
  7. /var/log/messages: general system information, like boot errors, application service errors, or hardware issues
  8. /var/log/daemon.log: information about background daemons running on the system
  9. /var.log/cups: printer and printing information
  10. var/log/mysqld.log: debugging, failure, and success of MySQL daemon
  11. /var/log/cron: record of all Crond-related messages (cron jobs) like when jobs are initiated or terminated
  12. /var/log/faillog: failed login attempts against the system, useful for security incident and credential attack investigations
  13. /var/log/btmp: failed login attempts by individual user, useful for security incident and credential attack investigations (more detailed log with IP, User
  14. /var/log/auth.log: system authorization information, like user login and authentication method
  15. /var/log/utmp: user current login state
  16. /var/log/wtmp: user login and logout records
  17. /var/log/httpd/: error and access log files for Apache httpd daemon, like memory issues or requests from HTTP
  18. /var/log/pureftp.log: FTP connections using pureftp process, like login successes and failures
  19. var/log/yum.log: record on package installations using Red Hat Enterprise yum command
  20. /var/log/dpkg.log: record on package installation or removal using the dpkg command
  21. /var/log/lastlog: every user’s most recent login
  22. /var/log/xferlog: FTP file transfer session information, like file names and user-initiated transfers
  23. ​​/var/log/Xorg.x.log: XWindows system messages
  24. /var/log/audit/audit.log: records user activity related to the Linux Audit daemon (auditd)
  25. /var/log/samba/: record of activity by the samba daemon that connects Windows/Linux filesystems

 

 

Graylog for Linux Logging

Various log shippers like Filebeat, Auditbeat, and NXLog enable fast easy configuration for shipping logs for aggregation and analytics. Using the Graylog Beats input for Filebeat and Auditbeat or Graylog GELF input using NXLog, shipping Linx logs becomes easy.

 

Using Graylog Sidecar configuration, you can manage logging levels for each Linux log shipper. All of these logging levels and configurations are managed centrally inside Graylog.

 

Graylog: Managing Linux Log to Improve Operations and Security

 

With Graylog, you can ingest, parse, normalize, and correlate log events from across your IT environment, including Linux and Windows. Correlating Active Directory Authentication with Linux, “/var/log/faillog”, and “/var/log/btmp” can help detect and investigate credential-based attacks.

With Graylog’s security analytics and anomaly detection capabilities, you get the cybersecurity platform you need without the complexity that makes your team’s job harder. With our powerful, lightning-fast features and intuitive user interface, you can lower your labor costs while reducing alert fatigue and getting the answers you need – quickly.

Our prebuilt search templates, dashboards, correlated alerts, and dynamic look-up tables enable you to get immediate value from your logs while empowering your security team.

The post 25 Linux Logs to Collect and Monitor appeared first on Graylog.