Let’s talk about a less discussed but critical aspect of Security Information and Event Management (SIEM) – data management. While the primary goals of SIEM include threat detection, regulatory compliance, and swift response, the backbone of these systems is log message ingestion and storage. The amount of machine data generated from various systems, applications, and security tools is staggering. Storing and processing this data can be costly and inefficient. Traditionally, SIEM systems adopt an all-or-nothing approach to data collection, but what if there was a better way?
This scenario creates a quandary – intuitively, some log messages hold more importance than others, yet we store all the log messages in fear of missing out on critical information in unforeseen circumstances.
Categorizing Data: Actionable vs. Standby
Imagine if we could classify SIEM data into two distinct categories:
- Actionable Data: These log messages are used for immediate threat detection, compliance dashboards, and anticipated investigations. For example, Windows Event 4648, which signals special privileges assigned to a new logon, is essential for real-time security alerts when monitoring for privilege escalation.
- Standby Data: These logs do not directly contribute to immediate threat responses but may prove invaluable later. An example is Windows Event 4689, noting when a process exists, which typically does not align directly with active security measures.
Rationale Behind Data Categorization
Grouping log messages as Actionable or Standby ensures that essential data is immediately accessible, while less critical information is stored cost-effectively, without compromising future accessibility. This approach enhances data management efficiency and significantly reduces data storage costs.
Challenges in Retrieving Standby Data
The effectiveness of this system hinges on our ability to access Standby Data when it becomes relevant. If retrieval processes are cumbersome or time-consuming, the benefits of this categorization are undermined. Ideally, the SIEM should seamlessly inform us when Standby Data correlates with our search criteria during an investigation.
SIEM Archives Not Suitable for Standby Data Availability
While most SIEMs offer an archival function to store data in less expensive storage, these do not differentiate between Actionable and Standby data, leading to investment misallocation due to SIEM licensing (regardless of if licensing is ingest-based, resource-based, etc.). Coupled with arduous and time-consuming data restoration processes, means SIEM Archives does not satisfy our needs for cost-effective storage without compromising future accessibility.
The Future of SIEM: Advanced Data Routing
For SIEM systems to truly support Actionable and Standby Data, they require advanced data routing functions, post-log processing and enrichment. This capability not only makes log processing more resource-efficient but also streamlines the retrieval process during investigations, ensuring that data is both accessible and appropriately priced.
Conclusion: A Call to Innovate in SIEM Technology
The next generation of SIEM technology must differentiate between Actionable and Standby data not only in functionality but also in licensing to ensure appropriate investment. This shifts the typical SIEM license model that is predicated on covering all ingested data, be it through traditional ingestion models or alternatives such as resource-based that essentially taxes on the compute required to process all data. A license model that distinguishes between Actionable and Standby data more accurately aligns to the overall value provided by the SIEM, without penalizing the needs for holding onto ‘standby’ data in case its ever needed.
The post Optimizing SIEM TCO: Smart Data Management Strategies appeared first on Graylog.