Security information and event management (SIEM) tools are indispensable in an organization’s cybersecurity framework. SIEM tools collect, analyze, and correlate log data from various devices and applications across an organization to identify suspicious activities, enhance overall security posture, and ensure compliance with industry regulations. However, the indiscriminate ingestion of logs into a SIEM tool can have detrimental consequences such as increased costs, system overload, and alert fatigue. Understanding the types of logs that are valuable to ingest and which ones to filter out is crucial in optimizing the functionality of SIEM systems.
Why Should You Manage Your Log Ingestion?
Cost Implications
SIEM tools operate under a licensing model that typically charges based on the volume of logs ingested. Ingesting a high volume of irrelevant or less significant logs can escalate costs exponentially. These unnecessary logs consume valuable resources, including storage and processing power, without contributing meaningful insights into the security landscape.
Alert Fatigue
Ingesting logs indiscriminately can result in an overwhelming number of alerts, many of which might be false positives or low-priority. This can lead to alert fatigue, where analysts are overwhelmed and may overlook or not properly investigate critical alerts due to the sheer volume of notifications.
System Overload
SIEM tools have limits to how much data they can process efficiently. Flooding the system with too many logs can lead to delays, performance degradation, and even system failures, making it challenging to identify and respond to threats in real time.
What Are Valuable Logs?
The selection of valuable logs is crucial for enhancing the effectiveness and precision of your SIEM tool. Here is a detailed look at valuable logs, complete with examples to guide you in making informed log ingestion decisions.
Security Logs
- Firewalls and Network Devices: Logs from firewalls and routers that monitor and control incoming and outgoing network traffic based on your organization’s previously established security policies. These can include logs indicating traffic from unknown IP addresses or regions, repeated login attempts, or suspicious URL requests.
- Intrusion Detection/Prevention Systems (IDS/IPS): Logs that monitor network and system activities for malicious exploits or security policy violations. These would include logs that capture detected patterns matching known attack signatures or unusual data packets in the network traffic.
- Antivirus and Anti-Malware Tools: Logs detailing the detection, blocking, and removal of malicious software. Examples of information you may want are detected malware names, locations, actions taken, and user accounts involved.
Authentication and Access Logs
- Active Directory (AD): Logs containing information on user logins/logouts, password changes, and group membership changes. These would include logs highlighting multiple failed login attempts or password changes.
- Virtual Private Network (VPN): Logs reflecting remote user connections to an organization’s network. Examples of information you may want would be logs capturing connection times, locations, user names, and login/logout events.
- Multi-Factor Authentication (MFA): Logs documenting additional authentication processes beyond just a username and password. Examples of this are logs of successful or failed MFA attempts and user account details.
System and Application Logs
- Operating Systems (OS): Logs that record events in the operating system. These would include logs noting system or application errors, updates, or user account changes.
- Databases: Logs monitoring access and operations performed in databases. Examples of this are logs illustrating data queries, modifications, or unauthorized access attempts.
- Web Servers: Logs keeping track of server activity, user access, and executed operations. These would include logs demonstrating HTTP status codes, user agents, and URLs accessed.
- Application Logs from Critical Applications: Helpful information specific to, say, a financial application could include user access records, transaction activities, and error messages.
Cloud Environment Logs
- Cloud Access Security Broker (CASB): Logs from CASBs that monitor activity and enforce security policies across cloud services. These could include logs detailing file uploads/downloads, shared files, and accessed cloud applications.
- Cloud Infrastructure and Services: Logs that oversee activities within cloud resources. Examples of these are logs of initiated instances, network configurations, and access control modifications.
What Logs Provide Limited Value?
While many logs are invaluable sources of information for security analysis and monitoring, some log types often offer limited actionable insights and can clutter the SIEM system. Here’s an expansion on logs of limited value with examples to facilitate judicious log ingestion decisions.
- Print Logs: These logs contain information regarding printer usage, including details about printed documents, users, and printer status. These are usually not critical for security analysis unless in very specific use cases where print log analysis is crucial.
- Detailed Debugging Logs: Debug logs help developers identify, trace, and fix issues in software applications. They might include error messages, code statuses, or variable values. However, these are often too granular and voluminous, contributing to log noise and making critical security information harder to discern.
- Routine System Notifications: These logs contain informational messages about benign system activities, updates, or status. These could be things like logs noting system startups, software installation, or scheduled tasks. However, unless these are specifically relevant, these logs often don’t contribute substantial insights into security monitoring and can generally be filtered out to reduce noise.
- Environmental Monitoring Logs: Logs that capture environmental conditions such as temperature, humidity, or power status in server rooms or data centers. These are generally not directly related to cybersecurity threats but might be useful in specific scenarios.
- Non-critical Application Logs: Logs from applications that are not business-critical and don’t process sensitive information. An example of this could be logs from a non-sensitive internal communication tool. Ingesting these logs might not provide substantial value in detecting or analyzing security threats.
- Redundant or Duplicate Logs: Logs that are repetitive or duplicate information available in other logs. Multiple tools or systems often generate similar logs, each capturing the same event or activity. These redundant logs contribute to data overload without adding unique value.
Best Practices for Log Ingestion
- Log Prioritization and Filtering: Implement strategies to prioritize and filter logs based on their relevance and significance to security. This may involve defining clear criteria for which logs to ingest.
- Regular Review and Tuning: Regularly review and tune the SIEM configurations to ensure that the system is operating efficiently and that the ingested logs are still relevant.
- Effective Alert Management: Develop strategies for managing alerts effectively to reduce the risk of alert fatigue. This could include classifying and prioritizing alerts based on severity and relevance.
In conclusion, a strategic approach to log ingestion is essential for optimizing the functionality and efficiency of SIEM tools. Prioritizing the ingestion of logs that are most relevant to security and filtering out less significant logs helps maintain a balanced, cost-effective, and responsive SIEM system.