Security is powered by data, including many machine-generated log sources — from IT infrastructure to cloud applications, personal laptops to Point of Sale endpoints, and internet of things (IoT) devices. While you can’t investigate threats without it, it is often impossible to know which data will be important for analysis at the time of collection because threats are still unknown at that point.
This is where data storage comes in. If data isn’t collected and stored long enough for analysis, there is never an opportunity for detection and response. But as anyone tasked with log management knows, data storage is not simply a matter of thinking, “I want to store all of my data in the same way for the same amount of time forever.”
Data storage decisions depend on a number of factors. So how do you ensure that you select the most appropriate option to best facilitate security analytics for your organization? Let’s explore how.
Determining Storage Needs
To understand your organization’s storage needs, you should evaluate the following three factors:
1. Understand your compliance requirements.
Many organizations are required to store data for a certain amount of time, such as those that comply with the Health Insurance Portability and Accountability Act (HIPAA) and therefore need to maintain information for at least seven years. If you aren’t in an obviously regulated industry, make sure you check local regulations, as well. There are a number of new and developing data protection rules. Even if you’re in a space that hasn’t needed to comply with anything in the past, you may have to now, and this may include maintaining data for a required number of months or years.
2. Determine how searchable your stored data should be.
Just because your data is in storage doesn’t mean you can’t easily use it. Frequently, organizations need certain types of stored data to be available for search and analysis quickly and efficiently, whereas other types of data can be sent to long-term storage that likely won’t need to be searched that often. Therefore, you need to understand your searchability requirements, or else you may end up wasting money on a solution that’s incompatible with your business needs.
3. Commit to a budget.
Next, you should identify a realistic budget to commit to data storage. This includes both financial and time investments. Storing log data requires a certain level of infrastructure that your team may need to build and deploy. Whether you need (or simply want) to store log data for a single month or several years, it is critical that you understand the number of log messages ingested by your SIEM so you can compare that with your desired storage time and determine the most appropriate budget.
Examining Storage Options: Hot, Cold, or Warm
Finally! Now that you have a full understanding of your needs and resources, you’re ready to start looking at data storage solutions. With this information in hand, you are far more likely to pick a solution that satisfies your day-to-day as well as long-term business needs. Of course, this begs the question: What types of data storage are there?
LogRhythm first addressed data storage challenges with archive data storage, featuring both a hot data tier and a cold data tier. To accommodate even more storage use cases, we introduced a new, third tier: the Warm Node Data Indexer. Each tier focuses on distinct types of data, and therefore, has different strengths:
Hot Data Tier
The hot data tier focuses on information you’re looking to store for a shorter period of time (i.e., a few months). You’ll likely want to review this data more frequently than your cold, archived data, so we designed the hot data tier to be searchable and analyzable, with searches taking only a few seconds to complete.
Cold Data Tier
The cold data tier focuses on that archived data you’re likely examining less frequently, so we architected this tier to maximize the data’s time to live (TTL). Data can be stored in this tier for years, which is especially valuable for companies that need to maintain data over the long-term for compliance reasons.
The Warm Node Data Indexer
Even with these options, LogRhythm recognized data use cases still exist that customers may need a solution for, resulting in our recent release of the Warm Node Data Indexer. This new solution combines the searchability benefits of the hot data tier with the long-term storability benefits of the cold data tier in a cost-effective manner. Essentially, if you’ve ever found yourself wishing there was a storage option somewhere in the middle of the hot and cold data tiers, this one’s for you.
Specifically, the Warm Node Data Indexer allows you to:
- Store data for at least a year.
- Search and analyze that data within minutes.
- 1 Save more than six times the cost of storing your data.
The Ultimate Benefit: Efficient Threat Detection
Adding a third data tier with the Warm Node Data Indexer provides more choices to ensure the best architecture for your environment, specifically when weighing the cost-benefit balance of useful data lifetime. We’re excited to offer you more flexibility to provide cost-effective, easily implemented solutions so they can focus on what truly matters: quick, efficient threat detection and response.
Do you need help determining the best data storage solution for you? Let us know!
1 Based on the price per gigabyte of a hot storage appliance compared to the Warm Node Data Indexer.