Temporal Chain Normalization: The Unsung Hero of Event Correlation

When it comes to correlation capabilities, LogRhythm has you covered. With AI Engine you can perform a variety of activities, from observing a single activity to applying advanced behavior rules across multiple dimensions (entities, devices, log sources, metadata, etc.). In addition to some of the more obvious capabilities, I’m here to tell you about one not so known feature of AI Engine called Temporal Chain Normalization (TCN).

Let’s investigate TCN a bit and understand just what it is (other than sounding extremely cool).

Temporal Chain Normalization

The best way to explain TCN is through an example. Take a simple correlation activity: If we see Event A followed by Event B, then we generate an alert.

Click on images to view larger

Figure 1. Temporal Chain Normalization

Note: The normalized timestamps (in green) are extracted from the Log Message itself. They do not rely on collection time.

This logic is used by many of the pre-built rules within the LogRhythm Security Intelligence and Analytics Platform (e.g., brute force attack: multiple login failure [Event A] followed by login success [Event B])

The Challenge

Computers Can (and Do) Lie

You’re not going to be surprised to hear this, but unfortunately, computers, users, and even the internet, aren’t always 100 percent consistent or reliable. Network latency, system downtime, hawking radiation, and other anomalies can impact the collection speed of audit and log data.

Imagine you have a correlation rule looking for an attack (Event A) followed by a compromise (Event B). But when an attack actually occurs, the logs are received out of order because of a network outage. What happens then? Nothing happens. No alarms fire and that’s a problem.

As you can see below the collection timestamp (in red) shows that Event B was received after Event A; therefore, no alert was generated.

Figure 2. Temporal Chain Normalization

Organizations should not assume that all data generated within their environment is assigned an accurate timestamp relative to the variety of data sources and logging mechanisms across the IT environment. Disparate timestamps can result in data being evaluated out of order, which impairs real-time event recognition and response. There are many reasons log and machine data collected for analysis may be attributed to a different time than when the event actually occurred!

The Solution

Normalise and Reorder Logs

Unless you have a security information and event management (SIEM) platform with the ability to normalise and reorder out of sequence log messages, you are going to miss important alerts.

The good news? The LogRhythm Security Intelligence Platform can do just that with TCN.

Temporal Chain Normalization in Action

Let’s start off with something simple. Here’s an AI Engine rule that looks for Event A followed by Event B. For simplicity, I’ve generated two unique Windows Events and expect to see Event ID 4321 followed by Event ID 4322.

Figure 3. Temporal Chain Normalization

Figure 4. Temporal Chain Normalization

Figure 5. Temporal Chain Normalization

Note: As this is a demonstration rule, it only groups on the Known Host for Rule Block 1 (as I’ll be generating Event A and Event B from two different hosts).

Using a simple script I generated Event A (on Host 3) and Event B (on Host 4) in the correct chronological order to validate the rule works as expected:

$now = get-date

Write-EventLog –LogName Application –Source “Application” –EntryType Information –EventID 4321 –Message $now

sleep 60

$now = get-date

Write-EventLog –LogName Application –Source “Application” –EntryType Information –EventID 4322 –Message $now

As expected, here’s the resultant alarm firing when Event A and Event B were generated and collected in the proper order.

Figure 6. Temporal Chain Normalization

Figure 7. Temporal Chain Normalization Click on images to view larger

Next I ran the same test again. But this time I stopped the LogRhythm Collector running on Host 3, then proceed to generate Event A (on Host 3) and then Event B (on Host 4), then waited a minute to start the Collector on Host 3 again. This caused the collected order of logs to effectively be Event B followed by Event A, rather than Event A followed by Event B.

Here’s a breakdown of the above with detailed steps:

Host 3

  1. Stop LogRhythm Collector service
  2. Generate Event A (vmid 4321)

Figure 8. Temporal Chain Normalization

Host 4

  1. Generate Event B (vmid 4322)

Figure 9. Temporal Chain Normalization

Host 3

  1. Wait 60 seconds
  2. Start LogRhythm Collector service

Here’s the resulting alarm.

Figure 10. Temporal Chain Normalization

Note the time it fired is 11:19:20 (EST).

Did it surprise you to find out that the alarm fired even though the events were collected out of order? This is all due to LogRhythm’s time normalization capabilities.

Something a Bit More Advanced

Let’s jump into a more complex example using some of the pre-built AI Engine rules within LogRhythm. I choose rule number 42: User Thread Detection: Internal Brute Force Auth. For those not familiar with this rule, here’s how it works:

Figure 11. Temporal Chain Normalization

Rather than create a network failure, or turn off an agent, I’ll demo this with a simple custom log source with authentication activity. In this sample log source, we’re collecting via text file and the first entry (bold) is our login success. Note its timestamp of 19:53:50. Next up, we leave it a little while and add the remaining dozen login failures—all which appear afterwards in the log file, but have an earlier timestamp:

2016-08-27T19:53:50+00:00,Login Success,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:32+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:33+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:34+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:35+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:36+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:37+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:38+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:39+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:40+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:41+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:42+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

2016-08-27T19:52:43+00:00,Login Failure,cmartin,VGBWKS03,VGBWKS01

After a minute or so, the following alarm is generated.

Figure 12. Temporal Chain Normalization

Yet again we’ve successfully detected the brute force attack, even though the log messages were collected out of sync.

Figure 13. Temporal Chain Normalization

Multiple Time Zones? No Problem

For organizations with a network that spans across multiple time zones, TCN is a basic and must have feature. With TCN, all logs are stored in Coordinated Universal Time (UTC) allowing for accurate event correlation and prioritization.

More from Chris Martin