Detecting and Ending Long-Running Processes

The Challenge: Processes Gone Wild

It is fairly straightforward to correlate and alert on activity you have a log message for, but what about in the scenario where there’s no log or audit message. How do you detect when someone does not do something?

This was the challenge that arose recently. Here’s the scenario—administrators connect to a variety of servers daily and launch a variety of tools. However, often the default action for administrators is to disconnect their session rather than log off. This results in applications left running that consume valuable server resources such as memory and CPU.

Now while there are configuration settings you can put in place to address these challenges, often in a large enterprise, they are not that straightforward as they may seem. So, how can you use LogRhythm to detect these long-running processes? And once you have found them, what can you do about them?

Well fortunately the LogRhythm platform has a powerful correlation feature of the LogRhythm Advanced Intelligence Engine (AIE) called Log Not Observed and inbuilt active response functionality called SmartResponse™ that we can use to meet this operational use case.

The Solution

First, enable the LogRhythm agent, Endpoint Process Monitoring, to detect a new process being launched. This features enables you to see, in real time, all of the process starting and stopping on Windows, Linux or UNIX hosts.

Note, while you used the LogRhythm Endpoint Process Monitoring agent, this use case can be achieved using native Operating System logs from Windows or Linux.

Figure 1. AI Engine

With Process Monitoring enabled, you then build out an AI Engine (AIE) correlation rule to detect this. As mentioned, you’re using a really powerful AIE feature—the Log Not Observed block.

With this feature, you can look for the lack of something happening, which is something uniquely powerful to the LogRhythm Security Intelligence Platform. Here’s how it all works:

  • The first AIE rule block looks for new applications starting. In this rule, you have a list of the specification process you want to monitor.
  • Then the rule waits for six hours.
  • And if it does not see the LogRhythm Process Monitor register a Process Stopping message for the process in question, you generate an AIE Alarm.

Figure 2. LogRhythm Process Monitor

Click image to expand

And here you are: fast forward six hours later and you have an AIE Alarm with an analyst-approvable SmartResponse to end the long-running application in question (as this is an operation example and not security focused we set the risk value to zero).

Figure 3. AIE SmartResponse Rule

Finally, approve the LogRhythm SmartResponse to end the long-running process and free up the server’s resources.

This use case could be augmented by having the SmartResponse run automatically or adding a secondary SmartResponse to email the user in question confirmation that their process has been automatically terminated and explaining why.

Figure 4. AI Engine

The functionality listed here can be expanded for a wide variety of use cases around the Log Not Observed functionality, such as:

  • Hosts being updated but not coming back after a reboot
  • Windows updates pending, but not being installed within a specific time frame
  • Privileges being granted and not revoked or a temporary accounts being enabled and not disabled in a specific time frame

There you have it, how to alarm and quickly see what is not there!

To learn more about creating AI Engine Rules, download the data sheet.

Download Now