AWS Defense Evasion and Centralized Multi-Account Logging

Two men working with code on computers

Amazon Web Services (AWS) is a widely known cloud service provider, but organizations that use AWS products face unique cybersecurity challenges. Leveraging techniques that reduce risk against new cloud security challenges will help you stay ahead of threats across your AWS environment.

In this blog, we will cover:

  • The basics of AWS multi-account security challenges.
  • Identify defense evasion techniques that attackers are leveraging.
  • Learn best practices for AWS security.
  • Investigate ways to improve visibility and detection of CloudTrail logs using a SIEM solution.

Let’s get started!

What are AWS Multiple Accounts?

AWS accounts can be created per application workloads (e.g., development account, staging account, production account, admin account). Therefore, a large organization can have a hundred-plus AWS accounts for its business units or applications.

As your organization grows and the technology architecture becomes more complex, AWS recommends having multiple accounts as a best practice; security posture can be improved logically by separating the accounts and resources used in each account. For example, if an account is potentially compromised, an attacker will only be able to use the resources in that compromised account. The logical isolation protects other AWS accounts from getting compromised, minimizing the blast radius. You can also centrally manage multiple accounts can using AWS organizations.

Graphic showing AWS multi-account for different workloads.
Figure 1: AWS multi-account for different workloads.

A Quick Synopsis of CloudTrail Logs

CloudTrail logs provide the event history of AWS account activity. The CloudTrail service records AWS API calls for an AWS account, and then the security team can monitor suspicious API calls. The CloudTrail is enabled within a single AWS account. Typically, CloudTrail API calls can be streamed into Amazon Simple Storage Service (Amazon S3) buckets locally for longer retention. If the AWS account is compromised, the attacker could tamper with the logs stored in the S3 bucket within that compromised account. Modifying the trail does not mean deleting and modifying the logs in CloudTrail event history, and by default, CloudTrail event history is stored for 90 days. You can learn more about CloudTrail here.

Multiple AWS accounts can write CloudTrail logs to a central CloudTrail S3 bucket on a separate AWS security account. The below screenshot shows three different accounts writing their CloudTrail logs to a separate AWS account S3 bucket.

Aggregation of CloudTrail logs from multiple accounts to one single account.
Figure: 2 Aggregation of CloudTrail logs from multiple accounts to one single account.

How to Create a Trail and Bucket Policy for Multiple Accounts

CloudTrail log files are stored in an S3 bucket as a gzip archive. The S3 bucket folders are formatted with the originating account ID, and the logs corresponding to each account are stored under the original account number. Learn more here.

Aggregated multi-account logs stored in security account S3 bucket.
Figure 3: Aggregated multi-account logs stored in security account S3 bucket.

How LogRhythm Collects CloudTrail Data

LogRhythm Open Collector brings modern logs, usually in JSON format, from cloud log sources, flat file, or other formats, into the LogRhythm NextGen SIEM Platform. It is designed for easy mapping of JSON fields to the LogRhythm schema.

Disruption of CloudTrail Logging by an Adversary

Let’s explore several use cases for how to detect and respond to suspicious activity across CloudTrail logging in your AWS environment.

Use Case: Detecting StopLogging with SIEM

Suppose an attacker has potentially compromised an AWS account with administrator privileges. The attacker can turn off logging for a trail using console or AWS Command Line Interface (CLI). Stopping the trail will stop logging CloudTrail data into an S3 bucket as part of defense evasion. The below API calls are suspicious and very important to monitor because an adversary may disrupt trails in attempt to evade defense. Learn more about StopLogging here.

Code showing Attacker using AWS CLI to stop the trail.
Figure: 4 Attacker using AWS CLI to stop the trail.

LogRhythm AI Engine (AIE) monitors suspicious AWS API calls using the message, classification, directionality, and several other factors in determining a risk-based priority (RBP) score. The RBP score is a number between 0 and 100 and characterizes the activity’s associated risk level: zero represents low risk, and 100 represents an extremely high risk. A RBP score is assigned to each activity processed by LogRhythm.

LogRhythm High-risk Alarm triggering StopLogging.
Figure 5: High-risk Alarm triggering StopLogging.

You can quickly drill down using the LogRhythm Alarm Drill Down to discover the Top Impacted AWS Accounts associated, then further to see all logs in detail (including metadata and raw log message) for the impacted AWS account. From there, drill down into the indexed log, with final navigation to the raw log message through a simple click on the “log message” tab of the Web UI.

StopLogging Alarm Drill Down.
Figure 6: StopLogging Alarm Drill Down.

The drill down of the alarm will help you identify the metadata quickly, what AWS accounts are compromised, and the user in question.

Extracted metadata for CloudTrail StopLogging.
Figure 7: Extracted metadata for CloudTrail StopLogging.

Use Case: Detecting DeleteTrail

If an attacker got the required IAM permission over the compromised account, they could delete the trail on the victim’s AWS account for defense evasion. Learn more about DeleteTrail here.

Attacker using AWS CLI to delete the trail.
Figure 8: Attacker using AWS CLI to delete the trail.
LogRhythm High-risk Alarm triggered for the compromised AWS trail delete alarm.
Figure 9: High-risk Alarm triggered for the compromised AWS trail delete alarm.

In this example, you can drill down on the alarm to get more information and proceed with the investigation.

Alarm Drill Down for DeleteTrail API call.
Figure 10: Alarm Drill Down for DeleteTrail API call.

The drill down of the alarm will help you investigate the suspicious delete trail API call.

DeleteTrail API call and the impacted account.
Figure 11: DeleteTrail API call and the impacted account.

Use Case: UpdateTrail

An attacker can create cloud instances in unused geographic regions to evade detection. For example, an adversary could use an unused AWS region for cryptocurrency mining, which can cost an organization a substantial amount of money. CloudTrail can send logs to S3 buckets from a single region or multi-region. The below screenshot shows the configuration is set for multi-region to be true in production. The organization is logging all the activities across all the regions in an S3 bucket. The attacker may disable the multi-region trail for all regions. The activities which happen in the other regions will not get logged by disabling the CloudTrail. Learn more about UpdateTrail here.

Multi-region trail turned on in the production environment.
Figure 12: Multi-region trail turned on in the production environment.

The below screenshot shows an example of how an attacker has disabled the multi-region trail part of defense evasion.

Attacker disabled multi-region trail.
Figure 13: Attacker disabled multi-region trail.

You can drill down on the alarm to get more information and proceed with the investigation.

LogRhythm Alarm Drill Down for UpdateTrail API call for multi-region.
Figure 14: Alarm Drill Down for UpdateTrail API call for multi-region.

Again, you can drill down on the alarm for further investigation.

Figure 15a: Update API call and the impacted account.

Update API call and the impacted account.
Figure 15b: Update API call and the impacted account.

LogRhythm’s Dashboard displays the event layer of log data and provides the ability to drill down to the underlying raw log data. The raw log shows the multi-region is set to be disabled by the attacker.

16 Multi-region is set to “false” by the attacker.
Figure: 16 Multi-region is set to “false” by the attacker.

AWS is all about shared responsibility model. Customers are responsible for the security in the cloud and AWS is responsible for the security of the cloud. CloudTrail event history is essential for incident response. An attacker with the right permissions can gather information about the target environment and then disrupt the logging of CloudTrail to stay under the radar. Modifying the trail does not mean deleting and modifying the logs in CloudTrail event history, and by default CloudTrail event history is stored for 90 days.

AWS Security Best Practices

It’s critical to continue to improve your cloud security defenses and leverage AWS security best practices to reduce risk to your organization. Here are couple of tips to live by moving forward:

  • When configuring CloudTrail, apply trail to all regions.
  • Consolidate all the region logs.
  • Configure central logging to one single bucket in a separate AWS security account.
  • Enable multi-factor authentication to delete the S3 buckets.
  • Configure AWS organization and apply service control policies where no one can tamper with CloudTrail logging.
  • Monitor suspicious API calls.

Download this data sheet to learn more about how LogRhythm can help you safeguard your AWS infrastructure. Plus, check out my other two blog for additional tips on defending your AWS environment: