CloudTrail Delivery Failing

Risk Level: Medium (should be achieved)

Rule ID: CT-011

Ensure that the log files created by your Amazon CloudTrail trails are delivered without any failures to the designated recipients in order to keep CloudTrail log data for security and compliance audits.

This rule can help you with the following compliance standards:

APRA
MAS
NIST4

For further details on compliance standards supported by Conformity, see here.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Security

Operational
excellence

When your Amazon CloudTrail trails are not able to deliver log files to their recipients due to delivery errors or misconfigurations (usually involving the access policies that you have in place), the logging data recorded by these trails can't be saved and used for future security audits.

Audit

Case A: To identify Amazon CloudTrail trails that are not able to deliver log files to the designated S3 bucket(s), perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon CloudTrail console at https://console.aws.amazon.com/cloudtrail/.

03 In the navigation panel, under CloudTrail, choose Trails.

04 Click on the name (link) of the Amazon CloudTrail trail that you want to examine.

05 In the General details section, verify the Last log file delivered attribute value (date/time). If the Last log file delivered value has not been updated recently (log files are generally published every 5 minutes), the selected Amazon CloudTrail trail failed to deliver the last log file to the designated S3 bucket.

06 Repeat steps no. 4 and 5 for each Amazon CloudTrail trail created within your AWS cloud account.

Using AWS CLI

01 Run list-trails command (OSX/Linux/UNIX) with custom query filters to list the names of all the Amazon CloudTrail trails created for your AWS cloud account:

aws cloudtrail list-trails
  --region us-east-1
  --query 'Trails[*].Name'

02 The command output should return an array with the requested CloudTrail trail names:

[
    "cc-main-cloud-trail",
    "cc-data-events-trail"
]

03 Run get-trail-status command (OSX/Linux/UNIX) using the name of the Amazon CloudTrail trail that you want to examine as the identifier parameter and custom query filters to expose any log file delivery errors available for the selected trail:

aws cloudtrail get-trail-status
  --region us-east-1
  --name cc-main-cloud-trail
  --query 'LatestDeliveryError'

04 The command output should return null if the selected trail delivered successfully its last log file or an error code if the specified trail failed to deliver the log file:

"AccessDenied"

If the get-trail-status command responds with an error code instead of null, as shown in the output example above, the selected Amazon CloudTrail trail failed to deliver the last log file to the target S3 bucket.

05 Repeat steps no. 3 and 4 for each Amazon CloudTrail trail available in your AWS cloud account.

Case B: To determine if your Amazon CloudTrail trails are able to send SNS notifications for log file delivery, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon CloudTrail console at https://console.aws.amazon.com/cloudtrail/.

03 In the navigation panel, under CloudTrail, choose Trails.

04 Click on the name (link) of the Amazon CloudTrail trail that you want to examine.

05 In the General details section, verify the Last SNS notification attribute value. If the Last SNS notification attribute does not have a value, the selected Amazon CloudTrail trail failed to send SNS notifications for log file delivery.

06 Repeat steps no. 4 and 5 for each Amazon CloudTrail trail created within your AWS cloud account.

Using AWS CLI

01 Run list-trails command (OSX/Linux/UNIX) with custom query filters to list the names of all the Amazon CloudTrail trails created for your AWS cloud account:

aws cloudtrail list-trails
  --region us-east-1
  --query 'Trails[*].Name'

02 The command output should return an array with the requested CloudTrail trail names:

[
    "cc-main-cloud-trail",
    "cc-data-events-trail"
]

03 Run get-trail-status command (OSX/Linux/UNIX) using the name of the Amazon CloudTrail trail that you want to examine as the identifier parameter and custom query filters to expose any SNS notification errors triggered for the selected trail:

aws cloudtrail get-trail-status
  --region us-east-1
  --name cc-main-cloud-trail
  --query 'LatestNotificationError'

04 The command output should return null if the selected trail is sending an SNS notification for each log file delivery or an error code if the trail failed to send SNS notifications:

"AuthorizationError"

If the get-trail-status command responds with an error code instead of null, as shown in the output example above, the selected Amazon CloudTrail trail failed to send SNS notifications for log file delivery.

05 Repeat steps no. 3 and 4 for each Amazon CloudTrail trail available in your AWS cloud account.

Remediation / Resolution

Case A: Usually, the Amazon CloudTrail trails fail to deliver log files when there is a problem with the target S3 bucket and will not occur for timeouts. To remediate this issue, create a new S3 bucket and associate your trail with the new bucket so that Amazon CloudTrail can send again log files to S3. To reconfigure your Amazon CloudTrail trails, perform the following actions:

Using AWS CloudFormation

01 CloudFormation template (JSON):

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Replace the Target S3 Bucket with a New Bucket",
    "Parameters": {
        "TrailName": {
            "Type": "String"
        }
    },
    "Resources": {
      "TargetS3Bucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                "BucketName": "cc-main-cloudtrail-bucket",
                "AccessControl": "Private",
                "PublicAccessBlockConfiguration": {
                "BlockPublicAcls": true,
                "IgnorePublicAcls": true,
                "BlockPublicPolicy": true,
                "RestrictPublicBuckets": true
              }
            }
        },
        "TargetS3BucketPolicy": {
            "Type": "AWS::S3::BucketPolicy",
            "Properties": {
                "Bucket": "cc-main-cloudtrail-bucket",
                "PolicyDocument": {
                   "Version": "2012-10-17",
                   "Statement": [
                      {
                          "Sid": "CloudTrailAclCheck",
                          "Effect": "Allow",
                          "Principal": {
                            "Service": "cloudtrail.amazonaws.com"
                          },
                          "Action": "s3:GetBucketAcl",
                          "Resource": "arn:aws:s3:::cc-main-cloudtrail-bucket"
                      },
                      {
                          "Sid": "AWSCloudTrailWrite",
                          "Effect": "Allow",
                          "Principal": {
                            "Service": "cloudtrail.amazonaws.com"
                          },
                          "Action": "s3:PutObject",
                          "Resource": "arn:aws:s3:::cc-main-cloudtrail-bucket/cc-trail-logs/AWSLogs/466594415815/*",
                          "Condition": {
                              "StringEquals": {
                                  "s3:x-amz-acl": "bucket-owner-full-control"
                              }
                          }
                      }
                   ]
                }
            }
        },
        "Trail": {
            "Type": "AWS::CloudTrail::Trail",
            "Properties": {
                "TrailName": {
                    "Ref": "TrailName"
                },
                "S3BucketName": "cc-main-cloudtrail-bucket",
                "S3KeyPrefix": "cc-trail-logs",
                "IsLogging": true
            }
        }
    }
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
Description: Replace the Target S3 Bucket with a New Bucket
Parameters:
  TrailName:
    Type: String
Resources:
  TargetS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: cc-main-cloudtrail-bucket
      AccessControl: Private
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        IgnorePublicAcls: true
        BlockPublicPolicy: true
        RestrictPublicBuckets: true
  TargetS3BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: cc-main-cloudtrail-bucket
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: CloudTrailAclCheck
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:GetBucketAcl
            Resource: arn:aws:s3:::cc-main-cloudtrail-bucket
          - Sid: AWSCloudTrailWrite
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:PutObject
            Resource: arn:aws:s3:::cc-main-cloudtrail-bucket/cc-trail-logs/AWSLogs/466594415815/*
            Condition:
              StringEquals:
                s3:x-amz-acl: bucket-owner-full-control
  Trail:
    Type: AWS::CloudTrail::Trail
    Properties:
      TrailName: !Ref 'TrailName'

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

data "aws_caller_identity" "current-account" {}

resource "aws_s3_bucket" "new-s3-bucket" {

  bucket        = "cc-main-cloudtrail-bucket"
  force_destroy = true
  policy = <<POLICY
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Sid": "AWSCloudTrailAclCheck",
              "Effect": "Allow",
              "Principal": {
                "Service": "cloudtrail.amazonaws.com"
              },
              "Action": "s3:GetBucketAcl",
              "Resource": "arn:aws:s3:::cc-main-cloudtrail-bucket"
          },
          {
              "Sid": "AWSCloudTrailWrite",
              "Effect": "Allow",
              "Principal": {
                "Service": "cloudtrail.amazonaws.com"
              },
              "Action": "s3:PutObject",
              "Resource": "arn:aws:s3:::cc-main-cloudtrail-bucket/cc-trail-logs/AWSLogs/${data.aws_caller_identity.current-account.account_id}/*",
              "Condition": {
                  "StringEquals": {
                      "s3:x-amz-acl": "bucket-owner-full-control"
                  }
              }
          }
      ]
  }
  POLICY

}

resource "aws_cloudtrail" "cloudtrail-trail" {

  name                          = "cc-main-cloud-trail"

  # Replace the Target S3 Bucket with a New Bucket
  s3_bucket_name                = aws_s3_bucket.new-s3-bucket.id
  s3_key_prefix                 = "cc-trail-logs"
  enable_logging                = true

}

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon CloudTrail console at https://console.aws.amazon.com/cloudtrail/.

03 In the navigation panel, under CloudTrail, choose Trails.

04 Click on the name (link) of the Amazon CloudTrail trail that you want to reconfigure.

05 In the General details section, choose Edit and perform the following operations:

Select Create new S3 bucket under Storage location to create the new target bucket.
Provide a unique name for the new S3 bucket in the Trail log bucket and folder box.
Choose Save changes to apply the changes. Once the target bucket is created, Amazon CloudTrail starts to deliver log files to the new S3 bucket and the Last log file delivered attribute value is updated.

06 Repeat steps no. 4 and 5 for each Amazon CloudTrail trail that you want to reconfigure, available in your AWS cloud account.

Using AWS CLI

01 Run create-bucket command (OSX/Linux/UNIX) to create a new target S3 bucket for your Amazon CloudTrail trail:

aws s3api create-bucket
  --bucket cc-main-cloudtrail-bucket
  --region us-east-1
  --acl private

02 The command output should return the name of the newly created S3 bucket:

{
  "Location": "/cc-main-cloudtrail-bucket"
}

03 Run put-public-access-block command (OSX/Linux/UNIX) to enable the S3 Public Access Block feature for the new Amazon S3 bucket (the command should not produce an output):

aws s3api put-public-access-block
  --region us-east-1
  --bucket cc-main-cloudtrail-bucket
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

04 Run update-trail command (OSX/Linux/UNIX) using the name of the Amazon CloudTrail trail that you want to reconfigure as the identifier parameter, to associate the new S3 bucket (i.e. target bucket) with the selected trail:

aws cloudtrail update-trail
  --region us-east-1
  --name cc-main-cloud-trail
  --s3-bucket-name cc-main-cloudtrail-bucket

05 The command output should return the metadata available for the reconfigured trail:

{
    "IncludeGlobalServiceEvents": true,
    "IsOrganizationTrail": false,
    "Name": "cc-main-cloud-trail",
    "TrailARN": "arn:aws:cloudtrail:us-east-1:123456789012:trail/cc-main-cloud-trail",
    "LogFileValidationEnabled": true,
    "IsMultiRegionTrail": true,
    "S3BucketName": "cc-main-cloudtrail-bucket"
}

06 Repeat steps no. 1 – 5 for each Amazon CloudTrail trail that you want to reconfigure, created within your AWS cloud account.

Case B: When an Amazon CloudTrail trail fails to send SNS notifications for log file delivery, it's usually because there is a problem with the associated SNS topic. To resolve this issue, create a new SNS topic and associate your trail with the new topic so that Amazon CloudTrail can send delivery notifications again. To reconfigure your Amazon CloudTrail trails, perform the following actions:

Using AWS CloudFormation

01 CloudFormation template (JSON):

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Replace the Associated SNS Topic with a New Topic",
    "Parameters": {
        "TrailName": {
            "Type": "String"
        },
        "BucketName": {
            "Type": "String"
        },
        "S3BucketKeyPrefix": {
            "Type": "String"
        }
    },
    "Resources": {
        "Trail": {
            "Type": "AWS::CloudTrail::Trail",
            "Properties": {
                "TrailName": {
                    "Ref": "TrailName"
                },
                "S3BucketName": {
                    "Ref": "BucketName"
                },
                "S3KeyPrefix": {
                    "Ref": "S3BucketKeyPrefix"
                },
                "IsLogging": true,
                "SnsTopicName": "arn:aws:sns:us-east-1:123456789012:cc-cloud-trail-new-topic"
            }
        }
    }
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
Description: Replace the Associated SNS Topic with a New Topic
Parameters:
  TrailName:
    Type: String
  BucketName:
    Type: String
  S3BucketKeyPrefix:
    Type: String
Resources:
  Trail:
    Type: AWS::CloudTrail::Trail
    Properties:
      TrailName: !Ref 'TrailName'
      S3BucketName: !Ref 'BucketName'
      S3KeyPrefix: !Ref 'S3BucketKeyPrefix'
      IsLogging: true
      SnsTopicName: arn:aws:sns:us-east-1:123456789012:cc-cloud-trail-new-topic

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

data "aws_caller_identity" "current-account" {}

resource "aws_s3_bucket" "trail-s3-bucket" {

  bucket        = "cc-main-cloudtrail-bucket"
  force_destroy = true
  policy = <<POLICY
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Sid": "AWSCloudTrailAclCheck",
              "Effect": "Allow",
              "Principal": {
                "Service": "cloudtrail.amazonaws.com"
              },
              "Action": "s3:GetBucketAcl",
              "Resource": "arn:aws:s3:::cc-main-cloudtrail-bucket"
          },
          {
              "Sid": "AWSCloudTrailWrite",
              "Effect": "Allow",
              "Principal": {
                "Service": "cloudtrail.amazonaws.com"
              },
              "Action": "s3:PutObject",
              "Resource": "arn:aws:s3:::cc-main-cloudtrail-bucket/cc-trail-logs/AWSLogs/${data.aws_caller_identity.current-account.account_id}/*",
              "Condition": {
                  "StringEquals": {
                      "s3:x-amz-acl": "bucket-owner-full-control"
                  }
              }
          }
      ]
  }
  POLICY

}

resource "aws_cloudtrail" "cloudtrail-trail" {

  name                          = "cc-main-cloud-trail"
  s3_bucket_name                = aws_s3_bucket.trail-s3-bucket.id
  s3_key_prefix                 = "cc-trail-logs"
  enable_logging                = true

  # Replace the Associated SNS Topic with a New Topic
  sns_topic_name = "arn:aws:sns:us-east-1:123456789012:cc-cloud-trail-new-topic"

}

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon CloudTrail console at https://console.aws.amazon.com/cloudtrail/.

03 In the navigation panel, under CloudTrail, choose Trails.

04 Click on the name (link) of the Amazon CloudTrail trail that you want to reconfigure.

05 In the General details section, choose Edit and perform the following operations:

Select New under Create a new SNS topic to create a new Amazon SNS topic and associate it with the selected trail.
Provide a unique name for the new topic in the SNS topic box.
Choose Save changes to apply the changes. The new Amazon SNS topic will get permissions to receive notifications whenever a trail log file is delivered to the target S3 bucket.

06 Navigate to Amazon SNS console at https://console.aws.amazon.com/sns/.

07 In the navigation panel, under Amazon SNS, choose Topics.

08 Click on the name of the SNS topic created at step no. 5, select the Subscriptions tab, and choose Create subscription.

09 On the Create subscription setup page, select Email from the Protocol dropdown list, provide the email address where you want to receive trail log notifications in the Endpoint box, then choose Create subscription to apply the new subscription to the selected Amazon SNS topic.

10 Use your preferred email client to open the subscription message from the AWS Notifications, then click on the appropriate link to confirm your SNS subscription.

11 Repeat steps no. 4 – 10 for each Amazon CloudTrail trail that you want to reconfigure, available in your AWS cloud account.

Using AWS CLI

01 Run create-topic command (OSX/Linux/UNIX) to create a new SNS topic for your Amazon CloudTrail trail:

aws sns create-topic
  --region us-east-1
  --name cc-cloud-trail-new-topic

02 The command output should return the Amazon Resource Name (ARN) of the newly created SNS topic:

{
  "TopicArn": "arn:aws:sns:us-east-1:123456789012:cc-cloud-trail-new-topic"
}

03 Run subscribe command (OSX/Linux/UNIX) to subscribe to the Amazon SNS topic created at the previous step using one or more email addresses as subscription endpoints:

aws sns subscribe
  --region us-east-1
  --topic-arn arn:aws:sns:us-east-1:123456789012:cc-cloud-trail-new-topic
  --protocol email
  --notification-endpoint alert@cloudconformity.com
  --return-subscription-arn

04 The command output should return the ARN of the new SNS subscription:

{
  "SubscriptionArn": "arn:aws:sns:us-east-1:123456789012:cc-cloud-trail-new-topic:abcdabcd-1234-abcd-1234-abcd1234abcd"
}

05 Run confirm-subscription command (OSX/Linux/UNIX) to confirm the new SNS subscription by validating the token sent to the subscription endpoint (i.e. your email address) specified at the previous step (the command should not produce an output):

aws sns confirm-subscription
  --region us-east-1
  --topic-arn arn:aws:sns:us-east-1:123456789012:cc-cloud-trail-new-topic
  --token 5014392f37fb687f5d51e6e241d7700ae02f7124d8268910b858cb4db727ceeb2474bb937929d3bdd7ce5d0cce19325d036bca58d3c217426bcafa9c501a2cac5646456gf1dd3797627467553dc438a8c974119496fc3eff026eaa5d15578ded6f9a5c43aec62d83ef5f49109da730881

06 Run update-trail command (OSX/Linux/UNIX) using the name of the Amazon CloudTrail trail that you want to reconfigure as the identifier parameter, to associate the selected trail with the newly created SNS topic:

aws cloudtrail update-trail
  --region us-east-1
  --name cc-main-cloud-trail
  --sns-topic-name cc-cloud-trail-new-topic

07 The command output should return the metadata available for the reconfigured trail:

{
    "IncludeGlobalServiceEvents": true,
    "IsOrganizationTrail": false,
    "Name": "cc-main-cloud-trail",
    "TrailARN": "arn:aws:cloudtrail:us-east-1:123456789012:trail/cc-main-cloud-trail",
    "LogFileValidationEnabled": false,
    "SnsTopicARN": "arn:aws:sns:us-east-1:123456789012:cc-cloud-trail-new-topic",
    "IsMultiRegionTrail": true,
    "S3BucketName": "aws-cloudtrail-log-bucket",
    "SnsTopicName": "cc-cloud-trail-new-topic"
}

08 Repeat steps no. 1 – 7 for each Amazon CloudTrail trail that you want to reconfigure, created within your AWS cloud account.

References

AWS Command Line Interface (CLI) Documentation
cloudtrail
list-trails
get-trail-status
update-trail
sns
create-topic
subscribe
confirm-subscription
s3api
create-bucket
put-public-access-block

CloudFormation Documentation
AWS CloudTrail resource type reference

Terraform Documentation
AWS Provider

Publication date Oct 17, 2017

Audit

Using AWS Console

Using AWS CLI

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS CloudFormation

Using Terraform (AWS Provider)

Using AWS Console

Using AWS CLI

Using AWS CloudFormation

Using Terraform (AWS Provider)

Using AWS Console

Using AWS CLI

References

Related CloudTrail rules