- Knowledge Base
- Amazon Web Services
- Amazon CloudFront
- Enable Real-Time Logging
Ensure that the Real-Time Logging monitoring feature is enabled for your Amazon CloudFront distributions in order to receive log information about requests made to your web content in real time. The real-time logs are delivered to the data stream of your choice in Amazon Kinesis Data Streams. You can build your own Kinesis data stream consumer or use Amazon Kinesis Data Firehose to send the log data to Amazon S3, Amazon Redshift, Amazon ElasticSearch Service, or a third-party log processing service. Amazon CloudFront real-time logs are configurable. You can choose:
- The sampling rate for your real-time logs, i.e. the percentage of requests for which you want to receive real-time log records.
- The specific fields that you want to record in real time.
- The specific cache behaviors (path patterns) that you want to receive real-time logs for.
Operationalexcellence
With Amazon CloudFront real-time logs you can get valuable information about requests made to your web content in real time (the log records are delivered within seconds of receiving the requests). You can use these real-time logs to monitor, analyze, and take action based on content delivery performance.
Audit
To determine if your Amazon CloudFront distributions are configured to record real-time logs, perform the following operations:
Using AWS Console
01 Sign in to the AWS Management Console.
02 Navigate to Amazon CloudFront console at https://console.aws.amazon.com/cloudfront/v3/.
03 In the left navigation panel, under CloudFront, choose Distributions.
04 Click on the name (link) of the active CloudFront distribution that you want to examine. An active distribution has the Status set to Enabled.
05 Select the Behaviors tab to access the cache behavior(s) configured for the selected distribution.
06 Select the cache behavior that you want to examine and choose Edit.
07 On the Edit behavior configuration page, click on Additional settings, and check the Enable real-time logs configuration setting. If the Enable real-time logs setting is set to No, the Real-Time Logging feature is not enabled for the selected cache behavior.
08 Repeat steps no. 6 and 7 to determine the feature status for each cache behavior associated with the selected distribution. If the Real-Time Logging feature is not enabled for the associated cache behavior(s), the selected Amazon CloudFront distribution is not configured to send real-time logs to an Amazon Kinesis data stream.
09 Repeat steps no. 4 – 8 for each active CloudFront distribution available within your AWS cloud account.
Using AWS CLI
01 Run list-distributions command (OSX/Linux/UNIX) with custom query filters to list the ID of each active Amazon CloudFront distribution created in your AWS account:
aws cloudfront list-distributions --output table --query 'DistributionList.Items[?(Enabled==`true`)].Id'
02 The command output should return a table with the requested distribution ID(s):
-------------------- |ListDistributions | +------------------+ | ABCDABCDABCDAB | | AABBCCDDAABBCC | +------------------+
03 Run get-distribution-config command (OSX/Linux/UNIX) using the ID of the active Amazon CloudFront distribution that you want to examine as the identifier parameter and custom query filters to describe the Amazon Resource Name (ARN) of the real-time log configuration attached to each custom cache behavior created for the selected distribution. A real-time log configuration contains information about which log fields to record, the sampling rate for log records, and the Kinesis data stream where the real-time logs are delivered:
aws cloudfront get-distribution-config --id ABCDABCDABCDAB --query 'DistributionConfig.CacheBehaviors.Items[*].RealtimeLogConfigArn'
04 The command output should return the requested configuration information:
[]
If the get-distribution-config command output returns an empty array (i.e. []), as shown in the example above, there are no real-time log configurations available for the associated cache behaviors, therefore the selected Amazon CloudFront distribution is not configured to send real-time logs to an Amazon Kinesis data stream.
05 Repeat steps no. 3 and 4 for each active CloudFront distribution deployed in your AWS cloud account.
Remediation / Resolution
To reconfigure your Amazon CloudFront distributions in order to enable the Real-Time Logging feature, perform the following operations:
Using AWS CloudFormation
01 CloudFormation template (JSON):
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"KinesisStreamArn": {
"Type": "String"
},
"KinesisStreamRoleArn": {
"Type": "String"
}
},
"Resources": {
"CloudFrontRealtimeLogConfig": {
"Type": "AWS::CloudFront::RealtimeLogConfig",
"Properties": {
"Name": "cc-real-time-log-config",
"EndPoints": [
{
"KinesisStreamConfig": {
"RoleArn": {
"Ref": "KinesisStreamRoleArn"
},
"StreamArn": {
"Ref": "KinesisStreamArn"
}
},
"StreamType": "Kinesis"
}
],
"Fields": [
"timestamp",
"c-ip",
"time-to-first-byte",
"x-edge-location",
"x-edge-request-id",
"x-host-header",
"x-forwarded-for",
"ssl-protocol",
"ssl-cipher"
],
"SamplingRate": 100
}
},
"CloudFrontDistribution": {
"Type": "AWS::CloudFront::Distribution",
"Properties": {
"DistributionConfig": {
"Enabled": true,
"Comment": "CloudFront CDN Web Distribution",
"Origins": [
{
"Id": "cc-cdn-origin",
"DomainName": "domain.com",
"CustomOriginConfig": {
"HTTPPort": 80,
"HTTPSPort": 443,
"OriginKeepaliveTimeout": 15,
"OriginSSLProtocols": [
"TLSv1.2"
],
"OriginProtocolPolicy": "https-only"
}
}
],
"DefaultRootObject": "index.html",
"DefaultCacheBehavior": {
"Compress": true,
"AllowedMethods": [
"HEAD",
"GET"
],
"DefaultTTL": 0,
"ForwardedValues": {
"Cookies": {
"Forward": "none"
},
"QueryString": false
},
"ViewerProtocolPolicy": "https-only",
"RealtimeLogConfigArn": {
"Ref": "CloudFrontRealtimeLogConfig"
}
},
"HttpVersion": "http2",
"ViewerCertificate": {
"CloudFrontDefaultCertificate": true
},
"PriceClass": "PriceClass_All"
}
}
}
}
}
02 CloudFormation template (YAML):
AWSTemplateFormatVersion: '2010-09-09' Parameters: KinesisStreamArn: Type: String KinesisStreamRoleArn: Type: String Resources: CloudFrontRealtimeLogConfig: Type: AWS::CloudFront::RealtimeLogConfig Properties: Name: cc-real-time-log-config EndPoints: - KinesisStreamConfig: RoleArn: !Ref 'KinesisStreamRoleArn' StreamArn: !Ref 'KinesisStreamArn' StreamType: Kinesis Fields: - timestamp - c-ip - time-to-first-byte - x-edge-location - x-edge-request-id - x-host-header - x-forwarded-for - ssl-protocol - ssl-cipher SamplingRate: 100 CloudFrontDistribution: Type: AWS::CloudFront::Distribution Properties: DistributionConfig: Enabled: true Comment: CloudFront CDN Web Distribution Origins: - Id: cc-cdn-origin DomainName: domain.com CustomOriginConfig: HTTPPort: 80 HTTPSPort: 443 OriginKeepaliveTimeout: 15 OriginSSLProtocols: - TLSv1.2 OriginProtocolPolicy: https-only DefaultRootObject: index.html DefaultCacheBehavior: Compress: true AllowedMethods: - HEAD - GET DefaultTTL: 0 ForwardedValues: Cookies: Forward: none QueryString: false ViewerProtocolPolicy: https-only RealtimeLogConfigArn: !Ref 'CloudFrontRealtimeLogConfig' HttpVersion: http2 ViewerCertificate: CloudFrontDefaultCertificate: true PriceClass: PriceClass_All
Using Terraform (AWS Provider)
01 Terraform configuration file (.tf):
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0"
}
}
required_version = ">= 0.14.9"
}
provider "aws" {
profile = "default"
region = "us-east-1"
}
resource "aws_cloudfront_realtime_log_config" "cloudfront-realtime-log-config" {
name = "cc-real-time-log-config"
sampling_rate = 100
fields = ["timestamp", "c-ip", "time-to-first-byte", "x-edge-location", "x-edge-request-id", "x-host-header", "x-forwarded-for", "ssl-protocol", "ssl-cipher"]
endpoint {
stream_type = "Kinesis"
kinesis_stream_config {
role_arn = "arn:aws:iam::123456789012:role/service-role/CloudFrontRealtimeLogConfigRole"
stream_arn = "arn:aws:kinesis:us-east-1:123456789012:stream/cc-real-time-log-stream"
}
}
}
resource "aws_cloudfront_distribution" "cloudfront-distribution" {
enabled = true
default_root_object = "index.html"
comment = "CloudFront CDN Web Distribution"
origin {
domain_name = "domain.com"
origin_id = "cc-cdn-origin"
custom_origin_config = {
http_port = 80
https_port = 443
origin_keepalive_timeout = 15
origin_ssl_protocols = ["TLSv1.2"]
origin_protocol_policy = "https-only"
}
}
default_cache_behavior {
compress = true
target_origin_id = "cc-domain-origin"
allowed_methods = ["GET", "HEAD"]
cached_methods = ["GET", "HEAD"]
forwarded_values {
query_string = false
cookies {
forward = "none"
}
}
min_ttl = 0
default_ttl = 3600
max_ttl = 86400
viewer_protocol_policy = "https-only"
realtime_log_config_arn = aws_cloudfront_realtime_log_config.cloudfront-realtime-log-config.arn
}
viewer_certificate {
cloudfront_default_certificate = true
}
price_class = "PriceClass_All"
}
Using AWS Console
01 Sign in to the AWS Management Console.
02 To be able to read and analyze your Amazon CloudFront real-time logs, you must send them to an Amazon Kinesis data stream. To set up a data stream, navigate to Amazon Kinesis console at https://console.aws.amazon.com/kinesis.
03 In the left navigation panel, under Amazon Kinesis, select Data streams.
04 Choose Create data stream to set up a new Amazon Kinesis data stream.
05 On the Create a data stream page, provide a unique name for the new stream in the Data stream name box, and enter the appropriate number of shards in the Number of open shards box to configure the data stream capacity. Choose Create data stream to create a new Amazon Kinesis data stream.
06 Navigate to Amazon CloudFront console at https://console.aws.amazon.com/cloudfront/v3/.
07 In the left navigation panel, under Telemetry, choose Logs.
08 Select the Real-time configurations tab and choose Create configuration to create a new real-time log configuration. The real-time log configuration contains information about which log fields you want to receive, the sampling rate for the log records, and the Kinesis data stream where you want to deliver the logs.
09 On the Create configuration setup page, perform the following actions:
- For Name, provide a unique name for the new real-time log configuration.
- For Sampling rate, enter the percentage of log records delivered, from 1% to 100%.
- Choose the fields that you want to include in the new configuration from the Fields dropdown list. By default, all fields are chosen. To remove a certain field from the configuration, click on the remove button (X icon) next to the field.
- For Endpoint, choose the Kinesis data stream where you want to receive real-time logs, created earlier in the Remediation process.
- For IAM role, choose Create new service role CloudFrontRealtimeLogConfigRole-name to enable the CloudFront service to create the required IAM role for you.
- For Distributions, select the ID of the Amazon CloudFront distribution that you want to monitor using the Real-Time Logging feature.
- For Cache behaviors, select the cache behavior(s) that you want to associate with the new real-time log configuration.
- Choose Create configuration to create the real-time log configuration and attach it to the specified Amazon CloudFront distribution. This will enable the Real-Time Logging feature for the selected distribution.
10 Repeat steps no. 7 – 9 to enable and configure real-time logging for each active Amazon CloudFront distribution available within your AWS cloud account.
Using AWS CLI
01 To read and analyze your Amazon CloudFront real-time logs, you must send the logs to an Amazon Kinesis data stream. Run create-stream command (OSX/Linux/UNIX) to create a new Amazon Kinesis data stream (the command does not produce an output):
aws kinesis create-stream --stream-name cc-real-time-log-stream --shard-count 3
02 Run describe-stream command (OSX/Linux/UNIX) with custom query filters to describe the Amazon Resource Name (ARN) of the Kinesis data stream created at the previous step:
aws kinesis describe-stream --stream-name cc-real-time-log-stream --query 'StreamDescription.StreamARN'
03 The command output should return the requested Amazon Resource Name (ARN):
"arn:aws:kinesis:us-east-1:123456789012:stream/cc-real-time-log-stream"
04 Run create-realtime-log-config command (OSX/Linux/UNIX) to create the real-time log configuration required to enable the Real-Time Logging feature for your Amazon CloudFront distribution:
aws cloudfront create-realtime-log-config
--name cc-real-time-log-config
--sampling-rate 100
--fields "timestamp" "c-ip" "time-to-first-byte" "x-edge-location" "x-edge-request-id" "x-host-header" "x-forwarded-for" "ssl-protocol" "ssl-cipher"
--end-points "StreamType=Kinesis,KinesisStreamConfig={RoleARN=arn:aws:iam::123456789012:role/service-role/CloudFrontRealtimeLogConfigRole,StreamARN=arn:aws:kinesis:us-east-1:123456789012:stream/cc-real-time-log-stream}"
--query 'RealtimeLogConfig.ARN'
05 The command output should return the ARN of the new real-time log configuration:
"arn:aws:cloudfront::123456789012:realtime-log-config/cc-real-time-log-config"
06 Run get-distribution-config command (OSX/Linux/UNIX) to extract all the configuration information from the active Amazon CloudFront distribution that you want to reconfigure:
aws cloudfront get-distribution-config --id ABCDABCDABCDAB --query 'DistributionConfig'
07 The command output should return the requested configuration information:
{
"CallerReference": "abcd1234-abcd-1234-abcd-1234abcd1234",
"Aliases": {
"Quantity": 0
},
"DefaultRootObject": "",
"Origins": {
"Quantity": 1,
"Items": [
{
"Id": "cloudconformity.com",
"DomainName": "cloudconformity.com",
"OriginPath": "",
"CustomHeaders": {
"Quantity": 0
},
"CustomOriginConfig": {
"HTTPPort": 80,
"HTTPSPort": 443,
"OriginProtocolPolicy": "http-only",
"OriginSslProtocols": {
"Quantity": 3,
"Items": [
"TLSv1",
"TLSv1.1",
"TLSv1.2"
]
},
"OriginReadTimeout": 30,
"OriginKeepaliveTimeout": 5
},
"ConnectionAttempts": 3,
"ConnectionTimeout": 10,
"OriginShield": {
"Enabled": false
}
}
]
},
"OriginGroups": {
"Quantity": 0
},
"DefaultCacheBehavior": {
"TargetOriginId": "cloudconformity.com",
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "allow-all",
"AllowedMethods": {
"Quantity": 2,
"Items": [
"HEAD",
"GET"
],
"CachedMethods": {
"Quantity": 2,
"Items": [
"HEAD",
"GET"
]
}
},
"SmoothStreaming": false,
"Compress": true,
"LambdaFunctionAssociations": {
"Quantity": 0
},
"FunctionAssociations": {
"Quantity": 0
},
"FieldLevelEncryptionId": "",
"ForwardedValues": {
"QueryString": false,
"Cookies": {
"Forward": "none"
},
"Headers": {
"Quantity": 0
},
"QueryStringCacheKeys": {
"Quantity": 0
}
},
"MinTTL": 0,
"DefaultTTL": 86400,
"MaxTTL": 31536000
},
"CacheBehaviors": {
"Quantity": 1,
"Items": [
{
"PathPattern": "/images",
"TargetOriginId": "cloudconformity.com",
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "allow-all",
"AllowedMethods": {
"Quantity": 2,
"Items": [
"HEAD",
"GET"
],
"CachedMethods": {
"Quantity": 2,
"Items": [
"HEAD",
"GET"
]
}
},
"SmoothStreaming": false,
"Compress": true,
"LambdaFunctionAssociations": {
"Quantity": 0
},
"FunctionAssociations": {
"Quantity": 0
},
"FieldLevelEncryptionId": "",
"ForwardedValues": {
"QueryString": false,
"Cookies": {
"Forward": "none"
},
"Headers": {
"Quantity": 0
},
"QueryStringCacheKeys": {
"Quantity": 0
}
},
"MinTTL": 0,
"DefaultTTL": 86400,
"MaxTTL": 31536000
}
]
},
"CustomErrorResponses": {
"Quantity": 0
},
"Comment": "",
"Logging": {
"Enabled": false,
"IncludeCookies": false,
"Bucket": "",
"Prefix": ""
},
"PriceClass": "PriceClass_100",
"Enabled": true,
"ViewerCertificate": {
"CloudFrontDefaultCertificate": true,
"MinimumProtocolVersion": "TLSv1",
"CertificateSource": "cloudfront"
},
"Restrictions": {
"GeoRestriction": {
"RestrictionType": "none",
"Quantity": 0
}
},
"WebACLId": "",
"HttpVersion": "http1.1",
"IsIPV6Enabled": false
}
08 Run get-distribution-config command (OSX/Linux/UNIX) to describe the current version of the configuration available for the selected distribution (i.e. eTag):
aws cloudfront get-distribution-config --id ABCDABCDABCDAB --query 'ETag'
09 The command output should return the requested information:
"AAAABBBBCCCCD"
10 Modify the configuration document returned at step no. 7 to attach the new real-time log configuration to each cache behavior associated with your CloudFront distribution, in order to send real-time logs to the specified Amazon Kinesis data stream. To attach a real-time log configuration to a custom cache behavior, add the"RealtimeLogConfigArn": "<real-time-log-config-arn>" attribute to the "CacheBehaviors.Items" array, where <real-time-log-config-arn> is the Amazon Resource Name (ARN) returned at step no. 5. Save the document with the modified distribution configuration to a JSON file named enable-real-time-logging.json:
{
"CallerReference": "abcd1234-abcd-1234-abcd-1234abcd1234",
"Aliases": {
"Quantity": 0
},
"DefaultRootObject": "",
"Origins": {
"Quantity": 1,
"Items": [
{
"Id": "cloudconformity.com",
"DomainName": "cloudconformity.com",
"OriginPath": "",
"CustomHeaders": {
"Quantity": 0
},
"CustomOriginConfig": {
"HTTPPort": 80,
"HTTPSPort": 443,
"OriginProtocolPolicy": "http-only",
"OriginSslProtocols": {
"Quantity": 3,
"Items": [
"TLSv1",
"TLSv1.1",
"TLSv1.2"
]
},
"OriginReadTimeout": 30,
"OriginKeepaliveTimeout": 5
},
"ConnectionAttempts": 3,
"ConnectionTimeout": 10,
"OriginShield": {
"Enabled": false
}
}
]
},
"OriginGroups": {
"Quantity": 0
},
"DefaultCacheBehavior": {
"TargetOriginId": "cloudconformity.com",
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "allow-all",
"AllowedMethods": {
"Quantity": 2,
"Items": [
"HEAD",
"GET"
],
"CachedMethods": {
"Quantity": 2,
"Items": [
"HEAD",
"GET"
]
}
},
"SmoothStreaming": false,
"Compress": true,
"LambdaFunctionAssociations": {
"Quantity": 0
},
"FunctionAssociations": {
"Quantity": 0
},
"FieldLevelEncryptionId": "",
"ForwardedValues": {
"QueryString": false,
"Cookies": {
"Forward": "none"
},
"Headers": {
"Quantity": 0
},
"QueryStringCacheKeys": {
"Quantity": 0
}
},
"MinTTL": 0,
"DefaultTTL": 86400,
"MaxTTL": 31536000
},
"CacheBehaviors": {
"Quantity": 1,
"Items": [
{
"PathPattern": "/images",
"TargetOriginId": "cloudconformity.com",
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"TrustedKeyGroups": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "allow-all",
"AllowedMethods": {
"Quantity": 2,
"Items": [
"HEAD",
"GET"
],
"CachedMethods": {
"Quantity": 2,
"Items": [
"HEAD",
"GET"
]
}
},
"SmoothStreaming": false,
"Compress": true,
"LambdaFunctionAssociations": {
"Quantity": 0
},
"FunctionAssociations": {
"Quantity": 0
},
"FieldLevelEncryptionId": "",
"ForwardedValues": {
"QueryString": false,
"Cookies": {
"Forward": "none"
},
"Headers": {
"Quantity": 0
},
"QueryStringCacheKeys": {
"Quantity": 0
}
},
"MinTTL": 0,
"DefaultTTL": 86400,
"MaxTTL": 31536000,
"RealtimeLogConfigArn": "<real-time-log-config-arn>"
}
]
},
"CustomErrorResponses": {
"Quantity": 0
},
"Comment": "",
"Logging": {
"Enabled": false,
"IncludeCookies": false,
"Bucket": "",
"Prefix": ""
},
"PriceClass": "PriceClass_100",
"Enabled": true,
"ViewerCertificate": {
"CloudFrontDefaultCertificate": true,
"MinimumProtocolVersion": "TLSv1",
"CertificateSource": "cloudfront"
},
"Restrictions": {
"GeoRestriction": {
"RestrictionType": "none",
"Quantity": 0
}
},
"WebACLId": "",
"HttpVersion": "http1.1",
"IsIPV6Enabled": false
}
11 Run update-distribution command (OSX/Linux/UNIX) using the document modified at the previous step (i.e. enable-real-time-logging.json) as the distribution configuration document, to reconfigure the selected Amazon CloudFront distribution in order to enable the Real-Time Logging feature. --if-match parameter represents the current version of the configuration, returned at step no. 9:
aws cloudfront update-distribution --id ABCDABCDABCDAB --if-match AAAABBBBCCCCD --distribution-config file://enable-real-time-logging.json --query 'Distribution.Status'
12 The command output should return the status of the modified CloudFront distribution:
"InProgress"
13 Repeat steps no. 6 – 12 to enable and configure real-time logging for each active Amazon CloudFront distribution deployed in your AWS cloud account.
References
- AWS Documentation
- Amazon CloudFront FAQs
- CloudFront logging
- Real-time logs
- AWS Command Line Interface (CLI) Documentation
- kinesis
- describe-stream
- create-stream
- cloudfront
- list-distributions
- get-distribution-config
- create-realtime-log-config
- update-distribution