Enable Dead Letter Queue for Lambda Functions

Risk Level: Low (generally tolerable level of risk)

Ensure that each Amazon Lambda function created within your AWS cloud account is configured to use a Dead Letter Queue (DLQ) in order to send unprocessed events from asynchronous invocations to an Amazon SQS queue or an Amazon SNS topic.

This rule can help you work with the AWS Well-Architected Framework.

Reliability

Operational
excellence

When an event fails all attempts or stays in the asynchronous invocation queue for too long, Amazon Lambda service discards it. Enabling Dead Letter Queues (DLQs) for your Amazon Lambda functions can make your serverless application more resilient by capturing and storing unprocessed events from asynchronous invocations for further analysis or reprocessing. Configuring Dead Letter Queues for Amazon Lambda functions will give you more control over message handling for all asynchronous invocations, including those delivered via AWS service events (S3, SNS, IoT, etc.).

Audit

To determine if your Amazon Lambda functions are configured to use Dead Letter Queues (DLQs), perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon Lambda console at https://console.aws.amazon.com/lambda/.

03 In the left navigation panel, under AWS Lambda, choose Functions.

04 Click on the name (link) of the function that you want to examine.

05 Select the Configuration tab and choose Asynchronous invocation from the left menu.

06 In the Asynchronous invocation section, check the Dead-letter queue service attribute value. If the attribute value is set to None, the selected Amazon Lambda function is not associated with a Dead-Letter Queue (DLQ) required to send discarded events to an Amazon SQS queue or an Amazon SNS topic.

07 Repeat steps no. 4 – 6 for each Lambda function available within the current AWS region.

08 Change the AWS cloud region from the console navigation bar and repeat the Audit process for other regions.

Using AWS CLI

01 Run list-functions command (OSX/Linux/UNIX) to list the names of all the Amazon Lambda functions available in the selected AWS region:

aws lambda list-functions
  --region us-east-1
  --output table
  --query 'Functions[*].FunctionName'

02 The command output should return a table with the requested function name(s):

--------------------------
|      ListFunctions     |
+------------------------+
|  cc-process-app-queue  |
|  cc-export-user-data   |
|  cc-get-s3-log-data    |
+------------------------+

03 Run get-function-configuration command (OSX/Linux/UNIX) using the name of the Amazon Lambda function that you want to examine as the identifier parameter and custom query filters to describe the Amazon Resource Name (ARN) of the SQS queue or SNS topic used as target by the Dead-Letter Queue:

aws lambda get-function-configuration
  --region us-east-1
  --function-name cc-process-app-queue
  --query 'DeadLetterConfig.TargetArn'

04 The command output should return the ARN of the associated SQS queue or SNS topic:

null

If the get-function-configuration command output returns null, as shown in the example above, the selected Amazon Lambda function is not associated with a Dead-Letter Queue (DLQ) required to send discarded events to an Amazon SQS queue or AN Amazon SNS topic.

05 Repeat step no. 3 and 4 for each Lambda function available in the selected AWS region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the Audit process for other regions.

Remediation / Resolution

To enable and configure Dead Letter Queues (DLQs) for your existing Amazon Lambda functions, perform the following actions:

As an example, this section demonstrates how to configure a Dead-Letter Queue to send failed (discarded) events to an Amazon SQS queue. The Amazon SQS queue will hold failed events until they are retrieved. You can retrieve events manually, or you can configure Amazon Lambda to read from the queue and invoke a function.

Using AWS CloudFormation

01 CloudFormation template (JSON):

{
	"AWSTemplateFormatVersion": "2010-09-09",
	"Resources": {
		"FunctionExecutionRole": {
			"Type": "AWS::IAM::Role",
			"Properties": {
				"RoleName": "LambdaExecutionRole",
				"AssumeRolePolicyDocument": {
					"Version": "2012-10-17",
					"Statement": [
						{
							"Effect": "Allow",
							"Principal": {
								"Service": [
									"lambda.amazonaws.com"
								]
							},
							"Action": [
								"sts:AssumeRole"
							]
						}
					]
				},
				"Path": "/",
				"Policies": [
					{
						"PolicyName": "AWSLambdaBasicExecutionRole",
						"PolicyDocument": {
							"Version": "2012-10-17",
							"Statement": [
								{
									"Effect": "Allow",
									"Action": [
										"logs:CreateLogGroup",
										"logs:CreateLogStream",
										"logs:PutLogEvents",
										"ec2:DescribeNetworkInterfaces",
										"ec2:CreateNetworkInterface",
										"ec2:DeleteNetworkInterface",
										"ec2:DescribeInstances",
										"ec2:AttachNetworkInterface"
									],
									"Resource": "*"
								}
							]
						}
					}
				]
			}
		},
		"LambdaFunctionDLQ": {
			"Type": "AWS::SQS::Queue",
			"Properties": {
				"QueueName": "cc-lambda-function-dlq",
				"MessageRetentionPeriod": 3600,
				"VisibilityTimeout": 10
			}
		},
		"LambdaFunction": {
			"Type": "AWS::Lambda::Function",
			"Properties": {
				"FunctionName": "cc-app-worker-function",
				"Handler": "lambda_function.lambda_handler",
				"Role": {
					"Fn::GetAtt": [
						"FunctionExecutionRole",
						"Arn"
					]
				},
				"Code": {
					"S3Bucket": "cc-lambda-functions",
					"S3Key": "worker.zip"
				},
				"Runtime": "python3.9",
				"MemorySize": 1024,
				"Timeout": 45,
				"VpcConfig": {
					"SecurityGroupIds": [
						"sg-0abcd1234abcd1234"
					],
					"SubnetIds": [
						"subnet-abcd1234",
						"subnet-1234abcd"
					]
				},
				"DeadLetterConfig": null,
				"TargetArn": {
					"Fn::GetAtt": [
						"LambdaFunctionDLQ",
						"Arn"
					]
				}
			}
		}
	}
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
	Resources:
	FunctionExecutionRole:
		Type: AWS::IAM::Role
		Properties:
		RoleName: LambdaExecutionRole
		AssumeRolePolicyDocument:
			Version: '2012-10-17'
			Statement:
			- Effect: Allow
				Principal:
				Service:
					- lambda.amazonaws.com
				Action:
				- sts:AssumeRole
		Path: /
		Policies:
			- PolicyName: AWSLambdaBasicExecutionRole
			PolicyDocument:
				Version: '2012-10-17'
				Statement:
				- Effect: Allow
					Action:
					- logs:CreateLogGroup
					- logs:CreateLogStream
					- logs:PutLogEvents
					- ec2:DescribeNetworkInterfaces
					- ec2:CreateNetworkInterface
					- ec2:DeleteNetworkInterface
					- ec2:DescribeInstances
					- ec2:AttachNetworkInterface
					Resource: '*'
	LambdaFunctionDLQ:
		Type: AWS::SQS::Queue
		Properties:
		QueueName: cc-lambda-function-dlq
		MessageRetentionPeriod: 3600
		VisibilityTimeout: 10
	LambdaFunction:
		Type: AWS::Lambda::Function
		Properties:
		FunctionName: cc-app-worker-function
		Handler: lambda_function.lambda_handler
		Role: !GetAtt 'FunctionExecutionRole.Arn'
		Code:
			S3Bucket: cc-lambda-functions
			S3Key: worker.zip
		Runtime: python3.9
		MemorySize: 1024
		Timeout: 45
		VpcConfig:
			SecurityGroupIds:
			- sg-0abcd1234abcd1234
			SubnetIds:
			- subnet-abcd1234
			- subnet-1234abcd
		DeadLetterConfig: null
		TargetArn: !GetAtt 'LambdaFunctionDLQ.Arn'

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
	required_providers {
		aws = {
			source  = "hashicorp/aws"
			version = "~> 4.0"
		}
	}

	required_version = ">= 0.14.9"
}

provider "aws" {
	profile = "default"
	region  = "us-east-1"
}

resource "aws_iam_role" "lambda-execution-role" {
	name = "LambdaExecutionRole"
	path = "/"
	managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"]

	assume_role_policy = <<EOF
	{
		"Version": "2012-10-17",
		"Statement": [
			{
				"Action": "sts:AssumeRole",
				"Principal": {
					"Service": "lambda.amazonaws.com"
				},
				"Effect": "Allow"
			}
		]
	}
	EOF
}

resource "aws_sqs_queue" "lambda-function-dlq" {
	name                       = "cc-lambda-function-dlq"
	message_retention_seconds  = 3600
	visibility_timeout_seconds = 10
}

resource "aws_lambda_function" "lambda-function" {
	function_name    = "cc-app-worker-function"
	s3_bucket        = "cc-lambda-functions"
	s3_key           = "worker.zip" 
	role             = aws_iam_role.lambda-execution-role.arn
	handler          = "lambda_function.lambda_handler"
	runtime          = "python3.9"
	memory_size      = 1024
	timeout          = 45   
	vpc_config {
		subnet_ids         = [ "subnet-01234abcd1234abcd", "subnet-0abcd1234abcd1234" ]
		security_group_ids = [ "sg-0abcd1234abcd1234" ]
	}
	dead_letter_config = {
		target_arn = aws_sqs_queue.lambda-function-dlq.arn
	}
}

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon SQS console at https://console.aws.amazon.com/sqs/.

03 In the left navigation panel, under Amazon SQS, select Queues.

04 Choose Create queue to initiate the SQS queue setup process.

05 On the Create queue setup page, select Standard for Type to create a standard SQS queue and provide a unique name for the new queue in the Name box. Leave the default configuration parameters unchanged, and choose Create queue to launch your new Amazon SQS queue.

06 Navigate to Amazon Lambda console at https://console.aws.amazon.com/lambda/.

07 In the left navigation panel, under AWS Lambda, choose Functions.

08 Click on the name of the function that you want to reconfigure.

09 Select the Configuration tab and choose Asynchronous invocation from the left menu.

10 Click on the Edit button available in the Asynchronous invocation section to modify the function's asynchronous configuration.

11 Select Amazon SQS from the Dead-letter queue service dropdown menu and choose the SQS queue created at step no. 5 from the Queue dropdown list. Choose Save to apply the changes.

12 Your Lambda function's execution role requires permission to write to the SQS queue configured as Dead Letter Queue (DLQ) at the previous step, therefore make sure that your function's execution role has the permissions to deliver a message to the specified queue, i.e. sqs:SendMessage action.

13 Repeat steps no. 8 – 12 to configure a Dead Letter Queue (DLQ) for each Amazon Lambda function available within the current AWS region.

14 Change the AWS cloud region from the navigation bar and repeat the Remediation process for the other regions.

Using AWS CLI

01 Run create-queue command (OSX/Linux/UNIX) to create the standard Amazon SQS queue that will serve as Dead Letter Queue (DLQ) for the specified Lambda function:

aws sqs create-queue
  --region us-east-1
  --queue-name cc-dead-letter-queue

02 The command output should return the complete URL of the new SQS queue:

{
	"QueueUrl": "https://queue.amazonaws.com/123456789012/cc-dead-letter-queue"
}

03 Run get-queue-attributes command (OSX/Linux/UNIX) to describe the Amazon Resource Name (ARN) of the newly created Amazon SQS queue:

aws sqs get-queue-attributes
  --region us-east-1
  --queue-url https://queue.amazonaws.com/123456789012/cc-dead-letter-queue
  --attribute-names QueueArn
  --query 'Attributes.QueueArn'

04 The command output should return the requested Amazon Resource Name (ARN):

"arn:aws:sqs:us-east-1:123456789012:cc-dead-letter-queue"

05 Run update-function-configuration command (OSX/Linux/UNIX) using the name of the Amazon Lambda function that you want to reconfigure as the identifier parameter, to update the function's asynchronous configuration with a Dead-Letter Queue (DLQ) that specifies the SQS queue where Amazon Lambda sends asynchronous events when they fail processing:

aws lambda update-function-configuration
  --region us-east-1
  --function-name cc-process-app-queue
  --dead-letter-config TargetArn=arn:aws:sqs:us-east-1:123456789012:cc-dead-letter-queue

06 The command output should return the configuration metadata for the modified Lambda function:

{
	"LastUpdateStatus": "Successful",
	"FunctionName": "cc-process-app-queue",
	"LastModified": "2020-10-08T10:10:10.000+0000",
	"RevisionId": "abcdabcd-1234-abcd-1234-abcd1234abcd",
	"MemorySize": 128,
	"State": "Active",
	"Version": "$LATEST",
	"Role": "arn:aws:iam::123456789012:role/service-role/cc-funct-role-st3x096p",
	"Timeout": 3,
	"DeadLetterConfig": {
		"TargetArn": "arn:aws:sqs:us-east-1:123456789012:cc-dead-letter-queue"
	},
	"Runtime": "nodejs12.x",
	"TracingConfig": {
		"Mode": "PassThrough"
	},
	"CodeSha256": "abcdabcdabcdabcdabcdabcdabcdabcdabcd",
	"Description": "",
	"CodeSize": 304,
	"FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:cc-process-app-queue",
	"Handler": "index.handler"
}

07 Your Lambda function's execution role requires permission to write to the SQS queue configured as Dead Letter Queue (DLQ) at the previous steps, therefore make sure that your function's execution role has the permissions to deliver a message to the specified queue, i.e. sqs:SendMessage action.

08 Repeat steps no. 5 – 7 to configure a Dead Letter Queue (DLQ) for each Amazon Lambda function available in the selected AWS region.

09 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 8 to perform the Remediation process for other regions.

References

AWS Command Line Interface (CLI) Documentation
lambda
list-functions
get-function-configuration
update-function-configuration
sqs
create-queue
get-queue-attributes

Publication date Sep 12, 2021

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS CloudFormation

Using Terraform (AWS Provider)

Using AWS Console

Using AWS CLI

References

Related Lambda rules