Enable Capacity Rebalancing

Risk Level: Medium (should be achieved)

Ensure that your Amazon Auto Scaling Groups (ASGs) are configured with the Capacity Rebalancing feature in order to monitor and automatically respond to changes that affect the availability of the Amazon EC2 Spot Instances running within your ASGs. Capacity Rebalancing helps you maintain application availability by proactively augmenting your ASG fleet with a new Spot Instance before a running instance is interrupted by AWS.

Security

Reliability

Amazon EC2 Auto Scaling is always aware of EC2 instance rebalance recommendation notifications. The cloud service emits these notifications when Spot Instances are at elevated risk of interruption. When the Capacity Rebalancing feature is enabled for your Auto Scaling group (ASG), Amazon EC2 attempts to proactively replace Spot Instances that have received a rebalance recommendation notification, providing the opportunity to rebalance your workload to new Spot Instances that are not at elevated risk of interruption. This means that your application can continue to process the requests while Amazon EC2 Auto Scaling launches a new Spot Instance before an existing instance is interrupted.

Audit

To determine if Capacity Rebalancing is enabled for your Auto Scaling Groups (ASGs), perform the following operations:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon EC2 console at https://console.aws.amazon.com/ec2.

03 In the navigation panel, under Auto Scaling, choose Auto Scaling Groups.

04 Select the Auto Scaling Group (ASG) that you want to examine.

05 Choose the Details tab from the console bottom panel and check the Capacity rebalance attribute value available in the Purchase options and instance types section. If Capacity rebalance is set to Off, the Capacity Rebalancing feature is not enabled for the selected Amazon Auto Scaling Group (ASG).

06 Repeat steps no. 4 and 5 for each Auto Scaling Group available in the current AWS cloud region.

07 Change the AWS region from the navigation bar and repeat steps no. 4 – 6 for other cloud regions.

Using AWS CLI

01 Run describe-auto-scaling-groups command (OSX/Linux/UNIX) to list the name of each Amazon Auto Scaling Group (ASG) provisioned in the selected AWS region:

aws autoscaling describe-auto-scaling-groups
  --region us-east-1
  --output table
  --query 'AutoScalingGroups[*].AutoScalingGroupName'

02 The command output should return a table with the requested ASG names:

---------------------------
|DescribeAutoScalingGroups|
+-------------------------+
|  cc-production-web-asg  |
|  cc-project5-stage-asg  |
+-------------------------+

03 Run describe-auto-scaling-groups command (OSX/Linux/UNIX) using the name of the Auto Scaling Group (ASG) that you want to examine as identifier parameter and custom query filters to describe the Capacity Rebalancing feature status available for the selected ASG:

aws autoscaling describe-auto-scaling-groups
  --region us-east-1
  --auto-scaling-group-names cc-production-web-asg
  --query 'AutoScalingGroups[*].CapacityRebalance'

04 The command output should return the requested configuration status (true for enabled, false for disabled):

[
	false
]

If the describe-auto-scaling-groups command output returns false, as shown in the example above, the Capacity Rebalancing feature is not enabled for the selected Amazon Auto Scaling Group (ASG).

05 Repeat step no. 3 and 4 for each Auto Scaling Group deployed in the selected AWS cloud region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the audit process for other regions.

Remediation / Resolution

When Capacity Rebalancing is enabled and a rebalance notification is sent to an EC2 instance, the Auto Scaling Group automatically attempts to replace the instance before it is interrupted. To enable the Capacity Rebalancing feature for existing Amazon Auto Scaling Groups (ASGs), perform the following actions:

Using AWS CloudFormation

01 CloudFormation template (JSON):

{
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "Enable Capacity Rebalancing",
	"Resources": {
		"ASGLaunchTemplate": {
			"Type": "AWS::EC2::LaunchTemplate",
			"Properties": {
				"LaunchTemplateName": "asg-launch-template",
				"LaunchTemplateData": {
					"ImageId": "ami-01234abcd1234abcd",
					"InstanceType": "c5.xlarge",
					"SecurityGroupIds": [
						"sg-01234123412341234"
					],
					"BlockDeviceMappings": [
						{
							"DeviceName": "/dev/sda1"
						},
						{
							"Ebs": {
								"VolumeSize": 30
							}
						}
					]
				}
			}
		},
		"EC2AutoScalingGroup": {
			"Type": "AWS::AutoScaling::AutoScalingGroup",
			"Properties": {
				"LaunchTemplate": {
					"LaunchTemplateId": {
						"Ref": "ASGLaunchTemplate"
					},
					"Version": {
						"Fn::GetAtt": [
							"ASGLaunchTemplate",
							"LatestVersionNumber"
						]
					}
				},
				"AvailabilityZones": [
					"us-east-1a",
					"us-east-1b"
				],
				"MinSize": "1",
				"MaxSize": "1",
				"DesiredCapacity": "1",
				"Cooldown": "350",
				"CapacityRebalance": true
			}
		}
	}
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
	Description: Enable Capacity Rebalancing
	Resources:
	ASGLaunchTemplate:
		Type: AWS::EC2::LaunchTemplate
		Properties:
		LaunchTemplateName: asg-launch-template
		LaunchTemplateData:
			ImageId: ami-01234abcd1234abcd
			InstanceType: c5.xlarge
			SecurityGroupIds:
			- sg-01234123412341234
			BlockDeviceMappings:
			- DeviceName: "/dev/sda1"
			- Ebs:
			VolumeSize: 30
	EC2AutoScalingGroup:
		Type: AWS::AutoScaling::AutoScalingGroup
		Properties:
		LaunchTemplate:
			LaunchTemplateId:
			Ref: ASGLaunchTemplate
			Version:
			Fn::GetAtt:
			- ASGLaunchTemplate
			- LatestVersionNumber
		AvailabilityZones:
		- us-east-1a
		- us-east-1b
		MinSize: '1'
		MaxSize: '1'
		DesiredCapacity: '1'
		Cooldown: '350'
		CapacityRebalance: true

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
	required_providers {
		aws = {
			source  = "hashicorp/aws"
			version = "~> 4.0"
		}
	}

	required_version = ">= 0.14.9"
}

provider "aws" {
	profile = "default"
	region  = "us-east-1"
}

resource "aws_launch_template" "asg-launch-template" {
	name_prefix            = "asg-launch-template"
	image_id               = "ami-01234abcd1234abcd"
	instance_type          = "c5.xlarge"
	vpc_security_group_ids = [ "sg-01234123412341234" ]
	block_device_mappings {
	device_name = "/dev/sda1"
		ebs {
		volume_size = 30
		}
	}

	capacity_rebalance = true

}

resource "aws_autoscaling_group" "ec2-auto-scaling-group" {
	availability_zones = [ "us-east-1a","us-east-1b" ]
	desired_capacity   = 1
	max_size           = 1
	min_size           = 1
	default_cooldown   = 350
	launch_template {
		id      = aws_launch_template.asg-launch-template.id
		version = "$Latest"
	}
}

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon EC2 console at https://console.aws.amazon.com/ec2.

03 In the navigation panel, under Auto Scaling, choose Auto Scaling Groups.

04 Select the Auto Scaling Group (ASG) that you want to reconfigure (see Audit section part I to identify the right ASG resource) and choose Edit from the console top-right menu.

05 On the Edit <auto-scaling-group-name> configuration page, in the Instances distribution section, select the Capacity rebalance checkbox available under Capacity optimized Spot settings to enable the Capacity Rebalancing feature for the selected Amazon Auto Scaling Group (ASG). Choose Update to apply the configuration changes.

06 Repeat steps no. 4 and 5 to enable Capacity Rebalancing for other Auto Scaling Groups available in the current AWS cloud region.

07 Change the AWS cloud region from the navigation bar and repeat the remediation process for other regions.

Using AWS CLI

01 Run update-auto-scaling-group command (OSX/Linux/UNIX) using the name of the Amazon Auto Scaling Group (ASG) that you want to reconfigure as the identifier parameter (see Audit section part II to identify the right resource), to enable the Capacity Rebalancing feature for the selected ASG (if successful, the command request does not produce an output):

aws autoscaling update-auto-scaling-group
  --region us-east-1
  --auto-scaling-group-name cc-production-web-asg
  --capacity-rebalance

02 Repeat step no. 1 to turn on Capacity Rebalancing for other Auto Scaling Groups deployed in the selected AWS region.

03 Change the AWS region by updating the --region command parameter value and repeat step no. 1 and 2 to perform the remediation process for other regions.

References

AWS Documentation
Auto Scaling groups
Use Capacity Rebalancing to handle Amazon EC2 Spot interruptions
EC2 instance rebalance recommendations

AWS Command Line Interface (CLI) Documentation
autoscaling
describe-auto-scaling-groups
update-auto-scaling-group

Publication date Sep 8, 2023

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS CloudFormation

Using Terraform (AWS Provider)

Using AWS Console

Using AWS CLI

References

Related EC2 rules