EC2 Instance Scheduled Events

Risk Level: High (act immediately)

Rule ID: EC2-020

Determine if there are any Amazon EC2 instances scheduled for retirement and/or maintenance in your AWS cloud account and take the necessary steps (reboot, restart, or re-launch) to resolve them. The Amazon EC2 instances support multiple types of scheduled events such as Reboot (instance-reboot or system-reboot) – the instance or the underlying host machine is rebooted, Instance Stop (instance-stop) – the instance is stopped and started to migrate it to a new host machine, Instance Retirement (instance-retirement) – the instance is terminated and System Maintenance (system-maintenance) – the instance is taken offline for a short period of time, then rebooted.

This rule can help you with the following compliance standards:

APRA
MAS

For further details on compliance standards supported by Conformity, see here.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Reliability

Monitoring Amazon EC2 scheduled events within your AWS cloud account will help you prevent unexpected downtime and data loss, improving the reliability and availability of your EC2 instance fleet.

Note: This guide assumes that your Amazon EC2 instances are associated with Elastic IPs. If your instances do not have Elastic IPs attached, you will have to update their public IP reference(s) in your application code or within the DNS zone file after you stop and start the necessary instances, as each restarted instance gets a new public IP address.

Audit

To determine if Amazon Web Services (AWS) had scheduled any events for your Amazon EC2 instances, perform the following operations:

Using AWS Console

01 Sign in to the AWS Management Console.

01 Navigate to Amazon EC2 console at https://console.aws.amazon.com/ec2/.

01 On the Amazon EC2 console, check the Scheduled events section for any Amazon EC2 instances that have scheduled events assigned. If the Scheduled events status is set to "No scheduled events", there are no Amazon EC2 instances scheduled for retirement or maintenance within the current AWS cloud region. If the Scheduled events section lists one or more Amazon EC2 instances, there are EC2 instances scheduled for retirement/maintenance in the current AWS region. Click on the status link to access the Events page and identify the type of the scheduled event for each EC2 instance listed the Scheduled events section.

01 Change the AWS cloud region from the console navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run describe-instance-status command (OSX/Linux/UNIX) with predefined query filters to list the IDs of all the Amazon EC2 instances that have scheduled events assigned, available in the selected AWS cloud region:

aws ec2 describe-instance-status
  --region us-east-1
  --filters "Name=event.code,Values=instance-reboot,system-reboot,system-maintenance,instance-retirement,instance-stop"
  --output table --query 'InstanceStatuses[*].InstanceId'

02 The command output should return a table with the requested information. If the table returned by the describe-instance-status command output is empty, there are no Amazon EC2 instances scheduled for retirement or maintenance within the selected AWS region, otherwise the table will contain the IDs of the Amazon EC2 instances that have scheduled events allocated (as shown in the output example below):

-------------------------
|   DescribeInstances   |
+-----------------------+
|  i-01234abcd1234abcd  |
|  i-0abcd1234abcd1234  |
+-----------------------+

03 Run describe-instance-status command (OSX/Linux/UNIX) using the ID of the Amazon EC2 instance that you want to examine as the identifier parameter and custom query filters to describe the type of the scheduled event allocated for selected EC2 instance:

aws ec2 describe-instance-status
  --region us-east-1
  --instance-id i-01234abcd1234abcd
  --query 'InstanceStatuses[*].Events.Code'

04 The command output should return the scheduled event type for the selected instance:

[
	"instance-stop"
]

05 Repeat steps no. 3 and 4 for each scheduled EC2 instance available in the selected AWS cloud region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat the audit process for other regions.

Remediation / Resolution

To resolve the Amazon EC2 instances scheduled for retirement or maintenance based on the allocated event type, perform the following operations:

Actions required for the instance-reboot events:
IMPORTANT: The following process assumes that the Amazon EC2 instance selected for reboot is NOT currently used in production or for critical operations.

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon EC2 console at https://console.aws.amazon.com/ec2/.

03 In the navigation panel, under Instances, choose Instances.

04 Select the running Amazon EC2 instance scheduled for the instance-reboot event.

05 Click on the Instance state dropdown button from the console top menu and select Reboot instance.

06 In the Reboot instance? confirmation box, review the instance details, then choose Reboot to confirm the action. Once the boot sequence is complete, the status of the selected Amazon EC2 instance should change from Pending to Running (this may take a few minutes).

Using AWS CLI

01 Run describe-instances command (OSX/Linux/UNIX) using the ID of the Amazon EC2 instance that you want to reboot as the identifier parameter, to reboot the selected instance (if successful, the command request does not produce an output):

aws ec2 reboot-instances
  --region us-east-1
  --instance-ids i-01234abcd1234abcd

Actions required for the system-reboot event:

No action is required on your end. The system reboot is managed by AWS and occurs during the scheduled maintenance window. AWS will send you an email prior to the system-reboot event with all the necessary details, including the start and the end date of the event.

Actions required for the instance-stop or instance-retirement events:

IMPORTANT: The following process assumes that the Amazon EC2 instance selected for restart is NOT currently used in production or for critical operations.

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon EC2 console at https://console.aws.amazon.com/ec2/.

03 In the navigation panel, under Instances, choose Instances.

04 Select the running Amazon EC2 instance scheduled for instance-stop or instance-retirement events.

05 Click on the Instance state dropdown button from the console top menu and select Stop instance.

06 In the Stop instance? confirmation box, review the instance details, then choose Stop.

07 Choose again the Instance state dropdown button from the console top menu and select Start instance. Once the boot sequence is complete, the status of the selected Amazon EC2 instance should change from Pending to Running(this may take a few minutes).

Using AWS CLI

01 Run stop-instances command (OSX/Linux/UNIX) to stop the Amazon EC2 instance scheduled for instance-stop or instance-retirement events:

aws ec2 stop-instances
  --region us-east-1
  --instance-ids i-01234abcd1234abcd

02 The output should return the stop-instances command request metadata:

{
	"StoppingInstances": [
		{
			"InstanceId": "i-01234abcd1234abcd",
			"CurrentState": {
				"Code": 64,
				"Name": "stopping"
			},
			"PreviousState": {
				"Code": 16,
				"Name": "running"
			}
		}
	]
}

03 Run start-instances command (OSX/Linux/UNIX) to restart the Amazon EC2 instance stopped at the previous steps (it may take a few minutes until the instance enters the running state):

aws ec2 start-instances
    --region us-east-1
    --instance-ids i-01234abcd1234abcd

04 The output should return the start-instances command request metadata:

{
	"StartingInstances": [
		{
			"InstanceId": "i-01234abcd1234abcd",
			"CurrentState": {
				"Code": 0,
				"Name": "pending"
			},
			"PreviousState": {
				"Code": 80,
				"Name": "stopped"
			}
		}
	]
}

Actions required for the system-maintenance events:

IMPORTANT: The following process assumes that the Amazon EC2 instance selected for restart is NOT currently used in production or for critical operations.

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon EC2 console at https://console.aws.amazon.com/ec2/.

03 In the navigation panel, under Instances, choose Instances.

04 Select the running Amazon EC2 instance scheduled for the system-maintenance event.

05 Click on the Instance state dropdown button from the console top menu and select Stop instance.

06 In the Stop instance? confirmation box, review the instance details, then choose Stop.

Using AWS CLI

01 Run stop-instances command (OSX/Linux/UNIX) to stop the Amazon EC2 instance scheduled for the system-maintenance event:

aws ec2 stop-instances
  --region us-east-1
  --instance-ids i-01234abcd1234abcd

02 The output should return the stop-instances command request metadata:

{
	"StoppingInstances": [
		{
			"InstanceId": "i-01234abcd1234abcd",
			"CurrentState": {
				"Code": 64,
				"Name": "stopping"
			},
			"PreviousState": {
				"Code": 16,
				"Name": "running"
			}
		}
	]
}

03 Run start-instances command (OSX/Linux/UNIX) to restart the Amazon EC2 instance stopped at the previous steps (it may take a few minutes until the instance enters the running state):

aws ec2 start-instances
  --region us-east-1
  --instance-ids i-01234abcd1234abcd

04 The output should return the start-instances command request metadata:

{
	"StartingInstances": [
		{
			"InstanceId": "i-01234abcd1234abcd",
			"CurrentState": {
				"Code": 0,
				"Name": "pending"
			},
			"PreviousState": {
				"Code": 80,
				"Name": "stopped"
			}
		}
	]
}

References

AWS Command Line Interface (CLI) Documentation
ec2
describe-instance-status
reboot-instances
stop-instances
start-instances

Publication date Jun 23, 2016

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS Console

Using AWS CLI

Actions required for the system-reboot event:

Actions required for the instance-stop or instance-retirement events:

Using AWS Console

Using AWS CLI

Actions required for the system-maintenance events:

Using AWS Console

Using AWS CLI

References

Related EC2 rules