OpenSearch Free Storage Space

Risk Level: High (not acceptable risk)

Rule ID: ES-014

Identify any Amazon OpenSearch clusters that appear to run low on disk space and scale them up to help mitigate any issues triggered by insufficient disk space and improve their I/O performance. The default threshold value set for the amount of free storage space is 10% because any value below this could have a serious impact on your OpenSearch clusters performance. For example, if the free storage space becomes dangerously low, your clusters can start blocking incoming write requests.

The Amazon CloudWatch metric used to detect OpenSearch clusters with low free storage space is:

FreeStorageSpace – the amount of available storage space for all data nodes within the cluster. Amazon OpenSearch throws an "ClusterBlockException" error when this metric reaches 0.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Performance
efficiency

Low disk space leads to instability and slowdowns. When OpenSearch clusters run out of free storage space, basic write operations like adding documents and creating indices begin to fail. Detecting Amazon OpenSearch clusters that run low on disk space is crucial, especially when these cloud resources are used in production.

Note: You can change the default threshold value (10%) for this rule on Conformity account console and set your own value for the amount of available storage space to configure the storage limits for your OpenSearch clusters.

Audit

To identify Amazon OpenSearch clusters that run low on disk space, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon OpenSearch console at https://console.aws.amazon.com/esv3/.

03 In the main navigation panel, under Dashboard, select Domains.

04 Click on the name (link) of the OpenSearch cluster that you want to examine.

05 Select the Cluster configuration tab and check the Number of nodes and EBS volume size attributes value available in the Data nodes section to determine the total amount of storage space allocated for your cluster data nodes.

06 Select the Cluster health tab and click on the Total free storage space graph thumbnail displayed in the Overall health section to open the panel with the total amount of free storage space across all data nodes. On the Total free storage space (FreeStorageSpace) panel, set the following parameters:

From the Period dropdown list, select 1 Hour.
From the Time range list, select 24h.
From the Statistic dropdown list, select Sum. Once the monitoring data is loaded, check the total free storage space available, in GiB, for the selected OpenSearch cluster. Choose x (close) to close the panel.

07 Based on storage information gathered at steps no. 5 and 6, if the amount of free storage space is less than 10%, the selected Amazon OpenSearch cluster has insufficient disk space to perform optimally, therefore adding EBS-based storage to the existing data nodes is highly recommended.

08 Repeat steps no. 4 – 7 for each Amazon OpenSearch domain available within the current AWS region.

09 Change the AWS cloud region from the navigation bar and repeat the Audit process for other regions.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the name of each Amazon OpenSearch domain available in the selected AWS region:

aws es list-domain-names
  --region us-east-1
  --query 'DomainNames[*].DomainName'

02 The command output should return the identifier (name) of each OpenSearch domain provisioned in the selected region:

[
	"trendmicro",
	"cloudconformity"
]

03 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the Amazon OpenSearch cluster that you want to examine as the identifier parameter and custom query filters to get information about the cluster data nodes and their allocated storage:

aws es describe-elasticsearch-domain
  --region us-east-1
  --domain-name trendmicro
  --query 'DomainStatus.[{"NodeCount":ElasticsearchClusterConfig.InstanceCount,"VolumeSize":EBSOptions.VolumeSize}]'

04 The command output should return the information required to compute the amount of storage space, in GiB, allocated for the selected OpenSearch cluster:

[
	{
		"NodeCount": 2,
		"VolumeSize": 15
	}
]

05 Run get-metric-statistics command (OSX/Linux/UNIX) to obtain the statistics recorded by Amazon CloudWatch for the FreeStorageSpace metric, representing the total amount of free storage space across all data nodes within the cluster. Change the--start-time (start recording date) and --end-time (stop recording date) parameters value to choose your own time frame for recording the FreeStorageSpace metric usage. Set the --period parameter value to define the granularity (in seconds) of the returned datapoints, based on your requirements. A period can be as short as 1 minute (60 seconds) or as long as 1 day (86400 seconds). The following command example returns the total amount of free storage space available for an OpenSearch domain named "trendmicro", usage data captured over a time period of 24 hours, using 1-hour period as the granularity for the returned datapoints:

aws cloudwatch get-metric-statistics
  --region us-east-1
  --metric-name FreeStorageSpace
  --start-time 2018-02-01T12:21:05
  --end-time 2018-02-02T12:21:05
  --period 3600
  --namespace AWS/ES
  --statistics Sum
  --dimensions Name=DomainName,Value=trendmicro

06 The command output should return the storage space details requested (in Megabytes):

{
	"Datapoints": [
		{
			"Timestamp": "2018-02-01T12:21:05Z",
			"Sum": 1,410.4770,
			"Unit": "Megabytes"
		},
		{
			"Timestamp": "2018-02-01T13:21:05Z",
			"Sum": 1,410.4765,
			"Unit": "Megabytes"
		},
		{
			"Timestamp": "2018-02-01T14:21:05Z",
			"Sum": 1,410.4760,
			"Unit": "Megabytes"
		},

		...

		{
			"Timestamp": "2018-02-02T10:21:05Z",
			"Sum": 1,410.4700,
			"Unit": "Megabytes"
		},
		{
			"Timestamp": "2018-02-02T11:21:05Z",
			"Sum": 1,410.4695,
			"Unit": "Megabytes"
		},
		{
			"Timestamp": "2018-02-02T12:21:05Z",
			"Sum": 1,410.4690,
			"Unit": "Megabytes"
		}
	],
	"Label": "FreeStorageSpace"
}

07 Based on storage information gathered at steps no. 4 and 6, if the amount of free storage space is less than 10%, the selected Amazon OpenSearch cluster has insufficient disk space to perform optimally, therefore adding EBS-based storage to the existing data nodes is strongly recommended.

08 Repeat steps no. 3 – 7 for each Amazon OpenSearch cluster available in the selected AWS region.

09 Change the AWS cloud region by updating the --region command parameter value and repeat the Audit process for other regions.

Remediation / Resolution

To add storage space to the Amazon OpenSearch clusters that run low on disk space, you can scale them up by adding storage to the existing data node volumes. To scale up and recover from the lack of free disk space, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon OpenSearch console at https://console.aws.amazon.com/esv3/.

03 In the main navigation panel, under Dashboard, select Domains.

04 Select the OpenSearch cluster that you want to reconfigure, choose Actions from the console top menu, and select Edit cluster configuration.

05 In the Data nodes section, set a new storage size value (in GiB) within the EBS storage size per node box to upgrade the storage space for each data node volume provisioned within the selected OpenSearch cluster. Choose Save changes to apply the configuration changes.

06 Repeat steps no. 4 and 5 to expand the storage space for other Amazon OpenSearch clusters available within the current AWS region.

07 Change the AWS cloud region from the navigation bar and repeat the Remediation process for other regions.

Using AWS CLI

01 Run update-elasticsearch-domain-config command (OSX/Linux/UNIX) using the name of the Amazon OpenSearch cluster that you want to reconfigure as the identifier parameter to upgrade the storage space for each data node volume provisioned for the selected OpenSearch cluster:

aws es update-elasticsearch-domain-config
  --domain-name trendmicro
  --region us-east-1
  --ebs-options EBSEnabled=true,VolumeType="gp2",VolumeSize=50

02 The command output should return the configuration information available for the reconfigured cluster:

{
	"DomainConfig": {
		"ElasticsearchVersion": {
			"Options": "7.9",
			"Status": {
				"CreationDate": "2021-12-21T14:44:37.462000+00:00",
				"UpdateDate": "2021-12-21T14:57:39.078000+00:00",
				"UpdateVersion": 5,
				"State": "Active",
				"PendingDeletion": false
			}
		},
		"ElasticsearchClusterConfig": {
			"Options": {
				"InstanceType": "m4.large.elasticsearch",
				"InstanceCount": 2,
				"DedicatedMasterEnabled": false,
				"ZoneAwarenessEnabled": true,
				"ZoneAwarenessConfig": {
					"AvailabilityZoneCount": 2
				},
				"WarmEnabled": false,
				"ColdStorageOptions": {
					"Enabled": false
				}
			},
			"EBSOptions": {
				"Options": {
					"EBSEnabled": true,
					"VolumeType": "gp2",
					"VolumeSize": 50
				},
				"Status": {
					"CreationDate": "2021-12-21T14:44:37.462000+00:00",
					"UpdateDate": "2021-12-21T14:57:39.078000+00:00",
					"UpdateVersion": 5,
					"State": "Active",
					"PendingDeletion": false
				}
			},

			...

			"SnapshotOptions": {
				"Options": {
					"AutomatedSnapshotStartHour": 0
				},
				"Status": {
					"CreationDate": "2021-12-21T14:44:37.462000+00:00",
					"UpdateDate": "2021-12-21T14:57:39.078000+00:00",
					"UpdateVersion": 5,
					"State": "Active",
					"PendingDeletion": false
				}
			},
			"VPCOptions": {
				"Options": {},
				"Status": {
					"CreationDate": "2021-12-21T22:17:52.963000+00:00",
					"UpdateDate": "2021-12-21T22:17:52.963000+00:00",
					"UpdateVersion": 17,
					"State": "Active",
					"PendingDeletion": false
				}
			},
			"CognitoOptions": {
				"Options": {
					"Enabled": false
				},
				"Status": {
					"CreationDate": "2021-12-21T22:17:52.963000+00:00",
					"UpdateDate": "2021-12-21T22:17:52.963000+00:00",
					"UpdateVersion": 17,
					"State": "Active",
					"PendingDeletion": false
				}
			}
		}
	}
}

03 Repeat steps no. 1 and 2 to expand the storage space for other Amazon OpenSearch clusters available in the selected AWS region.

04 Change the AWS cloud region by updating the --region command parameter value and repeat the Remediation process for other regions.

References

AWS Documentation
Amazon OpenSearch Service FAQs
Troubleshooting Amazon OpenSearch Service
AWS services that publish CloudWatch metrics

AWS Command Line Interface (CLI) Documentation
es
list-domain-names
describe-elasticsearch-domain
update-elasticsearch-domain-config
cloudwatch
get-metric-statistics

Publication date Feb 3, 2018

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS Console

Using AWS CLI

References

Related Elasticsearch rules