Idle OpenSearch Domains

01 Sign in to the AWS Management Console.

02 Navigate to Amazon OpenSearch console at https://console.aws.amazon.com/esv3/.

03 In the main navigation panel, under Dashboard, select Domains.

04 Click on the name (link) of the OpenSearch domain that you want to examine.

05 Select the Cluster health tab and click on the CPU utilization graph thumbnail displayed in the Data nodes section to open the panel with the CPU usage. On the CPU utilization (CPUUtilization) panel, set the following parameters:

1. From the Period dropdown list, select 1 Hour.
2. From the Time range list, select 1w (1 week).
3. From the Statistic dropdown list, select Average.

06 Determine the domain role and owner within your application stack by checking the Role and Owner tags values assigned to the selected OpenSearch domain in order to decide whether or not it's safe to delete the resource. To check for the required tags, perform the following actions:

1. Select the Tags tab to access the tag sets configured for the selected OpenSearch domain.
2. Check the Role tag value, available in the Value column, or any Role-like tag value that can provide information about the usage profile of the OpenSearch domain in order to decide if the resource can be deleted or not.
3. Check the Owner tag value, available in the Value column, or any Owner-like tag value that can provide the contact information (name, email, phone number, etc.) of the resource owner in order to get the confirmation to delete or not the selected OpenSearch resource.

07 If all the conditions outlined at step no. 5 and 6 are met, the selected Amazon OpenSearch domain is considered idle and can be terminated in order to stop incurring charges for the OpenSearch cluster.

08 Repeat steps no. 4 – 7 for each Amazon OpenSearch domain available within the current AWS region.

09 Change the AWS cloud region from the navigation bar and repeat the Audit process for other regions.

01 Run list-domain-names command (OSX/Linux/UNIX) to list the name of each Amazon OpenSearch domain available in the selected AWS region:

aws es list-domain-names
  --region us-east-1
  --query 'DomainNames[*].DomainName'

02 The command output should return the identifier (name) of each OpenSearch domain provisioned in the selected region:

[
	"trendmicro",
	"cloudconformity"
]

03 Run get-metric-statistics command (OSX/Linux/UNIX) to obtain the statistics recorded by Amazon CloudWatch for the CPUUtilization metric, representing the CPU usage for the selected OpenSearch domain (cluster). Change the--start-time (start recording date) and --end-time (stop recording date) parameters value to choose your own time frame for recording the CPUUtilization metric usage. Set the --period parameter value to define the granularity (in seconds) of the returned datapoints, based on your requirements. A period can be as short as 1 minute (60 seconds) or as long as 1 day (86400 seconds). The following command example returns the average CPU usage for a OpenSearch domain named "trendmicro", usage data captured over a time period of 7 days, using 1-hour period as the granularity for the returned datapoints:

aws cloudwatch get-metric-statistics
  --region us-east-1
  --metric-name CPUUtilization
  --start-time 2017-04-05T08:46:05
  --end-time 2017-04-12T08:46:05
  --period 3600
  --namespace AWS/ES
  --statistics Average
  --dimensions Name=DomainName,Value=trendmicro

04 The command output should return the CPUUtilization usage details requested:

{
    "Datapoints": [
        {
            "Timestamp": "2017-04-05T08:46:05Z",
            "Average": 0.031533333333333333,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-05T09:46:05Z",
            "Average": 0.033499999999999995,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-05T10:46:05Z",
            "Average": 0.10425,
            "Unit": "Percent"
        },

        ...

        {
            "Timestamp": "2017-04-12T05:46:05Z",
            "Average": 0.030999999999999993,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-12T06:46:05Z",
            "Average": 0.02833333333333333,
            "Unit": "Percent"
        },
        {
            "Timestamp": "2017-04-12T07:46:05Z",
            "Average": 0.02783333333333333,
            "Unit": "Percent"
        }
    ],
    "Label": "CPUUtilization"
}

05 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the selected OpenSearch domain as the identifier parameter and custom query filters to describe the domain's ARN:

aws es describe-elasticsearch-domain
  --domain-name trendmicro
  --region us-east-1
  --query 'DomainStatus.ARN'

06 The command output should return the requested OpenSearch domain ARN:

"arn:aws:es:us-east-1:123456789012:domain/trendmicro"

07 Run list-tags command (OSX/Linux/UNIX) using the domain ARN returned at the previous step as the identifier parameter to describe the tags applied to the selected OpenSearch domain (cluster):

aws es list-tags
  --region us-east-1
  --arn "arn:aws:es:us-east-1:123456789012:domain/trendmicro"

08 The command output should return the tag sets (key-value pairs) defined for the selected domain. The Roleand the Ownertags returned and their values (highlighted) can be used to determine the resource role within the application stack and to contact the domain owner for more information in order to decide if the OpenSearch resource can be deleted or not:

{
    "TagList": [
        {
            "Value": "dev-os-cluster",
            "Key": "Role"
        },
        {
            "Value": "trendmicro.com",
            "Key": "Owner"
        }
    ]
}

09 If the data returned for the steps no. 3 – 8 satisfy the conditions set by the conformity rule (domain owner, domain role, and CPU usage), the selected Amazon OpenSearch domain is considered idle and can be terminated in order to stop incurring charges for the OpenSearch cluster.

10 Repeat steps no. 3 – 9 for each Amazon OpenSearch domain available in the selected AWS region.

11 Change the AWS cloud region by updating the **--region** command parameter value and repeat the Audit process for other regions.

01 Sign in to the AWS Management Console.

02 Navigate to Amazon OpenSearch console at https://console.aws.amazon.com/esv3/.

03 In the main navigation panel, under Dashboard, select Domains.

04 Select the Amazon OpenSearch domain that you want to remove and choose Delete from the console top menu.

05 In the Delete domain? confirmation box, type the name of the idle domain in the required field, then choose Delete to remove the selected OpenSearch domain from your AWS cloud account.

06 Repeat steps no. 4 and 5 for each idle Amazon OpenSearch domain that you want to delete, available within the current AWS region.

07 Change the AWS cloud region from the navigation bar and repeat the Remediation process for other regions.

01 Run delete-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the idle domain as the identifier parameter, to remove the specified Amazon OpenSearch domain (cluster) from your AWS cloud account:

aws es delete-elasticsearch-domain
  --region us-east-1
  --domain-name trendmicro

02 The command output should return the delete-elasticsearch-domain command request metadata:

{
    "DomainStatus": {
        "DomainId": "123456789012/trendmicro",
        "DomainName": "trendmicro",
        "ARN": "arn:aws:es:us-east-1:123456789012:domain/trendmicro",
        "Created": true,
        "Deleted": true,
        "Endpoint": "search-trendmicro-abcd1234abcd1234abcd.us-east-1.es.amazonaws.com",
        "Processing": true,
        "UpgradeProcessing": false,
        "ElasticsearchVersion": "7.9",
        "ElasticsearchClusterConfig": {
            "InstanceType": "t3.small.elasticsearch",
            "InstanceCount": 3,
            "DedicatedMasterEnabled": false,
            "ZoneAwarenessEnabled": false,
            "WarmEnabled": false,
            "ColdStorageOptions": {
                "Enabled": false
            }
        },
        "EBSOptions": {
            "EBSEnabled": true,
            "VolumeType": "gp2",
            "VolumeSize": 30
        },
        "SnapshotOptions": {},
        "CognitoOptions": {
            "Enabled": false
        },
        "EncryptionAtRestOptions": {
            "Enabled": false
        },
        "NodeToNodeEncryptionOptions": {
            "Enabled": false
        },
        "AdvancedOptions": {
            "indices.fielddata.cache.size": "",
            "override_main_response_version": "false",
            "rest.action.multi.allow_explicit_index": "false"
        },
        "ServiceSoftwareOptions": {
            "CurrentVersion": "R20211203-P2",
            "NewVersion": "",
            "UpdateAvailable": false,
            "Cancellable": false,
            "UpdateStatus": "COMPLETED",
            "Description": "There is no software update available for this domain.",
            "AutomatedUpdateDate": "2021-12-15T22:07:00+00:00",
            "OptionalDeployment": false
        },
        "DomainEndpointOptions": {
            "EnforceHTTPS": false,
            "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07",
            "CustomEndpointEnabled": false
        },
        "AdvancedSecurityOptions": {
            "Enabled": false,
            "InternalUserDatabaseEnabled": false
        },
        "AutoTuneOptions": {
            "State": "ENABLE_IN_PROGRESS"
        }
    } 
}

03 Repeat steps no. 1 and 2 for each idle Amazon OpenSearch domain that you want to remove, available in the selected AWS region.

04 Change the AWS cloud region by updating the **--region** command parameter value and repeat the Remediation process for other regions.