AWS OpenSearch Slow Logs

Risk Level: Medium (should be achieved)

Rule ID: ES-011

Ensure that your Amazon OpenSearch clusters are configured to publish slow logs to CloudWatch Logs. This feature enables the service to publish slow logs from the indexing and search operations performed on your OpenSearch clusters and help you gain insight into the performance of these operations.

This rule can help you with the following compliance standards:

APRA
MAS
NIST4

For further details on compliance standards supported by Conformity, see here.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Operational
excellence

Performance
efficiency

Once enabled, Slow Logs (search and index slow logs) can help you identify performance issues caused by specific queries or due to changes in cluster usage. You can then use this information to optimize your queries or your index configuration to address the problem.

Note: If enabled, the standard Amazon CloudWatch Logs pricing does apply.

Audit

To determine if the Slow Logs feature is enabled for your Amazon OpenSearch clusters, perform the following operations:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon OpenSearch console at https://console.aws.amazon.com/esv3/.

03 In the main navigation panel, under Dashboard, select Domains.

04 Click on the name (link) of the OpenSearch cluster that you want to examine.

05 Select the Logs tab and check the value available in the Statuscolumn for Search slow logs and Index slow logs log types, listed in the CloudWatch Logssection. If the Status value for the Search slow logs and/or Index slow logs log types is set to Disabled, slow logs are not enabled for the selected Amazon OpenSearch cluster.

06 Repeat steps no. 4 and 5 for each Amazon OpenSearch cluster available within the current AWS region.

07 Change the AWS cloud region from the navigation bar and repeat the Audit process for other regions.

Using AWS CLI

01 Run list-domain-names command (OSX/Linux/UNIX) to list the name of each Amazon OpenSearch cluster (domain) available in the selected AWS region:

aws es list-domain-names
  --region us-east-1
 --query 'DomainNames[*].DomainName'

02 The command output should return the identifier (name) of each OpenSearch domain provisioned in the selected region:

[
    "trendmicro",
    "cloudconformity"
]

03 Run describe-elasticsearch-domain command (OSX/Linux/UNIX) using the name of the Amazon OpenSearch cluster that you want to examine as the identifier parameter and custom query filters to describe the Slow Logs feature configuration available for the selected cluster:

aws es describe-elasticsearch-domain
  --region us-east-1
  --domain-name trendmicro
  --query 'DomainStatus.LogPublishingOptions'

04 The command output should return the requested configuration information:

{
    "INDEX_SLOW_LOGS": {
        "Enabled": false
    },
    "SEARCH_SLOW_LOGS": {
        "Enabled": false
    }
}

If the describe-elasticsearch-domain command output returns null, the CloudWatch Logs are not enabled for the selected cluster, otherwise, check the "Enabled" property for each listed log type. If the "Enabled" value for the "INDEX_SLOW_LOGS" and/or "SEARCH_SLOW_LOGS" log types is set to false, as shown in the example above, the Slow Logs feature is not enabled for the selected Amazon OpenSearch cluster.

05 Repeat steps no. 3 and 4 for each Amazon OpenSearch cluster available in the selected AWS region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat the Audit process for other regions.

Remediation / Resolution

To enable the Slow Logs feature for your Amazon OpenSearch clusters (domains), perform the following operations:

Using AWS CloudFormation

01 CloudFormation template (JSON):

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Enable and Configure Slow Logs Feature for OpenSearch Domains",
    "Resources": {
    "CloudWatchLogGroup": {
        "Type": "AWS::Logs::LogGroup",
        "Properties": {
            "LogGroupName": "trendmicro",
            "RetentionInDays": 7
        }
    },
    "CloudWatchLogGroupPolicy": {
        "Type": "AWS::Logs::ResourcePolicy",
        "Properties": {
            "PolicyName": "cc-slow-logs-policy",
            "PolicyDocument": {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Principal": {
                            "Service": "es.amazonaws.com"
                        },
                        "Action": [
                            "logs:PutLogEvents",
                            "logs:PutLogEventsBatch",
                            "logs:CreateLogStream"
                        ],
                        "Resource": "arn:aws:logs:*"
                    }
                ]
            }
        }
    },
    "OpenSearchDomain": {
        "Type":"AWS::OpenSearchService::Domain",
        "Properties": {
            "DomainName": "cc-opensearch-domain",
            "EngineVersion": "OpenSearch_1.1",
            "ClusterConfig": {
                "InstanceType": "t3.small.search",
                "InstanceCount": "2"
            },
            "EBSOptions": {
                "EBSEnabled": true,
                "VolumeType": "gp2",
                "VolumeSize": "50"
            },
            "LogPublishingOptions": {
                "SEARCH_SLOW_LOGS": {
                    "CloudWatchLogsLogGroupArn": {"Fn::GetAtt": ["CloudWatchLogGroup","Arn"]},
                    "Enabled": true
                },
                "INDEX_SLOW_LOGS": {
                    "CloudWatchLogsLogGroupArn": {"Fn::GetAtt": ["CloudWatchLogGroup","Arn"]},
                    "Enabled": true
                }
            },
            "AccessPolicies": {
                "Version":"2012-10-17",
                "Statement":[
                {
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "arn:aws:iam::123456789012:user/cc-opensearch-user"
                    },
                    "Action":"es:*",
                    "Resource": "arn:aws:es:us-east-1:123456789012:domain/cc-opensearch-domain/*"
                }
                ]
            }
        }
        }
    }
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
    Description: Enable and Configure Slow Logs Feature for OpenSearch Domains
    Resources:
        CloudWatchLogGroup:
        Type: AWS::Logs::LogGroup
        Properties:
            LogGroupName: trendmicro
            RetentionInDays: 7
        CloudWatchLogGroupPolicy:
        Type: AWS::Logs::ResourcePolicy
        Properties:
            PolicyName: cc-slow-logs-policy
            PolicyDocument:
            Version: '2012-10-17'
            Statement:
                - Effect: Allow
                Principal:
                    Service: es.amazonaws.com
                Action:
                    - logs:PutLogEvents
                    - logs:PutLogEventsBatch
                    - logs:CreateLogStream
                Resource: arn:aws:logs:*
        OpenSearchDomain:
        Type: AWS::OpenSearchService::Domain
        Properties:
            DomainName: cc-opensearch-domain
            EngineVersion: OpenSearch_1.1
            ClusterConfig:
            InstanceType: t3.small.search
            InstanceCount: '2'
            EBSOptions:
            EBSEnabled: true
            VolumeType: gp2
            VolumeSize: '50'
            LogPublishingOptions:
            SEARCH_SLOW_LOGS:
                CloudWatchLogsLogGroupArn: !GetAtt 'CloudWatchLogGroup.Arn'
                Enabled: true
            INDEX_SLOW_LOGS:
                CloudWatchLogsLogGroupArn: !GetAtt 'CloudWatchLogGroup.Arn'
                Enabled: true
            AccessPolicies:
            Version: '2012-10-17'
            Statement:
                - Effect: Allow
                Principal:
                    AWS: arn:aws:iam::123456789012:user/cc-opensearch-user
                Action: es:*
                Resource: arn:aws:es:us-east-1:123456789012:domain/cc-opensearch-domain/*

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
    required_providers {
        aws = {
            source  = "hashicorp/aws"
            version = "~> 4.0"
        }
    }

    required_version = ">= 0.14.9"
}

provider "aws" {
    region  = "us-east-1"
}

resource "aws_cloudwatch_log_group" "cc-log-group" {
    name = "trendmicro"
}

resource "aws_cloudwatch_log_resource_policy" "cc-log-group-policy" {
    policy_name = "cc-slow-logs-policy"

    policy_document = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Effect": "Allow",
        "Principal": {
            "Service": "es.amazonaws.com"
        },
        "Action": [
            "logs:PutLogEvents",
            "logs:PutLogEventsBatch",
            "logs:CreateLogStream"
        ],
        "Resource": "arn:aws:logs:*"
    }
    ]
}
CONFIG
}

resource "aws_opensearch_domain" "opensearch-domain" {
    domain_name = "cc-opensearch-domain"
    engine_version = "OpenSearch_1.1"

    cluster_config {
        instance_type = "t3.small.search"
        instance_count = 2
    }

    ebs_options {
        ebs_enabled = true
        volume_size = 50
        volume_type = "gp2"
    }

    # Enable and Configure Slow Logs Feature for OpenSearch Domains
    log_publishing_options {
        cloudwatch_log_group_arn = aws_cloudwatch_log_group.cc-log-group.arn
        log_type                 = "SEARCH_SLOW_LOGS"
    }

    log_publishing_options {
        cloudwatch_log_group_arn = aws_cloudwatch_log_group.cc-log-group.arn
        log_type                 = "INDEX_SLOW_LOGS"
    }

    access_policies = <<POLICY
    {
        "Version": "2012-10-17",
        "Statement":[
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::123456789012:user/cc-opensearch-user"
                },
                "Action":"es:*",
                "Resource": "arn:aws:es:us-east-1:123456789012:domain/cc-opensearch-domain/*"
            }
        ]
    }
    POLICY

}

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon OpenSearch console at https://console.aws.amazon.com/esv3/.

03 In the main navigation panel, under Dashboard, select Domains.

04 Click on the name of the OpenSearch cluster that you want to reconfigure.

05 Select the Logs tab to access the cluster logging configuration settings.

06 In the CloudWatch Logs section, perform the following actions:

Select Search slow logs and choose Enable to enable publishing search slow logs to Amazon CloudWatch Logs. In the Select log group from CloudWatch logs section, select Create new group and provide a name for the new CloudWatch Logs log group in the New log group name box. In the Specify CloudWatch access policy section, choose Create new policy to create the policy required to successfully deliver the logs to your new CloudWatch Logs log group. Choose Enable to apply the configuration changes.
Select Index slow logsand choose Enable to enable publishing index slow logs to CloudWatch Logs. In the Select log group from CloudWatch logs section, select Create new group and provide a name for the new CloudWatch Logs log group in the New log group name box. In the Specify CloudWatch access policy section, choose Create new policy to create the access policy necessary to successfully deliver the logs to your CloudWatch Logs log group. Choose Enable to apply the configuration changes.

07 Repeat steps no. 4 – 6 to enable and configure search and index slow logs for other Amazon OpenSearch clusters available within the current AWS region.

08 Change the AWS cloud region from the navigation bar and repeat the Remediation process for other regions.

Using AWS CLI

01 Run create-log-group command (OSX/Linux/UNIX) to create a new Amazon CloudWatch Logs log group within the selected AWS region (the command does not produce an output):

aws logs create-log-group
  --region us-east-1
  --log-group-name /aws/OpenSearchService/domains/trendmicro

02 Run describe-log-groups command (OSX/Linux/UNIX) to describe the Amazon Resource Name (ARN) of the log group created at the previous step:

aws logs describe-log-groups
  --region us-east-1
  --log-group-name /aws/OpenSearchService/domains/trendmicro
  --query 'logGroups[*].arn'

03 The command output should return the requested log group ARN:

[
    "arn:aws:logs:us-east-1:123456789012:log-group:/aws/OpenSearchService/domains/trendmicro:*",
]

04 Run put-resource-policy command (OSX/Linux/UNIX) to give the Amazon OpenSearch service the permissions to write to the newly created log group:

aws logs put-resource-policy
  --region us-east-1
  --policy-name cc-slow-logs-policy
  --policy-document '{ "Version": "2012-10-17", "Statement": [{ "Sid": "", "Effect": "Allow", "Principal": { "Service": "es.amazonaws.com"}, "Action":[ "logs:PutLogEvents"," logs:PutLogEventsBatch","logs:CreateLogStream"],"Resource": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/OpenSearchService/domains/trendmicro:*"}]}'

05 The command output should return the command request metadata (including information about the access policy used):

{
    "resourcePolicy": {
        "policyName": "cc-slow-logs-policy",
        "policyDocument": "{ \"Version\": \"2012-10-17\", \"Statement\": [{ \"Sid\": \"\", \"Effect\": \"Allow\", \"Principal\": { \"Service\": \"es.amazonaws.com\"}, \"Action\":[ \"logs:PutLogEvents\",\" logs:PutLogEventsBatch\",\"logs:CreateLogStream\"],\"Resource\": \"arn:aws:logs:us-east-1:123456789012:log-group:/aws/OpenSearchService/domains/trendmicro:*\"}]}",
        "lastUpdatedTime": 1641218442256
    }
}

06 Run update-elasticsearch-domain-config command (OSX/Linux/UNIX) to enable and configure search and index slow logs for the selected Amazon OpenSearch cluster:

aws es update-elasticsearch-domain-config
  --region us-east-1
  --domain-name trendmicro
  --log-publishing-options "SEARCH_SLOW_LOGS={CloudWatchLogsLogGroupArn= arn:aws:logs:us-east-1:981005872766:log-group:/aws/OpenSearchService/domains/trendmicro:*,Enabled=true},INDEX_SLOW_LOGS={CloudWatchLogsLogGroupArn= arn:aws:logs:us-east-1:981005872766:log-group:/aws/OpenSearchService/domains/trendmicro:*,Enabled=true}"

07 The command output should return the new configuration information available for the modified OpenSearch cluster:

{
    "DomainConfig": {
        "ElasticsearchClusterConfig": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": {
                "WarmEnabled": false,
                "DedicatedMasterEnabled": false,
                "InstanceCount": 1,
                "ZoneAwarenessEnabled": false,
                "InstanceType": "t3.small.elasticsearch"
            }
        },
        "VPCOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641218607.34,
                "UpdateVersion": 21,
                "UpdateDate": 1641218607.34
            },
            "Options": {}
        },
        "CognitoOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641218607.34,
                "UpdateVersion": 21,
                "UpdateDate": 1641218607.34
            },
            "Options": {
                "Enabled": false
            }
        },
        "NodeToNodeEncryptionOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": {
                "Enabled": false
            }
        },
        "AdvancedSecurityOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": {
                "InternalUserDatabaseEnabled": false,
                "Enabled": false
            }
        },
        "DomainEndpointOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": {
                "EnforceHTTPS": false,
                "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07"
            }
        },
        "EBSOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": {
                "VolumeSize": 30,
                "VolumeType": "gp2",
                "EBSEnabled": true
            }
        },
        "SnapshotOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": {
                "AutomatedSnapshotStartHour": 0
            }
        },
        "ElasticsearchVersion": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": "7.9"
        },
        "LogPublishingOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Processing",
                "CreationDate": 1641216346.828,
                "UpdateVersion": 21,
                "UpdateDate": 1641218607.199
            },
            "Options": {
                "INDEX_SLOW_LOGS": {
                    "CloudWatchLogsLogGroupArn": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/OpenSearchService/domains/trendmicro:*",
                    "Enabled": true
                },
                "SEARCH_SLOW_LOGS": {
                    "CloudWatchLogsLogGroupArn": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/OpenSearchService/domains/trendmicro:*",
                    "Enabled": true
                }
            }
        },
        "AdvancedOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": {
                "override_main_response_version": "false",
                "rest.action.multi.allow_explicit_index": "false",
                "indices.fielddata.cache.size": ""
            }
        },
        "EncryptionAtRestOptions": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": {
                "Enabled": false
            }
        },
        "AccessPolicies": {
            "Status": {
                "PendingDeletion": false,
                "State": "Active",
                "CreationDate": 1641212317.965,
                "UpdateVersion": 5,
                "UpdateDate": 1641213076.849
            },
            "Options": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Deny\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:us-east-1:123456789012:domain/trendmicro/*\"}]}"
        }
    }
}

08 Repeat steps no. 6 and 7 to enable the Slow Logs feature for other Amazon OpenSearch clusters available in the selected AWS region.

09 Change the AWS cloud region by updating the --region command parameter value and repeat the Remediation process for other regions.

References

AWS Documentation
What is Amazon OpenSearch Service?
Creating and managing Amazon OpenSearch Service domains

AWS Command Line Interface (CLI) Documentation
es
list-domain-names
describe-elasticsearch-domain
update-elasticsearch-domain-config
logs
create-log-group
describe-log-groups
put-resource-policy

CloudFormation Documentation
Amazon OpenSearch Service resource type reference

Terraform Documentation
AWS Provider

Publication date Nov 8, 2017

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS CloudFormation

Using Terraform (AWS Provider)

Using AWS Console

Using AWS CLI

References

Related Elasticsearch rules