Logo of Trend Micro
Use the Conformity Knowledge Base AI to help improve your Cloud Posture

EMR Instances Counts

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: Medium (should be achieved)
Rule ID: EMR-003

Ensure that the number of Elastic MapReduce (EMR) cluster instances (master and core instances) provisioned in your AWS account has not reached the limit quota established by your organization for the EMR workload deployed. By default, Cloud Conformity sets a threshold value of 5 for the maximum number of provisioned instances, however, you have the capability to adjust this threshold based on your organization requirements upon enabling this rule. Once you define your own threshold for the maximum number of Elastic MapReduce instances that you need to run across all AWS regions, Cloud Conformity engine will start to continuously check your account for EMR instances and when the number of instances reach the specified count (threshold) you will get notified via communication channels configured within your Cloud Conformity account. If the EMR limit quota defined for your AWS account is reached, you can raise an AWS support case where you can request to limit the number of provisioned EMR instances based on your requirements.

This rule can help you with the following compliance standards:

  • APRA
  • MAS

For further details on compliance standards supported by Conformity, see here.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Security

Monitoring and setting limits for the maximum number of Elastic MapReduce cluster instances provisioned within your AWS account will help you to manage better your EMR compute resources, prevent unexpected charges on your AWS bill and act fast to mitigate attacks. For example, users within your organization can create more EMR instances than the number established in the company policy, exceeding the monthly budget allocated for cloud computing resources. Furthermore, if your AWS account security has been compromised and the attacker is able to create a large number of EMR resources within your account, you risk to accrue a lot of AWS charges in a short period of time and this can affect your business.

Note: The threshold for the maximum number of EMR instances per AWS account set for this conformity rule is 5 (default value).

Audit

To determine the number of Elastic MapReduce (EMR) instances (master and core instances) currently available within your AWS account, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to EMR dashboard at https://console.aws.amazon.com/elasticmapreduce/.

03 In the left navigation panel, under Amazon EMR, click Cluster list to access your AWS EMR clusters page.

04 Select the EMR cluster that you want to examine then click on the View details button from the dashboard top menu.

05 On the selected cluster configuration details page, click on the Hardware tab to expand the EMR cluster hardware panel.

06 Inside the Instance Groups section, verify the value available within the Instance Count column for each instance (master and core instance) provisioned in the cluster.

07 Go back to the Clusters page and repeat steps no. 4 – 6 to determine the total number of instances provisioned by other EMR clusters within the current region.

08 Change the AWS region from the navigation bar and repeat steps no. 3 – 7 for all other regions. If the total number of Elastic MapReduce cluster instances provisioned in your AWS account is greater than 5, the defined threshold was exceeded, therefore you must take action and raise an AWS support case to limit the number of EMR instances (see Remediation/Resolution section).

Using AWS CLI

01 Run list-clusters command (OSX/Linux/UNIX) using custom query filters to list the identifiers (IDs) of all active Amazon EMR clusters available in the selected region: 

aws emr list-clusters
    --region us-east-1
    --active
    --output table
    --query 'Clusters[*].Id'

02 The command output should return a table with the requested cluster IDs: 

---------------------
|   ListClusters    |
+-------------------+
|  j-1234aaabbb333  |
|  j-1234dddeee111  |
|  j-1234cccddd222  |
+-------------------+

03 Run describe-cluster command (OSX/Linux/UNIX) using the ID of the cluster that you want to examine as identifier, returned at the previous step, and custom query filters to expose the instance(s) type used by the selected Amazon EMR cluster: 

aws emr describe-cluster
    --region us-east-1
    --cluster-id j-1234aaabbb333
    --query 'Cluster.InstanceGroups[InstanceGroupType,RunningInstanceCount]'

04 The command output should return the number of instances for the selected AWS EMR cluster. The first value returned represents the number of core instances and second value the number of master instances: 

[
    [
        "CORE",
        2
    ],
    [
        "MASTER",
        1
    ]
]

05 Repeat step no. 3 and 4 to determine the number of instances provisioned by all other AWS EMR clusters, available in the current region.

06 Repeat steps no. 1 – 5 to perform the process for all other AWS regions. The describe-cluster command output should return an array with the current number of EMR cluster instances (core instances and master instances), available in the selected region. If the total number of EMR instances within all the arrays returned is greater than 5 (combined), the recommended limit threshold was exceeded, therefore you must take action and raise an AWS support case to limit the number of EMR cluster instances that can be provisioned in your account.

Remediation / Resolution

To build an AWS support case in order to limit the number of provisioned Elastic MapReduce cluster instances based on your requirements, perform the following actions:

Note: Requesting a limit for the number of EMR instances per region using the AWS API via Command Line Interface (CLI) is not currently supported.

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to AWS Support Center dashboard at https://console.aws.amazon.com/support/.

03 In the left navigation panel, choose Create Case to create a new AWS support case.

04 On the Create Case page, perform the following:

  1. Under Regarding, select Service Limit Increase.
  2. Choose EC2 Instances from the Limit Type dropdown list.
  3. In the Request 1 section, perform the following actions:
    • Select the AWS region where EMR instances limit increase is required from the Region dropdown list.
    • Select the necessary EMR instance type from the Primary Instance Type dropdown list.
    • Select Instance Limit from the Limit dropdown list.
    • In the New limit value box, enter the limit value to request for the number of provisioned EMR cluster instances.
  4. In the Use Case Description textbox, enter a small description where you explain the limit request so AWS support can evaluate your case faster.
  5. From Supported Language, choose your preferred correspondence language for the current case.
  6. Under Contact method, select a preferred contact method that AWS support team can use to respond to your request.
  7. Click Submit to send the limit request to Amazon Web Services. A customer support representative will contact you shortly.

References

Publication date Sep 28, 2017

Related EMR rules