AWS Redshift Reserved Nodes Purchase Recommendations

Risk Level: Medium (should be achieved)

Identify any Amazon Redshift nodes that can be upgraded to Reserved Nodes (RNs) and follow our recommendations for purchasing your RNs in order to receive a significant discount on the hourly charges. Reserved Nodes/Instances represent an AWS billing concept that enables discounted hourly rates for compute capacity in exchange of reserving a certain amount of compute usage upfront. The cost savings when using Reserved Nodes over On-Demand nodes are up to 75% depending on the commitment term (1-year or 3-years) and the payment option used (No Upfront, Partial Upfront or All Upfront). For example, for a ds2.xlarge (Dense Storage) Redshift node provisioned in the US-East region you can save roughly $126 per month by upgrading it to a Reserved Node for a period of 1 year with no upfront costs or save $438 per month (as of October 2016) by reserving it for a period of 3 years with full payment made upfront (All Upfront payment option).

Cost
optimisation

Reserved Nodes represent a good strategy to cut down on Redshift service costs but purchasing them without a plan can feel like a risky process. The Cloud Conformity purchasing recommendations outlined in this rule will help you optimize your RNs spending and save money by using only what you need.

Note: In order to verify the current status of purchased reserved capacity and provide accurate recommendations, Cloud Conformity needs access to the AWS payer account (the account designated to pay the bills) or to AWS individual accounts. The following IAM policy allows Cloud Conformity to view all the reserved Redshift cluster nodes available in your account(s):

{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Action": [
        "redshift:DescribeReservedNodes"
      ],
      "Resource": "*"
    }
  ]
}

Audit

To verify your current Amazon Redshift compute capacity and determine if there are any candidates for upgrading to Reserved Nodes (RNs), perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to Redshift dashboard at https://console.aws.amazon.com/redshift/.

03 In the left navigation panel, under Redshift Dashboard, click Clusters.

04 Choose the Redshift cluster that you want to examine then click on its identifier link:

listed in the Cluster column.

05 Select the Configuration tab, analyze the cluster attributes and determine its role within the application stack or data warehouse environment. If the selected Redshift cluster is not the component of an application stack or data environment that is about to be decommissioned and it will be in use for at least 6 months, the cluster is stable and its nodes qualify as candidates for upgrade to Reserved Nodes (RNs). To upgrade these nodes in an optimal manner you need to follow the recommendations outlined in the Remediation/Resolution section and purchase Redshift RNs based on the cluster active (On-Demand) nodes attributes.

06 Repeat step no. 4 and 5 to determine if other Redshift cluster nodes provisioned in the current region are eligible for the upgrade.

07 Change the AWS region from the navigation bar and repeat the process for the other regions.

Using AWS CLI

01 Run describe-clusters command (OSX/Linux/UNIX) using custom query filters to list the identifiers (names) of all Redshift clusters currently available in the selected region:

aws redshift describe-clusters
    --region us-east-1
    --output table
    --query 'Clusters[*].ClusterIdentifier'

02 The command output should return a table with the requested cluster names:

----------------------
|  DescribeClusters  |
+--------------------+
|  cc-cluster        |
|  dbu-cc-cluster    |
|  dcd-cc-cluster    |
+--------------------+

03 Run again describe-clusters command (OSX/Linux/UNIX) using the name of cluster that you want to examine as identifier and the necessary query filters to describe the selected resource attributes:

aws redshift describe-clusters
    --region us-east-1
    --cluster-identifier cc-cluster

04 The command output should return the requested information (i.e. cluster attributes, cluster node(s) specifications, etc):

{
    "Clusters": [
        {
            "PubliclyAccessible": true,
            "MasterUsername": "ccclusterdbusr",
            "VpcSecurityGroups": [
                {
                    "Status": "active",
                    "VpcSecurityGroupId": "sg-0e1e2e41"
                }
            ],
            "ClusterPublicKey": "ssh-rsa AAAAB3NzaD ... ",
            "NumberOfNodes": 1,
            "PendingModifiedValues": {},
            "VpcId": "vpc-2fb56548",

             ...


            "AllowVersionUpgrade": true,
            "ClusterCreateTime": "2016-04-06T16:22:21.654Z",
            "ClusterSubnetGroupName": "default",
            "ClusterSecurityGroups": [],
            "ClusterIdentifier": "cc-cluster",
            "ClusterNodes": [
                {
                    "NodeRole": "SHARED",
                    "PrivateIPAddress": "172.31.23.8",
                    "PublicIPAddress": "52.3.149.177"
                }
            ],
            "AvailabilityZone": "us-east-1a",
            "NodeType": "ds2.xlarge",
            "Encrypted": false,
            "ClusterRevisionNumber": "1106",
            "ClusterStatus": "available"
        }
    ]
}

05 Analyze the Redshift cluster attributes listed at the previous step and determine its role within the application stack or data warehouse environment. If the selected Redshift cluster is not the component of an application stack or big data environment that is about to be decommissioned and it will be in use for at least 6 months, the cluster is stable and its nodes qualify as candidates for upgrade to Reserved Nodes (RNs). To upgrade these nodes you need to follow the recommendations outlined in the Remediation/Resolution section and purchase Redshift RNs based on the cluster On-Demand nodes attributes.

06Repeat steps no. 3 - 5 to determine if other Redshift cluster nodes provisioned in the current region are eligible for the upgrade.

07 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 6 to perform the audit process for other regions.

Remediation / Resolution

To upgrade your On-Demand Redshift nodes to Reserved Nodes you need to get informed in order to avoid paying for compute resources that you do not need. To make the RN purchase based on the right pricing model, the right type at the right time, follow the recommendations outlined below:

Recommendations

01Commitment Term - choose carefully the commitment period based on your application(s) requirements because you will be charged for every hour during the entire reservation term that you select, regardless of whether the cluster node is running or not. AWS provides two types of commitment terms:

1-year term - the reservation is made for just 1 year and it’s useful for steady-state workloads (e.g. ETL based applications).
3-year term - the reservation is made for 3 years and it's useful for long-running, stable and predictable workloads (e.g. big data analytics).

02 Payment Options - decide how you would like to pay for your Redshift Reserved Nodes throughout the chosen commitment period. There are 3 payment options to choose from:

No Upfront - no upfront payment is required at purchase and but discount is lower (20% over On-Demand pricing model) when compared to other payment options. This option is not available for 3-year term reservations.
Partial Upfront - a portion of the cost must be paid upfront and the remaining hours in the term are billed at the established discounted hourly rate. This option balances the Reserved Node payments between upfront and hourly.
All Upfront - you are paying for the entire commitment term selected (one or three years) with one upfront payment and get the best effective hourly price when compared to On-Demand model and the biggest discount when compared to other payment options available.

03 Consolidated Billing - using Consolidated Billing is recommended when you own multiple accounts because the AWS billing engine treats all your accounts as a single account for billing purposes (designated payer account), resulting in the lowest possible bill as the applicable discounts are spread across all accounts. For instance, the Reserved Nodes purchased in one AWS account can share their reservations with other AWS accounts available in the same billing family. The purchase recommendations are based on the previous calendar month's hour-by-hour usage aggregated across all consolidated billing accounts. For example, if you have two Redshift RNs in the same region with the same attributes, the AWS billing system checks each hour the total number of nodes you have running that match those attributes. If there are two nodes or less, you will be charged the Reserved Node rate for each matching node running that hour. If more than two are running, you will be charged the On-Demand rate for the additional node(s).

04 Purchasing Process - is important to review carefully the checkout page details before the purchase. After your purchase is confirmed, it cannot be cancelled so make sure the parameters displayed on the checkout page are accurate.

To upgrade to Reserved Nodes just purchase Redshift RNs in the same region, using the same attributes as your active On-demand nodes and the billing will be automatically changed to reflect the reserved rates. To buy the required reserved Redshift nodes, perform the following actions:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Redshift dashboard at https://console.aws.amazon.com/redshift/.

03 In the navigation panel choose Reserved Nodes.

04 Click Purchase Reserved Nodes button from the dashboard top menu to start the purchasing process.

05 On the Purchase Reserved Nodes page, provide the following RN offering details:

From Node Type dropdown list, select the same node type (e.g. ds2.xlarge) used by the active (On-demand) Redshift node that you want to upgrade.
From Term dropdown list, select the length of time for the new reservation based on your requirements.
From Offering Type dropdown list, choose the payment type that you want to use for this reservation (e.g. Partial Upfront).
In the Number of Nodes box type 1.
Click Continue to proceed with the next page.

06 On the Redshift RN summary (checkout) page, verify the node configuration information to make sure that the RN parameters are correct, then click Purchase to buy the selected reserved Redshift node.

07 Repeat steps no. 4 - 6 to upgrade other On-Demand Redshift nodes provisioned in the current region by purchasing Reserved Nodes that match their configuration.

08 Change the AWS region from the navigation bar and repeat the process for other regions.

09 Login to the AWS Management Console.

Using AWS CLI

01 Before you purchase your Redshift Reserved Node(s) run describe-reserved-node-offerings command (OSX/Linux/UNIX) to get pricing details for the available Redshift RN offerings:

aws redshift describe-reserved-node-offerings
    --region us-east-1

02 The command output should return the requested offering(s) information:

{
    "ReservedNodeOfferings": [
        {
            "OfferingType": "Partial Upfront",
            "FixedPrice": 2500.0,
            "NodeType": "ds2.xlarge",
            "UsagePrice": 0.0,
            "RecurringCharges": [
                {
                    "RecurringChargeAmount": 0.500,
                    "RecurringChargeFrequency": "Hourly"
                }
            ],
            "Duration": 31536000,
            "ReservedNodeOfferingId":
            "e34716e3-a84e-56fe-99c3-d832c45ab5eb",
            "CurrencyCode": "USD"
        },


        ...


        {
            "OfferingType": "Partial Upfront",
            "FixedPrice": 750.0,
            "NodeType": "dc1.large",
            "UsagePrice": 0.0,
            "RecurringCharges": [
                {
                    "RecurringChargeAmount": 0.161,
                    "RecurringChargeFrequency": "Hourly"
                }
            ],
            "Duration": 31536000,
            "ReservedNodeOfferingId":
            "ea3016e3-a84f-65a1-99c3-2a473e3efcd5",
            "CurrencyCode": "USD"
        }
    ],
    "ResponseMetadata": {
       "RequestId": "b0151a43-75eb-11e2-3894-e142fe92a473"
    }
}

03 Run purchase-reserved-nodes-offering command (OSX/Linux/UNIX) to purchase the Redshift Reserved Node based on the offering information returned at the previous step. Make sure that you select the right offering, the one that match your cluster active nodes configuration. The following command example purchase a reserved Redshift node using an offering identified by the ID e34716e3-a84e-56fe-99c3-d832c45ab5eb:

aws redshift purchase-reserved-node-offering
    --reserved-node-offering-id e34716e3-a84e-56fe-99c3-d832c45ab5eb
    --node-count 1

04 The command output should return the new AWS Redshift Reserved Node metadata:

{
   "ReservedNode": {
      "OfferingType": "Partial Upfront",
      "FixedPrice": 2500.0,
      "NodeType": "ds2.xlarge",
      "ReservedNodeId": "5ea73e33-bc01-4d78-b35e-a4c3e9320c8b10",
      "UsagePrice": 0.0,
      "RecurringCharges": [
         {
            "RecurringChargeAmount": 0.500,
            "RecurringChargeFrequency": "Hourly"
         }
      ],
      "NodeCount": 1,
      "State": "payment-pending",
      "StartTime": "2016-10-29T16:08:29.051Z",
      "Duration": 31536000,
      "ReservedNodeOfferingId": "e34716e3-a84e-56fe-99c3-d832c45ab5eb"
   },
   "ResponseMetadata": {
      "RequestId": "a0b3a7bf-2100-11e2-b605-2754d7396c77"
   }
}

05 Repeat steps no. 1 - 4 to upgrade other On-Demand Redshift nodes provisioned in the current region by purchasing Reserved Nodes that match their configuration.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the entire process for other regions.

To renew your existing Redshift Reserved Nodes (RNs), you need to repurchase them using the same configuration attributes (AWS region, node type, term, etc). The Redshift RNs are not renewed automatically so you must repurchase your existing RNs in order to avoid On-Demand rates charges when the current reservations expire. To repurchase them perform the following:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Redshift dashboard at https://console.aws.amazon.com/redshift/.

03 In the navigation panel choose Reserved Nodes.

04 Select the Redshift Reserved Node that you want to renew and note its configuration attributes (i.e. node type, term and offering type).

05 Click Purchase Reserved Nodes button from the dashboard top menu to start the purchasing process.

06 On the Purchase Reserved Nodes page, provide the following RN offering details:

From Node Type dropdown list, select the same node type (e.g. ds2.xlarge) used by the Redshift Reserved Node that you want to renew.
From Term dropdown list, select the length of time for the new reservation based on your requirements.
From Offering Type dropdown list, choose the payment type that you want to use for this reservation (e.g. Partial Upfront).
In the Number of Nodes box type 1.
Click Continue to proceed with the next page.

07 On the Redshift RN summary page, verify the node configuration information then click Purchase to repurchase your reserved Redshift node.

08 Repeat steps no. 4 - 7 to renew (repurchase) other Redshift Reserved Nodes available in the current region.

09 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 First, run describe-reserved-nodes command (OSX/Linux/UNIX) using the ID of the Redshift RN that you want to renew as parameter to describe the node offering identifier. This ID will be used later as parameter to find the offering(s) that match the existing node configuration attributes:

aws redshift describe-reserved-nodes
    --region us-east-1
    --reserved-node-id ac10ab51-bc51-4d65-b35c-a4a3e93da753
    --query 'ReservedNodes[*].ReservedNodeOfferingId'

02 The command output should return the selected RN offering identifier:

[
    "eb9916e3-a84f-53ba-99c3-2a473e3ec51a"
]

03 Now run describe-reserved-node-offerings command (OSX/Linux/UNIX) to get pricing information for any available Redshift RN offerings. Use the identifier returned at the previous step as identifier to match the specifications of the RN that you want to renew (repurchase):

aws redshift describe-reserved-node-offerings
    --region us-east-1
    --reserved-node-offering-id eb9916e3-a84f-53ba-99c3-2a473e3ec51a

04 The command output should return the requested offering(s) information:

{
    "ReservedNodeOfferings": [
        {
            "OfferingType": "Partial Upfront",
            "FixedPrice": 2480.0,
            "NodeType": "ds2.xlarge",
            "UsagePrice": 0.0,
            "RecurringCharges": [
                {
                    "RecurringChargeAmount": 0.478,
                    "RecurringChargeFrequency": "Hourly"
                }
            ],
            "Duration": 31536000,
            "ReservedNodeOfferingId":
            "eb9916e3-a84f-53ba-99c3-2a473e3ec51a",
            "CurrencyCode": "USD"
        }
    ]
}

05 Run purchase-reserved-nodes-offering command (OSX/Linux/UNIX) to purchase the Redshift Reserved Node based on the offering information returned at the previous step. The following command example purchase a reserved Redshift node using an offering identified by the ID eb9916e3-a84f-53ba-99c3-2a473e3ec51a:

aws redshift purchase-reserved-node-offering
    --reserved-node-offering-id eb9916e3-a84f-53ba-99c3-2a473e3ec51a
    --node-count 1

06 The command output should return the new AWS Redshift Reserved Node metadata:

{
   "ReservedNode": {
      "OfferingType": "Partial Upfront",
      "FixedPrice": 2480.0,
      "NodeType": "ds2.xlarge",
      "ReservedNodeId": "3ac13e33-bc01-4d65-b35d-a4a3e9320eba0",
      "UsagePrice": 0.0,
      "RecurringCharges": [
         {
            "RecurringChargeAmount": 0.478,
            "RecurringChargeFrequency": "Hourly"
         }
      ],
      "NodeCount": 1,
      "State": "payment-pending",
      "StartTime": "2016-10-29T17:09:21.051Z",
      "Duration": 31536000,
      "ReservedNodeOfferingId": "eb9916e3-a84f-53ba-99c3-2a473e3ec51a"
   },
   "ResponseMetadata": {
      "RequestId": "ca53a7bf-2100-11e2-b605-2754d73977da"
   }
}

07 Repeat steps no. 1 - 6 to renew (repurchase) other Redshift Reserved Nodes available in the current region.

08 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 7 to perform the entire process for other regions.

References

AWS Command Line Interface (CLI) Documentation
redshift
describe-clusters
describe-reserved-nodes
describe-reserved-node-offerings
purchase-reserved-node-offering

Publication date Dec 3, 2016

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Recommendations

Using AWS Console

Using AWS CLI

Using AWS Console

Using AWS CLI

References

Related Redshift rules