Logo of Trend Micro
Use the Conformity Knowledge Base AI to help improve your Cloud Posture

Redshift Instance Generation

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: Medium (should be achieved)
Rule ID: RS-008

Ensure that all Redshift clusters provisioned within your AWS account are using the latest generation of nodes (instances) in order to get higher performance with lower costs. If you are currently using Amazon Redshift nodes from the previous generation (i.e. ds1.xlarge or ds1.8xlarge), Cloud Conformity encourages you to upgrade these nodes to their latest generation equivalents (ds2.xlarge or ds2.8xlarge).

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Sustainability
Performance
efficiency
Cost
optimisation

Using the latest generation of AWS Redshift nodes instead of the previous generation nodes will upgrade your clusters for higher hardware performance (more computing capacity and faster CPUs, more RAM memory and higher disk throughput using solid-state disks) at the same costs.

Audit

The previous generation of Amazon Redshift node types in use are ds1.xlarge and ds1.8xlarge. To determine if your Redshift clusters are using node types from the previous generation, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to Redshift dashboard at https://console.aws.amazon.com/redshift/.

03 In the left navigation panel, under Redshift Dashboard, click Clusters.

04 Choose the Redshift cluster that you want to examine then click on its identifier link:

listed in the Cluster column.

05 On the selected cluster Configuration tab, in the Cluster Properties section, verify the Node Type property value:

If the Node Type value is set to ds1.xlarge or ds1.8xlarge, the selected Amazon Redshift cluster is using nodes from a previous generation, therefore an upgrade is highly recommended (see Remediation/Resolution section for the upgrade process).

06 Repeat step no. 4 and 5 to verify the nodes type for other Redshift clusters provisioned in the current region.

07 Change the AWS region from the navigation bar and repeat the audit process for other regions.

Using AWS CLI

01 Run describe-clusters command (OSX/Linux/UNIX) using custom query filters to list the identifiers (names) of all Redshift clusters currently available in the selected region: 

aws redshift describe-clusters
	--region us-east-1
	--output table
	--query 'Clusters[*].ClusterIdentifier'

02 The command output should return a table with the requested cluster names: 

----------------------
|  DescribeClusters  |
+--------------------+
|  cc-cluster        |
|  dsg-cc-cluster    |
|  dsx-cc-cluster    |
+--------------------+

03 Run again describe-clusters command (OSX/Linux/UNIX) using the name of cluster that you want to examine as identifier and the necessary query filters to expose the node(s) type used by the selected AWS Redshift cluster: 

aws redshift describe-clusters
	--region us-east-1
	--cluster-identifier cc-cluster
	--query 'Clusters[*].NodeType'

04 The command output should return the Redshift node type currently used: 

[
    "ds1.xlarge"
]

If the value returned by the command output is ds1.xlarge or ds1.8xlarge, the selected Amazon Redshift cluster is using nodes from a previous generation, therefore the upgrade to the newest generation is highly recommended.

05 Repeat step no. 3 and 4 for each Redshift cluster available in the current region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the audit process for other regions.

Remediation / Resolution

The following table will help you choose the equivalent current generation node type required for the upgrade process:

Previous Generation Redshift Node Types Current Generation Redshift Node Types
ds1.xlarge ds2.xlarge
ds1.8xlarge ds2.8xlarge

To upgrade your previous generation Redshift nodes to their latest generation equivalents, perform the following:

Using AWS Console

01 Login to the AWS Management Console.

02 Navigate to Redshift dashboard at https://console.aws.amazon.com/redshift/.

03 In the left navigation panel, under Redshift Dashboard, click Clusters.

04 Select the Redshift cluster that you want to upgrade then click on its identifier link:

listed in the Cluster column.

05 Click the Backup dropdown button from the dashboard top menu and select Take Snapshot.

06 On the Create Snapshot dialog box, enter a unique name for your cluster snapshot in the Snapshot Identifier box then click Create to take the snapshot. The process may take several minutes. Once the snapshot is created it will appear on your Redshift Snapshots page.

07 In the navigation panel, under Redshift Dashboard, click Snapshots.

08 Select the Amazon Redshift cluster snapshot created at step no. 6.

09 Click the Actions dropdown button from the dashboard top menu and select Restore From Snapshot.

10 In the Restore From Snapshot dialog box, perform the following actions:

  1. Select the node type equivalent from the Node Type dropdown list: Select the node type equivalent from the Node Type dropdown list.
  2. In the Cluster Identifier box, enter a unique name for the new (upgraded) Redshift cluster.
  3. Configure the rest of the options (Cluster Parameter Group, Availability Zone, VPC Security Groups, etc) based on the configuration information taken from the existing cluster.
  4. Click Restore to create the new (upgraded) Redshift cluster.

11 As soon as the build process is complete, update your application configuration to refer to the new cluster endpoint, e.g:
cc-cluster-upgraded.cmfpsgvyjhfo.us-east-1.redshift.amazonaws.com.

12 Once the Redshift cluster endpoint is changed within your application configuration, you can remove the previous generation node type based cluster from your AWS account by performing the following actions:

  1. In the navigation panel, under Redshift Dashboard, click Clusters.
  2. Choose the Redshift cluster that you want to remove then click on its identifier link listed in the Cluster column.
  3. On the selected cluster Configuration tab, click the Cluster dropdown button from the dashboard main menu then select Delete.
  4. Inside the Delete Cluster dialog box, enter a unique name for the final snapshot in the Snapshot name box then click Delete to confirm the action. Once the snapshot is created the selected cluster removal process begins.

13 Repeat steps no. 3 - 12 to upgrade the nodes type for other AWS Redshift clusters provisioned in the current region.

14 Change the AWS region from the navigation bar and repeat the entire process for other regions.

Using AWS CLI

01 Run describe-clusters command (OSX/Linux/UNIX) to describe the configuration information of the Amazon Redshift cluster that you want to upgrade (see Audit section part II to identify the right cluster): 

aws redshift describe-clusters
	--region us-east-1
	--cluster-identifier cc-cluster

02 The command output should return the requested configuration information which will be useful later when the new Redshift cluster will be created: 

{
    "Clusters": [
        {
            "PubliclyAccessible": true,
            "MasterUsername": "ccclusterusr",
            "VpcSecurityGroups": [
                {
                    "Status": "active",
                    "VpcSecurityGroupId": "sg-061e2e7c"
                }
            ],
            "ClusterPublicKey": "ssh-rsa AAAAB3NzaC ... ",
            "NumberOfNodes": 1,
            "PendingModifiedValues": {},
            "VpcId": "vpc-2fb56548",
            "ClusterVersion": "1.0",
            "Tags": [],
            "AutomatedSnapshotRetentionPeriod": 1,
            "ClusterParameterGroups": [
                {
                    "ParameterGroupName": "default.redshift-1.0",
                    "ParameterApplyStatus": "in-sync"
                }
            ],
            "DBName": "ccclusterdb",
            "PreferredMaintenanceWindow": "fri:03:00-fri:03:30",
            "Endpoint": {
                "Port": 5439,
                "Address": "cc-cluster.cmfpsgvyjhfo ... "
            },
            "IamRoles": [],
            "AllowVersionUpgrade": true,
            "ClusterCreateTime": "2015-01-11T16:38:21.654Z",
            "ClusterSubnetGroupName": "default",
            "ClusterSecurityGroups": [],
            "ClusterIdentifier": "cc-cluster",
            "ClusterNodes": [
                {
                    "NodeRole": "SHARED",
                    "PrivateIPAddress": "172.31.23.5",
                    "PublicIPAddress": "52.3.149.186"
                }
            ],
            "AvailabilityZone": "us-east-1b",
            "NodeType": "ds1.xlarge",
            "Encrypted": false,
            "ClusterRevisionNumber": "1106",
            "ClusterStatus": "available"
        }
    ]
}

03 Run create-cluster-snapshot command (OSX/Linux/UNIX) to create a manual snapshot of the existing Redshift cluster: 

aws redshift create-cluster-snapshot
	--region us-east-1
	--cluster-identifier cc-cluster
	--snapshot-identifier cc-cluster-manual-snapshot

04 The command output should return the snapshot configuration metadata: 

{
    "Snapshot": {
        "EstimatedSecondsToCompletion": -1,
        "OwnerAccount": "123456789012",
        "CurrentBackupRateInMegaBytesPerSecond": 0.0,
        "ActualIncrementalBackupSizeInMegaBytes": -1.0,
        "NumberOfNodes": 1,
        "Status": "creating",
        "VpcId": "vpc-2fb56548",
        "ClusterVersion": "1.0",
        "Tags": [],
        "MasterUsername": "ccclusterusr",
        "TotalBackupSizeInMegaBytes": -1.0,
        "DBName": "ccclusterdb",
        "BackupProgressInMegaBytes": 0.0,
        "ClusterCreateTime": "2015-01-11T16:38:21.654Z",
        "EncryptedWithHSM": false,
        "ClusterIdentifier": "cc-cluster",
        "SnapshotCreateTime": "2016-10-17T18:15:49.041Z",
        "AvailabilityZone": "us-east-1b",
        "NodeType": "ds1.xlarge",
        "Encrypted": false,
        "ElapsedTimeInSeconds": 0,
        "SnapshotType": "manual",
        "Port": 5439,
        "SnapshotIdentifier": "cc-cluster-manual-snapshot"
    }
}

05 Run restore-from-cluster-snapshot command (OSX/Linux/UNIX) to create a new AWS Redshift cluster from the snapshot created at step no. 3, using the configuration information returned at step no. 2 and the current generation node type name: 

aws redshift restore-from-cluster-snapshot
	--region us-east-1
	--cluster-identifier cc-cluster-upgraded
	--snapshot-identifier cc-cluster-manual-snapshot
	--node-type ds2.xlarge
	--vpc-security-group-ids sg-061e2e7c
	--cluster-subnet-group-name default
	--availability-zone us-east-1b
	--cluster-parameter-group-name default.redshift-1.0
	--publicly-accessible

06 The command output should return the metadata of the new (upgraded) Redshift cluster: 

{
    "Cluster": {
        "IamRoles": [],
        "ClusterVersion": "1.0",
        "NumberOfNodes": 1,
        "VpcId": "vpc-2fb56548",
        "NodeType": "ds2.xlarge",
        "PubliclyAccessible": true,
        "Tags": [],
        "MasterUsername": "ccclusterusr",
        "ClusterParameterGroups": [
            {
                "ParameterGroupName": "default.redshift-1.0",
                "ParameterApplyStatus": "in-sync"
            }
        ],
        "Encrypted": false,
        "ClusterSecurityGroups": [],
        "AllowVersionUpgrade": true,
        "VpcSecurityGroups": [
            {
                "Status": "active",
                "VpcSecurityGroupId": "sg-061e2e7c"
            }
        ],
        "AvailabilityZone": "us-east-1b",
        "ClusterSubnetGroupName": "default",
        "AutomatedSnapshotRetentionPeriod": 1,
        "ClusterStatus": "creating",
        "ClusterIdentifier": "cc-cluster-upgraded",
        "DBName": "ccclusterdb",
        "PreferredMaintenanceWindow": "fri:03:00-fri:03:30",
        "PendingModifiedValues": {}
    }
}

07 Run describe-clusters command (OSX/Linux/UNIX) using the appropriate query filters to expose the new Redshift cluster endpoint: 

aws redshift describe-clusters
	--region us-east-1
	--cluster-identifier cc-cluster-upgraded
	--query 'Clusters[*].Endpoint.Address'

08 The command output should return the new cluster endpoint URL: 

[
    "cc-cluster-upgraded.cmfpsgvyjhfo.us-east-1.redshift.amazonaws.com"
]

09 As soon as the build process is complete, update your application configuration to point to the AWS Redshift cluster endpoint address returned at step no. 8.

10 Once the Redshift cluster endpoint is changed within your application configuration, run delete-cluster command (OSX/Linux/UNIX) to remove the previous generation node type based cluster from your AWS account: 

aws redshift delete-cluster
	--region us-east-1
	--cluster-identifier cc-cluster
	--final-cluster-snapshot-identifier cc-cluster-finalsnapshot

11 The command output should return the metadata of the cluster selected for deletion: 

{
    "Cluster": {
        "PubliclyAccessible": true,
        "MasterUsername": "ccclusteruser",
        "VpcSecurityGroups": [
            {
                "Status": "active",
                "VpcSecurityGroupId": "sg-061e2e7c"
            }
        ],
        "NumberOfNodes": 1,
        "PendingModifiedValues": {},
        "VpcId": "vpc-2fb56548",
        "ClusterVersion": "1.0",
        "Tags": [],
        "AutomatedSnapshotRetentionPeriod": 1,
        "ClusterParameterGroups": [
            {
                "ParameterGroupName": "default.redshift-1.0",
                "ParameterApplyStatus": "in-sync"
            }
        ],
        "DBName": "ccclusterdb",
        "PreferredMaintenanceWindow": "sun:06:00-sun:06:30",
        "Endpoint": {
            "Port": 5439,
            "Address": "cc-cluster.cmfpsgvyjhfo.us-east-1 ... "
        },
        "IamRoles": [],
        "AllowVersionUpgrade": true,
        "ClusterCreateTime": "2015-01-11T16:38:21.654Z",
        "ClusterSubnetGroupName": "default",
        "ClusterSecurityGroups": [],
        "ClusterIdentifier": "cc-cluster",
        "AvailabilityZone": "us-east-1b",
        "NodeType": "ds1.xlarge",
        "Encrypted": false,
        "ClusterStatus": "final-snapshot"
    }
}

12 Repeat steps no. 1 - 11 for each Redshift cluster that requires the node type upgrade, available in the current region.

13 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 12 to perform the entire process for other regions.

References

Publication date Oct 18, 2016

Related Redshift rules