Redshift Cluster Encrypted With KMS Customer Master Keys

01 Login to the AWS Management Console.

02 Navigate to IAM dashboard at https://console.aws.amazon.com/iam/.

03 In the left navigation panel, click Encryption Keys.

04 Select the appropriate AWS region from the Filter menu:

(must match the region where the Redshift cluster that you want to examine is provisioned).

05 Verify the ID:

of the KMS default key generated for the Redshift service, key identified by the aws/redshift alias.

06 Now navigate to Redshift dashboard at https://console.aws.amazon.com/redshift/.

07 In the navigation panel, under Redshift Dashboard, click Clusters.

08 Choose the Redshift cluster that you want to examine then click on its identifier link:

listed in the Cluster column.

09 On the selected cluster configuration page, in the Cluster Database Properties section, verify the Encrypted property status and configuration:

1. If the current status is set to No: , data-at-rest encryption is not enabled for the selected Redshift cluster. To enable encryption using AWS KMS CMK, follow the steps outlined in the Remediation/Resolution section of this rule.
2. If the current status is set to Yes and the KMS Key ID: is matching the key ID found at step no. 5, the selected cluster is encrypted using the default master key (AWS-managed key) instead of a KMS CMK key (customer-managed key). To enable encryption using KMS customer-managed key, follow the steps outlined in the Remediation/Resolution section.

10 Repeat steps no. 6 - 9 to verify the encryption status and configuration for other clusters provisioned in the current region.

11 Change the AWS region from the navigation bar and repeat the audit process for other regions.

01 Run describe-clusters command (OSX/Linux/UNIX) using custom query filters to list the identifiers (names) of all Redshift clusters currently available in the selected region:

aws redshift describe-clusters
	--region us-east-1
	--output table
	--query 'Clusters[*].ClusterIdentifier'

02 The command output should return a table with the requested names:

----------------------
|  DescribeClusters  |
+--------------------+
|  cc-cluster        |
|  dbe-cluster       |
|  dbx-cluster       |
+--------------------+

03 Run again describe-clusters command (OSX/Linux/UNIX) using the name of cluster that you want to examine as identifier and the necessary query filters to expose its encryption status and configuration:

aws redshift describe-clusters
	--region us-east-1
	--cluster-identifier cc-cluster
	--query 'Clusters[*].[Encrypted,KmsKeyId]'

04 The command output should return the requested status/configuration details:

1. If the value returned is false, the selected Amazon Redshift cluster is not encrypted at all:
   [
       false
   ]
   

2. If the output is an array which contains the encryption status set to true (encrypted) and the Amazon Resource Name (ARN) of the encryption key used (as shown in the example above), the selected Redshift cluster is currently encrypted using an AWS KMS key:
   [
       [
           true,
           "arn:aws:kms:us-east-1:123456789012:key/
            4302e0f2-ec36-4f3c-806a-89f454193a47"
       ]
   ]
   

05 Now run list-aliases command (OSX/Linux/UNIX) to list all the KMS keys aliases and their ARNs, available in the AWS region that hosts the Redshift cluster:

aws kms list-aliases
	--region us-east-1

06 The command output should return each available KMS key alias, ID and ARN. Now compare each key ID returned (TargetKeyId parameter value - highlighted below) with the KMS key ID returned at step no. 4.b (ARN subset) in order to determine the key type used for the selected cluster encryption. If the AliasName parameter value for the matched ID is alias/aws/redshift, the selected Redshift cluster is encrypted using the default key instead of an AWS KMS customer-managed key (recommended):

{
  "Aliases": [
    {
       "AliasArn": "arn:aws:kms:us-east-1:123456789012:alias/aws/ebs",
       "AliasName": "alias/aws/ebs",
       "TargetKeyId": "3883fccd-25df-4545-bc32-1171cabd3586"
    },
    {
      "AliasArn": "arn:aws:kms:us-east-1:123456789012:alias/aws/redshift",
      "AliasName": "alias/aws/redshift",
      "TargetKeyId": "4302e0f2-ec36-4f3c-806a-89f454193a47"
    }
  ]
}

07 Repeat steps no. 3 - 6 for each Redshift cluster available in the current region.

08 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 7 to perform the audit process for other regions.

01 Login to the AWS Management Console.

02 Navigate to IAM dashboard at https://console.aws.amazon.com/iam/.

03 In the left navigation panel click Encryption Keys.

04 Select the appropriate AWS region from the Filter menu:

(must match the region where the new Redshift cluster will be launched).

05 Click Create Key button from the dashboard top menu.

06 In the Alias (required) and Description fields, enter a name (alias) and a description for the new CMK then click the Next Step button.

07 Under Key Administrators section, select which IAM users and/or roles can administer the new CMK, then click Next Step.

08 Under This Account section, select which IAM users and/or roles can use the new CMK to encrypt/decrypt data with the AWS KMS API.

09 (Optional) Under External Accounts section, click Add an External Account and enter an external account ID in order to add another AWS account that can use this CMK to encrypt/decrypt the cluster data. The owners of the external AWS accounts must also provide access to this CMK by creating appropriate policies for their IAM users.

10 Click Next Step to continue.

11 Under Preview Key Policy section, review the key policy generated by AWS then click Finish to create your new CMK. Once the key is created, the KMS dashboard will display a confirmation message: “Your master key was created successfully. Alias: <the CMK display name>”.

12 Click on the newly created CMK alias link and copy the key full ARN displayed in the Summary section:

13 Now that the necessary KMS CMK customer-managed key is created, navigate to the Redshift dashboard at https://console.aws.amazon.com/redshift/ to launch the new cluster.

14 In the left navigation panel, under Redshift Dashboard, click Clusters.

15 Click Launch Cluster button from the dashboard top menu to start the cluster setup process.

16 On the Cluster Details configuration page, enter a unique name for your new cluster in the Cluster Identifier field and fill out the rest of the fields available on this page with the information taken from your existing (unencrypted or encrypted with the default key) cluster.

17 Click the Continue button to continue the setup process.

18 On the Node Configuration page, select the appropriate node type for the new cluster from the Node Type dropdown list and configure the number of nodes used to match the existing cluster configuration.

19 Click Continue.

20 On the Additional Configuration page, in the first configuration section, select KMS next to Encrypt Database. Choose the name of your KMS CMK key created earlier from the Master Key dropdown list to encrypt the new Redshift cluster using your own customer-managed key. Configure the rest of the options available on the Additional Configuration page to reflect the existing cluster configuration.

21 Click Continue to load the next page.

22 On the Review page, review the cluster properties, its database details, the security and encryption configuration, then click Launch Cluster to build the new AWS Redshift cluster.

23 On the confirmation page click Close to return to the dashboard. Once the Cluster Status value changes to available and the DB Health status changes to healthy, the new cluster can used to load the existing data.

24 Now unload your data from the former Redshift cluster and reload it into the newly created cluster using the Amazon Redshift Unload/Copyutility. With this utility tool you can unload (export) your data from the unencrypted cluster (source) to an AWS S3 bucket, encrypt it, then import the data into your new cluster (destination) and clean up the S3 bucket used. All the necessary instructions to install, configure and use the Amazon Redshift Unload/Copy tool can be found at this URL.

25 As soon as the migration process is completed and all the data is loaded into your new Redshift cluster (encrypted with the KMS CMK key), you can update your application configuration to refer to the new cluster endpoint:

26 Once the Redshift cluster endpoint is changed within your application configuration, you can remove the former (i.e. the unencrypted or encrypted with the default key) cluster from your AWS account by performing the following actions:

1. In the navigation panel, under Redshift Dashboard, click Clusters.
2. Choose the Redshift cluster that you want to remove then click on its identifier link listed in the Cluster column.
3. On the selected cluster Configuration tab, click the Cluster dropdown button from the dashboard main menu then select Delete from the dropdown list: .
4. Inside the Delete Cluster dialog box, enter a unique name for the final snapshot in the Snapshot name box then click Delete to confirm the action. Once the snapshot is created the former cluster removal process begins.

27 Repeat steps no. 2 - 26 to enable data-at-rest encryption for other Redshift clusters launched in the current region using KMS CMK customer-managed keys.

28 Change the AWS region from the navigation bar and repeat the entire process for other regions.

01 Create a policy that enables the selected IAM users and/or roles to administer the new CMK and the selected IAM users and/or roles to encrypt/decrypt data using the AWS KMS API. Create a new policy document called redshift-cmk-policy.json and paste the following content (replace the highlighted details - the ARNs for the IAM users and/or roles - with your details):

{
  "Version": "2012-10-17",
  "Id": "redshift-custom-key-policy",
  "Statement": [
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Allow access for Key Administrators",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/RedshiftManager"
      },
      "Action": [
        "kms:Create*",
        "kms:Describe*",
        "kms:Enable*",
        "kms:List*",
        "kms:Put*",
        "kms:Update*",
        "kms:Revoke*",
        "kms:Disable*",
        "kms:Get*",
        "kms:Delete*",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow use of the key",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/ClusterAdmin"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow attachment of persistent resources",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/ClusterAdmin"
      },
      "Action": [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ],
      "Resource": "*",
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": "true"
        }
      }
    }
  ]
}

02 Run create-key command (OSX/Linux/UNIX) using the file name of the policy document created at the previous step (i.e. redshift-cmk-policy.json) as parameter to create the new KMS CMK customer-managed key:

aws kms create-key
	--region us-east-1
	--description 'CMK for Amazon Redshift cluster encryption'
	--policy file://redshift-cmk-policy.json

03 The command output should return the new KMS CMK metadata. Copy the CMK unique ID (KeyID parameter value - highlighted) in order to use it later when you need to specify the CMK key required for cluster encryption:

{
    "KeyMetadata": {
        "Origin": "AWS_KMS",
        "KeyId": "33ccf6a5-39c7-48af-ba0c-d2b245798025",
        "Description": "CMK for Amazon Redshift cluster encryption",
        "Enabled": true,
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "CreationDate": 1475864212.692,
        "Arn": "arn:aws:kms:us-east-1:123456789012:key/
                33ccf6a5-39c7-48af-ba0c-d2b245798025",
        "AWSAccountId": "123456789012"
    }
}

04 Run create-alias command (OSX/Linux/UNIX) using the key ARN returned at the previous step to attach an alias (display name) to the CMK. The alias must start with the prefix "alias/":

aws kms create-alias
	--region us-east-1
	--alias-name alias/cc-cluster-cmk
	--target-key-id arn:aws:kms:us-east-1:123456789012:key/33ccf6a5-39c7-48af-ba0c-d2b245798025

05 Run describe-clusters command (OSX/Linux/UNIX) to reveal the configuration available for the existing (unencrypted or encrypted with the default key) Redshift cluster:

aws redshift describe-clusters
	--region us-east-1
	--cluster-identifier cc-cluster

06 The command output should return the requested configuration information which will be useful later when the new Redshift cluster will be created:

{
    "Clusters": [
        {
            "PubliclyAccessible": true,
            "MasterUsername": "ccclusteruser",
            "VpcSecurityGroups": [
                {
                    "Status": "active",
                    "VpcSecurityGroupId": "sg-58dc0a22"
                }
            ],
            "ClusterPublicKey": "ssh-rsa AAAAB4Bc46D5XtMu ... ",
            "NumberOfNodes": 1,
            "PendingModifiedValues": {},
            "VpcId": "vpc-2fb56548",
            "ClusterVersion": "1.0",
            "Tags": [],
            "AutomatedSnapshotRetentionPeriod": 1,
            "ClusterParameterGroups": [
                {
                    "ParameterGroupName": "default.redshift-1.0",
                    "ParameterApplyStatus": "in-sync"
                }
            ],
            "DBName": "ccclusterdb",
            "PreferredMaintenanceWindow": "sun:06:00-sun:06:30",
            "Endpoint": {
                "Port": 5439,
                "Address": "cc-cluster.cmfpsgvyjhfo.us-east-1 ... "
            },
            "IamRoles": [],
            "AllowVersionUpgrade": true,
            "ClusterCreateTime": "2016-09-01T12:24:54.550Z",
            "ClusterSubnetGroupName": "default",
            "ClusterSecurityGroups": [],
            "ClusterIdentifier": "cc-cluster",
            "ClusterNodes": [
                {
                    "NodeRole": "SHARED",
                    "PrivateIPAddress": "172.31.9.161",
                    "PublicIPAddress": "54.55.80.230"
                }
            ],
            "AvailabilityZone": "us-east-1a",
            "NodeType": "dc1.large",
            "Encrypted": true,
            "KmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/
                         4302e0f2-ec36-4f3c-806a-89f454193a47",
            "ClusterRevisionNumber": "1101",
            "ClusterStatus": "available"
        }
    ]
}

07 Run create-cluster command (OSX/Linux/UNIX) using the existing cluster configuration details returned at the previous step to create a new one. The new Amazon Redshift cluster will be encrypted with your own KMS CMK customer-managed key, referenced by its ID (e.g. 33ccf6a5-39c7-48af-ba0c-d2b245798025):

aws redshift create-cluster
	--region us-east-1
	--cluster-identifier cc-cluster-encrypted
	--cluster-type single-node
	--node-type dc1.large
	--db-name ccclusterdb
	--master-username ccclusteruser
	--master-user-password MYclusterPWD0
	--vpc-security-group-ids sg-58dc0a22
	--availability-zone us-east-1a
	--port 5439
	--cluster-subnet-group-name default
	--cluster-parameter-group-name default.redshift-1.0
	--automated-snapshot-retention-period 1
	--publicly-accessible
	--allow-version-upgrade
	--encrypted
	--kms-key-id 33ccf6a5-39c7-48af-ba0c-d2b245798025

08 The command output should return the new cluster configuration metadata:

{
    "Cluster": {
        "PubliclyAccessible": true,
        "MasterUsername": "ccclusteruser",
        "VpcSecurityGroups": [
            {
                "Status": "active",
                "VpcSecurityGroupId": "sg-58dc0a22"
            }
        ],
        "NumberOfNodes": 1,
        "PendingModifiedValues": {
            "MasterUserPassword": "****"
        },
        "VpcId": "vpc-2fb56548",
        "ClusterVersion": "1.0",
        "Tags": [],
        "AutomatedSnapshotRetentionPeriod": 1,
        "ClusterParameterGroups": [
            {
                "ParameterGroupName": "default.redshift-1.0",
                "ParameterApplyStatus": "in-sync"
            }
        ],
        "DBName": "ccclusterdb",
        "PreferredMaintenanceWindow": "thu:10:30-thu:11:00",
        "IamRoles": [],
        "AllowVersionUpgrade": true,
        "ClusterSubnetGroupName": "default",
        "ClusterSecurityGroups": [],
        "ClusterIdentifier": "cc-cluster-encrypted",
        "AvailabilityZone": "us-east-1a",
        "NodeType": "dc1.large",
        "Encrypted": true,
        "KmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/
                     33ccf6a5-39c7-48af-ba0c-d2b245798025",
        "ClusterStatus": "creating"
    }
}

09 Run again describe-clusters command (OSX/Linux/UNIX) using the appropriate query filters to expose the new Redshift cluster endpoint:

aws redshift describe-clusters
	--region us-east-1
	--cluster-identifier cc-cluster-encrypted
	--query 'Clusters[*].Endpoint.Address'

10 The command output should return the new cluster endpoint URL:

[
    "cc-cluster-encrypted.cmfpsgvyjhfo.us-east-1.redshift.amazonaws.com"
]

11 Now unload your data from the unencrypted Redshift cluster and reload it in the one created at the previous step by using the Amazon Redshift Unload/Copy utility. With this utility tool you can export your data from the unencrypted cluster (source) to an AWS S3 bucket, encrypt it, then import the data into your new Redshift cluster (destination) and remove the S3 bucket used. The necessary instructions to install, configure and use the Amazon Redshift Unload/Copy tool can be found on this page.

12 As soon as the transfer is completed and all the data is loaded into your new cluster, you can update your application configuration to point to the new AWS Redshift cluster endpoint address returned at step no. 10.

13 Once the Redshift cluster endpoint is changed within your application configuration, run delete-cluster command (OSX/Linux/UNIX) to remove the former cluster from your AWS account:

aws redshift delete-cluster
	--region us-east-1
	--cluster-identifier cc-cluster
	--final-cluster-snapshot-identifier cc-cluster-finalsnapshot

14 The command output should return the metadata of the cluster selected for deletion:

{
    "Cluster": {
        "PubliclyAccessible": true,
        "MasterUsername": "ccclusteruser",
        "VpcSecurityGroups": [
            {
                "Status": "active",
                "VpcSecurityGroupId": "sg-58dc0a22"
            }
        ],
        "NumberOfNodes": 1,
        "PendingModifiedValues": {},
        "VpcId": "vpc-2fb56548",
        "ClusterVersion": "1.0",
        "Tags": [],
        "AutomatedSnapshotRetentionPeriod": 1,
        "ClusterParameterGroups": [
            {
                "ParameterGroupName": "default.redshift-1.0",
                "ParameterApplyStatus": "in-sync"
            }
        ],
        "DBName": "ccclusterdb",
        "PreferredMaintenanceWindow": "sun:06:00-sun:06:30",
        "Endpoint": {
            "Port": 5439,
            "Address": "cc-cluster.cmfpsgvyjhfo.us-east-1 ... "
        },
        "IamRoles": [],
        "AllowVersionUpgrade": true,
        "ClusterCreateTime": "2016-09-01T12:24:54.550Z",
        "ClusterSubnetGroupName": "default",
        "ClusterSecurityGroups": [],
        "ClusterIdentifier": "cc-cluster",
        "AvailabilityZone": "us-east-1a",
        "NodeType": "dc1.large",
        "Encrypted": true,
        "KmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/
                     4302e0f2-ec36-4f3c-806a-89f454193a47",
        "ClusterStatus": "final-snapshot"
    }
}

15 Repeat steps no. 1 - 14 to enable data encryption for other Redshift clusters provisioned in the current region using KMS CMK customer-managed keys.

16 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 15 for other regions.