Output and Storage Volume Data Encrypted With KMS Customer Managed Keys

01 Sign in to the AWS Management Console.

02 Navigate to Amazon SageMaker console available at https://console.aws.amazon.com/sagemaker/.

03 In the main navigation panel, under Training, select Training jobs.

04 Click on the name (link) of the SageMaker training job that you want to examine, available in the Name column.

05 In the Job settings section, perform the following actions:

1. Under Algorithm, check the Volume encryption key attribute value to identify the Customer Managed Key (CMK) used to encrypt the data on the storage volume attached to the ML instance that runs your training job. If the Encryption key attribute does not have a value, the training and inference job data managed by the selected SageMaker training job is encrypted using an AWS-managed key (default key provided by AWS) instead of using a Customer Managed Key (CMK).
2. Under Output data configuration, check the Output encryption key attribute value to identify the Customer Managed Key (CMK) used to encrypt the training job output data. If the Output encryption key attribute does not have a value, the output data for the selected SageMaker training job is encrypted using an AWS-managed key (default key provided by AWS) instead of using a Customer Managed Key (CMK).

06 Repeat steps no. 4 and 5 for each Amazon SageMaker training job available within the current AWS region.

07 Change the AWS cloud region from the navigation bar to repeat the Audit process for other regions.

01 Run list-training-jobs command (OSX/Linux/UNIX) to list the name of each Amazon SageMaker training job available in the selected AWS cloud region:

aws sagemaker list-training-jobs
  --region us-east-1
  --query 'TrainingJobSummaries[*].TrainingJobName'

02 The command output should return the requested SageMaker training job names:

[
	"cc-ml-sampler-training-job",
	"cc-ml-project5-training-job"
]

03 Run describe-training-job command (OSX/Linux/UNIX) with the name of the Amazon SageMaker training job that you want to examine as the identifier parameter and custom output filters to describe the Amazon Resource Name (ARN) of the KMS key used to encrypt the training job output data and the data available on the storage volume attached to the ML instance that runs the training job:

aws sagemaker describe-training-job
  --region us-east-1
  --training-job-name cc-ml-sampler-training-job
  --query '{"OutputKmsKey":OutputDataConfig.KmsKeyId,"VolumeKmsKey":ResourceConfig.VolumeKmsKeyId}'

04 The command output should return the requested resource information:

{
	"VolumeKmsKey": null,
	"OutputKmsKey": ""
}

05 Repeat steps no. 3 and 4 for each Amazon SageMaker training job available in the selected AWS region.

06 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the Audit process for other regions.

01 Sign in to the AWS Management Console.

02 To create your own KMS Customer Managed Key (CMK), navigate to Key Management Service (KMS) console available at https://console.aws.amazon.com/kms/.

03 In the main navigation panel, choose Customer managed keys.

04 Choose Create Key to initiate the key setup process.

05 For Step 1 Configure key, perform the following actions:

1. Choose Symmetric for Key type.
2. Select KMS for Key usage.
3. Choose Advanced options, select KMS - recommended for Key material origin, and choose whether to allow your KMS key to be replicated into other AWS cloud regions. If Single-Region key is selected, the AWS region must match the region of your SageMaker training job.
4. Select Next to continue the key setup process.

06 For Step 2 Add labels, provide the following details:

1. Provide a unique name (alias) for your KMS key in the Alias box.
2. (Optional) Enter a short description in the Description box.
3. (Optional) Choose Add tag from the Tags - optional section to create any necessary tag sets. Tags can be used to categorize and identify your KMS keys and help you track your AWS costs.
4. Select Next to continue the setup.

07 For Step 3 Define key administrative permissions, perform the following operations:

1. For Key administrators, select which IAM users and/or roles can administer your new key through the KMS API. You may need to add additional permissions for the users or roles to administer the key from the AWS Management Console.
2. For Key deletion, choose whether to allow key administrators to delete your KMS key.
3. Select Next to continue the setup process.

08 For Step 4 Define key usage permissions, perform the following actions:

1. For Key users, select which IAM users and/or roles can use your KMS key in cryptographic operations.
2. (Optional) For Other AWS accounts section, specify the AWS accounts that can use your key. To configure cross-account access, choose Add another AWS account and enter the ID of the AWS cloud account that can use your KMS key for cryptographic operations. The administrators of the AWS accounts you specify at this step are responsible for managing the permissions that allow their IAM users and/or roles to use your key.
3. Select Next to continue the setup.

09 For Step 5 Review, review the key configuration and key policy, then choose Finish to create your new Amazon KMS Customer Managed Key (CMK).

10 Once your new KMS Customer Managed Key (CMK) is available, navigate to Amazon SageMaker console available at https://console.aws.amazon.com/sagemaker/.

11 In the main navigation panel, under Training, select Training jobs.

12 Click on the name (link) of the SageMaker training job that you want to re-create (i.e. source job) and note the training job configuration information such as algorithm, IAM role, network and output settings.

13 Navigate back to the Training jobs page, choose Actions, select Clone, and perform the following actions to create your new SageMaker training job:

1. For Job settings, perform the following operations:
   1. Provide a unique name for your new training job in the Job name box.
   2. Choose the IAM role used by the source training job from the IAM role dropdown list.
   3. Verify that the chosen algorithm and resource configuration options are set up correctly (must match the source training job configuration).
   4. For Encryption key - optional, select the name (alias) of the Amazon KMS Customer Managed Key (CMK) created earlier in the Remediation process.
2. For Network, perform the following actions:
   1. Select Enable network isolation to enable network isolation for the new Amazon SageMaker training job. When you enable network isolation, the containers are restricted from making any outbound network calls.
   2. Select the ID of the Virtual Private Cloud (VPC) where you want to deploy your resources, from the VPC - optional dropdown list. For better security, AWS recommends using a private VPC.
   3. Once the VPC network is selected, choose the ID of the appropriate VPC subnet(s) from the Subnet(s) dropdown list.
   4. Select one or more security groups from the Security group(s) list, based on your access policy requirements.
   5. Select Enable inter-container traffic encryption to protect the communication between ML compute instances in a distributed training job.
3. For Hyperparameters, ensure that the algorithm hyperparameters are set up correctly (must match the parameters used by the source training job).
4. For Input data configuration, ensure that input data channels are properly configured (must match the input data configuration used by the source job).
5. (Optional) For Checkpoint configuration - optional, select the appropriate location for algorithm-generated checkpoints.
6. For Output data configuration, ensure that output data location is properly configured (must match the output data configuration used by the source training job), and paste the ID/ARN of your new Amazon KMS Customer Managed Key (CMK) in the Encryption key - optional box.
7. (Optional) For Managed spot training, choose whether to enable enable managed spot training (must match the managed spot training settings used by the source job)
8. (Optional) For Tags - optional, create any required tag sets, according to the source training job tagging scheme.
9. Choose Create training job to create your new, compliant Amazon SageMaker training job.

14 Repeat steps no. 12 and 13 for each SageMaker training job that you want to re-create, available within the current AWS region.

15 Change the AWS cloud region from the navigation bar and repeat the Remediation process for other regions.

01 Define the IAM policy that enables the selected users and/or roles to manage your new KMS Customer Managed Key (CMK), and to encrypt/decrypt your SageMaker training job data using the KMS API. Create a new policy document (JSON format), name the file training-job-cmk-policy.json, and paste the following content (replace \<aws-account-id\> and \<role-name\> with your own AWS details):

{
	"Id": "aws-sagemaker-cmk-policy",
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Enable IAM User Permissions",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::<aws-account-id>:root"
			},
			"Action": "kms:*",
			"Resource": "*"
		},
		{
			"Sid": "Allow access for Key Administrators",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::<aws-account-id>:role/service-role/<role-name>"
			},
			"Action": [
				"kms:Create*",
				"kms:Describe*",
				"kms:Enable*",
				"kms:List*",
				"kms:Put*",
				"kms:Update*",
				"kms:Revoke*",
				"kms:Disable*",
				"kms:Get*",
				"kms:Delete*",
				"kms:TagResource",
				"kms:UntagResource",
				"kms:ScheduleKeyDeletion",
				"kms:CancelKeyDeletion",
				"kms:RotateKeyOnDemand"
			],
			"Resource": "*"
		},
		{
			"Sid": "Allow use of the key",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::<aws-account-id>:role/service-role/<role-name>"
			},
			"Action": [
				"kms:Encrypt",
				"kms:Decrypt",
				"kms:ReEncrypt*",
				"kms:GenerateDataKey*",
				"kms:DescribeKey"
			],
			"Resource": "*"
		},
		{
			"Sid": "Allow attachment of persistent resources",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::<aws-account-id>:role/service-role/<role-name>"
			},
			"Action": [
				"kms:CreateGrant",
				"kms:ListGrants",
				"kms:RevokeGrant"
			],
			"Resource": "*",
			"Condition": {
				"Bool": {
					"kms:GrantIsForAWSResource": "true"
				}
			}
		}
	]
}

02 Run create-key command (OSX/Linux/UNIX) with the policy document created at the previous step (i.e.training-job-cmk-policy.json) as value for the --policy parameter, to create your new Amazon KMS Customer Managed Key (CMK):

aws kms create-key
  --region us-east-1
  --description 'KMS CMK for encrypting SageMaker training job data'
  --policy file://training-job-cmk-policy.json
  --query 'KeyMetadata.Arn'

03 The command output should return the ARN of the new Customer Managed Key (CMK):

"arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"

04 Run create-alias command (OSX/Linux/UNIX) to attach an alias to your new Customer Managed Key (CMK). The alias must start with the prefix "alias/" (the command does not produce an output):

aws kms create-alias
  --region us-east-1
  --alias-name alias/TrainingJobCMK
  --target-key-id arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd

05 Run create-training-job command (OSX/Linux/UNIX) to re-create your Amazon SageMaker training job using a different encryption configuration. To encrypt the data on the storage volume attached to the ML instance that runs your training job with a customer-provided KMS key, provide the ARN of the Customer Managed Key (CMK) created earlier in the Remediation process as value for the --resource-config VolumeKmsKeyId parameter. To encrypt your training job output data using a customer-provided KMS key, provide the ARN of your new Customer Managed Key (CMK) as value for the --output-data-config KmsKeyId parameter:

aws sagemaker create-training-job
  --region us-east-1
  --training-job-name cc-new-sampler-training-job
  --algorithm-specification TrainingImage="123456789012.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.3-1",TrainingInputMode="File"
  --role-arn "arn:aws:iam::123456789012:role/service-role/cc-sagemaker-iam-role"
  --output-data-config S3OutputPath="s3://trendmicro.com",CompressionType="GZIP",KmsKeyId="arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"
  --resource-config InstanceType="ml.m5.large",InstanceCount=1,VolumeSizeInGB=20,KeepAlivePeriodInSeconds=300,VolumeKmsKeyId="arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd"
  --stopping-condition MaxRuntimeInSeconds=86400
  --vpc-config SecurityGroupIds="sg-0abcd1234abcd1234",Subnets="subnet-01234abcd1234abcd"
  --enable-network-isolation

06 The command output should return the Amazon Resource Name (ARN) of the new SageMaker training job:

{
	"TrainingJobArn": "arn:aws:sagemaker:us-east-1:123456789012:training-job/cc-new-sampler-training-job"
}

07 Repeat steps no. 5 and 6 for each SageMaker training job that you want to re-create, available in the selected AWS region.

08 Change the AWS cloud region by updating the --region command parameter value and repeat the Remediation process for other regions.