- Knowledge Base
- Google Cloud Platform
- GCP Google Kubernetes Engine Service
- Enable Encryption for Application-Layer Secrets for GKE Clusters
Enable encryption of Kubernetes secrets with Customer-Managed Keys (CMKs) for Google Kubernetes Engine (GKE) clusters in order to meet security and compliance requirements. Application-layer secrets encryption protects your Kubernetes secrets in etcd with an encryption key managed using the Cloud KMS service.
This rule resolution is part of the Conformity solution.
Application-layer secrets encryption provides an additional layer of security for sensitive data, such as Kubernetes secrets, stored in etcd. With this security feature, you can use an encryption key managed with Cloud KMS to encrypt data at the application layer and protect against attackers that gain access to an offline copy of etcd. Enabling application-layer secrets encryption for your GKE clusters is considered a security best practice for applications that store sensitive and confidential data.
Audit
To determine if application-layer secrets encryption is enabled for your Google Kubernetes Engine (GKE) clusters, perform the following actions:
Using GCP Console
01 Sign in to the Google Cloud Management Console.
02 Select the GCP project that you want to examine from the console top navigation bar.
03 Navigate to Google Kubernetes Engine (GKE) console at https://console.cloud.google.com/kubernetes.
04 In the main navigation panel, under Kubernetes Engine, select Clusters to access the list with the GKE clusters provisioned within the selected project.
05 Click on the name (link) of the GKE cluster that you want to examine.
06 Select the DETAILS tab to view the configuration information available for the selected cluster.
07 In the Security section, check the Application-layer secrets encryption attribute value to determine the encryption feature status. If the Application-layer secrets encryption value is set to Disabled, the encryption of the application-layer secrets is not enabled for the selected Google Kubernetes Engine (GKE) cluster.
08 If the cluster settings page displays a warning message regarding the CloudKMS key configuration, it is possible the key has been disabled or destroyed and will need further inspection before it can successfully encrypt the GKE cluster.
09 Repeat steps no. 5 – 8 for each GKE cluster created for the selected GCP project.
10 Repeat steps no. 2 – 9 for each project deployed within your Google Cloud account.
Using GCP CLI
01 Run projects list command (Windows/macOS/Linux) with custom query filters to list the ID of each project available in your Google Cloud account:
gcloud projects list --format="table(projectId)"
02 The command output should return the requested GCP project ID(s):
PROJECT_ID cc-bigdata-project-123123 cc-web-app-project-112233
03 Run container clusters list command (Windows/macOS/Linux) using the ID of the GCP project that you want to examine as the identifier parameter and custom query filters to describe the name and the region of each GKE cluster provisioned for the selected project:
gcloud container clusters list --project cc-bigdata-project-123123 --format="(NAME,LOCATION)"
04 The command output should return the requested cluster names and their regions:
NAME LOCATION cc-gke-backend-cluster us-central1 cc-gke-frontend-cluster us-central1
05 Run container clusters describe command (Windows/macOS/Linux) using the name of the GKE cluster that you want to examine as the identifier parameter and custom query filters to describe the resource ID of the Customer-Managed Key (CMK) used for application-layer secrets encryption:
gcloud container clusters describe cc-gke-backend-cluster --region=us-central1 --format="yaml(databaseEncryption.keyName,conditions)"
06 The command output should return the requested key identifier (ID):
null
If the container clusters describe command output returns null for the databaseEncryption attribute, there is no Customer-Managed Key (CMK) configured for Kubernetes secrets encryption. If the conditions attribute returns an error, for example CLOUD_KMS_KEY_ERROR, the CloudKMS key may be disabled or destroyed. In both cases, the encryption of the application-layer secrets is not correctly enabled for the selected Google Kubernetes Engine (GKE) cluster.
07 Repeat steps no. 5 and 6 for each GKE cluster available within the selected GCP project.
08 Repeat steps no. 3 – 7 for each GCP project deployed in your Google Cloud account.
Remediation / Resolution
To enable encryption of Kubernetes secrets at the application layer using Cloud KMS Customer-Managed Keys (CMKs), perform the following actions:
Using GCP Console
01 Sign in to the Google Cloud Management Console.
02 Select the GCP project that you want to examine from the console top navigation bar.
03 To create and configure your own Customer-Managed Key (CMK), perform the following:
- Before you can set up and manage any Customer-Managed Keys (CMKs), you must create a key ring. A Cloud KMS key ring is a grouping of cryptographic keys made available for organizational purposes in a specific location
- Navigate to Cloud KMS console at https://console.cloud.google.com/security/kms.
- Choose + CREATE KEY RING from the console top menu to create the required key ring for your new Customer-Managed Key (CMK).
- A key ring requires a name and location. On the Create key ring page, provide a unique name in the Key ring name box, choose the location type, and select the appropriate location from the Location type dropdown list. The location can be either global (Multi-region) or associated with a particular region (Region). If the Customer-Managed Key created later for this key ring will be used to encrypt/decrypt resources in a given region, select that region as the key ring location. Choose CREATE to deploy the new key ring.
- On the Create key setup page, perform the following:
- Under What type of key do you want to create?, select Generated key for the type of the CMK that you want to create.
- Provide a unique name for your new Customer-Managed Key in the Key name box.
- Choose the protection level that you want to use for the new key from the Protection level dropdown list. Protection level determines how cryptographic operations are performed.
- Select Symmetric encrypt/decrypt from the Purpose dropdown list to define the types of operations that your cryptographic key can perform.
- Choose the CMK rotation period from the Key rotation period dropdown list and select the starting date for key rotation.
- Choose CREATE to deploy your new Customer-Managed Key (CMK).
04 Navigate to Google Kubernetes Engine (GKE) console at https://console.cloud.google.com/kubernetes.
05 In the main navigation panel, under Kubernetes Engine, select Clusters.
06 Click on the name (link) of the GKE cluster that you want to reconfigure.
07 Select the DETAILS tab to access the configuration information available for the selected cluster.
08 In the Security section, click on the Edit button (pencil icon) next to Application-layer secrets encryption to modify the encryption configuration for application-layer secrets.
09 Inside the Edit Application-layer Secrets Encryption box, perform the following:
- Select the Encrypt secrets at the application layer checkbox to enable application-layer secrets encryption for the selected GKE cluster.
- Choose the Customer-Managed Key (CMK) created at step no. 3 from the Select a customer-managed key dropdown list. If your new CMK does not appear in the dropdown list, select DON'T SEE YOUR KEY? ENTER KEY RESOURCE NAME and provide the full resource ID of your Customer-Managed Key.
- There is a configuration box which shows: "The service-<project-number>@container-engine-robot.iam.gserviceaccount.com service account does not have the "cloudkms.cryptoKeyEncrypterDecrypter" role. Verify the service account has permission to encrypt/decrypt with the selected key". In the configuration box, choose GRANT to grant the specified service account the required IAM role on the selected CMK.
- Choose SAVE CHANGES to apply the changes.
10 Repeat steps no. 6 – 9 for each GKE cluster that you want to reconfigure, created for the selected GCP project.
11 Repeat steps no. 2 – 10 for each GCP project available in your Google Cloud account.
Using GCP CLI
01 Before you can set up and manage your Customer-Managed Key (CMK), you must create a key ring to store the CMK. Run kms keyrings create command (Windows/macOS/Linux) to create a new Cloud KMS key ring in the specified location. If the CMK created later for this key ring will be used to encrypt/decrypt resources in a given region, select that region as the key ring location:
gcloud kms keyrings create cc-cloud-gke-key-ring --location=us --project=cc-bigdata-project-123123 --format="table(name)"
02 The command output should return the identifier (name) of the new key ring:
NAME projects/cc-bigdata-project-123123/locations/us/keyRings/cc-cloud-gke-key-ring
03 Run kms keys create command (Windows/macOS/Linux) to create a new Customer-Managed Key (CMK) within the Cloud KMS key ring created at the previous steps:
gcloud kms keys create cc-cloud-gke-cmk --location=us-central1 --keyring=cc-cloud-gke-key-ring --purpose=encryption --protection-level=software --rotation-period=90d --next-rotation-time=2022-7-10T15:00:00.0000Z --format="table(name)"
04 The command output should return the full name of the new Customer-Managed Key (CMK):
NAME projects/cc-bigdata-project-123123/locations/us-central1/keyRings/cc-cloud-gke-key-ring/cryptoKeys/cc-cloud-gke-cmk
05 Run projects add-iam-policy-binding command (Windows/macOS/Linux) to assign the "CryptoKey Encrypter/Decrypter" role to the appropriate service account. Replace <kms-project-id>
with the ID of the Google Cloud project where the Customer-Managed Key has been created, and replace <project-number>
with the number of the Google Cloud project that is running your GKE cluster:
gcloud projects add-iam-policy-binding<kms-project-id>
--member serviceAccount:service-<project-number>
@container-engine-robot.iam.gserviceaccount.com --role roles/cloudkms.cryptoKeyEncrypterDecrypter
06 The command output should return the updated IAM policy (YAML format):
Updated IAM policy for project <kms-project-id>. bindings: - members: - serviceAccount:service-<project-number>@container-engine-robot.iam.gserviceaccount.com role: roles/cloudkms.cryptoKeyEncrypterDecrypter - members: - user:admin@cloudconformity.com role: roles/owner etag: abcdabcdabcd version: 1
07 Run container clusters update command (Windows/macOS/Linux) using the name of the GKE cluster that you want to reconfigure as the identifier parameter, to enable application-layer secrets encryption for the selected cluster, using the Customer-Managed Key (CMK) created at the previous steps:
gcloud container clusters update cc-gke-backend-cluster --region=us-central1 --project cc-bigdata-project-123123 --database-encryption-key projects/cc-bigdata-project-123123/locations/us-central1/keyRings/cc-cloud-gke-key-ring/cryptoKeys/cc-cloud-gke-cmk
08 The command output should return the full URL of the reconfigured GKE cluster:
Updating cc-gke-backend-cluster...done. Updated [https://container.googleapis.com/v1/projects/cc-bigdata-project-123123/regions/us-central1/clusters/cc-gke-backend-cluster].
09 Repeat steps no. 7 and 8 to for each GKE cluster that you want to reconfigure, available within the selected GCP project.
10 Repeat steps no. 1 – 9 for each GCP project deployed in your Google Cloud account.
References
- Google Cloud Platform (GCP) Documentation
- Secrets
- Harden your cluster's security
- Encrypt secrets at the application layer
- GCP Command Line Interface (CLI) Documentation
- gcloud projects list
- gcloud container clusters list
- gcloud container clusters describe
- gcloud container clusters update
- gcloud kms keyrings create
- gcloud kms keys create
- gcloud projects add-iam-policy-binding