Logo of Trend Micro
Use the Conformity Knowledge Base AI to help improve your Cloud Posture

Enable BigQuery Dataset Encryption with Customer-Managed Encryption Keys

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: High (not acceptable risk)
Rule ID: BigQuery-003

Ensure that all your Google Cloud BigQuery datasets are encrypted using Customer-Managed Encryption Keys (CMEKs) in order to have a more granular control over the dataset encryption/decryption process.

This rule resolution is part of the Conformity Security & Compliance tool for GCP.

Security

By default, Google Cloud BigQuery encrypts your data at rest using envelope encryption with Google-managed cryptographic keys. BigQuery manages this type of encryption without any additional actions from you and your application. However, if you have sensitive or confidential data to encrypt, and you need to fully control and manage the encryption process yourself, you can create and use Customer-Managed Encryption Keys (CMEKs). Instead of letting Google manage the encryption keys that protect your BigQuery datasets, you can choose to create and manage your own encryption keys using Google Cloud Key Management Service (Cloud KMS). With Cloud KMS, you can create, rotate, manage and destroy your own Customer-Managed Encryption Keys (CMEKs).

Audit

To determine if your Google Cloud BigQuery datasets are encrypted using default Customer-Managed Encryption Keys (CMEKs), perform the following operations:

Note: Verifying encryption settings for BigQuery datasets using Google Cloud Platform (GCP) Console is not currently supported.

Using GCP CLI and BQ CLI

01 Run projects list command (Windows/macOS/Linux) using custom query filters to list the ID of each Google Cloud Platform (GCP) project available in your cloud account: 

gcloud projects list
  --format="table(projectId)"

02 The command output should return the requested GCP project ID(s): 

PROJECT_ID
cc-big-data-project-112233
cc-anallytics-project-123123

03 Run bq ls command (using bq Python tool) to list the identifier (name) of each BigQuery dataset created for the specified Google Cloud project: 

bq ls
  --project_id cc-ml-project-112233
  --format=pretty

04 The command output should return a table with the requested dataset identifier(s): 

+------------------------+
|       datasetId        |
+------------------------+
|   production_dataset   |
|   ad_traffic_dataset   |
+------------------------+

05 Run bq show command (using bq tool) using the ID of the project and the name of the dataset that you want to examine as the identifier parameters, to describe the name of the default Customer-Managed Encryption Key (CMEK) configured for the selected BigQuery dataset: 

bq show
  --format=prettyjson cc-big-data-project-112233:production_dataset | jq '.defaultEncryptionConfiguration.kmsKeyName'

06 The command output should return the full name of the Customer-Managed Encryption Key used for dataset encryption: 

null

If the bq show command output returns null, as shown in the output example above, the selected Google Cloud BigQuery dataset is not encrypted using a Customer-Managed Encryption Key (CMEK).

07 Repeat steps no. 5 and 6 for each BigQuery dataset available in the selected GCP project.

08 Repeat steps no. 3 – 7 for each project deployed within your Google Cloud account.

Remediation / Resolution

To enable encryption at rest with default Customer-Managed Encryption Keys (CMEKs) for your BigQuery datasets, perform the following operations:

Note: Configuring default Customer-Managed Encryption Keys for BigQuery datasets using Google Cloud Platform (GCP) Console is not currently supported.

Using GCP CLI and BQ CLI

01 To set up and manage Customer-Managed Encryption Keys, you must first create a key ring. A Cloud KMS key ring is a grouping of cryptographic keys made available for organizational purposes in a specific Google Cloud location. Run kms keyrings create command (Windows/macOS/Linux) to create a new Cloud KMS key ring in the specified location (must match the BigQuery dataset location): 

gcloud kms keyrings create cc-bigdata-key-ring
  --location=us
  --project=cc-big-data-project-112233
  --format="table(name)"

02 The command output should return the identifier (name) of the newly created key ring: 

NAME
projects/cc-big-data-project-112233/locations/us/keyRings/cc-bigdata-key-ring

03 Run kms keys create command (Windows/macOS/Linux) to create a new Cloud KMS Customer-Managed Encryption Key (CMEK) within the KMS key ring provisioned at the previous steps: 

gcloud kms keys create cc-bigdata-cmek
  --location=us
  --keyring=cc-bigdata-key-ring
  --purpose=encryption
  --protection-level=software
  --rotation-period=90d
  --next-rotation-time=2022-11-11T12:00:00.0000Z
  --format="table(name)"

04 The command output should return the name of the new Customer-Managed Encryption Key: 

NAME
projects/cc-ml-project-112233/locations/us/keyRings/cc-bigdata-key-ring/cryptoKeys/cc-bigdata-cmek

05 Run projects add-iam-policy-binding command (Windows/macOS/Linux) to assign the Cloud KMS "CryptoKey Encrypter/Decrypter" role to the BigQuery system service account. Replace <kms-project-id> with the ID of the Google Cloud project where the Customer-Managed Encryption Key was created, and replace <project-number> with the project number (not the project ID) of the Google Cloud project that is running your BigQuery dataset: 

gcloud projects add-iam-policy-binding <kms-project-id>
  --member serviceAccount:bq-<project-number>@bigquery-encryption.iam.gserviceaccount.com
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter

06 The command output should return the updated IAM policy (YAML format): 

Updated IAM policy for project <kms-project-id>.
  bindings:
  - members:
    - serviceAccount:bq-<project-number>@bigquery-encryption.iam.gserviceaccount.com
    role: roles/cloudkms.cryptoKeyEncrypterDecrypter
  - members:
    - serviceAccount:bq-<project-number>@bigquery-encryption.iam.gserviceaccount.com
    - user:admin@trendmicro.com
    role: roles/owner
  etag: abcdabcdabcd
  version: 1

07 Run bq update command (using bq tool) to update the selected Google Cloud BigQuery dataset in order to enable encryption at rest using the default Customer-Managed Encryption Key (CMEK) created earlier in the Remediation process, as value for the --default_kms_key command parameter: 

bq update
  --default_kms_key projects/cc-ml-project-112233/locations/us/keyRings/cc-bigdata-key-ring/cryptoKeys/cc-bigdata-cmek
  --dataset cc-big-data-project-112233:production_dataset

08 The output should return the bq update command request status: 

Dataset 'cc-big-data-project-112233:production_dataset' successfully updated.

09 Repeat steps no. 7 and 8 for each BigQuery dataset available in the selected GCP project.

10 Repeat steps no. 1 – 9 for each project deployed within your Google Cloud account.

References

Publication date Aug 17, 2022

Related BigQuery rules