Azure Databricks Workspace Data Encrypted with Customer-Managed Keys

01 Sign in to the Microsoft Azure Portal.

02 Navigate to All resources blade available at https://portal.azure.com/#browse/all to access all your Microsoft Azure cloud resources.

03 Choose the Azure subscription that you want to access from the Subscription equals all filter box and choose Apply.

04 From the Type equals all filter box, choose Type for Filter, select Equals for Operator, choose Azure Databricks Service for Value, and select Apply to list only the Azure Databricks service workspaces available in the selected subscription.

05 Click on the name (link) of the Azure Databricks workspace that you want to examine.

06 In the resource navigation panel, under Settings, select Encryption to access the encryption configuration settings available for the selected workspace.

07 Under Customer-managed keys, verify the Managed Disks setting checkbox to determine if encryption at rest using Customer-Managed Keys (CMKs) is enabled for the managed disks associated with your Databricks workspace. If the Managed Disks setting checkbox is unchecked, your Azure Databricks workspace data is not encrypted at rest using Customer-Managed Keys (CMKs).

08 Repeat steps no. 5 - 7 for each Azure Databricks workspace available within the selected subscription.

09 Repeat steps no. 3 – 8 for each Azure subscription created in your Microsoft Azure cloud account.

01 Run account list command (Windows/macOS/Linux) with custom output filters to list the IDs of the cloud subscriptions available in your Azure cloud account:

az account list
	--query '[*].id'

02 The command output should return the requested subscription identifiers (IDs):

[
	"abcdabcd-1234-abcd-1234-abcdabcdabcd",
	"abcd1234-abcd-1234-abcd-abcd1234abcd"
]

03 Run account set command (Windows/macOS/Linux) with the ID of the Azure cloud subscription that you want to examine as the identifier parameter to set the selected subscription to be the current active subscription (the command does not produce an output):

az account set
	--subscription abcdabcd-1234-abcd-1234-abcdabcdabcd

04 Run databricks workspace list command (Windows/macOS/Linux) with custom output filters to list the identifier (ID) of each Azure Databricks workspace available in the selected Azure subscription:

az databricks workspace list
	--query '[*].id'

05 The command output should return the requested Databricks workspace IDs:

[
	"/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Databricks/workspaces/cc-project9-data-workspace",
	"/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Databricks/workspaces/cc-prod-databricks-workspace"
]

06 Run databricks workspace show command (Windows/macOS/Linux) with the ID of the Azure Databricks workspace that you want to examine as the identifier parameter and custom output filters to determine if encryption at rest using Customer-Managed Keys is enabled for the managed disks associated with the selected Databricks workspace:

az databricks workspace show
	--ids "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Databricks/workspaces/cc-project9-data-workspace"
	--query '{"encryptionKey":encryption.entities.managedDisk.keyVaultProperties.keyName}'

07 The command output should return the name of the Customer-Managed Key (CMK) used for workspace data encryption:

{
	"encryptionKey": null
}

08 Repeat steps no. 6 and 7 for each Azure Databricks workspace available in the selected Azure subscription.

09 Repeat steps no. 3 – 8 for each Azure subscription created in your Microsoft Azure cloud account.

01 Sign in to the Microsoft Azure Portal.

02 Navigate to Managed Identities blade available at https://portal.azure.com/#browse/Microsoft.ManagedIdentity%2FuserAssignedIdentities.

03 Choose the Azure subscription that you want to access from the Subscription equals all filter box and choose Apply.

04 Choose Create and perform the following actions to create the required user-assigned managed identity:

1. For Basics, choose the correct subscription and resource group, provide a unique name for the new managed identity, then select the Azure region where your cloud resources are deployed. Choose Next to continue the setup process.
2. For Tags, use the Name and Value fields to create tags that will help organize the identity of the identity. Choose Review + create to validate the identity setup.
3. For Review + create, review the resource configuration details, then choose Create to create your new user-assigned managed identity.

05 Navigate to Key vaults blade available at https://portal.azure.com/#browse/Microsoft.KeyVault%2Fvaults.

06 Choose Create and perform the following actions to create the Azure key vault that will store your new Customer-Managed Key (CMK):

1. For Basics, choose the correct Azure subscription and resource group, provide a unique name for the new key vault, then select the Azure cloud region where the vault will be deployed and the appropriate pricing tier. Configure the vault retention period and enable purge protection. Both soft delete and purge protection must be enabled on the key vault. Choose Next to continue the setup process.
2. For Access configuration, select Vault access policy for Permission model, choose Create under Access policies, and follow the setup wizard to create the policy that allows Azure Databricks to get, wrap, and unwrap encryption keys from the new vault. For the policy principal, choose the user-assigned managed identity created at step no. 4. Once the access policy is configured, choose Create to create and attach it to the key vault. Configure the Azure resource access under Resource access. Choose Next to continue the setup.
3. For Networking, configure the network access control for the new key vault. You can connect to your new key vault either publicly, via public IP addresses or service endpoints, or privately, using a private endpoint. Choose Next to continue.
4. For Tags, use the Name and Value fields to create tags that will help organize the identity of the key vault. Choose Review + create to validate the key vault setup.
5. For Review + create, review the resource configuration details, then choose Create to create your new Azure key vault.

07 Once the deployment is complete, choose Go to resource to access your new Microsoft Azure key vault.

08 In the resource navigation panel, under Objects, select Keys, then choose Generate/Import to create the Customer-Managed Key required for AKS cluster disks encryption.

09 On the Create a key setup page, provide a unique name for the encryption key in the Name box, set Key type to RSA, RSA key size to 2048, choose an activation and/or expiration date, set the Enabled flag to Yes, then choose Create to generate your new Customer-Managed Key (CMK).

10 Click on the name of the newly created Customer-Managed Key, choose the current version, and copy the key identifier listed in the Key Identifier box.

11 Navigate to Azure Databricks blade available at https://portal.azure.com/#browse/Microsoft.Databricks%2Fworkspaces.

12 Choose the Azure subscription that you want to access from the Subscription equals all filter box and choose Apply.

13 Click on the name (link) of the Azure Databricks workspace that you want to configure.

14 In the resource navigation panel, under Settings, select Encryption to access the encryption configuration settings available for the selected workspace.

15 In the Customer-managed keys section, perform the following actions to enable encryption at rest using Customer-Managed Keys (CMKs):

1. Check the Managed Disks setting checkbox.
2. Select the appropriate Azure subscription from the Subscription dropdown menu.
3. Paste the Customer-Managed Key (CMK) identifier (full URI) copied in step no. 10 in the Key Identifier box.
4. Choose Save to apply the changes.

16 Repeat steps no. 13 - 15 for each Azure Databricks workspace that you want to encrypt with a Customer-Managed Key (CMK), available in the selected subscription.

17 Repeat steps no. 2 – 16 for each Azure subscription created in your Microsoft Azure cloud account.

01 Run account list command (Windows/macOS/Linux) with custom output filters to list the IDs of the cloud subscriptions available in your Azure cloud account:

az account list
	--query '[*].id'

02 The command output should return the requested subscription identifiers (IDs):

[
	"abcdabcd-1234-abcd-1234-abcdabcdabcd",
	"abcd1234-abcd-1234-abcd-abcd1234abcd"
]

03 Run account set command (Windows/macOS/Linux) with the ID of the Azure cloud subscription that you want to examine as the identifier parameter to set the selected subscription to be the current active subscription (the command does not produce an output):

az account set
	--subscription abcdabcd-1234-abcd-1234-abcdabcdabcd

04 Run identity create command (OSX/Linux/UNIX) to create a new user-assigned managed identity, required to access the Azure key vault:

az identity create
	--name cc-project9-user-identity
	--resource-group cloud-shell-storage-westeurope
	--location westeurope
	--query '{id:id,principalId:principalId}'

05 The command output should return the resource ID and the principal ID of the new user-assigned managed identity:

{
	"id": "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourcegroups/cloud-shell-storage-westeurope/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cc-project9-user-identity",
	"principalId": "abcd1234-abcd-1234-abcd-abcd1234abcd"
}

06 Run keyvault create command (Windows/macOS/Linux) to create the Microsoft Azure key vault where the required Customer-Managed Key (CMK) will be placed. Both soft delete and purge protection must be enabled on the new key vault:

az keyvault create
	--name cc-project9-key-vault
	--resource-group cloud-shell-storage-westeurope
	--location westeurope
	--enable-rbac-authorization false
	--enabled-for-deployment true
	--enabled-for-template-deployment true
	--enable-purge-protection true
	--enabled-for-disk-encryption true
	--query 'properties.vaultUri'

07 The command output should return the URI for the new Azure key vault:

"https://cc-project9-key-vault.vault.azure.net/"

08 Run keyvault set-policy command (Windows/macOS/Linux) to assign the right permissions to your new Azure key vault. For the --object-id parameter, use "principalId" value returned in step no. 5:

az keyvault set-policy
	--name cc-project9-key-vault
	--object-id "abcd1234-abcd-1234-abcd-abcd1234abcd"
	--key-permissions create get recover wrapKey unwrapKey
	--query 'properties.accessPolicies'

09 The command output should return the modified key vault configuration information:

[
	{
		"applicationId": null,
		"objectId": "abcd1234-abcd-1234-abcd-abcd1234abcd",
		"permissions": {
			"certificates": null,
			"keys": [
				"recover",
				"create",
				"unwrapKey",
				"wrapKey",
				"get"
			],
			"secrets": null,
			"storage": null
		},
		"tenantId": "abcdabcd-1234-abcd-1234-abcdabcdabcd"
	}
]

10 Run keyvault key create command (Windows/macOS/Linux) to create the Customer-Managed Key (CMK) necessary to encrypt data for your Azure Databricks workspace:

az keyvault key create
	--name cc-project9-databricks-key
	--vault-name cc-project9-key-vault
	--kty RSA
	--size 2048
	--ops decrypt encrypt sign unwrapKey verify wrapKey
	--protection software
	--disabled false
	--query 'key.kid'

11 The command output should return the URL of the new Customer-Managed Key:

"https://cc-project9-key-vault.vault.azure.net/keys/cc-project9-databricks-key/abcd1234abcdabcd1234abcdabcd1234"

12 Run databricks workspace update command (OSX/Linux/UNIX) to enable encryption at rest using Customer-Managed Keys (CMKs) for your Azure Databricks workspace data:

az databricks workspace update
	--ids "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Databricks/workspaces/cc-project9-data-workspace"
	--disk-key-vault "https://cc-project9-key-vault.vault.azure.net/"
	--disk-key-name "cc-project9-databricks-key"
	--disk-key-version "abcd1234abcdabcd1234abcdabcd1234"
	--query 'encryption.entities.managedDisk'

13 The command output should return the encryption information available for the modified Databricks workspace:

{
	"keySource": "Microsoft.Keyvault",
	"keyVaultProperties": {
		"keyName": "cc-project9-databricks-key",
		"keyVaultUri": "https://cc-project9-key-vault.vault.azure.net/",
		"keyVersion": "abcd1234abcdabcd1234abcdabcd1234"
	},
	"rotationToLatestKeyVersionEnabled": false
}

14 Repeat steps no. 12 and 13 for each Azure Databricks workspace that you want to encrypt with a Customer-Managed Key (CMK), available in the selected subscription.

15 Repeat steps no. 3 – 14 for each Azure subscription created in your Microsoft Azure cloud account.