Private Kubernetes Clusters

Risk Level: High (not acceptable risk)

Ensure that your Azure Kubernetes Service (AKS) clusters are deployed as private clusters in order to enhance your Kubernetes workload's security and isolation. Private link-based AKS clusters are Kubernetes clusters where the API server is accessed through a private endpoint within your Virtual Network (VNet), rather than a public IP. This ensures that communication between the API server and the node pools occurs over a secure, private connection, without exposing the cluster to the Internet.

Security

Using private link-based AKS clusters in Azure cloud enhances security by ensuring that the Kubernetes API server is accessible only within your private network, preventing exposure to the public Internet. This reduces the attack surface, improves data privacy, and allows for stricter control over network traffic, aligning with security best practices for sensitive workloads.

Audit

To determine if your Azure Kubernetes Service (AKS) clusters are public or private, perform the following operations:

Using Azure Portal

01 Sign in to the Microsoft Azure Portal.

02 Navigate to All resources blade at https://portal.azure.com/#browse/all to access all your Microsoft Azure cloud resources.

03 Choose the Azure subscription that you want to access from the Subscription equalls all filter box and choose Apply.

04 From the Type equalls all filter box, choose Equals, select Kubernetes service, and choose Apply to list only the AKS clusters available in the selected Azure subscription.

05 Click on the name (link) of the AKS cluster that you want to examine.

06 In the resource navigation panel, choose Overview, and select the Properties tab to access the essential configuration properties available for the selected AKS cluster.

07 In the Networking section, check the Private cluster property value. If Private cluster is set to Not enabled, the selected Azure Kubernetes Service (AKS) cluster is not a private link-based AKS cluster.

08 Repeat steps no. 5 – 7 for each AKS cluster provisioned in the selected Azure subscription.

09 Repeat steps no. 3 – 8 for each subscription created in your Microsoft Azure cloud account.

Using Azure CLI

01 Run account list command (Windows/macOS/Linux) with custom output filters to list the IDs of the cloud subscriptions available in your Azure cloud account:

az account list
  --query '[*].id'

02 The command output should return the requested subscription identifiers (IDs):

[
	"abcdabcd-1234-abcd-1234-abcdabcdabcd",
	"abcd1234-abcd-1234-abcd-abcd1234abcd"
]

03 Run account set command (Windows/macOS/Linux) with the ID of the Azure cloud subscription that you want to examine as the identifier parameter to set the selected subscription to be the current active subscription (the command does not produce an output):

az account set
  --subscription abcdabcd-1234-abcd-1234-abcdabcdabcd

04 Run aks list command (Windows/macOS/Linux) with custom query filters to list the name and the associated resource group for each Azure Kubernetes Service (AKS) cluster available in the selected Azure subscription:

az aks list
  --output table
  --query '[*].{name:name, resourceGroup:resourceGroup}'

05 The command output should return the requested AKS cluster names:

Name                    ResourceGroup
----------------------  ------------------------------
cc-project5-aks-cluster  cloud-shell-storage-westeurope
cc-data-mining-cluster   cloud-shell-storage-westeurope

06 Run aks show command (Windows/macOS/Linux) with the name of the AKS cluster that you want to examine (and the associated resource group) as identifier parameters, to describe the Kubernetes API server access profile available for the selected AKS cluster:

az aks show
  --name cc-project5-aks-cluster
  --resource-group cloud-shell-storage-westeurope
  --query '{enablePrivateCluster:apiServerAccessProfile.enablePrivateCluster}'

07 The command output should return the requested configuration information:

{
	"enablePrivateCluster": false
}

If the command output returns null or false for the "enablePrivateCluster" configuration attribute, the selected Azure Kubernetes Service (AKS) cluster is not a private link-based AKS cluster.

08 Repeat steps no. 6 and 7 for each AKS cluster available within the selected Azure subscription.

09 Repeat steps no. 3 – 8 for each subscription created in your Microsoft Azure cloud account.

Remediation / Resolution

To ensure that your Azure Kubernetes Service (AKS) clusters are private, you have to re-create your clusters. To relaunch your AKS clusters with the required networking configuration, perform the following operations:

Using Azure Portal

01 Sign in to the Microsoft Azure Portal.

02 Navigate to All resources blade at https://portal.azure.com/#browse/all to access all your Microsoft Azure cloud resources.

03 Choose the Azure subscription that you want to access from the Subscription equalls all filter box and choose Apply.

04 From the Type equalls all filter box, choose Equals, select Kubernetes service, and choose Apply to list only the AKS clusters available in the selected Azure subscription.

05 Click on the name (link) of the AKS cluster that you want to re-create and copy the necessary configuration details such as the associated resource group, Azure region, Kubernetes version, node size and count, scaling and networking settings and so on. This configuration information will be required later when the new cluster will be deployed.

06 Navigate to Kubernetes Services blade available at https://portal.azure.com/#browse/Microsoft.ContainerService%2FmanagedClusters, choose Create, select Kubernetes cluster, and perform the following actions to deploy your private AKS cluster:

For Basics, provide a name for the new cluster, select the resource group where the resource will be created, choose the appropriate cluster preset configuration, select the Azure region, Availability Zone(s), and Kubernetes version, and configure the authentication and authorization settings with Microsoft Entra ID and Azure RBAC or Kubernetes RBAC. Choose Next to continue the setup process.
For Node pools, configure the primary and optional node pools required for the new cluster, and set the node pool OS disk encryption. Choose Next to continue the setup.
For Networking, perform the following actions:
1. For Private access, check the Enable private cluster checkbox to make the new AKS cluster private. A private AKS cluster uses an internal IP address to ensure that traffic between the API server and the node pools remains on a private network only.
2. For Container networking, choose the Azure Container Networking Interface (CNI) mode that you want to use, check the Bring your own Azure virtual network checkbox and provide your Virtual Network (VNet) details. For Network policy, choose whether to use network policies to allow you to define rules for ingress and egress traffic between the pods within your cluster.
3. Choose Next to continue the setup process.
For Integrations, connect your new AKS cluster with additional Azure cloud services such as Microsoft Defender for Cloud, Azure Container Registry, Istio, and Azure Policy. Choose Next to continue.
For Monitoring, you can enable and configure Container Insights, Prometheus, and Grafana for more comprehensive data on the overall performance and health of your AKS cluster. You can also configure any required alert rules. Choose Next to continue.
For Advanced, choose whether to use Azure Key Vault as a secrets store with your Kubernetes cluster via a CSI volume, provide the name of the necessary infrastructure resource group, and choose Next to continue the setup.
For Tags, create the necessary tags and choose Review + create to validate the configuration information provided during setup.
For Review + create, review the cluster configuration details, then choose Create to launch your new, private AKS cluster. The provisioning process should take a few minutes to complete.

07 Repeat steps no. 5 and 6 for each AKS cluster that you want to re-create, available in the selected subscription.

08 Repeat steps no. 3 – 7 for each subscription created in your Microsoft Azure cloud account.

Using Azure CLI

01 Run account list command (Windows/macOS/Linux) with custom output filters to list the IDs of the cloud subscriptions available in your Azure cloud account:

az account list
  --query '[*].id'

02 The command output should return the requested subscription identifiers (IDs):

[
	"abcdabcd-1234-abcd-1234-abcdabcdabcd",
	"abcd1234-abcd-1234-abcd-abcd1234abcd"
]

az account set
  --subscription abcdabcd-1234-abcd-1234-abcdabcdabcd

04 Run aks create command (OSX/Linux/UNIX) to launch a new Azure Kubernetes Service (AKS) cluster in the selected Azure subscription. Include the --enable-private-cluster command parameter to make the new AKS cluster private:

az aks create
  --name cc-project5-private-aks-cluster
  --resource-group cloud-shell-storage-westeurope
  --location "westeurope"
  --node-count 3
  --node-vm-size "Standard_B2s"
  --node-osdisk-size 50
  --kubernetes-version "1.29.8"
  --network-plugin azure
  --vnet-subnet-id "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-project5-vnet/subnets/subnet-001"
  --dns-service-ip 10.15.0.10
  --service-cidr 10.15.0.0/24
  --generate-ssh-keys
  --tier standard
  --load-balancer-sku standard
  --enable-private-cluster

05 The command output should return the configuration information available for the new AKS cluster:

{
	"aadProfile": null,
	"addonProfiles": {
		"azureKeyvaultSecretsProvider": {
			"config": null,
			"enabled": false,
			"identity": null
		},
		"azurepolicy": {
			"config": null,
			"enabled": false,
			"identity": null
		}
	},
	"apiServerAccessProfile": null,
	"autoUpgradeProfile": {
		"upgradeChannel": "none"
	},

	...

	"storageProfile": {
		"blobCsiDriver": null,
		"diskCsiDriver": {
			"enabled": true
		},
		"fileCsiDriver": {
			"enabled": true
		},
		"snapshotController": {
			"enabled": true
		}
	},
	"supportPlan": "KubernetesOfficial",
	"systemData": null,
	"type": "Microsoft.ContainerService/ManagedClusters",
	"windowsProfile": null,
	"workloadAutoScalerProfile": {
		"keda": null
	}
}

06 Repeat steps no. 4 and 5 for each AKS cluster that you want to re-create, available in the selected subscription.

07 Repeat steps no. 3 – 6 for each subscription created in your Microsoft Azure cloud account.

References

Azure Official Documentation
Create a private Azure Kubernetes Service (AKS) cluster
Access a private Azure Kubernetes Service (AKS) cluster using the command invoke or Run command feature

Publication date Oct 22, 2024

Audit

Using Azure Portal

Using Azure CLI

Remediation / Resolution

Using Azure Portal

Using Azure CLI

References

Related AKS rules