Logo of Trend Micro
Use the Conformity Knowledge Base AI to help improve your Cloud Posture

Use Azure CNI Add-On for Managing Network Resources

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: Medium (should be achieved)

Ensure that Azure Kubernetes Service (AKS) clusters are configured to use the Azure Container Networking Interface (CNI) add-on for dynamic allocation of IP addresses and enhanced subnet support within AKS.

Security
Performance
efficiency
Operational
excellence

The traditional Azure Container Networking Interface (CNI) faces pod IP exhaustion as AKS clusters grow, requiring rebuilding in larger subnets. Azure's new dynamic IP allocation solves this by allocating pod IPs from a separate subnet, offering better IP utilization, scalability, high performance, and separate virtual network policies for pods.

Audit

To determine if Azure CNI add-on is used for managing network resources, perform the following operations:

Using Azure Portal

01 Sign in to the Azure Management Console.

02 Navigate to All resources blade at https://portal.azure.com/#view/HubsExtension/BrowseAll to access all your Microsoft Azure cloud resources.

03 Choose the Azure subscription that you want to access from the Subscription filter box and choose Apply.

04 From the Type filter box, select Kubernetes service and choose Apply to list the Azure Kubernetes Service (AKS) clusters available in the selected subscription.

05 Click on the name (link) of the AKS cluster that you want to examine.

06 In the resource navigation panel, under Settings, select Networking to access the networking settings configured for the selected AKS cluster.

07 Under Network profile, check the Type (plugin) configuration attribute value. If the attribute value is set to Kubenet, the selected Azure Kubernetes Service (AKS) cluster is not configured to use the Azure Container Networking Interface (CNI) add-on for managing network resources.

08 Repeat steps no. 5 – 7 for each AKS cluster provisioned in the selected Azure subscription.

09 Repeat steps no. 3 – 8 for each subscription created in your Microsoft Azure cloud account.

Using Azure CLI

01 Run aks list command (Windows/macOS/Linux) using custom query filters to list the name and the associated resource group for each Azure Kubernetes Service (AKS) cluster available in the current subscription: 

az aks list
  --output table
  --query '[*].{name:name, resourceGroup:resourceGroup}'

02 The command output should return the requested AKS cluster names: 

Name                     ResourceGroup
----------------------   ------------------------------
cc-project5-aks-cluster  cloud-shell-storage-westeurope
cc-data-mining-cluster   cloud-shell-storage-westeurope

03 Run aks show command (Windows/macOS/Linux) using the name of the AKS cluster that you want to examine and its associated resource group as the identifier parameters to determine the networking mode configured for the selected AKS cluster: 

az aks show
  --name cc-project5-aks-cluster
  --resource-group cloud-shell-storage-westeurope
  --query 'networkProfile.networkPlugin'

04 The command output should return the name of the networking plugin used: 

"kubenet"

If the aks show command output returns "kubenet", as shown in the example above, the selected Azure Kubernetes Service (AKS) cluster is not configured to use the Azure Container Networking Interface (CNI) add-on for managing network resources.

05 Repeat step no. 3 and 4 for each AKS cluster available within the current Azure subscription.

06 Repeat steps no. 1 – 5 for each subscription created in your Microsoft Azure cloud account.

Remediation / Resolution

To configure your AKS clusters to use Azure CNI add-on for dynamic allocation of IPs and enhanced subnet support, you have to re-create your clusters. To provision the required Azure networking resources and relaunch your AKS clusters with the new networking configuration, perform the following operations:

Configuring AKS clusters to use Azure CNI add-on for dynamic allocation of IPs and enhanced subnet support using Azure Console (Azure Portal) is not currently supported.

Using Azure CLI

01 Run network vnet create command (Windows/macOS/Linux) to create a new Azure Virtual Network (VNet) for your AKS cluster: 

az network vnet create 
  --resource-group cloud-shell-storage-westeurope 
  --location westeurope 
  --name cc-containers-vnet 
  --address-prefixes 10.0.0.0/8

02 The command output should return the configuration information available for the new Azure Virtual Network (VNet): 

{
	"newVNet": {
		"addressSpace": {
			"addressPrefixes": [
			"10.0.0.0/8"
			]
		},
		"enableDdosProtection": false,
		"id": "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-containers-vnet",
		"location": "westeurope",
		"name": "cc-containers-vnet",
		"provisioningState": "Succeeded",
		"resourceGroup": "cloud-shell-storage-westeurope",
		"resourceGuid": "abcdabcd-1234-abcd-1234-abcdabcdabcd",
		"subnets": [],
		"type": "Microsoft.Network/virtualNetworks",
		"virtualNetworkPeerings": []
	}
}

03 Run network vnet subnet create command (Windows/macOS/Linux) to create the VNet subnet required for the AKS cluster nodes: 

az network vnet subnet create 
  --resource-group cloud-shell-storage-westeurope 
  --vnet-name cc-containers-vnet 
  --name cc-node-subnet 
  --address-prefixes 10.15.0.0/16

04 The command output should return the configuration information available for the new VNet subnet: 

{
	"addressPrefix": "10.15.0.0/16",
	"delegations": [],
	"id": "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-containers-vnet/subnets/cc-node-subnet",
	"name": "cc-node-subnet",
	"privateEndpointNetworkPolicies": "Disabled",
	"privateLinkServiceNetworkPolicies": "Enabled",
	"provisioningState": "Succeeded",
	"resourceGroup": "cloud-shell-storage-westeurope",
	"type": "Microsoft.Network/virtualNetworks/subnets"
}

05 Run network vnet subnet create command (Windows/macOS/Linux) to create the VNet subnet required for the AKS cluster pods: 

az network vnet subnet create 
  --resource-group cloud-shell-storage-westeurope 
  --vnet-name cc-containers-vnet
  --name cc-pod-subnet
  --address-prefixes 10.16.0.0/16

06 The command output should return the configuration information available for the new VNet subnet: 

{
	"addressPrefix": "10.16.0.0/16",
	"delegations": [],
	"id": "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-containers-vnet/subnets/cc-pod-subnet",
	"name": "cc-pod-subnet",
	"privateEndpointNetworkPolicies": "Disabled",
	"privateLinkServiceNetworkPolicies": "Enabled",
	"provisioningState": "Succeeded",
	"resourceGroup": "cloud-shell-storage-westeurope",
	"type": "Microsoft.Network/virtualNetworks/subnets"
}

07 Run aks show command (OSX/Linux/UNIX) using the name of the cluster that you want to re-create as the identifier parameter and custom query filters to get the specified AKS cluster configuration details. The information requested will be required later when the new Azure Kubernetes Service cluster will be launched: 

az aks show
  --name cc-project5-aks-cluster
  --resource-group cloud-shell-storage-westeurope

08 The command output should return the requested function configuration information: 

{
	"aadProfile": null,
	"addonProfiles": {
		"azureKeyvaultSecretsProvider": {
			"config": null,
			"enabled": false,
			"identity": null
		},
		"azurepolicy": {
			"config": null,
			"enabled": false,
			"identity": null
		}
	},
	"apiServerAccessProfile": null,
	"autoUpgradeProfile": {
		"upgradeChannel": "none"
	},
	"azureMonitorProfile": null,
	"azurePortalFqdn": "cc-project5-aks-cluster-dns-abcdabcd.portal.hcp.westeurope.azmk8s.io",
	"currentKubernetesVersion": "1.25.6",
	"disableLocalAccounts": false,
	"diskEncryptionSetId": null,
	"dnsPrefix": "cc-project5-aks-cluster-dns",
	"enablePodSecurityPolicy": null,
	"enableRbac": true,
	"extendedLocation": null,
	"fqdn": "cc-project5-aks-cluster-dns-abcdabcd.hcp.westeurope.azmk8s.io",
	"fqdnSubdomain": null,
	"httpProxyConfig": null,
	"kubernetesVersion": "1.25.6",
	"linuxProfile": null,
	"location": "westeurope",
	"maxAgentPools": 100,
	"name": "cc-project5-aks-cluster",
	"networkProfile": {
		"dnsServiceIp": "10.0.0.10",
		"ipFamilies": [
			"IPv4"
		],
		"loadBalancerProfile": {
			"enableMultipleStandardLoadBalancers": null,
			"idleTimeoutInMinutes": null,
			"managedOutboundIPs": {
			"count": 1,
			"countIpv6": null
			},
			"outboundIPs": null,
			"outboundIpPrefixes": null
		},
		"loadBalancerSku": "Standard",
		"natGatewayProfile": null,
		"networkDataplane": null,
		"networkMode": null,
		"networkPlugin": "kubenet",
		"networkPluginMode": null,
		"networkPolicy": null,
		"outboundType": "loadBalancer",
		"podCidr": "10.15.0.0/16",
		"podCidrs": [
			"10.15.0.0/16"
		],
		"serviceCidr": "10.0.0.0/16",
		"serviceCidrs": [
			"10.0.0.0/16"
		]
	},
	"nodeResourceGroup": "MC_cloud-shell-storage-westeurope_cc-project5-aks-cluster_westeurope",
	"oidcIssuerProfile": {
		"enabled": false,
		"issuerUrl": null
	},
	"podIdentityProfile": null,
	"powerState": {
		"code": "Running"
	},
	"privateFqdn": null,
	"privateLinkResources": null,
	"provisioningState": "Succeeded",
	"publicNetworkAccess": null,
	"resourceGroup": "cloud-shell-storage-westeurope",
	"securityProfile": {
		"azureKeyVaultKms": null,
		"defender": null,
		"imageCleaner": null,
		"workloadIdentity": null
	},
	"servicePrincipalProfile": {
		"clientId": "msi"
	},
	"sku": {
		"name": "Base",
		"tier": "Free"
	},
	"storageProfile": {
		"blobCsiDriver": null,
		"diskCsiDriver": {
			"enabled": true
		},
		"fileCsiDriver": {
			"enabled": true
		},
		"snapshotController": {
			"enabled": true
		}
	},
	"supportPlan": "KubernetesOfficial",
	"systemData": null,
	"type": "Microsoft.ContainerService/ManagedClusters",
	"windowsProfile": null,
	"workloadAutoScalerProfile": {
		"keda": null
	}
}

09 Run aks create command (OSX/Linux/UNIX) to launch a new Azure Kubernetes Service (AKS) cluster with Azure Container Networking Interface (CNI) add-on, using the cluster configuration information returned at step no. 8. For --vnet-subnet-id parameter use the ID of the node subnet. For --pod-subnet-id parameter use the ID of the pod subnet (the command does not produce an output): 

az aks create
  --name cc-new-project5-aks-cluster
  --resource-group cloud-shell-storage-westeurope
  --location "westeurope"
  --node-count 2
  --max-pods 250
  --node-vm-size "Standard_B2s"
  --node-osdisk-size 150
  --kubernetes-version "1.25.6"
  --network-plugin azure 
  --vnet-subnet-id /subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-containers-vnet/subnets/cc-node-subnet
  --pod-subnet-id /subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-containers-vnet/subnets/cc-pod-subnet
  --generate-ssh-keys

10 Run network vnet subnet create command (Windows/macOS/Linux) to create the node subnet required for the new AKS cluster node pool: 

az network vnet subnet create 
  --resource-group cloud-shell-storage-westeurope 
  --vnet-name cc-containers-vnet 
  --name cc-node-pool-subnet 
  --address-prefixes 10.17.0.0/16

11 The command output should return the configuration information available for the new subnet: 

{
	"addressPrefix": "10.17.0.0/16",
	"delegations": [],
	"id": "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-containers-vnet/subnets/cc-node-pool-subnet",
	"name": "cc-node-pool-subnet",
	"privateEndpointNetworkPolicies": "Disabled",
	"privateLinkServiceNetworkPolicies": "Enabled",
	"provisioningState": "Succeeded",
	"resourceGroup": "cloud-shell-storage-westeurope",
	"type": "Microsoft.Network/virtualNetworks/subnets"
}

12 Run network vnet subnet create command (Windows/macOS/Linux) to create the pod subnet required for the new AKS cluster node pool: 

az network vnet subnet create 
  --resource-group cloud-shell-storage-westeurope 
  --vnet-name cc-containers-vnet 
  --name cc-node-pool-pod-subnet 
  --address-prefixes 10.18.0.0/16

13 The command output should return the configuration information available for the new subnet: 

{
	"addressPrefix": "10.18.0.0/16",
	"delegations": [],
	"id": "/subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-containers-vnet/subnets/cc-node-pool-pod-subnet",
	"name": "cc-node-pool-pod-subnet",
	"privateEndpointNetworkPolicies": "Disabled",
	"privateLinkServiceNetworkPolicies": "Enabled",
	"provisioningState": "Succeeded",
	"resourceGroup": "cloud-shell-storage-westeurope",
	"type": "Microsoft.Network/virtualNetworks/subnets"
}

14 Run aks nodepool add command (Windows/macOS/Linux) to create a new node pool for your AKS cluster using the node and pod subnets provisioned at the previous step (the command does not produce an output): 

az aks nodepool add 
  --name newnodepool 
  --cluster-name cc-new-project5-aks-cluster 
  --resource-group cloud-shell-storage-westeurope 
  --max-pods 250 
  --node-count 2 
  --vnet-subnet-id /subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-containers-vnet/subnets/cc-node-pool-subnet 
  --pod-subnet-id /subscriptions/abcdabcd-1234-abcd-1234-abcdabcdabcd/resourceGroups/cloud-shell-storage-westeurope/providers/Microsoft.Network/virtualNetworks/cc-containers-vnet/subnets/cc-node-pool-pod-subnet
  --no-wait

15 Repeat steps no. 9 – 14 for each Azure Kubernetes Service cluster that you want to re-create, available within the current subscription.

16 Repeat steps no. 1 – 15 for each subscription created in your Microsoft Azure cloud account.

References

Publication date Aug 9, 2023

Related AKS rules