Logo of Trend Micro

ENI Multiple IP Mode

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: Medium (should be achieved)
Rule ID: AlibabaCloud-ACK-009

Ensure that the ENI multiple IP mode is enabled for your ACK clusters in order to make your clusters more scalable and interoperable. Alibaba Cloud's Elastic Network Interface (ENI) allows the assignment of ranges of internal IP addresses as aliases to the network interfaces of a single virtual machine (VM). This feature proves beneficial when managing numerous services on a VM and necessitates assigning distinct IP addresses to each service, eliminating any quota restrictions. In Container Service for Kubernetes (ACK), the ENI multiple IP mode is implemented by the Terway network plugin.

Performance
efficiency
Operational
excellence

ENI multiple IP mode in ACK clusters allocates IP addresses from a Terway-recognized CIDR block, enhancing scalability and interaction with Alibaba Cloud products. Benefits include proactive Pod IP reservation, independent firewall controls for Pods, and direct access to hosted services via Alias IPs, bypassing NAT gateways.

Audit

To determine if the ENI multiple IP mode is enabled for your ACK clusters, perform the following operations:

Using Alibaba Cloud Console

01 Sign in to your Alibaba Cloud account.

02 Navigate to Container Service for Kubernetes (ACK) console at https://cs.console.aliyun.com.

03 In the left navigation panel, under Overview, choose Clusters.

04 Click on the name (link) of the ACK cluster that you want to examine, listed in the Cluster Name/ID column.

05 In the ACK resource navigation panel, under the cluster name, choose Cluster information.

06 Choose the Basic Information tab and check the Network Plug-in attribute value. If the Network Plug-in value is not Terway, the selected ACK cluster is not using the Terway network plugin to enable support for the ENI multiple IP mode.

07 Repeat steps no. 4 – 6 for each Container Service for Kubernetes (ACK) cluster available within your Alibaba Cloud account.

Using Alibaba Cloud CLI

01 Run GET /clusters command (OSX/Linux/UNIX) to describe the configuration details for each Container Service for Kubernetes (ACK) cluster provisioned in your Alibaba Cloud account: 

aliyun cs GET /clusters
  --header "Content-Type=application/json;"
  --body "{}"

02 The command output should return the configuration information available for each available ACK cluster: 

[
	{
		"cluster_id": "abcd1234abcd1234abcd1234abcd1234a",
		"cluster_spec": "ack.standard",
		"cluster_type": "ManagedKubernetes",
		"created": "2024-02-05T17:44:26+08:00",
		"current_version": "1.28.3-aliyun.1",
		"deletion_protection": false,
		"init_version": "1.28.3-aliyun.1",
		"parameters": {
		"ALIYUN::AccountId": "1234567890123456",
		"ALIYUN::NoValue": "None",
		"ALIYUN::Region": "eu-west-1",
		"ALIYUN::TenantId": "1234567890123456",
		"AdjustmentType": "TotalCapacity",
		"BetaVersion": "",
		"CloudMonitorFlags": "False",
		"CloudMonitorVersion": "1.3.7",
		"ClusterDns": "192.168.0.10",
		"ClusterId": "abcd1234abcd1234abcd1234abcd1234a",
		"ContainerCIDR": "",
		"CustomK8sWorkerRole": "",
		"DisableAddons": "True",
		"DisableAutoCreateK8sWorkerRole": "False",
		"DisableAutoCreateK8sWorkerRolePolicy": "True",
		"DockerVersion": "17.06.2-ce-3",
		"ESSDeletionProtection": "True",
		"Eip": "False",
		"EipAddress": "",
		"EtcdVersion": "v3.5.9",
		"ExecuteVersion": "922993032",
		"HealthCheckType": "NONE",
		"IPStack": "ipv4",
		"ImageId": "aliyun_3_9_x64_20G_alibase_20231219.vhd",
		"KubernetesVersion": "1.28.3-aliyun.1",
		"MasterSLBPrivateIP": "172.23.38.234",
		"NatGateway": "False",
		"NatGatewayId": "",
		"NatGatewayType": "Enhanced",
		"NatGatewayVswitchId": "",
		"Network": "Flannel",
		"NodeNameMode": "nodeip",
		"NumOfNodes": "0",
		"OSType": "Linux",
		"Password": "******",
		"ProtectedInstances": "",
		"ProxyMode": "ipvs",
		"RemoveInstanceIds": "",
		"SNatEntry": "False",
		"ServiceCIDR": "192.168.0.0/16",
		"UserData": "",
		"VpcCidrWithSecondaryCidrs": "[\"172.16.0.0/12\"]",
		"WorkerAutoRenew": "False",
		"WorkerAutoRenewPeriod": "1",
		"WorkerDataDisk": "False",
		"WorkerDataDisks": "[]",
		"WorkerDeletionProtection": "True",
		"WorkerDeploymentSetId": "",
		"WorkerHpcClusterId": "",
		"WorkerInstanceChargeType": "PostPaid",
		"WorkerInstanceTypes": "ecs.ic5.xlarge",
		"WorkerKeyPair": "",
		"WorkerLoginPassword": "******",
		"WorkerPeriod": "3",
		"WorkerPeriodUnit": "Month",
		"WorkerSnapshotPolicyId": "******",
		"WorkerSystemDiskCategory": "cloud_essd",
		"WorkerSystemDiskPerformanceLevel": "PL0",
		"WorkerSystemDiskSize": "20",
		"ZoneId": ""
		},
		"profile": "Default",
		"region_id": "eu-west-1",
		"size": 1,
		"state": "running",
		"updated": "2024-02-05T17:46:49+08:00",
		"zone_id": "eu-west-1a"
	},

	...

	{
		"cluster_id": "1234abcd1234abcd1234abcd1234abcd1",
		"cluster_spec": "ack.standard",
		"cluster_type": "ManagedKubernetes",
		"created": "2024-02-05T16:40:31+08:00",
		"current_version": "1.28.3-aliyun.1",
		"deletion_protection": false,
		"init_version": "1.28.3-aliyun.1",
		"parameters": {
		"ALIYUN::AccountId": "1234567890123456",
		"ALIYUN::NoValue": "None",
		"ALIYUN::Region": "eu-west-1",
		"ALIYUN::StackName": "k8s-for-cs-1234abcd1234abcd1234abcd1234abcd1",
		"ALIYUN::TenantId": "1234567890123456",
		"AdjustmentType": "TotalCapacity",
		"BetaVersion": "",
		"CloudMonitorFlags": "False",
		"CloudMonitorVersion": "1.3.7",
		"ClusterDns": "10.0.0.10",
		"ClusterId": "1234abcd1234abcd1234abcd1234abcd1",
		"ContainerCIDR": "10.65.0.0/16",
		"CustomK8sWorkerRole": "",
		"DisableAddons": "True",
		"DisableAutoCreateK8sWorkerRole": "False",
		"DisableAutoCreateK8sWorkerRolePolicy": "True",
		"DockerVersion": "17.06.2-ce-3",
		"ESSDeletionProtection": "True",
		"Eip": "False",
		"EipAddress": "",
		"EtcdVersion": "v3.5.9",
		"ExecuteVersion": "460378767",
		"HealthCheckType": "NONE",
		"IPStack": "ipv4",
		"KeyPair": "",
		"KubernetesVersion": "1.28.3-aliyun.1",
		"MasterSLBPrivateIP": "172.23.38.230",
		"NatGateway": "False",
		"NatGatewayId": "",
		"NatGatewayType": "Enhanced",
		"NatGatewayVswitchId": "",
		"Network": "Flannel",
		"NodeNameMode": "nodeip",
		"NumOfNodes": "0",
		"OSType": "Linux",
		"Password": "******",
		"PodVswitchIds": "[]",
		"ProtectedInstances": "",
		"ProxyMode": "ipvs",
		"RemoveInstanceIds": "",
		"SNatEntry": "False",
		"ServiceCIDR": "10.0.0.0/16",
		"SnatTableId": "",
		"UserData": "",
		"VpcCidrWithSecondaryCidrs": "[\"172.16.0.0/12\"]",
		"WorkerAutoRenew": "False",
		"WorkerAutoRenewPeriod": "1",
		"WorkerDataDisk": "False",
		"WorkerDataDisks": "[]",
		"WorkerDeletionProtection": "True",
		"WorkerDeploymentSetId": "",
		"WorkerHpcClusterId": "",
		"WorkerInstanceChargeType": "PostPaid",
		"WorkerInstanceTypes": "ecs.u1-c1m1.xlarge",
		"WorkerKeyPair": "",
		"WorkerLoginPassword": "******",
		"WorkerPeriod": "3",
		"WorkerPeriodUnit": "Month",
		"WorkerSnapshotPolicyId": "******",
		"WorkerSystemDiskCategory": "cloud_essd",
		"WorkerSystemDiskPerformanceLevel": "PL0",
		"WorkerSystemDiskSize": "120",
		"ZoneId": ""
		},
		"profile": "Default",
		"region_id": "eu-west-1",
		"size": 1,
		"state": "running",
		"subnet_cidr": "10.65.0.0/16",
		"updated": "2024-02-05T16:42:53+08:00",
		"zone_id": "eu-west-1a"
	}
]

Check the network plugin configured for each ACK cluster by examining the "Network" attribute listed within the cluster parameters (i.e. "parameters" object). If the "Network" attribute value is not "terway-eniip", the selected ACK cluster is not using the Terway network plugin to enable support for the ENI multiple IP mode.

Remediation / Resolution

To ensure that the ENI multiple IP mode is enabled for your ACK clusters, your clusters must use the Terway network plugin. To re-create your ACK clusters with Terway, perform the following operations:

Using Alibaba Cloud Console

01 Sign in to your Alibaba Cloud account.

02 Navigate to Container Service for Kubernetes (ACK) console at https://cs.console.aliyun.com.

03 In the left navigation panel, under Overview, choose Clusters.

04 Choose Create Kubernetes Cluster and follow the setup wizard to create a new ACK cluster.

05 On the Cluster Configurations setup page, choose Terway for Network Plug-in and select both IPVLAN and Support for NetworkPolicy checkboxes to install the Terway network plugin for the selected ACK cluster.

06 After all the required settings are configured, choose Create Cluster to deploy your new ACK cluster.

07 Repeat steps no. 4 - 6 for each ACK cluster that you want to re-create, available in your Alibaba Cloud account.

Using Alibaba Cloud CLI

01 Run POST /clusters command (OSX/Linux/UNIX) to create a new ACK cluster with support for the ENI multiple IP mode. Include the following parameters in the command request: "addons":[{"name":"terway-eniip"}] and "pod_vswitch_ids":["vsw-1234abcd1234abcd1234a"] to install the Terway network plugin on the new cluster. "pod_vswitch_ids" parameter is essential when Terway is installed on the cluster, as it ensures that each pod within the cluster is assigned a distinct IP address: 

aliyun cs POST /clusters
  --header "Content-Type=application/json;"
  --body "{\"name\":\"tm-terway-eniip-cluster\",\"region_id\":\"eu-west-1\",\"cluster_type\":\"Kubernetes\",\"vpcid\":\"vpc-abcd1234abcd1234abcda\",\"service_cidr\":\"192.168.0.0/16\",\"cluster_spec\":\"ack.standard\",\"kubernetes_version\":\"1.28.3-aliyun.1\",\"vswitch_ids\":[\"vsw-1234abcd1234abcd1234a\"],\"nodepools\":[{\"auto_scaling\":{\"enable\":false,\"max_instances\":2,\"min_instances\":1,\"type\":\"cpu\"},\"kubernetes_config\":{\"cms_enabled\":true,\"runtime\":\"containerd\",\"runtime_version\":\"1.6.28\"},\"nodepool_info\":{\"name\":\"tm-default-node\"},\"scaling_group\":{\"instance_charge_type\":\"PostPaid\",\"instance_types\":[\"ecs.n1.medium\"],\"vswitch_ids\":[\"vsw-1234abcd1234abcd1234a\"]}}],\"addons\":[{\"name\":\"terway-eniip\"}],\"pod_vswitch_ids\":[\"vsw-1234abcd1234abcd1234a\"]}"

02 If successful, the output should return the new ACK cluster ID: 

{
	"cluster_id": "1234abcd1234abcd1234abcd1234abcd1",
	"request_id": "ABCDABCD-1234-ABCD-1234-ABCD1234ABCD",
	"task_id": "T-abcdabcdabcdabcdabcdabcd"
}

03 Repeat steps no. 1 and 2 for each ACK cluster that you want to re-create, available within your Alibaba Cloud account.

References

Publication date Feb 22, 2024

Related AlibabaCloud-ACK rules