Prevent Assigning External IPs to Workbench Instances

Risk Level: Medium (should be achieved)

Rule ID: VertexAI-011

Ensure that external IP addresses are not assigned to your Google Cloud Vertex AI notebook instances, in order to help prevent data exfiltration, maintain network isolation, and meet stringent compliance requirements.

This rule resolution is part of the Conformity Security & Compliance tool for GCP.

Security

For certain AI workloads, your organization might have essential requirements that include security and network restrictions. Vertex AI notebook instances with an assigned external IP address can communicate with the public internet or resources in other VPC networks. To prioritize security, the notebook instances are designed for internal use within a VPC network. Assigning an external IP exposes them directly to the Internet, bypassing certain security controls. This increases the risk of unauthorized access and potential vulnerabilities.

Audit

To determine if your Vertex AI notebook instances are configured with external IP addresses, perform the following operations:

Using GCP Console

01 Sign in to the Google Cloud Management Console.

02 Select the GCP project that you want to examine from the console top navigation bar.

03 Navigate to Vertex AI console available at https://console.cloud.google.com/vertex-ai.

04 In the main navigation panel, under NOTEBOOKS, choose Workbench, and select the INSTANCES tab.

05 Choose View: INSTANCES to list the Vertex AI notebook instances created for the selected GCP project.

06 Click on the name (link) of the notebook instance that you want to examine.

07 Select the SOFTWARE AND SECURITY tab and check the External IP feature status. If the External IP status is set to Enabled, an external IP address is assigned to the selected Vertex AI notebook instance.

08 Repeat steps no. 6 and 7 for each Vertex AI notebook instance launched for the selected GCP project.

09 Repeat steps no. 2 – 8 for each project deployed within your Google Cloud account.

Using GCP CLI

01 Run projects list command (Windows/macOS/Linux) with custom query filters to list the ID of each project available in your Google Cloud account:

gcloud projects list
  --format="table(projectId)"

02 The command output should return the requested GCP project ID(s):

PROJECT_ID
cc-vertex-project-123123
cc-appdata-project-112233

03 Run workbench instances list command (Windows/macOS/Linux) with the ID of the GCP project that you want to examine as the identifier parameter, to describe the name of each Vertex AI notebook instance created for the selected project:

gcloud workbench instances list
  --project cc-vertex-project-123123
  --location=us-central1-a
  --format="(NAME)"

04 The command output should return the requested notebook instance names:

NAME: tm-vertex-ai-notebook-instance
NAME: tm-development-notebook-instance

05 Run workbench instances describe command (Windows/macOS/Linux) with the name of the Vertex AI notebook instance that you want to examine as the identifier parameter and custom output filters to determine if the selected instance is configured with an external (public) IP address:

gcloud workbench instances describe tm-vertex-ai-notebook-instance
  --location=us-central1-a
  --format="json(gceSetup.disablePublicIp)"

06 The command output should return null if the instance is configured with an external IP or "disablePublicIp": true if not:

null

If the workbench instances describe command output returns null, as shown in the output example above, an external IP address is assigned to the selected Vertex AI notebook instance.

07 Repeat steps no. 5 and 6 for each Vertex AI notebook instance provisioned for the selected GCP project.

08 Repeat steps no. 3 – 7 for each GCP project deployed in your Google Cloud account.

Remediation / Resolution

To ensure that external IP addresses are not assigned to your Vertex AI notebook instances, you must re-create the instances with the appropriate IP configuration, by performing the following operations:

Using GCP Console

01 Sign in to the Google Cloud Management Console.

02 Select the GCP project that you want to access from the console top navigation bar.

03 Navigate to Vertex AI console available at https://console.cloud.google.com/vertex-ai.

04 In the main navigation panel, under NOTEBOOKS, choose Workbench, and select the INSTANCES tab.

05 Choose CREATE NEW, select ADVANCED OPTIONS, and perform the following actions to create your new notebook instance:

For Details, provide the following information:
1. For Name, enter a unique name for your new notebook instance.
2. For Region and Zone, select the GCP location where the instance will be deployed.
3. (Optional) Check the Enable Dataproc Serverless Interactive Sessions setting checkbox to enable access to Dataproc Spark kernels.
4. (Optional) For Labels, choose ADD LABEL, and use the Key and Value fields to create labels for the new instance.
5. (Optional) Use Network tags to assign network tags to your Workbench instance.
6. For Workbench type, choose Instance.
7. Choose Continue to continue the instance setup.
For Environment, perform the following actions:
1. Choose whether to use a custom container or the latest version of the Vertex AI Workbench for the instance environment.
2. (Optional) For Post-startup script, you can select a script that automatically runs after the instance boots up.
3. (Optional) For Metadata, choose ADD METADATA to add metadata keys to your Workbench instance.
4. Choose Continue to continue the setup.
For Machine type, perform the following operations:
1. For Machine type, choose the appropriate machine type for your workload.
2. For Shielded VM, check the Secure Boot, Virtual Trusted Platform Module (vTPM), and Integrity monitoring checkboxes for the most secure instance configuration.
3. For Idle shutdown, check the Enable Idle Shutdown checkbox to enable the Idle Shutdown feature for the new instance. Enter the preferred idle timeout value (in minutes) in the Time of inactivity before shutdown (Minutes) box.
4. Choose Continue to continue the setup process.
For Disks, perform the following operations:
1. For Disks, choose the boot disk type and boot disk size (GB) for the instance disks. (Optional) Check the Delete to trash checkbox if you want to use the operating system's trash behavior.
2. For Encryption, choose Cloud KMS key, and select the Cloud KMS Customer-Managed Encryption Key (CMEK) that you want to use for data encryption (recommended).
3. Choose Continue to continue the setup.
For Networking, choose Network in this project, and select the appropriate VPC network and subnetwork. Ensure that a custom, non-default VPC network is selected (recommended). For network isolation and stringent compliance, uncheck the Assign external IP address checkbox to prevent adding an external IP address to the new instance. Choose Continue to continue the setup process.
For IAM and security, perform the following actions:
1. For IAM and security, configure who can use the instance's JupyterLab interface. Choose Service account for default instance access or choose Single user to restrict access to one user only. Choose whether to use the default Compute Engine service account or a custom service account.
2. For Security options, uncheck the Root access to the instance checkbox to disable the root access to the new instance, and choose whether to allow terminal access and file downloads from JupyterLab.
3. Choose Continue to continue the setup.
For System health, perform the following operations:
1. For System health, check the Environment auto-upgrade checkbox to enable automatic upgrades. Choose whether to upgrade your new instance Weekly or Monthly.
2. For Reporting, check the Install Cloud Monitoring checkbox to install the Cloud Monitoring agent and enable the Cloud Monitoring feature. You can also check the Report custom metrics to Cloud Monitoring checkbox to collect system status and JupyterLab metrics. Ensure that Report system health and Report DNS status for required Google domains checkboxes are also checked for core service and DNS status verification.
3. Choose CREATE to launch your new Google Cloud Vertex AI notebook instance.

06 (Optional) If required, you can also enforce a constraint policy at the GCP organization level and define the virtual machine (VM) instances that are allowed to use external IP addresses or just configure an empty list to prevent any VM instances from obtaining external IPs. To enforce the "Define Allowed External IPs for VM Instances" policy, follow the steps outlined on this page.

07 Repeat step no. 5 for each Vertex AI notebook instance that you want to re-create, launched for the selected GCP project.

08 Repeat steps no. 2 – 7 for each project deployed within your Google Cloud account.

Using GCP CLI

01 Run workbench instances create command (Windows/macOS/Linux) to deploy your new Google Cloud Vertex AI notebook instance without an external (public) IP address by adding the --disable-public-ip configuration parameter to the command request:

gcloud workbench instances create tm-vertex-ai-notebook-instance
  --project=cc-vertex-project-123123
  --container-repository=gcr.io/deeplearning-platform-release/base-cpu
  --container-tag=latest
  --machine-type=e2-standard-2
  --location=us-central1-a
  --shielded-integrity-monitoring=true
  --shielded-secure-boot=true
  --shielded-vtpm=true
  --disable-public-ip
  --format="json(gceSetup.disablePublicIp)"

02 The command output should return the return the disablePublicIp flag for the new notebook instance:

Waiting for operation on Instance [tm-vertex-ai-notebook-instance] to be updated with [projects/cc-vertex-project-123123/locations/us-central1-a/operations/operation-abcd1234abcd-abcd1234abcd-abcd1234-abcd1234]...done.
Created workbench instance tm-vertex-ai-notebook-instance [https://notebooks.googleapis.com/v2/projects/cc-vertex-project-123123/locations/us-central1-a/operations/operation-abcd1234abcd-abcd1234abcd-abcd1234-abcd1234].

{
	"gceSetup": {
		"disablePublicIp": true
	}
}

03 (Optional) If required, you can also enforce a constraint policy at the GCP organization level and define the virtual machine (VM) instances that are allowed to use external IP addresses or just configure an empty list to prevent any VM instances from obtaining external IPs. To enforce the "Define Allowed External IPs for VM Instances" policy, follow the steps outlined on this page.

04 Repeat steps no. 1 and 2 for each Vertex AI notebook instance that you want to re-create, provisioned for the selected GCP project.

05 Repeat steps no. 1 – 4 for each GCP project deployed in your Google Cloud account.

References

Google Cloud Platform (GCP) Documentation
Introduction to Vertex AI Workbench
Vertex AI Workbench Jupyter Notebook tutorials
Create a Vertex AI Workbench instance
Organization policy constraints
Configure static external IP addresses

GCP Command Line Interface (CLI) Documentation
gcloud projects list
gcloud workbench instances list
gcloud workbench instances describe
gcloud workbench instances create

Publication date Jul 8, 2024

Audit

Using GCP Console

Using GCP CLI

Remediation / Resolution

Using GCP Console

Using GCP CLI

References

Related VertexAI rules