Notebook in VPC Only Mode Can Access Required Resources

01 Sign in to the AWS Management Console.

02 Navigate to Amazon SageMaker console available at https://console.aws.amazon.com/sagemaker/.

03 In the main navigation panel, under Applications and IDEs, select Notebooks.

04 Click on the name (link) of the notebook instance that you want to examine, available in the Name column.

05 In the Network section, copy the ID of the VPC subnet associated with the instance, listed under Subnet(s). If there is no VPC subnet information listed in the Network section, instead the following status is displayed: "No custom VPC settings applied.", the selected notebook instance is not running within a Virtual Private Cloud (VPC) and the Audit process ends here. Otherwise, you can continue the Audit process with the next step.

06 Navigate to VPC dashboard available at https://console.aws.amazon.com/vpc/ and perform the following actions to check for associated NAT gateways or valid interface VPC endpoints:

1. To check for NAT gateways:
   1. In the left navigation panel, under Virtual private cloud, select Subnets.
   2. Click inside the filter box, choose Subnet ID, paste the subnet ID copied at step no. 5, and press Enter.
   3. Select the VPC subnet returned by the console and choose the Route table tab.
   4. Check for any route with a NAT gateway (e.g nat-0abcd1234abcd1234) configured as Target. If there is no route with a NAT gateway configured as Target, the selected Amazon SageMaker notebook instance does not use a NAT gateway for Internet access. If there is a route with a NAT gateway configured as Target, continue the Audit process with the next step.
   5. Click on the name of the configured NAT gateway, select the resource, check the Connectivity type attribute value, and click on the ID (link) of the associated subnet, listed under Subnet. Select the subnet, choose the Route table tab and check for any route with an Internet Gateway (e.g igw-0abcd1234abcd1234) configured as Target. If there is no route with an Internet Gateway configured as target, and the gateway's Connectivity type is not Public, the selected Amazon SageMaker notebook instance does not use a public NAT gateway for Internet access.
2. To check for interface VPC endpoints. A valid interface VPC endpoint is a VPC resource with the Endpoint type set to Interface and Service name set to one of the following: com.amazonaws.\<region\>.sagemaker.api, com.amazonaws.\<region\>.sagemaker.runtime, com.amazonaws.\<region\>.s3, and com.amazonaws.\<region\>.servicecatalog:
   1. In the left navigation panel, under Virtual private cloud, select Endpoints.
   2. Click inside the filter box, choose Endpoint type, select = (Equals) from Operators, type Interface, and press Enter. If one or more results are returned, continue the Audit process with the next step.
   3. Click inside the filter box, choose Subnets, select = (Equals) from Operators, paste the subnet ID copied at step no. 5, and press Enter. If one or more results are returned, continue the Audit process with the next step.
   4. Click inside the filter box, choose Service name, select = (Equals) from Operators, type com.amazonaws.\<region\>.sagemaker.api (where \<region\> is the name of the AWS region), and press Enter. Repeat this step for the following services: com.amazonaws.\<region\>.sagemaker.runtime, com.amazonaws.\<region\>.s3, and com.amazonaws.\<region\>.servicecatalog. If no matching resources are found, there are no valid interface VPC endpoints, therefore, the selected Amazon SageMaker notebook instance does not use an interface VPC endpoint to access the required resources.

07 Repeat steps no. 4 - 6 for each Amazon SageMaker notebook instance available within the current AWS region.

08 Change the AWS cloud region from the navigation bar to repeat the Audit process for other regions.

01 Run list-notebook-instances command (OSX/Linux/UNIX) to list the name of each SageMaker notebook instance provisioned in the selected AWS region:

aws sagemaker list-notebook-instances
  --region us-east-1
  --query 'NotebookInstances[*].NotebookInstanceName'

02 The command output should return the requested SageMaker notebook instance names:

[
	"cc-sagemaker-notebook-instance",
	"cc-ml-application-instance"
]

03 Run describe-notebook-instance command (OSX/Linux/UNIX) with the name of the Amazon SageMaker notebook instance that you want to examine as the identifier parameter and custom output filters to describe the ID of the VPC subnet where the selected instance was deployed:

aws sagemaker describe-notebook-instance
  --region us-east-1
  --notebook-instance-name cc-sagemaker-notebook-instance
  --query 'SubnetId'

04 The command output should return the requested subnet ID or null if the instance was not created within a VPC subnet:

"subnet-01234abcd1234abcd"

05 To determine if your notebook instance can access required resources, perform the following commands to check for associated NAT gateways or valid interface VPC endpoints:

1. To check for NAT gateways:
   1. Run describe-route-tables command (OSX/Linux/UNIX) with the ID of the associated VPC subnet as the identifier parameter and custom output filters to describe the routing configuration available for the selected subnet:
      aws ec2 describe-route-tables
        --region us-east-1
        --filters "Name=association.subnet-id,Values=subnet-01234abcd1234abcd"
        --query 'RouteTables[].{Routes:Routes,RouteTableId:RouteTableId}'
      

   2. The command output should return the requested routing configuration (i.e. the routes configured for the specified VPC subnet and the ID of the route table):
      [
      	{
      		"Routes": [
      			{
      				"DestinationCidrBlock": "10.0.0.0/16",
      				"GatewayId": "local",
      				"Origin": "CreateRouteTable",
      				"State": "active"
      			}
      		],
      		"RouteTableId": "rtb-0abcdabcdabcdabcd"
      	}
      ]
      

      
      Check the describe-route-tables command output for any active route with a NAT gateway configured as target, i.e. "NatGatewayId": "\<nat-gateway-id\>". If there is no route with a NAT gateway configured as target, the selected Amazon SageMaker notebook instance does not use a NAT gateway for Internet access. If there is a route with a NAT gateway (e.g. "NatGatewayId": "nat-0abcd1234abcd1234") configured as target, continue the Audit process with the next step.
   3. Run describe-nat-gateways command (OSX/Linux/UNIX) to describe the connectivity type configured for the specified NAT gateway and the ID of the associated VPC subnet:
      aws ec2 describe-nat-gateways
        --nat-gateway-id nat-0abcd1234abcd1234
        --query 'NatGateways[].{ConnectivityType:ConnectivityType,SubnetId:SubnetId}'
      

   4. The command output should return the requested configuration information:
      [
      	{
      		"ConnectivityType": "private",
      		"SubnetId": "subnet-0abcd1234abcd1234"
      	}
      ]
      

   5. Run describe-route-tables command (OSX/Linux/UNIX) to describe the routing configuration of the VPC subnet associated with your NAT gateway:
      aws ec2 describe-route-tables
        --region us-east-1
        --filters "Name=association.subnet-id,Values=subnet-0abcd1234abcd1234"
        --query 'RouteTables[].{Routes:Routes,RouteTableId:RouteTableId}'
      

   6. The command output should return the requested routing configuration:
      [
      	{
      		"Routes": [
      			{
      				"DestinationCidrBlock": "10.0.0.0/16",
      				"GatewayId": "local",
      				"Origin": "CreateRouteTable",
      				"State": "active"
      			}
      		],
      		"RouteTableId": "rtb-01234123412341234"
      	}
      ]
      

      
      Check the describe-route-tables command output for any active route with an Internet Gateway configured as target, i.e. "GatewayId": "\<internet-gateway-id\>". If there is no route with an Internet Gateway (e.g "igw-0abcd1234abcd1234") configured as target, and at step 4, the gateway's "ConnectivityType" is not "public", the selected Amazon SageMaker notebook instance does not use a public NAT gateway for Internet access.
2. To check for interface VPC endpoints. A valid interface VPC endpoint is a VPC resource with the Endpoint type set to Interface and Service name set to one of the following: com.amazonaws.\<region\>.sagemaker.api, com.amazonaws.\<region\>.sagemaker.runtime, com.amazonaws.\<region\>.s3, and com.amazonaws.\<region\>.servicecatalog:
   1. Run describe-vpc-endpoints command (OSX/Linux/UNIX) with custom and predefined filters to describe the ID of the VPC subnet associated with the selected interface VPC endpoint. Replace \<region\> with the name of your AWS region:
      aws ec2 describe-vpc-endpoints
        --region us-east-1
        --filters "Name=vpc-endpoint-type,Values=Interface"
        --filters "Name=service-name,Values=com.amazonaws.<region>.sagemaker.api,com.amazonaws.<region>.sagemaker.runtime,com.amazonaws.<region>.s3,com.amazonaws.<region>.servicecatalog"
        --query 'VpcEndpoints[].{VpcEndpointId:VpcEndpointId,SubnetIds:SubnetIds}'
      

   2. The command output should return the requested configuration information. If the describe-vpc-endpoints command output returns an empty array (i.e. []), no matching VPC endpoints are found, therefore, the selected Amazon SageMaker notebook instance does not use an interface VPC endpoint to access the required resources. If the command returns an output, as shown in the example below, check the "SubnetIds" attribute value to determine if the VPC endpoint is associated with the notebook's subnet, returned at step no. 4. If the subnet IDs do not match, the selected Amazon SageMaker notebook instance does not use an interface VPC endpoint to access required resources.
      [
      	{
      		"VpcEndpointId": "vpce-0abcd1234abcd1234",
      		"SubnetIds": [
      			"subnet-0abcdabcdabcdabcd"
      		]
      	}
      ]
      

06 Repeat steps no. 3 - 5 for each SageMaker notebook instance available in the selected AWS region.

07 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 6 to perform the Audit process for other regions.

01 Sign in to the AWS Management Console.

02 Navigate to Amazon VPC console available at https://console.aws.amazon.com/vpc/.

03 To ensure your Amazon SageMaker notebook instance can access required resources, provision either a NAT gateway or an interface VPC endpoint by following these steps:

1. To create the public NAT gateway required for Internet access, perform the following actions:
   1. In the main navigation panel, under Virtual private cloud, select NAT gateways.
   2. Choose Create NAT gateway from the console top menu and provide the following information:
      1. For Name – optional, provide a unique name for the new NAT gateway.
      2. For Subnet, choose the public VPC subnet in which to create the NAT gateway (must match the VPC network configured for your Amazon SageMaker notebook instance).
      3. For Connectivity type, choose Public.
      4. For Elastic IP allocation ID, select an existing Elastic IP address (EIP) or create a new one by using the Allocate Elastic IP button.
      5. (Optional) For Tags, use the Add new tag button to create and apply tags to the new gateway. You can use the tags to organize, track, or control access for your AWS cloud resource.
      6. Choose Create NAT gateway to create your new, managed NAT gateway.
   3. Select the newly created gateway, choose the Details tab, and copy the resource identifier (ID), listed under NAT gateway ID.
   4. In the main navigation panel, under Virtual private cloud, select Subnets.
   5. Select the private VPC subnet configured for your notebook instance, choose the Details tab, and click on the ID (link) of the associated route table, listed under Route table.
   6. Update the route table of the private VPC subnet associated with your notebook instance, to route outbound traffic to the newly created NAT gateway. Select the route table, choose the Routes tab, and select Edit routes to add the necessary route.
   7. On the Edit routes configuration page, choose Add route to set up a new route for your NAT gateway. Type the CIDR that you want to use in the Destination box, choose NAT Gateway from the Target dropdown list, and paste the NAT gateway ID into the Target box. Choose Save changes to apply the configuration changes.
2. To create the interface VPC endpoint required to access services such as "com.amazonaws.\<region\>.sagemaker.api", "com.amazonaws.\<region\>.sagemaker.runtime", "com.amazonaws.\<region\>.s3", and "com.amazonaws.\<region\>.servicecatalog"), perform the following actions:
   1. In the main navigation panel, under Virtual private cloud, select Endpoints.
   2. Choose Create endpoint from the console top menu and provide the following information:
      1. For Name tag - optional, provide a unique name for the new VPC endpoint.
      2. For Service category, choose AWS services, and select one of the following services, depending on your specific requirements: com.amazonaws.\<region\>.sagemaker.api, com.amazonaws.\<region\>.sagemaker.runtime, com.amazonaws.\<region\>.s3, and com.amazonaws.\<region\>.servicecatalog.
      3. For VPC, select the VPC from which you'll access the AWS service.
      4. For Subnets, choose the VPC subnet(s) that will use the interface VPC endpoint (must match the subnet(s) configured for your Amazon SageMaker notebook instance).
      5. For Security groups, select the security group(s) to associate with the endpoint network interface(s) for the new VPC endpoint.
      6. For Policy, select Full access to allow all operations by all principals on all resources over the interface VPC endpoint, or select Custom to attach a VPC endpoint policy that controls the permissions that principals have for performing actions on resources over the new VPC endpoint.
      7. (Optional) For Tags, use the Add new tag button to create and apply tags to the new endpoint. You can use the tags to organize, track, or control access for your AWS cloud resource.
      8. Choose Create endpoint to create your interface VPC endpoint. The new VPC endpoint can be used by your Amazon SageMaker notebook instance to access required resources. No route table changes are required for interface VPC endpoints. Traffic directed to the VPC endpoint will use the DNS resolution provided by AWS.

04 Navigate to Amazon SageMaker console at https://console.aws.amazon.com/sagemaker/.

05 In the main navigation panel, under Applications and IDEs, select Notebooks.

06 Click on the name (link) of the notebook instance that you want to re-create (i.e. source instance) and note the instance configuration information such as instance type, platform identifier, IAM permissions, and encryption configuration.

07 Choose Create notebook instance and perform the following operations to create your new SageMaker notebook instance:

1. For Notebook instance name, provide a unique name for your new notebook instance.
2. For Notebook instance type, select the instance type for your notebook instance (must match the instance type of the source notebook instance).
3. For Platform identifier, select the appropriate software platform (must match the platform used by the source notebook instance).
4. Choose Additional configuration, select the notebook lifecycle configuration (optional), choose the minimum IMDS version, and specify the volume size of the notebook instance in GB (must match the volume size of the source notebook instance).
5. For IAM role, choose the IAM role used by the source, non-compliant notebook instance. If you want to create a new role, choose Create role using the role creation wizard and follow the setup process to create your new IAM role.
6. (Optional) For Root access - optional, choose Disable - Don't give users root access to the notebook to deny root access to your new SageMaker netbook instance.
7. For Encryption key - optional, select the name (alias) of the Amazon KMS Customer Managed Key (CMK) that you want to use to encrypt your notebook data. If no KMS key is selected, the notebook data is encrypted using an AWS-managed key (default).
8. Choose Network - optional and perform the following actions:
   1. Select the ID of the Virtual Private Cloud (VPC) where you want to deploy your new notebook instance.
   2. Once the VPC network is selected, choose the ID of your private VPC subnet from the Subnet dropdown list.
   3. Select one or more security groups from the Security group(s) list, based on your access policy requirements. Ensure that the security group rules allow outbound traffic. Allowing outbound connections is required for downloading packages, and train or host models.
   4. For Direct internet access, select Disable — Access the internet through a VPC to disable direct internet access to your notebook instance.
9. (Optional) Choose Git repositories - optional and select any required Git repositories. Repositories are added to your home directory.
10. (Optional) Choose Tags - optional and create any required tag sets, according to the source instance tagging scheme.
11. Choose Create notebook instance to launch your new Amazon SageMaker notebook instance.

08 (Optional) You can delete the source notebook instance to avoid further AWS charges. To remove the unneeded SageMaker notebook instance, perform the following actions:

1. Select the SageMaker notebook instance that you want to remove.
2. Choose Actions and select Stop to stop the instance.
3. Once the instance is stopped, choose again Actions and select Delete.
4. In the confirmation box, choose Delete to remove the notebook instance from your AWS cloud account.

09 Repeat steps no. 6 – 8 for each Amazon SageMaker notebook instance that you want to relaunch, available within the current AWS region.

10 Change the AWS cloud region from the navigation bar and repeat the Remediation process for other regions.

01 To provide your Amazon SageMaker notebook instance with access to necessary resources, provision either a NAT gateway or an interface VPC endpoint by following these steps:

1. To create the public NAT gateway required for Internet access, run the following commands:
   1. Run allocate-address command (OSX/Linux/UNIX) to create a new Elastic IP address (EIP) in the selected AWS region:
      aws ec2 allocate-address
        --region us-east-1
        --domain vpc
      

   2. The command output should return the information available for the new EIP:
      {
      	"PublicIp": "<public-ip>",
      	"Domain": "vpc",
      	"AllocationId": "eipalloc-01234abcd1234abcd"
      }
      

   3. Run create-nat-gateway command (OSX/Linux/UNIX) to create a new managed NAT gateway in the specified VPC subnet (must match the VPC network configured for your Amazon SageMaker notebook instance). For --allocation-id, use the ID of the Elastic IP (EIP) address created at the previous steps:
      aws ec2 create-nat-gateway
        --region us-east-1
        --connectivity-type public
        --subnet-id subnet-01234abcd1234abcd
        --allocation-id eipalloc-01234abcd1234abcd
      

   4. The command output should return the configuration information available for the new NAT gateway:
      {
      	"NatGateway": {
      		"CreateTime": "2024-07-22T16:23:27+00:00",
      		"NatGatewayAddresses": [
      			{
      				"AllocationId": "eipalloc-01234abcd1234abcd",
      				"IsPrimary": true,
      				"Status": "associating"
      			}
      		],
      		"NatGatewayId": "nat-0abcdabcd12341234",
      		"State": "pending",
      		"SubnetId": "subnet-01234abcd1234abcd",
      		"VpcId": "vpc-0abcd1234abcd1234",
      		"ConnectivityType": "public"
      	}
      }
      

   5. Update the route table of the private VPC subnet associated with your SageMaker notebook instance, to route outbound traffic to the newly created NAT gateway. Run create-route command (OSX/Linux/UNIX) with the ID of VPC subnet route table as the identifier parameter, to create a new route that sends traffic to your NAT gateway. As an example, the following command creates a route that matches all IPv4 traffic (i.e. 0.0.0.0/0):
      aws ec2 create-route
        --region us-east-1
        --route-table-id rtb-0abcdabcdabcdabcd
        --destination-cidr-block 0.0.0.0/0
        --gateway-id nat-0abcdabcd12341234
      

   6. The command output should return true if the request succeeds, otherwise, it should return an error:
      {
      	"Return": true
      }
      

2. To create the interface VPC endpoint required to access services such as com.amazonaws.\<region\>.sagemaker.api, com.amazonaws.\<region\>.sagemaker.runtime, com.amazonaws.\<region\>.s3, and com.amazonaws.\<region\>.servicecatalog, run the following commands:
   1. Run create-vpc-endpoint command (OSX/Linux/UNIX) to create a new interface VPC endpoint for the specified VPC subnet (must match the subnet configured for your Amazon SageMaker notebook instance). The following example uses com.amazonaws.\.s3 for the AWS service:
      aws ec2 create-vpc-endpoint
        --region us-east-1
        --vpc-id vpc-0abcd1234abcd1234
        --vpc-endpoint-type Interface
        --subnet-ids subnet-01234abcd1234abcd
        --security-group-id sg-01234abcd1234abcd
        --service-name com.amazonaws.us-east-1.s3
      

   2. The command output should return the configuration information available for your interface VPC endpoint. The new VPC endpoint can be used by your notebook instance to access required resources. No route table changes are required for interface VPC endpoints. Traffic directed to the VPC endpoint will use the DNS resolution provided by AWS:
      {
      	"VpcEndpoint": {
      		"VpcEndpointId": "vpce-01234123412341234",
      		"VpcEndpointType": "Interface",
      		"VpcId": "vpc-0abcd1234abcd1234",
      		"ServiceName": "com.amazonaws.us-east-1.s3",
      		"State": "pending",
      		"RouteTableIds": [],
      		"SubnetIds": [
      			"subnet-01234abcd1234abcd"
      		],
      		"Groups": [
      			{
      				"GroupId": "sg-01234abcd1234abcd",
      				"GroupName": "default"
      			}
      		],
      		"IpAddressType": "ipv4",
      		"DnsOptions": {
      			"DnsRecordIpType": "ipv4"
      		},
      		"NetworkInterfaceIds": [
      			"eni-01234abcd1234abcd"
      		],
      		"DnsEntries": [
      			{
      				"DnsName": "*.vpce-01234123412341234-abcd1234.s3.us-east-1.vpce.amazonaws.com",
      				"HostedZoneId": "ABCDABCDABCD"
      			},
      			{
      				"DnsName": "*.vpce-01234123412341234-abcd1234-us-east-1a.s3.us-east-1.vpce.amazonaws.com",
      				"HostedZoneId": "ABCDABCDABCD"
      			}
      		],
      		"CreationTimestamp": "2024-09-17T09:40:00.605000+00:00",
      		"OwnerId": "123456789012"
      	}
      }
      

02 Run describe-notebook-instance command (OSX/Linux/UNIX) with the name of the SageMaker notebook instance that you want to re-create as the identifier parameter, to describe the configuration information available for the selected notebook instance:

aws sagemaker describe-notebook-instance
  --region us-east-1
  --notebook-instance-name cc-sagemaker-notebook-instance

03 The command output should return the requested configuration details. This information is required when you launch your new SageMaker notebook instance:

{
	"NotebookInstanceArn": "arn:aws:sagemaker:us-east-1:123456789012:notebook-instance/cc-sagemaker-notebook-instance",
	"NotebookInstanceName": "cc-sagemaker-notebook-instance",
	"NotebookInstanceStatus": "InService",
	"Url": "cc-sagemaker-notebook-instance-paeo.notebook.us-east-1.sagemaker.aws",
	"InstanceType": "ml.t3.large",
	"RoleArn": "arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionPolicy-20240720T204001",
	"VolumeSizeInGB": 50,
	"RootAccess": "Disabled",
	"PlatformIdentifier": "notebook-al2-v2",
	"InstanceMetadataServiceConfiguration": {
		"MinimumInstanceMetadataServiceVersion": "2"
	}
}

04 Run create-notebook-instance command (OSX/Linux/UNIX) with the configuration information returned at the previous step to relaunch your Amazon SageMaker notebook instance using a different network configuration. To deploy your notebook instance to the Virtual Private Cloud (VPC) configured with a NAT gateway or with an interface VPC endpoint, provide the ID of the private VPC subnet that you want to use for your instance and the ID(s) of the security group(s) required for access control. Ensure that the security group rules allow outbound traffic. Allowing outbound connections is required for downloading packages, and train or host models:

aws sagemaker create-notebook-instance
  --region us-east-1
  --notebook-instance-name cc-new-sagemaker-notebook-instance
  --instance-type ml.t3.large
  --role-arn arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole-20240720T204001
  --kms-key-id arn:aws:kms:us-east-1:123456789012:key/1234abcd-1234-abcd-1234-abcd1234abcd
  --subnet-id subnet-01234abcd1234abcd
  --security-group-ids sg-aabbccdd012345678

05 The command output should return the ARN of the new Amazon SageMaker notebook instance:

{
	"NotebookInstanceArn": "arn:aws:sagemaker:us-east-1:123456789012:notebook-instance/cc-new-sagemaker-notebook-instance"
}

06 (Optional) You can delete the source notebook instance to avoid further charges. To remove the unnecessary SageMaker notebook instance, run delete-notebook-instance command (OSX/Linux/UNIX), with the name of the notebook instance that you want to delete as the identifier parameter (the command does not produce an output):

aws sagemaker delete-notebook-instance
  --region us-east-1
  --notebook-instance-name cc-sagemaker-notebook-instance

07 Repeat steps no. 2 – 6 for each Amazon SageMaker notebook instance that you want to re-create, available in the selected AWS region.

08 Change the AWS cloud region by updating the --region command parameter value and repeat the Remediation process for other regions.