- Knowledge Base
- Amazon Web Services
- Amazon SageMaker
- Check for Missing Execution Role
Ensure that your Amazon SageMaker notebook instances are associated with active (available) execution roles in order to have permissions to access required AWS cloud services and resources.
An instance's execution role is an AWS Identity and Access Management (IAM) role that provides the notebook instance with the necessary permissions to access specific AWS services and resources. For example, notebook instances need permissions to call other services such as Amazon S3. If your Amazon SageMaker notebook instances are no longer associated with active execution roles, they will lose the ability to perform essential operations.
Audit
To determine if your Amazon SageMaker notebook instances are referencing active execution roles, perform the following operations:
Using AWS Console
01 Sign in to the AWS Management Console.
02 Navigate to Amazon SageMaker console available at https://console.aws.amazon.com/sagemaker/.
03 In the main navigation panel, under Notebook, select Notebook instances.
04 Click on the name (link) of the notebook instance that you want to examine, available in the Name column.
05 In the Permissions and encryption section, click on the ARN (link) of the execution role associated with the notebook instance, available under IAM role ARN, to open the role page in Amazon IAM. If the role page is not available anymore, instead the following error message is displayed: One of the entities that you specified for this operation does not exist. The role with name \
06 Repeat steps no. 4 and 5 for each Amazon SageMaker notebook instance available within the current AWS region.
07 Change the AWS cloud region from the navigation bar to repeat the Audit process for other regions.
Using AWS CLI
01 Run list-notebook-instances command (OSX/Linux/UNIX) to list the name of each SageMaker notebook instance provisioned in the selected AWS region:
aws sagemaker list-notebook-instances --region us-east-1 --query 'NotebookInstances[*].NotebookInstanceName'
02 The command output should return the requested SageMaker notebook instance names:
[ "cc-sagemaker-ml-instance", "cc-ml-application-instance" ]
03 Run describe-notebook-instance command (OSX/Linux/UNIX) with the name of the Amazon SageMaker notebook instance that you want to examine as the identifier parameter and custom output filters to describe the Amazon Resource Name (ARN) of the execution role associated with the selected instance:
aws sagemaker describe-notebook-instance --region us-east-1 --notebook-instance-name cc-sagemaker-ml-instance --query 'RoleArn'
04 The command output should return the requested information:
"arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole-20240605T156785"
05 Run get-role command (OSX/Linux/UNIX) to obtain information about the IAM role associated with your Amazon SageMaker notebook instance:
aws iam get-role --role-name AmazonSageMaker-ExecutionRole-20240605T156785
06 The command output should return the requested resource configuration information:
An error occurred (NoSuchEntity) when calling the GetRole operation: The role with name AmazonSageMaker-ExecutionRole-20240605T156785 cannot be found.
If the get-role command output returns a NoSuchEntity error message instead of the role's information, as shown in the output example above, the execution role associated with the selected Amazon SageMaker notebook instance is no longer available, therefore, the instance's capability to access other AWS services and resources is disabled.
07 Repeat steps no. 3 - 6 for each SageMaker notebook instance available in the selected AWS region.
08 Change the AWS cloud region by updating the --region command parameter value and repeat steps no. 1 – 7 to perform the Audit process for other regions.
Remediation / Resolution
To configure any Amazon SageMaker notebook instances associated with missing AWS IAM roles, perform the following operations:
Using AWS Console
01 Sign in to the AWS Management Console.
02 Navigate to Amazon SageMaker console available at https://console.aws.amazon.com/sagemaker/.
03 In the main navigation panel, under Notebook, select Notebook instances.
04 Select the SageMaker notebook instance that you want to configure, choose Actions, and select Stop to stop the instance.
05 Once the instance is stopped, choose again Actions, and select Update settings.
06 On the Edit notebook instance page, in the Permissions and encryption section, under IAM role, choose Create role using the role creation wizard, and follow the setup wizard to create the required execution role. Provide role information such as role name, role description, network and encryption conditions, select the necessary ML activities and enable customizations of the activity settings, and choose any additional IAM policies that you want to add to this role. Once all the required settings are configured, choose Submit to create the execution role.
07 Navigate back to the Edit notebook instance page, refresh the page, and choose the name of the new execution role from the IAM role dropdown list, available under Use existing role.
08 Choose Update notebook instance to apply the configuration changes.
09 Once the instance status becomes Stopped, choose Actions, and select Start to start the notebook instance.
10 Repeat steps no. 4 – 9 for each SageMaker notebook instance that you want to configure, available within the current AWS region.
11 Change the AWS cloud region from the navigation bar and repeat the Remediation process for other regions.
Using AWS CLI
01 Define the trust relationship policy for the execution role. This trust policy allows Amazon SageMaker to use the role's permissions by giving the service principal "sagemaker.amazonaws.com" permission to call the AWS Security Token Service "AssumeRole" action. To create the required trust policy for the new role, save the following policy document to a JSON file named cc-role-trust-policy.json:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
02 Run create-role command (OSX/Linux/UNIX) to create the necessary execution role using the trust relationship policy defined at the previous step:
aws iam create-role --role-name cc-sagemaker-new-execution-role --assume-role-policy-document file://cc-role-trust-policy.json
03 The command output should return the information available for the new IAM role:
{ "Role": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com" } } ] }, "RoleId": "AAAABBBBCCCCDDDDEEEE", "CreateDate": "2024-06-05T10:00:00Z", "RoleName": "cc-sagemaker-new-execution-role", "Path": "/", "Arn": "arn:aws:iam::123456789012:role/service-role/cc-sagemaker-new-execution-role" } }
04 Run attach-role-policy command (OSX/Linux/UNIX) to attach one or more IAM policies to the newly created execution role, according to your use case. The following example makes use of the "AmazonSageMakerFullAccess" managed policy, which grants the execution role permission to perform certain Amazon S3 actions on buckets or objects with SageMaker, Sagemaker, sagemaker, or aws-glue in the name (the command does not produce an output). If your use case requires more granular permissions, consult this page to create an execution role that meets your business needs:
aws iam attach-role-policy --role-name cc-lambda-stream-new-execution-role --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
05 Run stop-notebook-instance command (OSX/Linux/UNIX) to stop the Amazon SageMaker notebook instance that you want to configure (the command does not produce an output):
aws sagemaker stop-notebook-instance --region us-east-1 --notebook-instance-name cc-sagemaker-ml-instance
06 Once the selected instance is stopped, run update-notebook-instance command (OSX/Linux/UNIX) with the name of the SageMaker notebook instance that you want to configure as the identifier parameter, to replace the missing execution role with the new role created and configured at the previous steps (the command does not return an output):
aws sagemaker update-notebook-instance --region us-east-1 --notebook-instance-name cc-sagemaker-ml-instance --role-arn arn:aws:iam::123456789012:role/service-role/cc-sagemaker-new-execution-role
07 Run start-notebook-instance command (OSX/Linux/UNIX) to stop the Amazon SageMaker notebook instance that you want to configure (the command does not produce an output):
aws sagemaker start-notebook-instance --region us-east-1 --notebook-instance-name cc-sagemaker-ml-instance
08 Repeat steps no. 1 – 7 for each SageMaker notebook instance that you want to configure, available in the selected AWS region.
09 Change the AWS cloud region by updating the --region command parameter value and repeat the Remediation process for other regions.
References
- AWS Documentation
- Amazon SageMaker FAQs
- Control root access to a SageMaker notebook instance
- SageMaker Roles
- AWS Managed Policies for Amazon SageMaker
- AWS Command Line Interface (CLI) Documentation
- list-notebook-instances
- describe-notebook-instance
- update-notebook-instance
- stop-notebook-instance
- start-notebook-instance