Use the Conformity Knowledge Base AI to help improve your Cloud Posture

Disable Preemptibility for VM Instances

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: High (not acceptable risk)

Ensure that your Google Cloud Platform (GCP) projects are not using preemptible virtual machine instances for production and business-critical applications. A preemptible virtual machine (VM) is an instance that you can create and run at a much lower price than normal instances but it can be terminated sooner due to system demands.

Reliability

Preemptible virtual machine instances are excess Compute Engine capacity, therefore their availability varies with usage. These instances may be preempted to free up compute resources for standard VM instances, and will only be able to run for a limited amount of time. Preemptible instances costs much less than standard instances, however, these resources are not suitable for production workloads as Google Cloud Compute Engine can terminate (preempt) the instances when it needs the resources for other tasks. Unless your applications are fault-tolerant and can withstand possible instance preemptions, it is strongly recommended to disable preemptibility for virtual machine (VM) instances.


Audit

To determine if there are any preemptible VM instances running within your GCP projects, perform the following actions:

Using GCP Console

01 Sign in to Google Cloud Management Console.

02 Select the production Google Cloud Platform (GCP) project that you want to examine from the console top navigation bar.

03 Navigate to Google Compute Engine dashboard at https://console.cloud.google.com/compute.

04 In the navigation panel, select VM instances to access the list with the virtual machine (VM) instances created for the selected project.

05 Click on the name of the VM instance that you want to examine.

06 Select the Details tab to access the instance configuration details.

07 On the Details panel, in the Availability policies section, check the Preemptibility configuration setting status. If Preemptibility is set to On, the selected Google Cloud virtual machine instance is preemptible, therefore the verified instance is not suitable for production workloads.

08 Repeat step no. 5 – 7 for each virtual machine (VM) instance created within the selected project.

09 Repeat steps no. 2 – 8 for each project deployed in your Google Cloud account.

Using GCP CLI

01 Run projects list command (Windows/macOS/Linux) using custom query filters to list the IDs of all the Google Cloud Platform (GCP) projects available in your Google Cloud account:

gcloud projects list
	--format="table(projectId)"

02 The command output should return the requested GCP project IDs:

PROJECT_ID
cc-web-stack-project-123123
cc-app-stack-project-112233

03 Run compute instances list command (Windows/macOS/Linux) using the ID of the GCP project that you want to examine as identifier parameter and custom query filters to describe the name and zone for each VM instance provisioned for the selected project:

gcloud compute instances list
	--project cc-web-stack-project-123123
	--format="table(name,zone)"

04 The command output should return the name(s) of the instance(s) within the selected GCP project:

NAME                      ZONE
cc-internal-vm-instance   us-central1-a
cc-custom-vm-instance     us-central1-a

05 Run compute instances describe command (Windows/macOS/Linux) using the name and the zone of the instance that you want to examine as identifier parameter and custom filtering to describe the preemptible status of the selected virtual machine instance:

gcloud compute instances describe cc-internal-vm-instance
	--zone us-central1-a
	--format="value(scheduling.preemptible)"

06 The command output should return the requested configuration status (True for enabled, False for disabled):

True

If the compute instances describe command output returns True, the selected Google Cloud virtual machine instance is preemptible, therefore the verified virtual machine is not suitable for production workloads.

07 Repeat step no. 5 and 6 for each virtual machine instance launched within the selected project.

08 Repeat steps no. 3 – 7 for each GCP project deployed in your Google Cloud account.

Remediation / Resolution

Avoid using preemptible virtual machine instances for production cloud environments. To disable preemptibility for existing Google Cloud virtual machine (VM) instances you have to re-create those instances and change their availability policy by performing the following actions:

Using GCP Console

01 Sign in to Google Cloud Management Console.

02 Select the GCP project that you want to access from the console top navigation bar.

03 Navigate to Google Compute Engine dashboard at https://console.cloud.google.com/compute.

04 In the navigation panel, select VM instances to access the list with all the Compute Engine instances provisioned for the selected project.

05 Choose the virtual machine (VM) instance that you want to re-create (see Audit section part I to identify the right resource), then click on the 3-dot button for instance menu options and select New machine image to create an image from the selected instance.

06 On the Create a machine image page, provide a name for your new machine image in the Name box, choose the location at which to store the image and the encryption key required to encrypt the data, then click Create to create the new virtual machine image.

07 Once the machine image is successfully created (i.e. the image status is set to Ready), click on the 3-dot button for image menu options and select Create VM to create a virtual machine instance from the newly created image.

08 On the Create an instance page, perform the following operations:

  1. Provide a unique name for the new VM instance in the Name box.
  2. In the Machine configuration section, choose the right machine family and type (must match the configuration of the instance that you want to re-create).
  3. Click on the Management, security, disks, networking, sole tenancy, select the Management tab, and perform the following:
    • Under Availability policy, choose Off (recommended) from the Preemptibility dropdown list to disable preemptibility for the new virtual machine instance.
    • Select Migrate VM instance (recommended) from the On host maintenance dropdown list to live migrate the selected instance during maintenance events.
    • Select On (recommended) from the Automatic restart dropdown list to enable automatic restart for the new virtual machine (VM) instance.
  4. Click Create to launch your new VM instance.

09 To avoid unnecessary charges on your Google Cloud monthly bill, you can remove the source (preemptible) instance from your GCP project. To remove the required instance, perform the following:

  1. In the navigation panel, select VM instances.
  2. Choose the VM instance that you want to remove, then click on the 3-dot button for instance menu options and select Delete to initiate the removal process.
  3. Within the Delete an instance confirmation box, select DELETE to confirm the action and remove the selected instance from your project.

10 Repeat steps no. 5 – 9 to disable preemptibility for other virtual machine instances provisioned for the selected project.

11 Repeat steps no. 2 – 10 for each GCP project created within your Google Cloud account.

Using GCP CLI

01 Run compute instances describe command (Windows/macOS/Linux) using the name of the virtual machine (VM) instance that you want to re-create as identifier parameter (see Audit section part II to identify the right resource), to describe the configuration metadata available for the selected instance:

gcloud compute instances describe cc-internal-vm-instance
	--zone us-central1-a

02 The command output should return the requested configuration information. This data will be used later to launch the new virtual machine instance:

cpuPlatform: Intel Haswell
deletionProtection: false
description: ''
disks:
- autoDelete: true
  boot: true
  deviceName: cc-internal-vm-instance
  diskSizeGb: '100'
  guestOsFeatures:
  - type: UEFI_COMPATIBLE
  - type: VIRTIO_SCSI_MULTIQUEUE
  index: 0
  interface: SCSI
  kind: compute#attachedDisk
  mode: READ_WRITE
  source: https://www.googleapis.com/compute/v1/projects/cc-web-stack-project-123123/zones/us-central1-a/disks/cc-internal-vm-instance
  type: PERSISTENT
displayDevice:
  enableDisplay: false

...

name: cc-internal-vm-instance
reservationAffinity:
  consumeReservationType: ANY_RESERVATION
scheduling:
  automaticRestart: false
  onHostMaintenance: TERMINATE
  preemptible: true
shieldedInstanceConfig:
  enableIntegrityMonitoring: true
  enableSecureBoot: false
  enableVtpm: true
shieldedInstanceIntegrityPolicy:
  updateAutoLearnPolicy: true
startRestricted: false
status: RUNNING
tags:
  items:
  - http-server
  - https-server
zone: https://www.googleapis.com/compute/v1/projects/cc-web-stack-project-123123/zones/us-central1-a

03 Run compute images create command (Windows/macOS/Linux) to create a machine image from the Google Compute Engine instance that you want to re-create. Include the --force flag to create the image from a running instance (by default, you cannot create images from running VM instances):

gcloud compute images create cc-internal-vm-image
	--source-disk cc-internal-vm-instance
	--source-disk-zone us-central1-a
	--storage-location us-central1
	--force

04 The command output should return the metadata for the newly created machine image:

Created [https://www.googleapis.com/compute/v1/projects/cc-web-stack-project-123123/global/images/cc-internal-vm-image].

NAME                    PROJECT                         FAMILY    STATUS
cc-internal-vm-image    cc-web-stack-project-123123               READY

05 Run compute instances create command (Windows/macOS/Linux) using the name of the machine image created at the previous steps and the configuration details returned at step no. 2 as configuration parameters, to create a new virtual machine (VM) instance from the selected image. To disable preemptibility for the new VM instance, avoid adding the --preemptible configuration parameter to the command request:

gcloud compute instances create cc-new-internal-vm-instance
	--image-project=cc-web-stack-project-123123
	--image=cc-internal-vm-image
	--zone=us-central1-a
	--machine-type=n1-standard-8
	--restart-on-failure
	--maintenance-policy=MIGRATE

06 The command output should return the metadata available for the new VM instance:

Created [https://www.googleapis.com/compute/v1/projects/cc-web-stack-project-123123/zones/us-central1-a/instances/cc-new-internal-vm-instance].

NAME                         ZONE           MACHINE_TYPE   INTERNAL_IP      EXTERNAL_IP      STATUS
cc-new-internal-vm-instance  us-central1-a  n1-standard-8  xxx.xxx.xxx.xxx  xxx.xxx.xxx.xxx  RUNNING

07 To avoid unnecessary charges on your Google Cloud bill, you can remove the source (preemptible) instance from your GCP project. To remove the required VM instance, run compute instances delete command (Windows/macOS/Linux) using the name of the source instance that you want to remove as identifier parameter, to remove the selected resource from your GCP project:

gcloud compute instances delete cc-internal-vm-instance
	--zone us-central1-a

08 Press Y at the command prompt to confirm the resource removal:

The following instances will be deleted. Any attached disks configured
 to be auto-deleted will be deleted unless they are attached to any
other instances or the `--keep-disks` flag is given and specifies them
 for keeping. Deleting a disk is irreversible and any data on the disk
 will be lost.
 - [cc-internal-vm-instance] in [us-central1-a]
Do you want to continue (Y/n)?

09 The command output should return the URL of the deleted VM instance:

Deleted [https://www.googleapis.com/compute/v1/projects/cc-web-stack-project-123123/zones/us-central1-a/instances/cc-internal-vm-instance].

10 Repeat steps no. 1 – 9 to disable preemptibility for other virtual machine instances available within the selected project.

11 Repeat steps no. 1 – 10 for each GCP project deployed within your Google Cloud account.

References

Publication date May 4, 2021