Use the Conformity Knowledge Base AI to help improve your Cloud Posture

Enable Lifecycle Management for Cloud Storage Objects

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: Medium (should be achieved)

Ensure that your Google Cloud Storage buckets are using lifecycle management rules to manage objects during their lifetime and help saving on Google Cloud Platform (GCP) storage costs. Lifecycle management rules apply actions to current and future objects when they meet user-defined conditions such as downgrading or deleting older objects, in order to help manage costs.

Cost
optimisation

Google Cloud Storage service transitions and expires data objects automatically according to the lifecycle management rules that you define at the bucket level. A lifecycle management configuration (i.e. rule or set of rules) lets you automatically delete objects at the end of their lifecycle and migrate objects to a cold storage class in order to optimize your data for storage costs. For example, you can define a Cloud Storage lifecycle management rule to save costs and achieve compliance (with the law, with your organization standards, or your business requirements) by changing the storage class of your objects to Nearline Storage if their age is greater than 365 days (1 year) and their current storage class is Standard Storage, Multi-Regional Storage, or Durable Reduced Availability (DRA) Storage. Nearline Storage class is best suited for backups and data accessed less than once a month. You can also configure a rule that changes the storage class of the objects to Coldline Storage if their age is greater than 1095 days (3 years) and their current storage class is Nearline Storage. Coldline Storage is mostly used for disaster recovery and data accessed less than once a quarter.


Audit

To determine if your Google Cloud Storage objects are using lifecycle management configurations, perform the following actions:

Using GCP Console

01 Sign in to Google Cloud Management Console.

02 Select the Google Cloud Platform (GCP) project that you want to examine from the console top navigation bar.

03 Navigate to Cloud Storage dashboard at https://console.cloud.google.com/storage.

04 In the navigation panel, select Browser to access the list with all the Cloud Storage buckets created for the selected project.

05 Choose the storage bucket that you want to examine and check the number of lifecycle management rules created for the objects inside the bucket, available in the Lifecycle rules column. If there are no lifecycle management rules created and the value is set to None, the objects stored within the selected Google Cloud Storage bucket are not using a lifecycle configuration to help manage storage costs.

06 Repeat step no. 5 for each storage bucket provisioned for the selected Google Cloud Platform (GCP) project.

07 Repeat steps no. 1 – 6 for each GCP project deployed in your Google Cloud account.

Using GCP CLI

01 Run projects list command (Windows/macOS/Linux) using custom query filters to list the IDs of all the Google Cloud Platform (GCP) projects available in your cloud account:

gcloud projects list
	--format="table(projectId)"

02 The command output should return the requested GCP project IDs:

PROJECT_ID
cc-web-project-112233
cc-data-project-111222

03 Run gsutil ls command (using gsutil Python tool) to list the identifier of each storage bucket created for the specified GCP project:

gsutil ls -p cc-web-project-112233

04 The command output should return the requested storage resource name(s):

gs://cc-audit-log-bucket/
gs://cc-data-backup-bucket/

05 Run gsutil lifecycle get command (using gsutil tool) using the name of the Cloud Storage bucket that you want to examine as identifier parameter, to describe the lifecycle management configuration available for the selected bucket:

gsutil lifecycle get gs://cc-audit-log-bucket

06 The command output should return the requested lifecycle configuration metadata:

gs://cc-audit-log-bucket/ has no lifecycle configuration.

If the gsutil lifecycle get command output returns "<storage-bucket> has no lifecycle configuration.", where <storage-bucket> is the name of the verified bucket, the objects stored inside the selected Google Cloud Storage bucket are not using a lifecycle management configuration to help optimize storage costs.

07 Repeat step no. 5 and 6 for each storage bucket available in the selected Google Cloud Platform (GCP) project.

08 Repeat step no. 1 – 7 for each GCP project created within your Google Cloud account.

Remediation / Resolution

To enable object lifecycle management for your Google Cloud Storage buckets, define a lifecycle rule to transition your data to the appropriate storage class or expire/delete it at the end of the specified lifecycle. To create and configure a lifecycle rule, perform the following actions:

Note: As example, this conformity rule will demonstrate how to downgrade the storage class of a bucket's objects older than 365 days to Nearline Storage, where the current storage class is Standard Storage, Multi-Regional Storage, and Durable Reduced Availability (DRA) Storage. Nearline Storage is a low-cost, highly durable storage class designed for storing infrequently accessed data. This class is most suitable for backups and data accessed once a month for analysis.

Using GCP Console

01 Sign in to Google Cloud Management Console.

02 Select the GCP project that you want to access from the console top navigation bar.

03 Navigate to Cloud Storage dashboard at https://console.cloud.google.com/storage.

04 In the navigation panel, select Browser to access all the buckets created for the selected GCP project.

05 Choose the storage bucket that you want to reconfigure (see Audit section part I to identify the right bucket), then click on None value (link) available in the Lifecycle rules column.

06 On the View object lifecycle rules page, perform the following commands:

  1. Click Add rule to add a new rule to the bucket's lifecycle management configuration. The new rule moves the objects stored inside the selected bucket to the Nearline Storage class 365 days after the objects are created.
  2. For 1 Select object conditions, perform the following:
    • Select the Age checkbox and set the number of days after which the objects are moved to the Nearline Storage class. In this case, 365 days.
    • Select the Storage class checkbox and choose the storage class(es) of the objects that you want to transition. For our example, the object classes are Standard, Multi-Regional and Durable Reduced Availability.
    • Click Continue to continue the setup process.
  3. For 2 Select action, choose Set to Nearline and click Continue.
  4. Click Save to save the changes and create the new lifecycle management rule. After you create a lifecycle rule, it may take up to 24 hours to take effect.

07 If required, repeat step no. 5 and 6 to create and configure lifecycle rules for other Cloud Storage buckets available within the selected project.

08 Repeat steps no. 2 – 7 for each GCP project deployed in your Google Cloud account.

Using GCP CLI

01 Define the necessary lifecycle management rule that you want to apply to the objects stored within the your Cloud Storage bucket, and save the rule contents to a JSON file named cc-object-lifecycle-rule.json. The following rule moves the objects stored inside the specified bucket to the Nearline Storage class 365 days after the objects are created:

{
  "lifecycle": {
    "rule": [
      {
        "action": {
          "type": "SetStorageClass",
          "storageClass": "NEARLINE"
        },
        "condition": {
          "age": 365,
          "matchesStorageClass": [
            "MULTI_REGIONAL",
            "STANDARD",
            "DURABLE_REDUCED_AVAILABILITY"
          ]
        }
      }
    ]
  }
}

02 Run gsutil lifecycle set command (using gsutil Python tool) using the name of the Google Cloud Storage bucket that you want to reconfigure as identifier parameter (see Audit section part II to identify the right resource), to add the lifecycle rule defined at the previous step (i.e. cc-object-lifecycle-rule.json) to the lifecycle management configuration of the selected bucket:

gsutil lifecycle set cc-lifecycle-rule.json gs://cc-audit-log-bucket

03 If successful, the command output should return the gsutil lifecycle set request status:

Setting lifecycle configuration on gs://cc-audit-log-bucket/...

04 If required, repeat steps no. 1 – 3 to define and set lifecycle rules for other Cloud Storage buckets available in the selected project.

05 Repeat steps no. 1 – 4 for each GCP project created within your Google Cloud account.

References

Publication date Apr 21, 2021