Use the Conformity Knowledge Base AI to help improve your Cloud Posture

S3 Buckets Lifecycle Configuration

Trend Cloud One™ – Conformity is a continuous assurance tool that provides peace of mind for your cloud infrastructure, delivering over 1000 automated best practice checks.

Risk Level: Low (generally tolerable level of risk)
Rule ID: S3-020

Ensure that your Amazon S3 buckets are using lifecycle configurations for security and cost optimization purposes. An S3 lifecycle configuration is a set of one or more rules, where each rule defines an action (transition or expiration action) for Amazon S3 to apply to a group of objects. A lifecycle configuration is used to manage Amazon S3 data during its lifetime.

This rule can help you with the following compliance standards:

  • APRA
  • MAS
  • NIST4

For further details on compliance standards supported by Conformity, see here.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Security
Cost
optimisation
Sustainability

With an S3 lifecycle configuration, you can enable Amazon S3 to downgrade the storage class for your objects, archive or delete S3 objects during their lifecycle. For example, you can define S3 lifecycle configuration rules to achieve compliance (with the law, with your organization standards or business requirements) by automatically transitioning your objects to S3 Standard-Infrequent Access (S3 Standard-IA) storage class one month after creation, or archive S3 objects with Amazon S3 Glacier using Glacier or Glacier Deep Archive storage class one year after creation. You can also implement lifecycle configuration rules to expire (delete) objects based on your retention requirements or clean up incomplete multipart uploads in order to optimize your Amazon S3 costs.


Audit

To determine if your Amazon S3 buckets are using lifecycle configurations, perform the following operations:

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon S3 console at https://console.aws.amazon.com/s3/.

03 Click on the name of the S3 bucket that you want to examine to access the bucket configuration settings.

04 Select the Management tab from the console menu to access the management settings available for the selected bucket.

05 In the Lifecycle rules section, search for existing lifecycle configuration rules. If there are no rules listed in the Lifecycle rules section or the status of the existing rule(s) is set to Disabled, the S3 lifecycle configuration is not enabled for the selected Amazon S3 bucket.

06 Repeat steps no. 3 – 5 to determine the S3 lifecycle configuration status for other Amazon S3 buckets available within your AWS cloud account.

Using AWS CLI

01 Run list-buckets command (OSX/Linux/UNIX) using custom query filters to list the names of all Amazon S3 buckets available in your AWS cloud account:

aws s3api list-buckets
	--query 'Buckets[*].Name'

02 The command output should return the names of your S3 buckets:

[
  "cc-prod-web-data",
  "cc-project5-logs"
]

03 Run get-bucket-lifecycle-configuration command (OSX/Linux/UNIX) using the name of the Amazon S3 bucket that you want to examine as the identifier parameter to describe the lifecycle configuration rules defined for the selected S3 bucket:

aws s3api get-bucket-lifecycle-configuration
  --bucket cc-prod-web-data
  --query 'Rules'

04 The command output should return the requested configuration information:

  1. If get-bucket-lifecycle-configuration command output returns the NoSuchLifecycleConfiguration error message, as shown in the output example below, there are no lifecycle rules defined for the bucket, therefore the S3 lifecycle configuration is not enabled for the selected Amazon S3 bucket:
    An error occurred (NoSuchLifecycleConfiguration) when calling the GetBucketLifecycleConfiguration operation: The lifecycle configuration does not exist
    
  2. If get-bucket-lifecycle-configuration command output returns one or more lifecycle rules but the "Status" of these rules is set to "Disabled", as shown in the output example below, the S3 lifecycle configuration is not enabled for the selected Amazon S3 bucket:
    [
      {
        "ID": "cc-transition-to-standard-ia",
        "Filter": {},
        "Status": "Disabled",
        "Transitions": [
          {
            "Days": 30,
            "StorageClass": "STANDARD_IA"
          }
        ]
      }
    ]
    

05 Repeat steps no. 3 and 4 to determine the S3 lifecycle configuration status for other Amazon S3 buckets available in your AWS cloud account.

Remediation / Resolution

To enable the lifecycle configuration for your Amazon S3 buckets by creating lifecycle rules, perform the following operations:

Note: As an example, this conformity rule describes how to utilize Amazon S3 lifecycle configuration to tier down the storage class of S3 objects over their lifetime in order to help reduce the storage costs and retain data for compliance purposes. The transition actions for the lifecycle configuration rule used as an example are:
  • Transition objects to the S3 Standard-Infrequent Access (S3 Standard-IA) storage class 30 days after creation.
  • Transition objects to the Glacier storage class 60 days after creation.
  • One expiration action that enables Amazon S3 to delete the objects a year after creation.

Using AWS CloudFormation

01 CloudFormation template (JSON):

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Set Lifecycle Configuration",
  "Resources": {
    "S3Bucket": {
      "Properties": {
        "BucketName": "cc-prod-web-data",
        "AccessControl": "Private",
        "LifecycleConfiguration": {
          "Rules": [
            {
              "Id": "cc-transition-access-log-data",
              "Prefix": "log",
              "Status": "Enabled",
              "ExpirationInDays": 365,
              "Transitions": [
                {
                  "TransitionInDays": 30,
                  "StorageClass": "STANDARD_IA"
                },
                {
                  "TransitionInDays": 60,
                  "StorageClass": "GLACIER"
                }
              ]
            }
          ]
        }
      },
      "Type": "AWS::S3::Bucket"
    }
  }
}

02 CloudFormation template (YAML):

AWSTemplateFormatVersion: '2010-09-09'
Description: Set Lifecycle Configuration
Resources:
  S3Bucket:
  Properties:
    BucketName: cc-prod-web-data
    AccessControl: Private
    LifecycleConfiguration:
      Rules:
      - Id: cc-transition-access-log-data
        Prefix: log
        Status: Enabled
        ExpirationInDays: 365
        Transitions:
        - TransitionInDays: 30
          StorageClass: STANDARD_IA
        - TransitionInDays: 60
          StorageClass: GLACIER
  Type: AWS::S3::Bucket

Using Terraform (AWS Provider)

01 Terraform configuration file (.tf):

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "default"
  region = "us-east-1"
}

resource "aws_s3_bucket" "data-bucket" {
  bucket = "cc-prod-web-data"
  acl = "private"

  lifecycle_rule {
    id = "cc-transition-access-log-data"
    enabled = true

    prefix = "log/"

    tags = {
      rule = "log"
      autoclean = "true"
    }

    transition {
      days = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days = 60
      storage_class = "GLACIER"
    }

    expiration {
      days = 365
    }
  }

}

Using AWS Console

01 Sign in to the AWS Management Console.

02 Navigate to Amazon S3 console at https://console.aws.amazon.com/s3/.

03 Click on the name of the S3 bucket that you want to reconfigure.

04 Select the Management tab from the console menu to access the management settings available for the selected bucket.

05 In the Lifecycle rules section, choose Create lifecycle rule to set up a new lifecycle configuration rule.

06 On the Create lifecycle rule setup page, perform the following actions:

  1. For Lifecycle rule name, provide a unique name for your new lifecycle configuration rule.
  2. Under Choose a rule scope, select This rule applies to all objects in the bucket, then select the I acknowledge that this rule will apply to all objects in the bucket checkbox to confirm the option. If you want the new rule to apply to specific objects, you must use a filter to identify those objects. In this case, choose Limit the scope of this rule using one or more filters.
  3. For Lifecycle rule actions, perform the following actions:
    • Select Transition current versions of objects between storage classes and:
      • Choose Standard-IA from the Storage class transitions dropdown list, then enter 30 in the Days after object creation box. This will transition your objects to the S3 Standard-IA storage class 30 days after creation.
      • Choose Add transition, select Glacier from the Storage class transitions dropdown list, then enter 60 in the Days after object creation box. This will transition your S3 objects to the Glacier storage class 60 days after creation. Select I acknowledge that this lifecycle rule will increase the one-time lifecycle request cost if it transitions small objects for Transitioning small objects to Glacier or Glacier Deep Archive will increase costs.
    • Select Expire current versions of objects and enter 365 in the Number of days after object creation box to enable Amazon S3 to delete your objects a year after creation.
  4. Review the rule configuration details listed in the Timeline summary section.
  5. Choose Create rule to save your S3 lifecycle configuration rule.

07 Repeat steps no. 3 – 6 to enable S3 lifecycle configuration for other Amazon S3 buckets available in your AWS cloud account.

Using AWS CLI

01 Run put-bucket-lifecycle-configuration command (OSX/Linux/UNIX) using the name of the Amazon S3 bucket that you want to reconfigure as the identifier parameter, to enable S3 lifecycle configuration for the selected Amazon S3 bucket by creating a new lifecycle rule that moves all the objects within the specified bucket to Standard-IA and Glacier storage classes 30 and 60 days after creation, and deletes the S3 objects a year after creation (if successful, the command does not produce an output):

aws s3api put-bucket-lifecycle-configuration
  --bucket cc-prod-web-data
  --lifecycle-configuration '{
  "Rules": [
    {
      "ID": "cc-transition-access-log-data",
      "Status": "Enabled",
      "Filter": {},
      "Expiration": {
        "Days": 365
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 60,
          "StorageClass": "GLACIER"
        }
      ]
    }
  ]
}'

02 Repeat step no. 1 to enable S3 lifecycle configuration for other Amazon S3 buckets available within your AWS cloud account.

References

Publication date Dec 8, 2017