Elastic Beanstalk Persistent Logs

Risk Level: Medium (should be achieved)

Rule ID: ElasticBeanstalk-003

Ensure that AWS Elastic Beanstalk (EB) environment logs are retained and uploaded to Amazon S3 in order to keep the logging data for future audits, historical purposes or to track and analyze the EB application environment behavior for a long period of time.

This rule can help you with the following compliance standards:

GDPR
APRA
MAS
NIST4

For further details on compliance standards supported by Conformity, see here.

This rule can help you work with the AWS Well-Architected Framework.

This rule resolution is part of the Conformity Security & Compliance tool for AWS.

Operational
excellence

The log files generated by AWS Elastic Beanstalk are all stored within an S3 bucket configured by the service when the environment is created. Most of these log files (trail and bundle logs) are removed from the S3 bucket 15 minutes after these are created, so in order to retain the logs you can configure your EB environment to publish logs to Amazon S3 automatically after these have been rotated. The retained (persistent) logs can be easily retrieved using the EB Management Console or the EB CLI. You can also configure your Elastic Beanstalk environment to stream logs to Amazon CloudWatch in real time.

Audit

To determine if your Elastic Beanstalk environments publish log data to Amazon S3 or stream it to Amazon CloudWatch (optional), perform the following:

Using AWS Console

01 Sign in to AWS Management Console.

02 Navigate to Elastic Beanstalk (EB) dashboard at https://console.aws.amazon.com/elasticbeanstalk/.

03 Choose the EB application environment that you want to examine.

04 In the left navigation panel choose Configuration to access the environment settings.

05 Inside Web Tier section, within Software Configuration box, check Log publication and Log streaming attributes value. If the value for Log publication configuration attribute is set to Off and the value for Log streaming attribute is set to disabled, the selected Amazon Elastic Beanstalk application environment does not publish its logs to AWS S3 nor stream the log data to AWS CloudWatch.

06 Repeat steps no. 3 – 5 to check the logging configuration for other Amazon EB environments provisioned in the current region.

07 Change the AWS region from the navigation bar and repeat the process for the other regions.

Using AWS CLI

01 Run describe-environments command (OSX/Linux/UNIX) using custom query filters to list the names of the Elastic Beanstalk application environments available in the selected AWS region:

aws elasticbeanstalk describe-environments
  --region us-east-1
  --output table
  --query 'Environments[*].EnvironmentName'

02 The command output should return a table with the requested EB environment names:

----------------------
|DescribeEnvironments|
+--------------------+
| CcProdWebsite-env  |
| CcWebStaging-env   |
+--------------------+

03 Execute again describe-environments command (OSX/Linux/UNIX) using custom query filters to list the name of the selected Elastic Beanstalk environment application:

aws elasticbeanstalk describe-environments
  --region us-east-1
  --environment-names CcProdWebsite-env
  --query 'Environments[*].ApplicationName'

04 The command output should return the requested EB application name:

[
    "cc-production-app"
]

05 Run describe-configuration-settings command (OSX/Linux/UNIX) using the name of the EB environment and the name of the application as identifiers and custom query filters to describe the logging configuration (publishing logs to S3 and streaming log data to CloudWatch) for the selected Elastic Beanstalk environment:

aws elasticbeanstalk describe-configuration-settings
  --region us-east-1
  --environment-name CcProdWebsite-env
  --application-name cc-production-app
  --query 'ConfigurationSettings[*].OptionSettings[?OptionName==`LogPublicationControl` || OptionName==`StreamLogs`] | []'

06 The command output should return the environment logging configuration set:

[
    {
        "OptionName": "LogPublicationControl",
        "Namespace": "aws:elasticbeanstalk:hostmanager",
        "Value": "false"
    },
    {
        "OptionName": "StreamLogs",
        "Namespace": "aws:elasticbeanstalk:cloudwatch:logs",
        "Value": "false"
    }
]

If the Value attributes value set for both LogPublicationControl and StreamLogs logging options is false, (as shown in the output example above), the selected AWS Elastic Beanstalk application environment does not publish or stream the log data to Amazon S3 and Amazon CloudWatch.

07 Repeat steps no. 3 – 6 to verify the logging configuration for other Amazon EB environments provisioned in the current region.

08 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 7 to perform the audit process for other regions.

Remediation / Resolution

To enable Elastic Beanstalk log file rotation to Amazon S3 and stream log data in real time to Amazon CloudWatch (optional), perform the following:

Using AWS Console

01 Sign in to AWS Management Console.

02 Navigate to Elastic Beanstalk (EB) dashboard at https://console.aws.amazon.com/elasticbeanstalk/.

03 Choose the EB resource that you want to reconfigure (see Audit section part I to identify the right EB environment).

04 In the left navigation panel choose Configuration to access the environment settings.

05 Inside Web Tier section, within Software Configuration box, click the edit settings button to access the environment software configuration page.

06 On the software configuration page, within Log Options section, click Enable log file rotation to Amazon S3 to publish the EB service logs to the S3 bucket configured by Elastic Beanstalk during the environment creation. (Optional) To configure the EC2 instances provisioned within your environment to stream logs to AWS CloudWatch and keep the log data for historical purposes, select Log streaming setting checkbox, choose a retention period (days) from the Retention dropdown list and select Keep logs after terminating environment from Lifecycle dropdown list. Click Apply to confirm the changes and enable persistent logging for the selected Elastic Beanstalk application environment.

07 Repeat steps no. 3 – 6 to enable persistent logging for other Amazon EB environments available in the current region.

08 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run update-environment command (OSX/Linux/UNIX) using the name of the Elastic Beanstalk environment that you want to reconfigure (see Audit section part II to identify the right EB environment) to enable log file rotation to AWS S3 for the selected application environment:

aws elasticbeanstalk update-environment
  --region us-east-1
  --environment-name CcProdWebsite-env
  --option-settings Namespace=aws:elasticbeanstalk:hostmanager,OptionName=LogPublicationControl,Value=true

02 The command output should return the update-environment command request metadata:

{
    "ApplicationName": "cc-production-app",
    "EnvironmentName": "CcProdWebsite-env",
    "VersionLabel": "Production Web Application",
    "Status": "Updating",
    "EnvironmentArn": "arn:aws:elasticbeanstalk:us-east-1:123456789012:environment/cc-production-app/CcProdWebsite-env",
    "PlatformArn": "arn:aws:elasticbeanstalk:us-east-1::platform/PHP 7.1 running on 64bit Amazon Linux/2.5.0",

    ...

    "AbortableOperationInProgress": true,
    "Tier": {
        "Version": "2",
        "Type": "Standard",
        "Name": "WebServer"
    },
    "Health": "Grey",
    "DateUpdated": "2017-08-11T17:46:46.311Z",
    "DateCreated": "2017-10-24T09:22:59.267Z"
}

03 (Optional) Execute update-environment command (OSX/Linux/UNIX) using the name of the Elastic Beanstalk environment that you want to reconfigure as identifier to enable log data streaming to Amazon CloudWatch for the selected application environment:

aws elasticbeanstalk update-environment
  --region us-east-1
  --environment-name CcProdWebsite-env
  --option-settings Namespace=aws:elasticbeanstalk:cloudwatch:logs,OptionName=StreamLogs,Value=true

04 The command output should return the request metadata:

{
    "ApplicationName": "cc-production-app",
    "EnvironmentName": "CcProdWebsite-env",
    "VersionLabel": "Production Web Application",
    "Status": "Updating",
    "EnvironmentArn": "arn:aws:elasticbeanstalk:us-east-1:123456789012:environment/cc-production-app/CcProdWebsite-env",
    "PlatformArn": "arn:aws:elasticbeanstalk:us-east-1::platform/PHP 7.1 running on 64bit Amazon Linux/2.5.0",

    ...

    "AbortableOperationInProgress": true,
    "Tier": {
        "Version": "2",
        "Type": "Standard",
        "Name": "WebServer"
    },
    "Health": "Grey",
    "DateUpdated": "2017-08-11T17:55:33.211Z",
    "DateCreated": "2017-10-24T09:22:59.267Z"
}

05 Repeat steps no. 1 – 4 to enable persistent logging for other Amazon EB environments available in the current region.

06 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 - 5 to perform the entire process for other regions.

References

AWS Command Line Interface (CLI) Documentation
elasticbeanstalk
describe-environments
describe-configuration-settings
update-environment

Publication date Nov 1, 2017

Audit

Using AWS Console

Using AWS CLI

Remediation / Resolution

Using AWS Console

Using AWS CLI

References

Related ElasticBeanstalk rules