Logo of Trend Micro
Use the Knowledge Base AI to help improve your Cloud Posture

Sufficient Kinesis Data Retention Period

Trend Vision One™ provides continuous assurance that gives peace of mind for your cloud infrastructure, delivering over 1100 automated best practice checks.

Risk Level: Medium (should be achieved)

Ensure that your AWS Kinesis streams have a sufficient data retention period configured for reliability and compliance purposes. The retention period represents the number of days to retain stream data. Prior to running this rule by the Cloud Conformity engine, the data retention period for your Amazon Kinesis streams need to be defined in the rule settings, on your Cloud Conformity account dashboard. The retention period can be configured to any value between 24 and 168 hours.

Reliability

A sufficient data retention period allows more time for your Kinesis stream data consumers to recover. The default retention period for an AWS Kinesis stream is 24 hours. To ensure that your consumers are able to read stream data before it expires if any problems occur, you can extend your data retention period up to 168 hours (7 days).

Audit

To determine if your Amazon Kinesis streams have a sufficient data retention period, perform the following actions:

Using AWS Console

01 Sign in to AWS Management Console.

02 Navigate to Kinesis dashboard at https://console.aws.amazon.com/kinesis/.

03 In the navigation panel, under Amazon Kinesis, choose Data Streams.

04 Select the Kinesis stream that you want to examine, click on the Actions dropdown button and select Details to access the stream configuration details.

05 Choose the Details tab from the top panel and check the Data retention period attribute value available in the Data retention period section.

06 Sign in to your Cloud Conformity account, access Sufficient Kinesis Data Retention Period conformity rule settings and compare the retention period value found at the previous step against the one set within the rule configuration section. If the number of hours configured as data retention period is less than the number defined in your Cloud Conformity account, the selected Amazon Kinesis stream does not have a sufficient data retention period configured.

07 Repeat step no. 4 – 6 for each Amazon Kinesis stream available in the current AWS region.

08 Change the AWS region from the navigation bar to repeat the audit process for other regions.

Using AWS CLI

01 Run list-streams command (OSX/Linux/UNIX) to list the names of all Kinesis streams available within the selected AWS region - US East (N. Virginia): 

aws kinesis list-streams 
	--region us-east-1 --query 'StreamNames'

02 The command output should return the names of the data streams available in the region: 

[
    "cc-iot-data-stream",
    "cc-project5-stream"
]

03 Run describe-stream command (OSX/Linux/UNIX) using the name of the stream that you want to examine as identifier parameter and custom query filters to return the data retention period defined for the selected Kinesis stream: 

aws kinesis describe-stream
	--region us-east-1
	--stream-name cc-iot-data-stream
	--query 'StreamDescription.RetentionPeriodHours'

04 The command output should return the requested value, i.e. the data retention period in number of hours: 

24

05 Sign in to your Cloud Conformity account, access Sufficient Kinesis Data Retention Period conformity rule settings and compare the retention period value returned by the describe-stream command output at the previous step against the one defined in the rule settings. If the data retention period for the verified stream is less than the one configured within your Cloud Conformity account settings, the selected Amazon Kinesis stream does not have set a sufficient data retention period.

06 Repeat step no. 3 – 5 for each Amazon Kinesis stream available in the current AWS region.

07 Change the AWS region by updating the --region command parameter value and repeat steps no. 1 – 5 to perform the entire audit process for other regions.

Remediation / Resolution

To update your Amazon Kinesis streams configuration in order to set up a sufficient data retention period, perform the following actions:

Using AWS Console

01 Sign in to your Cloud Conformity account, access Sufficient Kinesis Data Retention Period conformity rule settings, and copy the retention period value (number of days) defined for your AWS Kinesis streams.

02 Sign in to AWS Management Console.

03 Navigate to Kinesis dashboard at https://console.aws.amazon.com/kinesis/.

04 In the navigation panel, under Amazon Kinesis, choose Data Streams.

05 Select the Kinesis stream that you want to reconfigure, click on the Actions dropdown button and select Details to open the stream configuration page.

06 Choose the Details tab from the top panel, locate Data retention period section and click on the Edit button next to it.

07 Paste the value copied at step no. 1 (i.e. the sufficient retention period) into the Data retention period box, then click Save to apply the changes. Once the request is made, a pop-up message will be displayed: "Updating data retention period from <original-value> to <new-value>. This will take up to 30 seconds". The selected stream transitions through a PENDING state. Once the Kinesis data stream returns to the ACTIVE state, the new retention period becomes active.

08 Repeat steps no. 5 – 7 to reconfigure data retention period for other Amazon Kinesis streams available in the current AWS region.

09 Change the AWS region from the navigation bar and repeat the process for other regions.

Using AWS CLI

01 Run increase-stream-retention-period command (OSX/Linux/UNIX) using the name of the data stream that you want to reconfigure as identifier parameter value, to extend the retention period for the selected AWS Kinesis data stream. The maximum value that can be set for a stream's retention period is 168 hours (7 days). The following command example sets a retention period of 3 days to a stream named "cc-iot-data-stream", provisioned in the US East (N. Virginia) region (the command does not return an output): 

aws kinesis increase-stream-retention-period
	--region us-east-1
	--stream-name cc-iot-data-stream
	--retention-period-hours 72

02 Repeat step no. 1 to reconfigure data retention period for other Amazon Kinesis streams available in the selected AWS region.

03 Change the AWS region by updating the --region command parameter value and repeat the process for other regions.

References

Publication date Apr 23, 2019

Related Kinesis rules