Export All Log Entries Using Sinks

01 Sign in to the Google Cloud Management Console.

02 Select the GCP project that you want to examine from the console top navigation bar.

03 Navigate to Google Cloud Logging console at https://console.cloud.google.com/logs.

04 In the left navigation panel, select Log Router to access all the log sinks created for the selected GCP project.

05 Select the log sink that you want to examine, click on the 3-dot button to open the sink options menu, and select View sink details.

06 In the Sink details information box, check for the Inclusion filter configured for the selected sink. The Inclusion filter defines which log entries to include in logs routing sink. If the Inclusion filter box is not empty, the selected log sink is not configured to export all the log entries created for the selected GCP project.

07 Repeat steps no. 5 and 6 for all the log sinks listed in the Log Router Sinks section. If none of these sinks are using blank (empty) inclusion filters, there are no log sinks configured to export all the log entries for the selected Google Cloud Platform (GCP) project.

08 Repeat steps no. 2 – 7 for each GCP project deployed within your Google Cloud account.

01 Run projects list command (Windows/macOS/Linux) using custom query filters to list the IDs of all the projects available in your Google Cloud account:

gcloud projects list
  --format="table(projectId)"

02 The command output should return the requested GCP project IDs:

PROJECT_ID
cc-project5-123123
cc-internal-111222
cc-web-prod-123456

03 Run logging sinks list command (Windows/macOS/Linux) using the ID of the GCP project that you want to examine as the identifier parameter and custom query filters to describe the name and the filter for each log sink created within the selected GCP project:

gcloud logging sinks list
  --project=cc-project5-123123
  --format="json(name,filter)"

04 The command output should return the requested log sink information:

[
	{
		"name": "cc-data-access-log-sink",
		"filter": "resource.type=\"project\"\nresource.labels.project_id=\"cc-project5-123123\"\nlogName=\"projects/cc-project5-123123/logs/cloudaudit.googleapis.com%2Fdata_access\""
	},
	{
		"name": "cc-service-account-sink",
		"filter": "resource.type=\"service_account\"\nresource.labels.email_id=\"retreterter@cc-project5-123123.iam.gserviceaccount.com\""
	}
]

05 Repeat steps no. 3 and 4 for each GCP project deployed within your Google Cloud account.

01 Sign in to the Google Cloud Management Console.

02 Select the GCP project that you want to examine from the console top navigation bar.

03 Navigate to Google Cloud Logging console at https://console.cloud.google.com/logs.

04 In the left navigation panel, select Log Router, and choose Create sink from the top menu to create a new log router sink for the selected GCP project.

05 On the Create logs routing sink setup page, perform the following operations:

1. For Sink details, provide a name and description for the new logs routing sink. Choose NEXT to continue the setup.
2. For Sink destination, click inside the Select sink service box, and select the service type for the new sink. The service selected at this step will store the exported logs. As an example, this conformity rule will use the Cloud Storage service to send the logs to a logging bucket. However, sinks can be configured to export logs to BigQuery datasets, Pub/Sub topics, or custom destinations. For logs destination, select the destination bucket. If you need to create a new bucket for your logs, choose the Create new log bucket option from the Select a log bucket dropdown menu, then follow the setup wizard to create a new destination bucket. Choose NEXT to continue.
3. For Choose logs to include in sink, leave the Build inclusion filter box empty to skip defining an inclusion filter for the logs sink. If an inclusion filter is not provided for the sink, all ingested logs will be routed to the destination configured above. Choose NEXT to continue the setup.
4. For Choose logs to filter out of sink (optional), skip configuring exclusion filters in order to export all the log entries generated within the selected GCP project.
5. Choose Create sink to create your new logs routing sink. If the request is successful, the following confirmation message should be displayed: Your log sink was successfully created. Data should be available soon..

06 Repeat steps no. 2 – 5 to create logs routing sinks for other GCP projects deployed in your Google Cloud account.

01 Create the storage resource that will be used as destination for the exported logs. As example, this conformity rule will use a Google Cloud Storage bucket as logs destination. However, log sinks can be configured to export logs to BigQuery datasets, Pub/Sub topics, or custom destinations. Run gsutil mb command (using gsutil tool) to create the Cloud Storage bucket where the log sink will export all the log entries created for the selected GCP project. The following example, creates a standard bucket named "cc-project5-logs-bucket", within the US East-1 region, for a project named "cc-project5-123123":

gsutil mb -p cc-project5-123123 -c STANDARD -l US-EAST1 -b on gs://cc-project5-logs-bucket/

02 The command output should return the gsutil mb request status:

Creating gs://cc-project5-logs-bucket/...

03 Run logging sinks create command (Windows/macOS/Linux) to create the logs routing sink required to export all the log entries generated for the selected Google Cloud project, to the destination bucket created at the previous steps:

gcloud logging sinks create cc-project5-log-sink storage.googleapis.com/cc-project5-logs-bucket

04 Type Y to confirm that the new sink is configured with an empty filter that matches all logs:

Sink with empty filter matches all entries.
Do you want to continue (Y/n)?  Y

05 The command request should return the information available for the new sink (including the sink URL and the writer identity ID):

Created [https://logging.googleapis.com/v2/projects/cc-project5-123123/sinks/cc-project5-log-sink].
Please remember to grant `serviceAccount:abcdabcdabcd-123123@gcp-sa-logging.iam.gserviceaccount.com` the Storage Object Creator role on the bucket.
More information about sinks can be found at https://cloud.google.com/logging/docs/export/configure_export

06 Run gsutil iam ch command (using gsutil Python tool) to add the sink's writer identity returned at the previous step to the destination bucket, and give this identity the Storage Object Creator role in order to allow it to create objects within that bucket (the command does not produce an output):

gcloud iam ch serviceAccount:abcdabcdabcd-123123@gcp-sa-logging.iam.gserviceaccount.com:objectCreator gs://cc-project5-logs-bucket

07 Repeat steps no. 1 – 6 to create logs routing sinks for other GCP projects created within your Google Cloud account.