Configure Gmail Message Tracking

This guide outlines the process of configuring the Google Cloud Console to prepare for setting up the Gmail Message Tracking Beat.

Set Up Project and Roles for Gmail Message Tracking Beat

Step 1: Create or Choose a GCP Project

1. Login to the Google Cloud Console.

2. Click the Project dropdown (top navigation bar) and either:

   • Select an existing project, or

   • Click New Project, provide a Project Name, and click Create.

3. Enable billing for this project (required for BigQuery).

As a best practice, we recommend creating a separate project for Gmail logs. 

Step 2: Create a Custom Role for the Project

1. Open the GCP Console: https://console.cloud.google.com

2. In the left navigation menu, click IAM & Admin, and then click Roles.

3. Click Create Role at the top.

4. Fill in the following:

   • Title: (e.g., BigQueryLogReader)

   • ID: Auto-generated or customize.

   • Description: (e.g., Read access to Gmail logs in BigQuery)

5. Under Permissions, click Add Permissions.

6. Add the following permissions:

   • bigquery.datasets.get

   • bigquery.datasets.list

   • bigquery.tables.get

   • bigquery.tables.list

   • bigquery.tables.getData

   • bigquery.jobs.create

7. Click Add.

8. Click Create to save the custom role.

Step 3: Enable BigQuery API

1. Go to the BigQuery API Page within the Google Cloud Console.

2. Click Enable.

Step 4: Grant IAM Roles to Service Account

1. From the Google Cloud Console, go to IAM & Admin > IAM.

2. Click Grant Access at the top of the IAM table.

3. In the New Principals field, enter:

gapps-reports@system.gserviceaccount.com

4. Under Role, select BigQuery > BigQuery Data Editor.

5. Click Save.

Generate a Credential File

To collect Gmail Message Tracking logs with the Open Collector, a credentials file is required in the Beats configuration.

To configure and generate the credentials file:

1. Open the GCP Console.

2. Navigate to IAM & Admin > Service Accounts.

3. Click Create Service Account.

4. Enter a Name and an optional Description.

5. Click Create and Continue.

6. In the Grant access to this service account step, click Select a role.

7. Choose Custom, and then select the role created in Step 2.

8. Click Continue and then Done.

9. On the Service Accounts page, find your newly created account and click the three-dot menu, and then Manage keys.

10. Click Add Key, and then Create new key.

11. Select JSON and click Create.

12. The JSON key file is downloaded automatically.

Keep this JSON key file secure, as it contains API credentials used for authentication.

To export data to BigQuery, VPC service controls must be turned off (default setting) in the BigQuery project for Gmail. For more information, refer to Google’s VPC Service Controls documentation.

Assign Gmail Logs to a BigQuery Dataset

Gmail logs store records for each stage of a message in the Gmail delivery process. To analyze Gmail flow through the delivery process, assign Gmail logs to a dataset in a BigQuery project. After you assign Gmail logs, you can review reports.

To assign Gmail logs to a dataset in BigQuery:

1. Sign in to the Google Admin console using a super administrator account.

2. From the Admin console Home page, go to Reporting > Data integrations.

3. On the BigQuery Export card, click Edit.

4. Check the box for Enable Google Workspace data export to Google BigQuery.

5. (Optional) To export sensitive content from DLP rule logs, check Allow export of sensitive content from DLP rule logs.

6. Under BigQuery project ID, select the project where logs will be stored.
   Ensure gapps-reports@system.gserviceaccount.com has the Editor role in this project.

7. Under the New dataset within project, enter a unique name for the dataset (e.g., workspace_logs).

Google will automatically create the new dataset.

8. Click Save to apply the settings.

If you can’t save the project, go to the Google Cloud console, delete the new dataset, then save it again in the Admin console.

9. Google automatically creates the dataset in the selected BigQuery project within 24 hours.

10. You’ll start seeing logs under tables like:

activity_*
gmail_log_*

11. Once the dataset is created, go to BigQuery in GCP Console and:

    1. Confirm the dataset exists.

    2. Confirm gapps-reports@system.gserviceaccount.com has Editor access:
       BigQuery > your dataset > ⋮ (3-dot menu) > Share > confirm gapps-reports@system.gserviceaccount.com has Editor role.

    3. Confirm log tables start populating after ~24 hours.