Configure Gmail Message Tracking

Set Up Project and Roles for Gmail Message Tracking Beat

This section provides instructions for creating a project you want to use for Gmail logs. If you have already done this, skip to Create a Role for a Project.

Best practice is to create a separate project for Gmail logs. 

1. Open the GCP Console: https://console.cloud.google.com
2. Create a project for Gmail logs.

Create a Role for a Project

To create a custom role to fetch logs from BigQuery:

1. Open the GCP Console: https://console.cloud.google.com
   
2. On the left-side menu, click IAM & Admin, and then click Roles.
3. In the center pane, click Create Role.
4. Enter the basic information requested for the role.
5. Click Add Permissions and select the following permissions:
   • bigquery.datasets.get
   • bigquery.jobs.create
   • bigquery.tables.get
   • bigquery.tables.getData
   • bigquery.tables.list
     
6. Click Add, and then click Create.

Generate a Credential File

To collect GMT logs with Beats, a credentials file is required in the Beat configuration.

To configure and generate the credentials file:

1. Open the GCP Console: https://console.cloud.google.com

2. On the left-side menu, click IAM & Admin, and then click Service Accounts.

3. On the top menu bar, click Create Service Account.

4. Enter a service account name and an optional description, and then click Create.
5. In the Role field, select Custom, and then select the Custom role you created above.
   

   

6. Click Continue, and then click Create Key.
   A Create key panel appears.
7. Select JSON, and then click Create.
8. After the required credentials file is downloaded in JSON, click Done.

To export data to BigQuery, VPC service controls must be turned off (default setting) in the BigQuery project for Gmail.

Assign Gmail Logs to a BigQuery Dataset

Gmail logs store records for each stage of a message in the Gmail delivery process. To analyze Gmail flow through the delivery process, assign Gmail logs to a dataset in a BigQuery project. After you assign Gmail logs, you can review reports.

To assign Gmail logs to a dataset in BigQuery:

1. Open the Google Admin console: https://admin.google.com
2. From the home page, click Apps, G Suite, Gmail, and then click Setup.
3. Scroll to and click Email Logs in BigQuery.
4. Click Enable.
5. In the Add setting window, enter a description.
6. Select the BigQuery project you want to use for Gmail logs. Make sure to select a project with write access.
7. Enter a data set name where the Gmail logs are stored.
   

   

8. Click Save.

   

   If an error occurs, try to add the setting again. You might need to go to the BigQuery console and remove the previously created dataset.

9. After saving settings, go back to the BigQuery project in the GCP Console.
   A dataset with the following information is now in the project:
   • Three standard roles:
     • Project owners
     • Project editors
     • Project viewers
   • Four service accounts that are designated dataset editors: 
     • gmail-for-work-logs-writer1@system.gserviceaccount.com. Writes the logs. 
     • gmail-for-work-logs-writer2@system.gserviceaccount.com. Writes the logs.
     • gmail-for-work-logs-recoverer@system.gserviceaccount.com. Automatically restores the template table if it is accidentally removed.
     • gmail-for-work-logs-schema-updater@system.gserviceaccount.com. Updates the schema in the future.
     

     Do not remove these service accounts or change their roles. These are required accounts.

10. To verify that these service accounts are added, point to the new dataset, and then click Down next to the dataset name.

11. Click Share dataset.
    Daily email logs are now exported to BigQuery. It can take up to an hour for your changes to take effect.

    

    Email logs created before this procedure is completed cannot be exported to BigQuery.

daily_ Table

After you turn on email logs in BigQuery, a new table named daily_ is added to the dataset. Note the following:

• This table is always empty and never expires.
• This table is a template that provides the schema for the daily tables.
• After you create the daily_ template, daily tables are automatically created in your dataset. The logs are then available for use.
• Do not remove, modify, rename, or add data to this table.
• Use GMT time to determine the date change line.

Create Gmail Log Queries

You can compose your own custom queries using the schema for Gmail logs in BigQuery to fetch the metadata logs from BigQuery. For information on how to do this, see the documentation from Google: https://support.google.com/a/answer/7230050.

Verify Outputs

After following the procedures on this page and before moving on, ensure that you have the following three items, which are required to configure the Beat:

1. Project Id. The Project Id is associated with every project. You can also find this in your credential file { "project_id": "gmtlogsautomation-test" },).
2. JSON credentials file
3. Dataset Name. Make sure you have the dataset name, such as gmail_logs_dataset or gmail_logs_dataset_*