Configure the PubSub Beat

Collect Account Information from the GCP Console

The following information is required to configure Google for Beats:

• Project ID

• Credential file for a particular GCP account

• Topic Name

• Subscription Name (also referred to as Subscription ID)

Create a Role For a Project 

1. Open the GCP Console: https://console.cloud.google.com.

2. On the left-side menu, click IAM & Admin, and then click Roles.

3. In the center pane, click Create Role.

4. Enter the basic information requested for the role.

5. Click Add Permissions and select the following permissions:

   • pubsub.subscriptions.consume

   • pubsub.subscriptions.get

   • pubsub.subscriptions.create

   • pubsub.topics.attachSubscription

   If you have set subscription.create: false in pubsubbeat.yml file (the default value for subscription.create is true) and you are using an existing subscription, you can add the following permissions:

   • pubsub.subscriptions.consume

   • pubsub.subscriptions.get

   • pubsub.subscriptions.update

6. Click Add, and then Create.

Generate a Credential File

1. Open the GCP Console: https://console.cloud.google.com.

2. On the top menu bar, select the project to collect logs from.

   

The Select a project dialog box appears

3. Make a note of the Project ID.

   

4. From the left-side menu, click APIs & Services, and then click Credentials.

5. In the center pane, click Create credentials, and then click Service account.

6. In the Create service account section, enter a service account name and optional description.

7. In the Permissions section, select the role you created.

8. Click Done.

9. Locate your service account on the Service accounts page, click the vertical ellipsis (⋮) at the end of the row, and select Manage Keys

10. Click Add key, then Create new key, then select Key type: JSON

11. Click Create.

12. Save the file somewhere you can easily access it.

Configure a Topic and Subscription 

1. In the GCP console, select the project in which you want to create a topic.

2. Expand the main menu on the left and scroll to the Big Data section. 

3. Click Pub/Sub, and then click Topics.

4. On the top of the center pane, click Create Topic.

   If you already have a topic configured, you can skip these steps. Make a note of the Topic ID you want to use.

5. In the Topic ID field, type a unique name.
   Make a note of this Topic ID.

6. Click Create Topic.

   

7. In the topics grid view, click the three-dot (...) menu to the right of the topic to which you want to add a subscription.

8. Click Create subscription.

9. In the Subscription ID field, type a unique name.

10. Make a note of this Subscription ID.

11. Configure the subscription with the default settings, and then click Create.

Log Routing Architecture

This section describes how to capture, filter, and export logs from any Google Cloud service (including Vertex AI, Model Armor, and Cloud Discovery Engine) into a Pub/Sub topic.
Depending on your organizational requirements, you can choose between a Single-Project or Multi-Project architecture.

Architecture Options

Option 1: Single-Project Architecture

Use this when the Application, the Pub/Sub Topic, and the Log Sink all reside in the same project.

Step 1: Create the Pub/Sub Topic

1. Navigate to Pub/Sub in the Google Cloud Console.

2. Click Create Topic.

3. Enter a unique Topic ID (e.g., app-logs-internal).

4. Click Create.

Step 2: Create the Log Sink

1. Navigate to Logging > Log Router.

2. Click Create Sink.

3. Sink Name: Enter a unique name (e.g., app-to-pubsub-sink).

4. Sink Destination: Select Pub/Sub and select the Topic created in Step 1.

5. Inclusion Filter: Add appropriate filters or leave it blank to fetch all ingested logs.

6. (Optional.) Choose logs to filter out of sink.

7. Click Create.

Step 3: Grant Permissions

1. Go to the Pub/Sub page, select your Topic, and click the vertical ellipsis (⋮) at the end of the row.

2. Click View Permissions > Add Principle.

3. New Principals: Paste the Writer Identity email from the Log Router (find it in the "View Sink Details" of the sink you just created; it should look like this: service-XXXXXXXXXX@gcp-sa-logging.iam.gserviceaccount.com).

4. Role: Select Pub/Sub Publisher.

5. Click Save.

Option 2: Multi-Project Architecture

Use this when the Application is in one project, but the Pub/Sub Topic is in a dedicated "Logging" or "Security" project.

Step 1: Create the Topic in the Logging Project

1. Navigate to the Logging Project in the Google Cloud Console.

2. Navigate to Pub/Sub and create a new Topic.

3. Note the Topic ID.

Step 2: Create the Sink in the Source Project

1. Navigate to the Source Project (where the Google Cloud Services lives).

2. Navigate to Logging > Log Router.

3. Click Create Sink.

4. Sink Name: Enter a unique name.

5. Sink Destination: Select Pub/Sub. Select the Topic created in the Logging Project.

6. Inclusion Filter: Add appropriate filters or leave it blank to fetch all ingested logs

7. (Optional) Choose logs to filter out of sink

8. Click Create.

Step 3: Grant Cross-Project Permissions

Because the data is moving between two different projects, you must authorize the Source Project to write to the Logging Project:

1. Go to the Logging Project and select the Topic.

2. Click Permissions > Grant Access.

3. New Principals: Paste the Writer Identity email from the Sink created in the Source Project (Should look like service-XXXXXXXXXX@gcp-sa-logging.iam.gserviceaccount.com).

4. Role: Select Pub/Sub Publisher.

5. Click Save.

Export Logs with the Log Viewer

1. In the GCP console, from the left-side menu, scroll to the Operations section.

2. Click Logging, and then click Logs Router.

3. In the center pane, select an existing GCP project.
   The following screenshot shows an example of the Logs Router page, where several log sinks have already been configured:

   

Logs Router User Interface

The Logs Router interface contains a table summary of exports. Each table row contains information corresponding to some of the sink properties described in Logs Exports:

• Sink Name. The sink's identifier in the current project.

• Destination. The location where the exported log entries are written.

• Writer Identity. The service account that logging uses to write log entries to the destination. This service account must have permission to write to the export destination.

• Creation Time. The timestamp of the sink when it was created.

Each row has a menu on the right with the following options:

• Edit Sink. Opens an Edit Sink pane where you can change the sink's parameters.

• Delete Sink. Allows you to delete the sink and stop the logs export.

• View Filter. Displays the sink's query. Click Edit to change the sink's properties or query.

The search-query box above the table allows you to query your sinks by text search, or by Sink Name, Destination, or Writer Identity. For example, the following shows a search on Destination: bigquery with options to concatenate other sink properties using the argument OR (AND is the default):

Create a Log Sink at the Organization Level and the Project Level 

Instead of creating a log sink on each and every service, you can create a log sink at the organization and project level. This enables you to automatically redirect logs from all resources in that project to a desired destination.

For more information, see https://cloud.google.com/logging/docs/export/aggregated_sinks .

Configure Gemini Enterprise Model Armor Logs via PubSub

Prerequisites

• IAM role roles/modelarmor.admin assigned to your account on the GCP project.

• Model Armor API enabled:

  

  OR

  gcloud services enable modelarmor.googleapis.com --project=YOUR_PROJECT_ID
  

Step 1: Create a Model Armor Template

1. In the GCP Console, navigate to Security → Model Armor.

2. Click Create template.

3. Specify a Template ID and select a Region.
   Make a note of the region; your Gemini Enterprise app must be created in the same region.

4. Choose Stable for Filter Versions

5. In the Detections section, enable Prompt injection and jailbreak detection, Malicious URL Detection, and Sensitive data protection based on your requirements.

6. Under Responsible AI, configure content filters based on your requirements.

7. In Configure logging, select Prompts and responses.

8. Click Create.

9. After creation, open the template details page. In the Resource name field, click the copy icon to copy the full resource path.
   Save this value; you will need it in the next step.

Step 2: Create the Gemini Enterprise App

1. In the GCP Console, navigate to Gemini Enterprise (Agent Builder / Vertex AI Agent Platform).

2. Click Create app.

3. Enter an App name and ensure the selected Location matches the region you used for the Model Armor template.

4. Click Create.

5. Once the app is created, open Configurations → Assistant tab.

6. Enable the option Enable Model Armor.

7. In the Gemini Model Armor template field, paste the resource name path you copied in Step 1.

8. Complete any remaining required configurations.

9. Click Save and publish.

Step 3: Record the App ID

1. Navigate to the Apps page in Gemini Enterprise / Agent Builder.

2. Locate your newly created app in the list.

3. Copy the value in the ID column.
   Make a note of this App ID; you will need it when configuring the PubSub beat subscription.

Step 4: Enable Data Write Audit Logs for Cloud Discovery Engine

By default, only Admin Activity audit logs are enabled. Gemini Enterprise app interactions are captured under the Cloud Discovery Engine API and require Data Write logging to be turned on.

1. In the GCP Console, navigate to IAM & Admin → Audit Logs.

2. Search for Cloud Discovery Engine API in the service list.

3. Enable the Data write audit log type.

4. Click Save.

Step 5: Verify Model Armor Logs in Cloud Logging

In Cloud Logging → Logs Explorer, confirm events are flowing:

resource.labels.template_id="Endpoints-ModelArmor-Template"

Configure Gemini Enterprise Agent Space (Vertex AI Agent) Logs via Pubsub

Step 1: Enable APIs on Agent Platform

In the GCP Console, go to Agent Platform and click on Enable APIs:

Step 2: Enable Data Access Audit Logs (Crucial)

By default, only Admin Activity audit logs are enabled. To also capture DATA_READ and DATA_WRITE events for Vertex AI:

1. In the GCP Console, navigate to IAM & Admin → Audit Logs.

2. Find Vertex AI API in the service list.

3. Enable the Data Read and Data Write checkboxes.

4. Click Save.

Step 3: Verify Agent Space Logs in Cloud Logging

In Cloud Logging → Logs Explorer, confirm events are flowing:

protoPayload.serviceName="aiplatform.googleapis.com"

Configure a Combined Log Router Sink for Model Armor and Agent Space

Create a new, dedicated PubSub topic and subscription for this sink rather than reusing an existing one. Adding Model Armor and Agent Space logs, especially the broad logName:”cloudaudit.googleapis.com” clause, to an existing topic will inflate its message volume, mix incompatible log shapes, and may cause partial ingestion if other Beat instances share the same subscription.

Next Steps

Follow the topic and subscription creation steps in Configure a Topic and Subscription above, then create a new Log Router sink pointing to that topic using the inclusion filter below. Follow the sink creation steps in Configure and Export Logs for Multiple Projects in One Topic as a reference.

Combined Inclusion Filter

Paste this filter string into the Inclusion Filter field when creating or editing your Log Router sink:

(
  resource.type="audited_resource"
  AND protoPayload.serviceName="aiplatform.googleapis.com"
  AND protoPayload.resourceName=~"<yourappname>"
)
OR
(
  resource.type="audited_resource"
  AND resource.labels.service="discoveryengine.googleapis.com"
  AND resource.labels.method="google.cloud.discoveryengine.v1main.AssistantService.StreamAssist"
  AND protoPayload.resourceName=~"<yourappname>"
)
OR
(
  resource.type="modelarmor.googleapis.com/SanitizeOperation"
  AND jsonPayload.operationType="SANITIZE_USER_PROMPT"
  AND labels."modelarmor.googleapis.com/client_correlation_id"=~"<yourappname>"
)

Optional. Independent Log Streams (Recommended for High-Volume/Security Teams)

For large-scale deployments, we recommend routing Model Armor and Agent Space logs to separate Pub/Sub topics. This ensures that security guardrail events (Model Armor) do not mix with conversation and reasoning data (Agent Space), allowing for cleaner monitoring and distinct alerting rules.

Option 1: Security & Guardrail Stream (Model Armor)

Use this filter for monitoring PII leaks, prompt injections, and safety violations.

(resource.type="audited_resource" 
 AND (resource.labels.service="discoveryengine.googleapis.com" 
      OR resource.type="modelarmor.googleapis.com/SanitizeOperation")
 AND protoPayload.resourceName=~"<yourappname>")
OR
(resource.type="modelarmor.googleapis.com/SanitizeOperation"
 AND jsonPayload.operationType="SANITIZE_USER_PROMPT"
 AND labels."modelarmor.googleapis.com/client_correlation_id"=~"<yourappname>")

Option 2: Agent Reasoning & Interaction Stream (Agent Space)

Use this for monitoring how the agent is responding, which tools it is calling, and its reasoning steps.

When monitoring the Agent Space (Vertex AI), you will see two primary types of interactions. Depending on your needs, you may want to filter for one or both.

GenerateContent (The Conversation)

This is the actual "Intelligence" of the agent. It captures the prompt sent by the user and the response generated by the Gemini model.

Use this filter to audit what the AI is saying, to debug why an agent is giving a wrong answer, or to provide "History" to your users.

resource.type="audited_resource"
AND protoPayload.serviceName="aiplatform.googleapis.com"
AND protoPayload.methodName=~"GenerateContent"
AND protoPayload.resourceName=~"<yourappname>"

CountTokens (The Pre-flight Check)

Before the AI generates a response, the system calculates the "Token Count" to ensure the request fits within the model's limits.

Use this filter for Cost Monitoring and Input Validation. If a user sends a prompt that is too long, this log will show you how many tokens were attempted.

resource.type="audited_resource"
AND protoPayload.serviceName="aiplatform.googleapis.com"
AND protoPayload.methodName=~"CountTokens"
AND protoPayload.resourceName=~"<yourappname>"