Troubleshooting

Information about monitoring and logging tools of Flows.

Tools

Duo Flows uses the following logging and monitoring tools:

  1. LangSmith - collects logs scoped to underlaying graph execution, including information like: LLM completions or tool calls
  2. Google Cloud (GCP) logs explorer
  3. Sentry error tracking collects error traces for:
    1. Duo Workflow Service
    2. Duo Workflow Executor
  4. Runway monitoring dashboard - this a grafana dashboard that tracks hardware resource consumption for Duo Workflow Service
  5. Tableau dashboard for internal events tracking - displays aggregated data collected with internal event tracking, showing additional product metrics like total number of workflows, or distribution between differnt workflow outcomes

Google Cloud (GCP) Logs explorer

Following projects hold logs for different pieces of Runway deployments:

  1. gitlab-runway-production - holds logs for production runway deployments
  2. gitlab-runway-staging - holds logs for staging runway deployments

When browsing Runway logs, you can narrow scope to piece of infrastructure that is of interest to you using following filters:

  1. To filter only load balancer logs use:

    resource.type="http_load_balancer"
resource.labels.forwarding_rule_name="duo-workflow-https"

  2. To filter only Duo Workflow Service deployment logs use:

    resource.labels.service_name="duo-workflow-svc"

gRPCurl

The grpcurl is a cli tool that enables you to interact with gRPC servers just like curl does for http ones.

An example usage of grpcurl for Agent Foundations is shown in the example below:

  1. Agent Foundations credentails can be obtained via curl
curl -X POST -H "Authorization: Bearer $GITLAB_API_PRIVATE_TOKEN" https://gitlab.com/api/v4/ai/duo_workflows/direct_access
  1. With credentials assigned to environment variables grpcurl can be used to start bidirectional channel to Duo Workflow Service
grpcurl -keepalive-time 20 -H "x-gitlab-global-user-id":"$GLOBAL_USER_ID" \
   -H "x-gitlab-instance-id":"ea8bf81......." -H "x-gitlab-realm":"saas" \
   -H "x-gitlab-authentication-type":"oidc" \
   -H authorization:"bearer $GRPC_TOKEN" -d @ -vv -proto ../duo-workflow-service/contract/contract.proto
   -import-path ../duo-workflow-service/contract cloud.gitlab.com:443 DuoWorkflow/ExecuteWorkflow


Resolved method descriptor:
rpc ExecuteWorkflow ( stream .ClientEvent ) returns ( stream .Action );

Request metadata to send:
authorization: bearer eyJhbGc.....
x-gitlab-authentication-type: oidc
x-gitlab-global-user-id: Rf9.........
x-gitlab-instance-id: ea8bf810-..........
x-gitlab-realm: saas
  1. With channel being established messages can be sent via stdin
{
  "startRequest": {
    "workflowID": "12344",
    "goal": "create hello world in go",
    "workflowMetadata":  "{\"extended_logging\":true,\"git_sha\":\"e621c52bb0f3af0a102a06cf2e485aa961f60d8c\",\"git_url\":\"gitlab.com/gitlab-org/analytics-section/analytics-instrumentation/metric-dictionary.git\"}"
  }
}

Enhanced Logging for Team Members

Enabling Enhanced Logging

To enable enhanced logging for better troubleshooting, use the following Slack command within the #production slack channel:

/chatops run feature set duo_workflow_extended_logging --user=your_user_name true

Replace your_user_name with your actual GitLab username.

This will enable detailed tracing in LangSmith, which provides the most comprehensive view of workflow execution including LLM completions, tool calls, and execution flow.

Important Privacy and Security Considerations

⚠️ Data Privacy Warning: Extended logging captures detailed workflow execution data in LangSmith, including LLM completions, tool calls, prompts, and model responses.

  • NO RED DATA: Do not use Agentic Chat with RED data when the duo_workflow_extended_logging feature flag is enabled
  • Forward-only logging: This feature only logs new interactions after it’s enabled. If you experienced an issue before enabling the flag, you’ll need to reproduce the issue after enabling the enhanced logging
  • Access restrictions: Only GitLab AI Engineering team members have access to the LangSmith logs for troubleshooting purposes

Getting Your Workflow ID

When reporting issues or requesting assistance, you’ll need to provide your workflow_id. Here’s how to find it in different interfaces:

The following is an example log output from Agentic Chat:

2025-08-29T18:27:28:088 [debug]: [DuoWorkflowNodeExecutor][1234567] Running workflow: {
        "goal": "This is an example question for Agentic Chat to show what happens in the logs",
        "type": "chat",
        "existingWorkflowId": "1234567",
        "metadata": {
            "projectId": "7654321",
            "namespaceId": "7654321"
        },
        "additionalContext": [],
        "workspaceFolderPath": "/Users/johnsmith/dev/handbook",
        "workflowId": "1234567",
        "connectionType": "grpc",
        "enabledFeatureFlags": [
            "duo_workflow_work_item_tools",
            "duo_workflow_prompt_registry",
            "use_duo_context_exclusion"
        ]
    }

The workflow ID can be found in the workflowId field.

From IDEs (VS Code, JetBrains, etc.)

  1. Open the Duo Chat panel in your IDE
VS Code
  1. Enable Debug Logging - Under the GitLab Workflow settings menu enable debugging by checking the Gitlab: Debug box
  2. Ask Agentic Chat a question
  3. Open the GitLab Language Server log outputs in the VSCode Output panel.
    1. Search for workflowId in the Output panel.
      • Ensure that the goal matches the question you were asking.
JetBrains IDEs
  1. Enable Debug Logging - In the GitLab Duo tools setting menu under Advanced expand the GitLab Language Server section set the Logging->Log Level setting to debug
  2. Ask Agentic Chat a question
  3. Open the idea.log log file in your preferred log viewer
    1. Search for workflowId in the logs.
      • Ensure that the goal matches the question you were asking.

From Web Interface (GitLab.com)

  1. Go to your GitLab project
  2. Open the Duo Chat interface (chat icon in the sidebar or top navigation)
  3. Open browser dev tools (F12) → Network tab → look for GraphQL requests
  4. Ask Agentic Chat a question
  5. Open the graphql request
    1. Verify the goal in the variable section of the Payload of the GraphQL call matches the question you asked agentic chat.
    2. Switch to the Response tab
    3. The workflow ID will follow the gid://gitlab/Ai::DuoWorkflows::Workflow/ portion of the id (see example response below)
{
    "data": {
        "aiDuoWorkflowCreate": {
            "workflow": {
                "id": "gid://gitlab/Ai::DuoWorkflows::Workflow/1234567",
                "__typename": "DuoWorkflow"
            },
            "errors": [],
            "__typename": "AiDuoWorkflowCreatePayload"
        }
    }
}

The workflow ID in this example is 1234567

What to Share with AI Engineers

When requesting assistance, please provide:

  • Workflow ID: Essential for tracing the specific execution
  • Expected vs. Actual behavior: What you expected to happen vs. what actually happened
  • Steps to reproduce: If the issue is reproducible
  • Timestamp: When the issue occurred (helps narrow down logs)
  • Any error messages: Screenshots or copied text of error messages

Tips and tricks

A typical investigation around problematic Agent Foundations execution follows steps listed below:

Based on a user report:

  1. Ask the user for the workflow_id for the problematic workflow which is displayed in the list of workflows
  2. Use the workflow_id from previous step to filter down langsmith traces by applying a filter for metadata and thread_id=[workflow_id]
  3. Use the workflow_id from 1st step to filter down logs in gcp logs explorer jsonPayload.workflow_id="123456789"

Based on a Sentry issue:

  1. Use Agent Foundations Sentry issue to locate problematic workflow’s correlation_id.
  2. Use the correlation_id from previous step to filter down logs in gcp logs explorer, example filter: jsonPayload.correlation_id="e7171f28-706d-4a47-be25-29d9b3751c0e"

In addition one can use a workflow’s workflow_id that is being recorded either in sentry or in log explorer to filter down LangSmith logs using thread_id filter in metadata and comparing it against workflow_id.

Past in depth investigations

  1. Faulty network proxy via Cloudflare investigation issue
Last modified September 8, 2025: Add LangSmith Logging Setup Steps (7d8ef5d6)
View page source - Edit this page - please contribute. Creative Commons License