Secret Detection Runbooks

This page contains answers to the general questions about the Secret Detection Service. This runbook can be used by anyone who want to understand the technical aspects about the service.

General FAQs

1. Where is the service deployed?

   The service is deployed on Runway which internally uses Google Cloud Run to manage containers.

2. In how many environments will the service be deployed?

   The service is deployed in Staging (https://secret-detection.staging.runway.gitlab.net) and Production (https://secret-detection.production.runway.gitlab.net).

When to use this runbook?

This runbook is intended to be used when monitoring the Secret Detection Service to identify and mitigate any reliability issues or performance regressions that may occur when it is enabled on Gitlab.com and/or Dedicated.

What to monitor?

We primarily need to monitor system metrics and recurrent errors raised within the service. Here are the narrowed down list of monitoring targets:

• Resource Saturation: Saturation is a measure of what ratio of a finite resource is currently being utilized.
• Aggregated Service Level Indicators(SLIs)
  • Apdex Score: Apdex is a measure of requests that complete within a tolerable period of time for the service.
  • Error Ratio: Error rates are a measure of unhandled service exceptions per second. Client errors are excluded when possible.
  • Request Rate: The operation rate is the sum total of all requests being handle for all components within this service. Note that a single user request can lead to requests to multiple components.
• Recurrent appplication errors raised by the service.

How to monitor the service?

Most of above-mentioned monitoring targets i.e. Resource Saturation and Aggregated SLIs, are available in the Service Overview Dashboard.

When to use this runbook?

This runbook is intended to be used when monitoring the secret push protection feature to identify and mitigate any reliability issues or performance regressions that may occur when it is enabled on Gitlab.com. The runbook can also be used to understand more about relevant dashboards below and how to improve them:

• Secret Push Protection – Overview Dashboard

What to monitor?

While the feature, in its current form, doesn’t have any external components and is entirely encapsulated within the application server as a dependency, it does interact with a number of components as can be seen in this push event sequence diagram. Those components are:

When to use this runbook?

Use this runbook for:

• Running GPT tests - for running tests and comparing with previous benchmarks
• Deploying a new version of GitLab to GET - for updating a GET instance, most likely to test out changes related to secret push protection
• Setting up a new GET environment - for testing different reference architectures

Prerequisites

• gcloud (official instructions) - for running various commands, and for logging in to the test runner vm
• The Static Analysis GCP Project (see Resources section) - access required to make changes to the infrastructure

Running GPT tests

Manual testing

Get the url and password for the root user from 1password by searching for Static Analysis in the Engineering Vault. Please don’t delete projects, groups, or users, but feel free to create any of those, or anything else you’d like to test with.

When to use this runbook?

Use this runbook for troubleshooting Production issues related to the secret push protection feature.

Relevant settings

Monitoring

Secret push protection monitoring is the preferred dashboard for monitoring the feature to help identify and mitigate any reliability issues or performance regressions that may occur when it is enabled on GitLab.com.