Security Department Performance Indicators

Executive Summary

KPI Health Status
Age of current open application and container vulnerabilities by severity Confidential
  • S1 open age has increased to the target level due to fixes for container scanner findings being unavailable in the base container, exception requests have been submitted and approved. Automation and process improvements are in progress to address the open age for S2 vulnerabilities, which currently exceeds the target threshold as a result of issues being created for additional container scanner findings. This KPI is currently considered healthy.
Security Engineer On-Call Page Volume Confidential
  • A 16% decrease in on-call pages was experienced over the last 3 months as compared to the last 12 months. Notably, this shows continued stability in the on-call page volume. The number of S1 paged incidents is zero over the last 3 months, compared to ~.5 S1 page per month over the last 12 months average. These volumes are well within the acceptable threshold and we attribute this stability to security improvements made through rapid action groups and working groups.
  • Short term fluctuations are to be expected. Long term trends should be identified and actions should be taken to correct negative trends and to continue promoting positive trends.
Security Control Risk by System Confidential
  • Security Compliance performs regular testing of controls by system and uses the System Risk Scoring methodology to determine the overall risk of each system in scope of GitLab’s compliance and regulatory programs. A system risk score of 1 means all controls evaluated for that system are operating fully effectively with no open observations and a higher system risk score means there are an increasing number of observations opened against that system that require remediation. 57% (or 8 out of 14) systems assessed met the target. There are 114 observations open for these 14 systems.
Security Impact on Net ARR Confidential
  • In FY24-Q2, Field Security observed a 74% increase in security influenced revenue compared to FY24-Q1.
Estimated Cost of Abuse Confidential
  • Confidential metric - See notes in Key Review agenda
Security Budget Plan vs Actuals Okay
  • This chart was recently updated and now reflective of current state. Security are within budget expectations.
Security Handbook MR Rate Attention
  • Security was highly focused on operations and hiring in FY23. Action = Security expect to see a marked increase in handbook updates once project work picks back up.
Security Team Member Retention Confidential
  • Action = Security leadership have created OKRs focused on addressing team member concerns contributing to attrition rates.
Security Average Age of Open Positions Problem
  • High demand for Security professionals has lead to an extremely competitive hiring market. Security leadership are actively involved in the recruiting processes to reduce burden on recruiters and shorten time to hire.
Security Department Discretionary Bonus Rate Attention
  • Security had observed a downward trend for bonus nominations recently. The previous action for Managers to actively encourage team members to submit nominations where appropriate was effective at bringing the metric back on target.
Security Incidents by Category Okay
  • While the total number of security incidents is 19% greater than the 12 month average over the last 3 months, all incident categories are within expected and acceptable thresholds. We believe this higher number of incidents is due to higher fidelity detection, alerting and improved security awareness.
  • Category labels are applied in accordance to the SIRT handbook page. In the event that "NotApplicable" keeps growing, we'll be adding additional categories.
Operational Security Risk Management (Tier 2 Risks) Okay
  • Security operational risk management enables organizations to proactively identify and mitigate operational security risks that may impact Organizational Output, Brand Reputation, Business Continuity, Customers & Stakeholders, Legal & Regulatory and/or Financials. This heatmap has been generated from ZenGRC. Numbers within each box indicate the total number of documented security operational risks. Red boxes indicate the risk level is HIGH. Orange boxes indicate the risk level is MODERATE. Green boxes indicate the risk level is LOW. The heatmap shows risks that are currently open and accepted, in remediation or planned for remediation. Security’s Annual Risk Assessment was completed in June and will be presented to the Board of Directors in their Q2 meeting. As part of the ARA and standard operational risk activities (such as risk treatment or trends in the IT environment), risk scores displayed in the heatmap are subject to change over time. Our reporting cadence will change from annual to quarterly in Q3 to drive risk treatment through iteration and transparency.
Security Observations (Tier 3 Risks) Okay
  • 71% (or 805 of 1130) of all observations opened to date have been resolved, 114 of the 805 were high-risk observations. Of observations identified this fiscal year, ~28% (or 46 of 167) have been resolved. 33 high-risk observations have been identified.
Third Party Risk Management Okay
  • TPRM has fielded 236 vendor intake requests fiscal year to date (5.4% YoY increase). These requests have resulted in 70 new third party assessments which is a -12.5% decrease YoY.
Security Automation Iteration Velocity Average Okay
  • This PI is considered healthy.
Security Department Promotion Rate Attention
  • Security have recently announced multiple promotions. Action = Security leadership are now executing Individual Development Plans as part of one on ones with a direct focus on career development.

Key Performance Indicators

Age of current open application and container vulnerabilities by severity

The age of current open vulnerabilities gives us an at the moment snapshot in time of how fast we are scheduling and fixing the vulnerabilities found post-Production deploy. The age is measured in days, and the targets for each severity are defined in the Security Handbook. For Security purposes, please view this chart directly in Sisense.

Target: Time to remediate

Health: Confidential

  • S1 open age has increased to the target level due to fixes for container scanner findings being unavailable in the base container, exception requests have been submitted and approved. Automation and process improvements are in progress to address the open age for S2 vulnerabilities, which currently exceeds the target threshold as a result of issues being created for additional container scanner findings. This KPI is currently considered healthy.

URL(s):

Security Engineer On-Call Page Volume

This metric is focused around the volume and severity of paged incidents to the Security Engineer On-Call. This data can be used to track and identify trends associated with disruption work, which if and when possible, should be minimized. For Security purposes, please view this chart directly in Sisense.

Target: Number of pages/month does not exceed +50% of monthly average of the last 12 months for 3 consecutive months

Health: Confidential

  • A 16% decrease in on-call pages was experienced over the last 3 months as compared to the last 12 months. Notably, this shows continued stability in the on-call page volume. The number of S1 paged incidents is zero over the last 3 months, compared to ~.5 S1 page per month over the last 12 months average. These volumes are well within the acceptable threshold and we attribute this stability to security improvements made through rapid action groups and working groups.
  • Short term fluctuations are to be expected. Long term trends should be identified and actions should be taken to correct negative trends and to continue promoting positive trends.

URL(s):

Security Control Risk by System

Security Compliance performs regular testing of controls for in scope systems and uses the System Risk Score methodology to determine the system risk of each. A System risk rating of 1 means all controls evaluated for that system are fully effectively (very low risk) and there are no open observations associated with that system and the higher the system risk score the more risk that particular system carries.

Target: Confidential

Health: Confidential

  • Security Compliance performs regular testing of controls by system and uses the System Risk Scoring methodology to determine the overall risk of each system in scope of GitLab’s compliance and regulatory programs. A system risk score of 1 means all controls evaluated for that system are operating fully effectively with no open observations and a higher system risk score means there are an increasing number of observations opened against that system that require remediation. 57% (or 8 out of 14) systems assessed met the target. There are 114 observations open for these 14 systems.

URL(s):

Security Impact on Net ARR

The Field Security organization functions as a sales and customer enablement team therefore a clear indicator of success is directly reflected in the engagement of their assessment services by Legal, Sales, TAMs and customers themselves. Assessment services include completing security questionnaires, participating in customer calls, creating and providing security documentation, and facilitating customer audits. The dashboard is calculated as (assessments completed by customer + total contract value by customer = monthly dollar security impact).

Target: Confidential

Health: Confidential

  • In FY24-Q2, Field Security observed a 74% increase in security influenced revenue compared to FY24-Q1.

URL(s):

Estimated Cost of Abuse

This metric tracks the estimated cost of abuse in terms of CI Compute Cost & Storage costs from blocked accounts. It also includes aggregated Networking Cost data when it is over the baseline spend for known skus that only trend up during periods of elevated abuse, although this is not tracked at the user level. It does not include reputation damage costs or labor costs of having to manually prevent certain types of abuse.

Target: less than $10K/Month

This KPI cannot be public

Health: Confidential

  • Confidential metric - See notes in Key Review agenda

URL(s):

Security Budget Plan vs Actuals

We need to spend our investors' money wisely. We also need to run a responsible business to be successful. For Security purposes, please view this chart directly in Sisense. Latest data is in Adaptive, data team importing to Sisense in FY22Q2

Target: See Sisense for target

Health: Okay

  • This chart was recently updated and now reflective of current state. Security are within budget expectations.

URL(s):

Security Handbook MR Rate

The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/security` over time.

Target: 1

Health: Attention

  • Security was highly focused on operations and hiring in FY23. Action = Security expect to see a marked increase in handbook updates once project work picks back up.
Chart (Sisense↗)

Security Team Member Retention

We need to be able to retain talented team members. Retention measures our ability to keep them sticking around at GitLab. Team Member Retention = (1-(Number of Team Members leaving GitLab/Average of the 12 month Total Team Member Headcount)) x 100. GitLab measures team member retention over a rolling 12 month period.

Target: at or above 84%

This KPI cannot be public

Health: Confidential

  • Action = Security leadership have created OKRs focused on addressing team member concerns contributing to attrition rates.

URL(s):

Security Average Age of Open Positions

Measures the average time job openings take from open to close. This metric includes sourcing time of candidates compared to Time to Hire or Time to Offer Accept which only measures the time from when a candidate applies to when they accept.

Target: at or below 50 days

Health: Problem

  • High demand for Security professionals has lead to an extremely competitive hiring market. Security leadership are actively involved in the recruiting processes to reduce burden on recruiters and shorten time to hire.
Chart (Sisense↗)

Security Department Discretionary Bonus Rate

The number of discretionary bonuses given divided by the total number of team members, in a given period as defined. This metric definition is taken from the People Success Discretionary Bonuses KPI.

Target: at or above 10%

Health: Attention

  • Security had observed a downward trend for bonus nominations recently. The previous action for Managers to actively encourage team members to submit nominations where appropriate was effective at bringing the metric back on target.
Chart (Sisense↗)

Security Incidents by Category

The metric groups security incidents by incident category and provides visibility into possible trends in attack types or targeted systems. Tracking of this metrics allows GitLab to adjust security control strategies, identify opportunities for improvements, and address security controls needing attention. For Security purposes, please view this chart directly in Sisense.

Target: Number of security incidents in any category does not exceed +50% of the individual category's 3-month average.

Health: Okay

  • While the total number of security incidents is 19% greater than the 12 month average over the last 3 months, all incident categories are within expected and acceptable thresholds. We believe this higher number of incidents is due to higher fidelity detection, alerting and improved security awareness.
  • Category labels are applied in accordance to the SIRT handbook page. In the event that "NotApplicable" keeps growing, we'll be adding additional categories.

URL(s):

Operational Security Risk Management (Tier 2 Risks)

Operational risk management enables organizations to proactively identify and mitigate operational security risks that may impact Organizational Output, Brand Reputation, Business Continuity, Customers & Stakeholders, Legal & Regulatory and/or Financials. This heatmap has been generated from ZenGRC. Numbers within each box indicate the total number of potential risks. Red boxes indicate the risk level is HIGH. Orange boxes indicate the risk level is MODERATE. Green boxes indicate the risk level is LOW. The heatmap shows risks that are currently open and accepted, in remediation or planned for remediation.

Target: TBD, this will be determined upon Sisense integration for detailed dashboarding

Health: Okay

  • Security operational risk management enables organizations to proactively identify and mitigate operational security risks that may impact Organizational Output, Brand Reputation, Business Continuity, Customers & Stakeholders, Legal & Regulatory and/or Financials. This heatmap has been generated from ZenGRC. Numbers within each box indicate the total number of documented security operational risks. Red boxes indicate the risk level is HIGH. Orange boxes indicate the risk level is MODERATE. Green boxes indicate the risk level is LOW. The heatmap shows risks that are currently open and accepted, in remediation or planned for remediation. Security’s Annual Risk Assessment was completed in June and will be presented to the Board of Directors in their Q2 meeting. As part of the ARA and standard operational risk activities (such as risk treatment or trends in the IT environment), risk scores displayed in the heatmap are subject to change over time. Our reporting cadence will change from annual to quarterly in Q3 to drive risk treatment through iteration and transparency.
Chart (Sisense↗)

Security Observations (Tier 3 Risks)

An indicator of information system and process risk, there are multiple inputs that lead to identification of Observations to include Security Compliance continuous control testing, third party (vendor) assessments, external audits and customer assessments.

Target: Confidential

Health: Okay

  • 71% (or 805 of 1130) of all observations opened to date have been resolved, 114 of the 805 were high-risk observations. Of observations identified this fiscal year, ~28% (or 46 of 167) have been resolved. 33 high-risk observations have been identified.
Chart (Sisense↗)

Third Party Risk Management

An indicator of third party risk, third party risk assessments proactively identify potential vendor security risks as part of onboarding or contracting, enabling business owners to make risk based decisions throughout the vendor lifecycle.

Target: TBD, this will be determined upon Sisense integration for detailed dashboarding

Health: Okay

  • TPRM has fielded 236 vendor intake requests fiscal year to date (5.4% YoY increase). These requests have resulted in 70 new third party assessments which is a -12.5% decrease YoY.
Chart (Sisense↗)

Security Automation Iteration Velocity Average

We attempt to complete 7 weighted issues or more every two weeks. The measurement indicates how well the Security Automation team is scoping iterations over the last 4 biweekly iterations and provides a view of the average team velocity.

Target: 7

Health: Okay

  • This PI is considered healthy.
Chart (Sisense↗)

Security Department Promotion Rate

The total number of promotions over a rolling 12 month period divided by the month end headcount. The target promotion rate is 12% of the population. This metric definition is taken from the People Success Team Member Promotion Rate PI.

Target: 12%

Health: Attention

  • Security have recently announced multiple promotions. Action = Security leadership are now executing Individual Development Plans as part of one on ones with a direct focus on career development.
Chart (Sisense↗)

Legends

Health

Value Level Meaning
3 Okay The KPI is at an acceptable level compared to the threshold
2 Attention This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1 Problem We'll prioritize our efforts here
-1 Confidential Metric & metric health are confidential
0 Unknown Unknown

How pages like this work

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property Type Description
name Required String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path Required Relative path to the performance indicator page that this (K)PI should live on
definition Required refer to Parts of a KPI
parent Optional should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target Required The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org Required the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section Optional the product section (Ex: dev) as defined in sections.yml
stage Optional the product stage (Ex: release) as defined in stages.yml
group Optional the product group (Ex: progressive_delivery) as defined in stages.yml
category Optional the product group (Ex: feature_flags) as defined in categories.yml
is_key Required boolean value (true/false) that indicates if it is a (key) performance indicator
health Required indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
health.level Optional indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI.
health.reasons Optional indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls Optional list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel Optional indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data Optional allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart Optional indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard Optional indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard Optional indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed Optional indicates the Sisense embed version. For example: v2
sisense_data_secondary Optional allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart Optional Same as sisense_data.chart
sisense_data_secondary.dashboard Optional Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard Optional Same as sisense_data.shared_dashboard
sisense_data_secondary.embed Optional Same as sisense_data.embed
public Optional boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type Optional indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type Optional indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary Optional boolean flag that indicates if this is the Primary PI for the Product Group.
implementation Optional indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
implementation.status Optional indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI.
implementation.reasons Optional indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons Optional indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
lessons.learned Optional learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus Optional indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
monthly_focus.goals Optional indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name Optional indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.
Last modified September 6, 2023: Fix markdown errors and broken links in security (9c30d052)