Product Customer Satisfaction (CSAT) Survey

• Within the Product division, we have adopted Customer Satisfaction Score (CSAT) and are conducting this survey on a quarterly basis.
  • We previously surveyed customers using the NPS, which we moved away from starting in FY25 Q1 (see proposal deck).
• We are using CSAT because it allows us to:
  • Measure satisfaction directly vs. indirectly.
  • Connect satisfaction ratings and open ended responses back to changes in our product.
  • Compare and contrast CSAT survey data against our other company wide metrics (System Usability Scale (SUS)).

How the Product CSAT survey is run

UX Research determines our Product CSAT score for paid users of GitLab.com, self-managed GitLab, and GitLab Dedicated on a quarterly basis through a survey launched through Qualtrics. A CSAT template issue is created by the UX Research DRI at the beginning of each quarter. The issue template contains background information, research goals, and processes on conducting the survey from start to finish. Data is collected over a period of four weeks starting in the beginning of a quarter and the survey stays open until we have met our data collection goals. The next two weeks after data collection are used to clean, analyze, and report on the survey responses. All documents created are stored in an internal only Google Drive within UX Research.

Current workflow

Sampling goals

UX Research aims for a sample size of 400 total responses with plan type proportions for our sample to be +/- 3% compared to the population proportion. Self-managed users are excluded from this proportion because we’ve traditionally had difficulty recruiting them for research. Thus, we aim for 80-100 responses from self-managed users, which is based on the number of responses to past System Usability Scale surveys.

Data analysis

Data analysis consists of two components: quantitative analysis and open response coding.

1. Quantitative analysis: The majority of the quantitative analysis is accomplished by using the analysis template. The final cleaned data inserted in the completed tab automatically updates information in the summary tab, which shows the overall results. From there, the UX Research DRI can transpose the results from the summary tab and insert them into the CSAT Google Slide deck template.
2. Open response coding: The UX Research DRI will review the open response feedback left by CSAT respondents in the open ended feedback tab of the analysis template and categorize the data into a list of pre-defined themes (see theme glossary tab for descriptions and examples for each theme).

Deployment

Generating a list of eligible users

At the beginning of each quarter, the UX Research DRI will generate a list of eligible users to distribute the CSAT survey through the following steps:

1. Generate a new list of paid GitLab.com users to contact by using the following query in Snowflake. Query for approximately 22,000 user IDs and email addresses. Note: You will need access to Snowflake and SAFE data to query the data tables in this step. If you do not have access to one or both, please create an Access Request issue.
2. Contact the Research Operations Coordinator through a recruiting request issue template to generate a list of paid self-managed GitLab users.
3. Contact Marketing Analytics in the #mktg-analytics channel in Slack to get help with generating a list of GitLab Dedicated users.
4. Create a list of users you’ve compiled in steps 1-3 within the CSAT user Google Sheet template to track contacts for the quarter.
5. Remove users that were contacted more than 12 months ago from the list of previously contacted users.
6. Create a copy of the sheet containing the users you will contact this quarter and insert it in the CSM / UX Survey Participants Google Drive Folder, so Customer Success can avoid contacting the same users in their CSAT survey.
7. Calculate the proportions by plan type for the current quarter. Add these percentages to the user list to help calculate how many users for each plan type you need to invite. The end goal is to achieve a sample breakdown that roughly matches the population’s breakdown (+/- 3%). Note: You will need access to Tableau and SAFE data to view the link in this step. If you do not have access to one or both, please create an Access Request issue.

Sending an email wave

1. Using the percentages you calculated in the percentages tab of the CSAT user list Google Sheet template, determine how many users for each plan type you need to contact for the wave. If it’s the first wave, use the population proportions. If it’s a subsequent wave, use the proportions you calculate based on the responses so far (see next point).
2. To calculate your current sample plan proportions, download your survey results from Qualtrics. Calculate the percentage breakdown of plans so far. Then subtract that number from the population percentage and add the result to the population percentage. An example with fake numbers:

• Population percentage for Ultimate = 73%
• Percentage of Ultimate plan types after sending wave 1 = 65%
• Wave 2 percentage for Ultimate: (73% - 65%) + 73% = 81% In this example, the sample is under the population, hence the next wave percentage is higher than the population to try to get within 3% of the population percentage for Ultimate.

3. Waves should be ~2,000 users. Mark the desired number of users out of that 2,000 that fit your percentages for each plan type with the name of the wave you are sending.
4. Paste all the emails for that wave into a new sheet and upload them into the participant list for your CSAT project in Rally.
5. Once the mailing list has been uploaded to Rally, filter out @gitlab.com email addresses and people contacted about previous CSAT surveys from the past 12 months.
6. Create a new email distribution using the CSAT survey template email in Rally.
7. Typically emails are scheduled to go out Monday - Friday early in the morning US time (8 - 9am Eastern Time) with the goal of maximizing visibility and responses.

Once all email waves have been sent, add the user IDs and email that were used this quarter to the previously contacted sheet, noting the quarter they were contacted. This allows us to avoid contacting the same users too frequently.

Outreach to CSAT responders

When filling in the survey, UX Research gives CSAT respondents the option to indicate whether they would be open to a follow up interview. As part of the analysis, the UX Research DRI will compile a list of users who agreed to the follow up interview (see Google Sheet template for CSAT follow up users).

After the research report is shared out, the UX Research DRI will notify Product Managers, Product Designers, and Customer Success Managers about CSAT responders who opted into contact via this CSAT Responder Outreach issue template. The CSAT responder outreach workflow is described in more detail here.

Displaying the data

UX Research creates and communicates the CSAT reports and partners with Product Data Insights to maintain the CSAT dashboards for the UX department.

CSAT dashboards and maintenance

There are two internal dashboards in Tableau meant for presenting CSAT results:

• CSAT Scores: This dashboard shows the number and percentage of responses between those who are satisfied with GitLab the product (ratings of satisfied and very satisfied) and those who are not (ratings of neutral, dissatisfied, and very dissatisfied).
• CSAT Line Chart: This dashboard shows the CSAT score on a quarterly basis in order to track our score over time.

These dashboards can also be found in the UX Department Performance Indicators page.

The survey data in the analysis template gets connected to Tableau to show data from the current and previous quarters. UX Research is responsible for working with Product Data Insights to update these Tableau dashboards each quarter.

Follow up questions

For past reports, UX Research has gotten requests to answer follow up questions about the CSAT survey data (for example: What is the breakdown of company size across the survey responses?). When these requests arise, we’ve partnered with Product Data Insights to get support connecting CSAT respondents to internal data sources (i.e., Snowflake). We create an ad hoc request issue in the Product Data Insights project in GitLab and submit this request to a member of their team.

Past CSAT reports

FY24 reports

• FY24 Q3 Pilot

View page source - Edit this page - please contribute.