Forrester Total Economic Impact™ (TEI) Study Commissioned By GitLab, June 2020
GitLab commissioned Forrester Consulting to conduct a Total Economic Impact™ (TEI) study examining the potential return on investment (ROI) enterprises may realize by using GitLab.
GitLab customers were interviewed and related data was collected independently by Forrester Consulting in this process. The data collected, resulting model, and study itself was reviewed independently by Forrester Research analysts. GitLab stakeholders were also interviewed as part of the data gathering and review process.
This study looked at the cost savings and business benefits of enterprises using GitLab for Version Control & Collaboration (VC&C)/SCM, Continuous Integration (CI), and Continuous Delivery (CD).
Key benefits customers realized found in the study include:
- Improved development and delivery efficiency
- Increased revenue from increased number of releases
- Improved quality of code
- Savings from reducing the number of tools in use
The results speak for themselves - over 3 years, the TEI model estimated that a composite enterprise would potentially realize (all $ figures in USD):
- 407% Return on Investment (ROI)
- 87% improved development and delivery efficiency (reduced time), resulting in over $23 Million in savings
- 12x increase in the number of revenue generating application releases in a year, resulting in $12.3 million in additional revenue
- 80% reduction in code defects, resulting in over $16.8 million in savings
- $3.7 million in savings due to reducing the number of tools (and their associated costs) in use each year by 4.
Links to the TEI Study & Related Resources
How to use the TEI Study (Sales)
Introduce early in your conversations with prospects and customers to add 3rd party validation credibility and jumpstart business justification conversations.
- Use as a vehicle to discover additional information and opportunities to show value beyond an initial use case.
- Use as a first step towards a more comprehensive and detailed ROI value discovery engagement that would utilize our more comprehensive ROI model.
- Use as a way to validate our enterprise adoption and success by highlighting the size and scale of the interviewed customers.
- Use as a way to help focus the kinds of additional external validation support (customer reference calls, case studies, analyst reports, competitive intelligence, etc.) you may need to help your prospect/customer speed evaluation and procurement.
Also coming soon are an online calculator/custom report generator and Forrester delivered-enablement session on how to best understand the study and use it with your customers and prospects.
Quick Facts about the study
Four GitLab customers were interviewed for this TEI study, including a Software and Technology firm with 1,700 GitLab users using GitLab on-prem, a Financial Services company with 4,000 GitLab users on SaaS, a Telecommunications firm with 1,600 GitLab users on SaaS, and an Information Technology firm, with 4,000 users using GitLab on-prem.
The composite (model) organization created based on the interviewed customer set, used to generate the potential ROI and Value estimates is a:
- Global enterprise, with annual revenue of $37B, 3000 initial GitLab users, and increases their user count by 25% year over year.
- This enterprise has multiple users, working in different groups, using different toolchains, and each toolchain is comprised of multiple users tools with limited overlap.
- This enterprise chooses to use the Premium ($19 per user) Tier of GitLab SaaS.
- This enterprise uses GitLab for Version Control & Collaboration (SCM), Continuous Integration (CI), and Continuous Delivery (CD).
This TEI™ study explicitly incorporated the impact of costs into the model based on actual customer experiences, including Licensing costs, Implementation and ongoing management costs, and training costs.
Summarized quantified benefits include
- 407% Return of Investment (ROI)
- 87% improved development and delivery efficiency (reduced time), resulting in over $23 Million in savings
- 12x increase in the number of revenue generating application releases in a year, resulting in $12.3 million in additional revenue
- 80% reduction in code defects, resulting in over $16.8 million in savings
- $3.7 million in savings due to reducing the number of tools (and their associated costs) in use each year by 4
- 65% improved efficiency in the planning phase of development and delivery
- 92% improved efficiency in the creating/coding and review phase of development and delivery
- 90% improved efficiency in the Build/Verify (CI) phase of development and delivery
- 85% improved efficiency in the Package, Release, Configure and Monitor (CD) phase of development and delivery
This TEI study also incorporated significant risk adjustments and productivity recapture discounting to better mirror realistic outcomes. Example: in the model used to discover Improved Development and Delivery Efficiency, productivity recapture - being able to actually use the time saved by using GitLab by applying it to other work - is effectively discounted by 50%.
This study was commissioned by GitLab and delivered by Forrester Consulting, customer names and contact information was provided to Forrester Consulting, who in turn interviewed GitLab customers independently. The data collected, resulting model, and study itself was reviewed independently by Forrester Research analysts. GitLab stakeholders were also interviewed as part of the data gathering and review process.
Like all ROI models, while they are useful for discovering and articulating potential value, they are still “just” models, effectively a set of assumptions through which we can put data in and get a picture of what potential results might look like. Neither Forrester nor GitLab makes any assumptions as to the potential ROI that other organizations will receive. Both Forrester and GitLab strongly advise that readers, prospects, and customers, use their own estimates within the framework provided in the report to determine the appropriateness of an investment in GitLab.
Frequently Asked Questions
Q: What use cases does this TEI study cover?
A: This study looked at customers using GitLab for Version Control & Collaboration (SCM), Continuous Integration (CI), and Continuous Delivery (CD).
Q: Why are the results what they are? Why weren’t they higher?
A: The results of the TEI study were based on our actual customer’s experiences and included a realistic assessment of costs involved and realistic estimates of risk involved in an investment. Models should be adjusted and the data put into the model should reflect each individual’s unique situation inclusive of costs and risk to get a more realistic estimate of their potential results.
Q: Why was this study focused on enterprises and can this study’s results be used by smaller organisations?
A: At the time this study was commissioned it was determined that our most pressing, immediate need for a 3rd party validated ROI model was in support of our enterprise growth efforts. To get the most confident, useful results when creating a model it is critical to have the core sources of information (in this case customers) be as relatively similar as possible. All of that said, yes we believe that the TEI study can still provide useful context and insights for midmarket and smaller organisations.
Q: Is the TEI Study a standalone piece of content/tool or are there other parts or things it can be used with?
A: The study itself certainly can be used independently, but we also have an online calculator coming soon that allows anyone to input some of their own information and generate a custom report and also download a copy of the core study. We also will be hosting a special enablement session focused on understanding the study and best practices in using it delivered by Forrester soon, so keep any eye out for that invitation! Lastly, using the TEI can also be a great way to kick start a value conversation that can ultimately lead to a more thorough benefit justification analysis using our own GitLab ROI model.
Q: Is this the only TEI study Forrester Consulting has performed?
A: No, many vendors across the industry (including several of our competitors like GitHub and Atlassian) have and continue to commission these studies as they have become a trusted, useful mechanism for helping organizations articulate and prioritize the problems they are trying to solve in terms of business value so that they can make the best decision they can when choosing to invest in products and services.
Q: How does our TEI compare to those of competitors? Do competitors having their own TEI mean I can’t or shouldn’t use ours - especially if they show different or higher numbers for some things?
A: While the TEI uses a standardized methodology, each one ends up effectively unique, as it is shaped by the unique experiences of a specific combination of customers, situations, solutions, over a period of time. In other words, they end up different enough that TEI studies in general, should not be compared to each other.
That said there are situations where some amount of overlap in metric/value type may occur, and that’s where a second look at the totality of the model, the scope of the study, should be reviewed. For example, not all TEI studies incorporate the same precise type or amounts of costs or risk in their calculations of ROI as we do. We did incorporate these costs as well as reasonable risk adjustments and productivity recapture discounting because we believe that they are critical to getting a more realistic value picture, but as we know, not everyone shares our values and/or takes our approach to the market. As a result when you review and/or your customers and prospects review ours and others’ TEI studies, be sure to look for/draw attention to omissions like that.
Q: How long is this study good for? Are there any more like it coming as it only covers VC&C, CI, and CD and we do much more?
A: As this was a commissioned study, we retain the rights to the study itself and can use it in market as long as we (all of us) believe it to be valid/useful. We are currently exploring opportunities for additional, similar studies that may look at different organization sizes, use cases, etc.
Q: If I have other questions about the TEI, who do I reach out to?
A: Your Industry Analyst Relations team (Handbook Page, Team Members) is always here to help answer any and all questions you may have, be sure to reach out if you need any help!
Competitive Positioning Guidance
- Keep the conversation focused on our own TEI study - To minimize confusion and maximize the use of our customer and prospective customer’s time we should keep focus on our TEI study, model, and potential results. We should feel comfortable enough with our own study to be able to contrast/highlight the differences between a competitior’s TEI or similar study if it comes up, we should be aware that they may exist, but we should not assume everyone understands, is familiar with, or is even aware of a competitior’s TEI or similar study.
- ROIs are unique - We should remember and help those we have the opportunity to discuss our TEI study with that ROI’s are not all the same, they are not interchangable - you can’t and shouldn’t just compare one ROI figure directly to another as they won’t be based on the same data, the same assumptions, the same constraints.
- Emphasize the important qualities of our TEI study - We should be quick, comfortable, and confident to draw attention to key qualities of our TEI study that enhance both its credibility and its usefulness. Doing so makes the lack of these qualities in other’s studies all the more apparent.
- Depth & Breadth: The depth and breadth of our study (spanning 3 Use Cases - Version Control & Collaboration/SCM, CI, and CD) is objectively greater than most - we cover more and deeper, than many.
- Realistic Costs: Our study takes into account realistic costs like implementation, training, ongoing maintenance requirements that our interviewed customers actually experienced - many studies do not or only do so minimally, leading to unrealistic expectations and disappointment.
- Consistent Customer Sample: Our sample of customers (ours range from 1,600-4,000 users - a good, medium range for an enterprise investment, with similar experiences) is reasonably consistent, resulting, in our opinion, to better analysis, a better model, and ultimately better results for our customers.
bc83f2be)
