ClickHouse Working Group

Attributes

Context

ClickHouse is an open-source column-oriented database management system. It can efficiently filter, aggregate, and sum across large numbers of rows.

In FY23 ClickHouse was selected as GitLab’s standard datastore for features with big data and insert-heavy requirements (e.g. Observability, Analytics, etc.) ClickHouse is not intended to replace Postgres or Redis in GitLab’s stack.

In FY23-Q2 the Monitor:Observability team developed and shipped a ClickHouse data platform to store and query data for Error Tracking and other observability features. Other teams have also begun to incorporate ClickHouse into their current or planned architectures.

We want ensure teams can efficiently leverage the ClickHouse data platform when developing new features and that we can maintain and support this functionality effectively for SaaS and self-managed customers.

WG Exit

In April 2024 the ClickHouse Working Group was closed out. The group made major progress introducing ClickHouse to GitLab’s architecture to support GitLab’s analytics and monitoring features.

• Enabled usage of ClickHouse Cloud with GitLab, including security and legal approval for storing customer RED data and production readiness reviews.
• Consolidated our multiple ClickHouse hosting setups into ClickHouse Cloud, reducing operational load for our teams and improving reliability.
• Released multiple features that would not have been possible without ClickHouse’s capabilities.
• Fixed features such as Contributor Analytics which were no longer performant on GitLab.com.
• Developed documentation and tooling for using ClickHouse in GitLab’s development process.

Due to the overlap with the recently created Product Usage Data Architecture Working Group, the decision was made to wind down the ClickHouse group, and roll those concerns into that group’s work defining the architecture for analytical features.

Exit Criteria

This Working Group has the following goals:

1. Document and review the current uses/capabilities and future use cases.
2. Develop & Socialize Architecture Strategy for ClickHouse usage.
   1. Architecture Blueprint for ClickHouse usage within GitLab.
   2. Architecture Blueprint for scalable data ingestion pipeline for ClickHouse within GitLab.
   3. Provides guidance which sets expectations on scaling and query design/debugging/guardrails/etc.
   4. Define sane and secure defaults. Provide security recommendations and guardrails.
3. Develop a Rollout Strategy and Plan (e.g. feature flags, would customers ever want to disable/remove it, etc.).
   1. SaaS rollout strategy for identified use cases.
   2. Lay the groundwork to add ClickHouse to our reference architectures through self-managed support.
      • Packaging ClickHouse for our self-managed installations.
      • Research and understand the component costs and maintenance requirements of running a ClickHouse instance.
      • Determine the scale and deployment targets where adding ClickHouse makes sense.
      • Consider an abstraction layer to interacting with ClickHouse or alternatives.
4. Develop and implement a communication plan for the outcomes of the working group.

Exit Criteria Progress

Current and Planned Use Cases

• Current status of ClickHouse and GitLab’s implementation
• ClickHouse Use Cases

Roles and Responsibilities

View page source - Edit this page - please contribute.