Introducing a Messaging Layer to the GitLab Application

Summary

This document proposes design, architecture and a rollout roadmap for adopting & using a messaging layer to support data messaging & queuing needs at GitLab scale.

From some of our recent initiatives such as building the Data Insights Platform or Project Siphon, it has become evident that we need a scalable & reliable queueing system within our technology stack to be able to ingest & process large amounts of data. Having gone through multiple discussions around this, we have narrowed down our choices to using NATS as a solution to these needs - as explained later in this document.

Motivation

A primary driver for having a scalable messaging layer within our tech-stack is building our ability to ingest large amounts of data, especially analytical or monitoring data. While it is totally possible to persist data directly into our databases, it’s riddled with scalability challenges. For example, when ingesting data into ClickHouse, the database given its architecture - performs much better when ingesting large batches of data across fewer writes than ingesting a large number of small writes. Some past context around our experience with this.

Another key requirement is to be able to process incoming data before it lands within a database. At minimum, we need the ability to perform operations such as:

• dynamically enrich incoming data with metadata from other systems & data catalog,
• ensure ingested data does not contain sensitive information and anonymize, pseudonymize, or redact data as needed,
• batch data outside of main storage while certain (logical) conditions are met,
• fan-out ingested/processed data to multiple destinations, etc.

From an architectural perspective, having a data buffer available upstream to our persistent stores would help alleviate resource pressure downstream by absorbing large spikes in ingested data, which might happen quite frequently as we continue to generate and/or gather more data. We outline other significant benefits of building such an abstraction later in this document.

Considering our forward-looking use-cases, we also need such a system to be consistently available across all our deployment-models for GitLab instances, i.e. GitLab.com, Cells, Dedicated or Self-Managed. This would ensure we consolidate how we deal with any data stream generated across the product. With such a large deployment surface, it’s important we minimize all distribution & operational complexities of running such a system while ensuring its reliable, scalable and capable of delivering performance at GitLab scale.

As elucidated later in this document, NATS stands out given its minimal footprint, ease of distribution and its ability to both be embeddable within the product and scale out as a cluster when needed.

Goals

• Establish scalable & reliable data queueing infrastructure for a few first adopters: Siphon, Data Insights Platform.
• Provide authenticated & authorized access to all ingested data into the messaging layer.
• Provide necessary documentation to allow developers to interact with the messaging layer.
• Integrate messaging layer implementation with existing GitLab infrastructure to enable the aforementioned use-cases.

Non-goals

• Not aim to build a general-purpose event-bus for all our queueing/eventing needs just yet. Rather, the blueprint aims to lay the foundation for a future comprehensive event-bus with design decisions that should preserve compatibility with broader applications or use-cases looking forward.
• Not cover application-specific implementation details within the context of a messaging layer.

Proposal

The core of this proposal is to establish a foundational messaging piece within our tech-stack and build out necessary infrastructure well-integrated with GitLab installation(s).

For this first iteration, we do not expect to have all GitLab services or applications interacting with this messaging layer directly. The only planned usage right now is the following:

• For Siphon to buffer Postgres replication events before landing them in ClickHouse.

• Applications sending Snowplow-instrumented events to Data Insights Platform via event instrumentation layer which are dynamically enriched and landed into ClickHouse and AWS S3.

Looking forward

Once the aforementioned messaging layer is generally available, we aim to position it as the data queueing backbone for a more general-purpose events-based data platform within the product. Its existence helps ensure reliability and scalability for the various cross-platform data-related features at GitLab.

Benefits of building an events-based platform within GitLab

Having a centralized events-based data platform within the product helps improve GitLab’s logical architecture and its ability to scale well with time. As we build & continue to adopt such an architecture, the following key benefits come to mind:

• Enables loose-coupling between data producers and consumers across GitLab allowing them to scale independently of each other.
• Encourages asynchronous communication patterns between participating systems improving their scalability with increasing traffic volumes.
• Makes our architecture extensible wherein new consumers of existing data can be added with trivial time & effort.
• Provides a centralized, common architecture for sharing data useful for integrations across different parts of the product. This also leads to building over time singular sources of truth for important data across the product bringing consistency to product data.

The following discussions also explain why building an events-based abstraction is important for GitLab’s architecture at its current scale:

• Product Event Platform
• GitLab Events Platform
• GitLab Structured Events

Note, while the existence of a centralized data-sharing platform helps alleviate scalability & reliability concerns from other parts of our infrastructure, esp. databases, we will also need to iron out a few other concerns such as authorising clients, routing data efficiently and decoupling away from the monolith to ensure such a system proves valuable to our logical architecture.

Identified use-cases

Following is a detailed set of use-cases that benefit from the existence of a centralized data platform within the product.

Considered alternatives

As a potential solution to the aforementioned use-cases, a few popular backends were considered throughout our discussions. The following comparison matrix helps assess different possible backends for messaging needs within GitLab.

Note, we considered the following four deployment targets (for GitLab) when assessing potential solutions:

• GitLab.com SaaS: Multi-tenant instance.
• GitLab Dedicated: Single-tenant, dedicated resources per instance.
• Self-Managed on-cloud: For this environment, we assume a GCP-environment to prefer using GCP resources, an AWS-environment to prefer using AWS resources, etc. accounting for deployment-homogeneity.
• Self-Managed on-premise: For this environment, we assume all environments to be air-gapped so as to not have to make other assumptions on behalf of our customers.

Cost estimation & analysis

To ensure a comparable analysis for the different systems, we made some assumptions around how much data we need to host in each of the analysed systems.

For example, accounting for all Snowplow-instrumented data originating from .com SaaS, we estimate to generate 500GB of event data per day. If we then intend to retain this data for a week, we’ll roughly accumulate 3.5TB data which will need to be hosted on the underlying infrastructure at any given point in time. It can be assumed that data lifecycle policies kick-in correctly and this remains our maximum storage footprint within the context of this example.

• Daily generated data: 500GB
• Retention: 7 days
• Maximum stored data: 500GB * 7 days = 3.5TB

The following is how all analysed backends fair with that amount of data.

Cost analysis with respect to GitLab reference architectures

Read for more details: https://docs.gitlab.com/administration/reference_architectures/

Before estimating resources, we’ll need a measure of message traffic across the different reference architectures:

• Message volume: Estimating how many messages per second.
• Message size: Estimating message payload size in bytes.
• Number of publishers/consumers
• Persistence/storage: How much data needs to be stored and/or retained over how much time.

Conclusions from a features & cost perspective

• A significant factor worth noting from analysing these backends is their high operational, distribution & support overheads both for running them ourselves OR expecting them to be available inside non-hosted environments such as Self-Managed.
• All things considered, NATS appears to be the least expensive across all backends, considering it ships as a single Go-binary and can be installed in close-proximity to user services/applications with zero external dependencies. When needed, it can be scaled/sharded out across multiple servers/clusters subject to which reference architecture we run it within.
• The only factor favoring managed cloud-solutions instead is any support costs involved but their operational costs seem to outweigh any benefits we derive from using them, especially as their usage and/or adoption grows with our scale.

The rest of this document focuses on building out the proposal to use NATS as the messaging solution within a GitLab instance.

Components

• One or more JetStream-enabled NATS servers
• Persistent volumes, preferably SSDs, for each deployed NATS server.
• Decentralized authentication callout server (long-term only) : For authentication/authorisation, we aim to start with leveraging hard-coded roles & credentials within NATS (centralized) but in the long-term, we expect to use a decentralized server-side auth-callout implementation for authenticating inbound traffic.

Deployments

With the intention to make NATS available to every GitLab installation, we aim to build necessary support for the following deployment models:

Tenancy

For all deployment-types, we expect a given NATS cluster to be multi-tenant, i.e. multiple users/services hosted on the deployment to share the same underlying NATS infrastructure for their messaging needs.

Rollout roadmap

Design & Implementation

Architecture

The following diagram depicts a simple 3-node NATS cluster deployed as a StatefulSet with dedicated persistent volumes. A given application can then discover the service:

• using a headless service, or

nats.default.svc.cluster.local

• addressing the pods directly

nats-0.nats.default.svc.cluster.local
nats-1.nats.default.svc.cluster.local
nats-2.nats.default.svc.cluster.local

Setup

• Setup an N-nodes cluster subject to data volumes & retention.
• Enable data persistence via NATS JetStream.
• Configure stream replication for subjects as needed.

Connectivity

• NATS comes with support for both plaintext and TLS connections. We intend to use TLS-enabled connections to help authenticate all incoming traffic.

• Access to the NATS service will only be available to internal network clients. External clients will not be able to connect to the NATS service.

• External load-balancing is therefore not needed, which is also highly advised against. Each NATS server in a given cluster must be reachable individually with all loadbalancing deferred to NATS itself.

• For Cells-based installations, we can leverage the use of NATS Gateways to connect clusters should we need to, but we expect each cell to be serviceable in isolation for majority of our applications. While we believe that NATS Gateways may help with interconnection in future, this functionality is out-of-scope for this iteration of the blueprint.

Topology

• For all deployment-types, NATS will be deployed region-local only similar to Redis or Sidekiq and not Geo-replicated like Postgres. While it is possible to mesh NATS clusters together as a feature, this functionality is out-of-scope for this iteration of the blueprint.

• For GitLab.com, we intend to setup & run NATS clusters cloud-natively on Kubernetes with the assumption that the following potential overheads can be well-managed:

  • running stateful workloads within Kubernetes.
  • performance overheads from routing traffic within Kubernetes services.
  • load-balancing traffic via Kubernetes service-topology.

• From an operational perspective and considering we prefer new services to be built Kubernetes-based, it’ll also be trivial for us to use the same configurations for GitLab.com, Dedicated, Cells and Self-Managed going forward.

• If running it cloud-natively does incur overheads, we can resort to running NATS directly on VMs within the same VPC/network-boundaries to leverage better utilization of the underlying hardware and reduce any operational complexity.

• We have also prototyped both of these deployment models:

  • using a Helm chart for cloud native installations, initial POC
  • using Terraform to install NATS on cloud VMs directly - initial POC

Data persistence

• NATS allows all ingested data to be stored in-memory or persisted on-disk.
• We intend to leverage NATS JetStream to persist all ingested data durably.
• We expect to use SSDs to improve the performance of all read/write operations.
• We intend to enforce retention policies for ingested data and provision underlying storage with sufficient headroom to begin with especially as we tune our retention needs gradually. From an operational perspective, it is necessary we have the ability to increase underlying storage trivially when needed.
• We expect to enable data compression to reduce storage footprint further.

Integration with GitLab

• We can add the aforementioned NATS connection string to the instance configuration to be used by services/applications. For example:
  • Add nats['address'] = 'nats://nats.default.svc.cluster.local:4222' to /etc/gitlab/gitlab.rb, or
  • Override NATS_URL=nats://nats.default.svc.cluster.local:4222 in gitlab.yml.

Security

Authentication/Authorization

• NATS has prebuilt support for connections encrypted over TLS.
• It also comes with centralized auth support via JWT/NKEYS.
• We expect to promote the usage of separate principals (users/accounts) across distinct systems, e.g. producers/consumers of a given stream.
• NATS offers grouping of clients and subject space with accounts.

Example authentication/authorization setup

We take Siphon as an example use-case here, wherein we cover:

• Creation of accounts to isolate clients.
  • Adding users to these accounts with specific permissions for available subjects. Authentication will be achieved via nkeys.
  • Producers and consumers have their own users and permissions.
• Producer/Consumer nkeys are to be treated with the same security practices as we currently do for our database secrets.

Example authorization.conf

listen: 127.0.0.1:4222
jetstream: enabled

producer_permissions = {
  publish = ">"
  subscribe = ">"
}

consumer_permissions = {
  publish = {
    deny = ">"
  }
  subscribe = ">"
}

accounts: {
    siphon: {
        users: [
            {nkey: Uxx, permissions: $producer_permissions},
            {nkey: Uxx, permissions: $consumer_permissions}
        ]
    }
}

In the above configuration, producer is allowed to publish and subscribe over all the subject space for siphon account while consumer is only allowed to subscribe to available subjects.

We can also apply further granularity on subject space if desired. The use of permissions map allows for such fine-grained control.

Example usage of this configuration:

func TestServerConfiguration(t *testing.T) {
    server, _ := RunServerWithConfig("authorization.conf")
    t.Logf(server.ClientURL())

    // producer_nkey.txt holds the nkey seed
    opt, err := nats.NkeyOptionFromSeed("producer_nkey.txt")
    if err != nil {
        t.Error(err)
        return
    }
    nc, err := nats.Connect(server.ClientURL(), opt)
    if err != nil {
        t.Error(err)
        return
    }
    js, err := nc.JetStream()
    if err != nil {
        t.Error(err)
        return
    }
    _ = js.Streams()
    defer nc.Close()
}

Encryption

• While NATS has support for encryption at rest, it recommends the use of filesystem encryption when available.

• Considering our deployment-targets, we intend to use disk-encryption - available on both GCP and AWS with support for additional customer-managed key options.

Auditing/Logging

• We intend to ship necessary NATS logs to our centralized logging infrastructure to enable any auditing/monitoring purposes.

Operations

Scalability

NATS is extremely lightweight and can support ingesting & digesting high amounts of messages with sub-millisecond latencies. Given its architecture, it’s also optimized for handling backpressure and exercise flow-control subject to traffic volumes.

We ran the following preliminary tests against a Kubernetes-based NATS cluster with 3 servers, each such server (pod) running running on a c2d-standard-16 GKE node and attached to a 100GB pd-balanced SSD persistent volume for data storage.

Note, the underlying GKE cluster is a regional cluster with cluster-nodes spread in 3 distinct AZs. NATS servers (pods) were carefully spread across the 3 AZs at all times during our tests.

Key insights

• CPU usage is directly proportional to cluster usage with large spikes in the case of asynchronous publishers producing a very large number of events in a short period of time.
• Memory usage remained stable regardless of the amount of events ingested.
• Cross-AZ replication does affect cluster throughput but overall performance remains well-beyond our immediate needs.

Synchronous publisher with stream replication

CPU usage remained consistently low while write-throughput takes a notable hit.

➜  platform-pre-stg kubectl -n nats exec -it nats-box-6888bbc55c-kd6tm -- nats --server=nats://nats.nats.svc.cluster.local:4222 bench foobar --pub 1 --sub 5 --msgs=1000000 --js --maxbytes 20GB --purge --replicas=2 --syncpub
10:55:59 JetStream ephemeral ordered push consumer mode, subscribers will not acknowledge the consumption of messages
10:55:59 Starting JetStream benchmark [subject=foobar, multisubject=false, multisubjectmax=100000, js=true, msgs=1,000,000, msgsize=128 B, pubs=1, subs=5, stream=benchstream, maxbytes=20 GiB, storage=file, syncpub=true, pubbatch=100, jstimeout=30s, pull=false, consumerbatch=100, push=false, consumername=natscli-bench, replicas=2, purge=true, pubsleep=0s, subsleep=0s, dedup=false, dedupwindow=2m0s]

NATS Pub/Sub stats: 7,957 msgs/sec ~ 994.75 KB/sec
 Pub stats: 1,326 msgs/sec ~ 165.81 KB/sec
 Sub stats: 6,631 msgs/sec ~ 828.96 KB/sec
  [1] 1,326 msgs/sec ~ 165.80 KB/sec (1000000 msgs)
  [2] 1,326 msgs/sec ~ 165.80 KB/sec (1000000 msgs)
  [3] 1,326 msgs/sec ~ 165.80 KB/sec (1000000 msgs)
  [4] 1,326 msgs/sec ~ 165.79 KB/sec (1000000 msgs)
  [5] 1,326 msgs/sec ~ 165.81 KB/sec (1000000 msgs)
  min 1,326 | avg 1,326 | max 1,326 | stddev 0 msgs

Asynchronous publisher with stream replication, publishing batches in sizes 100, 1000, 10000

CPU usage is proportional to batch-size with write-throughput improving with moderately sized batches.

➜  platform-pre-stg kubectl -n nats exec -it nats-box-6888bbc55c-kd6tm -- nats --server=nats://nats.nats.svc.cluster.local:4222 bench foobar --pub 1 --sub 5 --msgs=1000000 --js --maxbytes 20GB --purge --replicas=2
11:13:02 JetStream ephemeral ordered push consumer mode, subscribers will not acknowledge the consumption of messages
11:13:02 Starting JetStream benchmark [subject=foobar, multisubject=false, multisubjectmax=100000, js=true, msgs=1,000,000, msgsize=128 B, pubs=1, subs=5, stream=benchstream, maxbytes=20 GiB, storage=file, syncpub=false, pubbatch=100, jstimeout=30s, pull=false, consumerbatch=100, push=false, consumername=natscli-bench, replicas=2, purge=true, pubsleep=0s, subsleep=0s, dedup=false, dedupwindow=2m0s]

NATS Pub/Sub stats: 274,880 msgs/sec ~ 33.55 MB/sec
 Pub stats: 46,073 msgs/sec ~ 5.62 MB/sec
 Sub stats: 229,066 msgs/sec ~ 27.96 MB/sec
  [1] 46,021 msgs/sec ~ 5.62 MB/sec (1000000 msgs)
  [2] 45,866 msgs/sec ~ 5.60 MB/sec (1000000 msgs)
  [3] 45,901 msgs/sec ~ 5.60 MB/sec (1000000 msgs)
  [4] 45,813 msgs/sec ~ 5.59 MB/sec (1000000 msgs)
  [5] 45,986 msgs/sec ~ 5.61 MB/sec (1000000 msgs)
  min 45,813 | avg 45,917 | max 46,021 | stddev 76 msgs

➜  platform-pre-stg kubectl -n nats exec -it nats-box-6888bbc55c-kd6tm -- nats --server=nats://nats.nats.svc.cluster.local:4222 bench foobar --pub 1 --sub 5 --msgs=1000000 --js --maxbytes 20GB --purge --replicas=2 --no-progress --pubbatch=1000
11:17:38 JetStream ephemeral ordered push consumer mode, subscribers will not acknowledge the consumption of messages
11:17:38 Starting JetStream benchmark [subject=foobar, multisubject=false, multisubjectmax=100000, js=true, msgs=1,000,000, msgsize=128 B, pubs=1, subs=5, stream=benchstream, maxbytes=20 GiB, storage=file, syncpub=false, pubbatch=1,000, jstimeout=30s, pull=false, consumerbatch=100, push=false, consumername=natscli-bench, replicas=2, purge=true, pubsleep=0s, subsleep=0s, dedup=false, dedupwindow=2m0s]

NATS Pub/Sub stats: 524,251 msgs/sec ~ 64.00 MB/sec
 Pub stats: 152,262 msgs/sec ~ 18.59 MB/sec
 Sub stats: 436,876 msgs/sec ~ 53.33 MB/sec
  [1] 100,985 msgs/sec ~ 12.33 MB/sec (1000000 msgs)
  [2] 88,393 msgs/sec ~ 10.79 MB/sec (1000000 msgs)
  [3] 88,367 msgs/sec ~ 10.79 MB/sec (1000000 msgs)
  [4] 87,896 msgs/sec ~ 10.73 MB/sec (1000000 msgs)
  [5] 87,375 msgs/sec ~ 10.67 MB/sec (1000000 msgs)
  min 87,375 | avg 90,603 | max 100,985 | stddev 5,204 msgs

➜  platform-pre-stg kubectl -n nats exec -it nats-box-6888bbc55c-kd6tm -- nats --server=nats://nats.nats.svc.cluster.local:4222 bench foobar --pub 1 --sub 5 --msgs=1000000 --js --maxbytes 20GB --purge --replicas=2 --no-progress --pubbatch=10000
11:17:57 JetStream ephemeral ordered push consumer mode, subscribers will not acknowledge the consumption of messages
11:17:57 Starting JetStream benchmark [subject=foobar, multisubject=false, multisubjectmax=100000, js=true, msgs=1,000,000, msgsize=128 B, pubs=1, subs=5, stream=benchstream, maxbytes=20 GiB, storage=file, syncpub=false, pubbatch=10,000, jstimeout=30s, pull=false, consumerbatch=100, push=false, consumername=natscli-bench, replicas=2, purge=true, pubsleep=0s, subsleep=0s, dedup=false, dedupwindow=2m0s]

NATS Pub/Sub stats: 424,064 msgs/sec ~ 51.77 MB/sec
 Pub stats: 70,985 msgs/sec ~ 8.67 MB/sec
 Sub stats: 353,386 msgs/sec ~ 43.14 MB/sec
  [1] 71,156 msgs/sec ~ 8.69 MB/sec (1000000 msgs)
  [2] 70,899 msgs/sec ~ 8.65 MB/sec (1000000 msgs)
  [3] 70,757 msgs/sec ~ 8.64 MB/sec (1000000 msgs)
  [4] 70,887 msgs/sec ~ 8.65 MB/sec (1000000 msgs)
  [5] 70,812 msgs/sec ~ 8.64 MB/sec (1000000 msgs)
  min 70,757 | avg 70,902 | max 71,156 | stddev 137 msgs

Asynchronous publisher, testing pull vs push consumers

Nothing noteworthy about CPU usage with pull consumers performing better than push ones.

➜  platform-pre-stg kubectl -n nats exec -it nats-box-6888bbc55c-kd6tm -- nats --server=nats://nats.nats.svc.cluster.local:4222 bench foobar --pub 1 --sub 5 --msgs=1000000 --js --maxbytes 20GB --purge --replicas=2 --no-progress --pubbatch=100 --push
11:24:53 JetStream durable push consumer mode, subscriber(s) will explicitly acknowledge the consumption of messages
11:24:53 JetStream ephemeral ordered push consumer mode, subscribers will not acknowledge the consumption of messages
11:24:53 Starting JetStream benchmark [subject=foobar, multisubject=false, multisubjectmax=100000, js=true, msgs=1,000,000, msgsize=128 B, pubs=1, subs=5, stream=benchstream, maxbytes=20 GiB, storage=file, syncpub=false, pubbatch=100, jstimeout=30s, pull=false, consumerbatch=100, push=true, consumername=natscli-bench, replicas=2, purge=true, pubsleep=0s, subsleep=0s, dedup=false, dedupwindow=2m0s]

NATS Pub/Sub stats: 65,697 msgs/sec ~ 8.02 MB/sec
 Pub stats: 32,912 msgs/sec ~ 4.02 MB/sec
 Sub stats: 32,848 msgs/sec ~ 4.01 MB/sec
  [1] 6,581 msgs/sec ~ 822.69 KB/sec (200000 msgs)
  [2] 6,586 msgs/sec ~ 823.28 KB/sec (200000 msgs)
  [3] 6,574 msgs/sec ~ 821.78 KB/sec (200000 msgs)
  [4] 6,575 msgs/sec ~ 821.90 KB/sec (200000 msgs)
  [5] 6,579 msgs/sec ~ 822.48 KB/sec (200000 msgs)
  min 6,574 | avg 6,579 | max 6,586 | stddev 4 msgs

➜  platform-pre-stg kubectl -n nats exec -it nats-box-6888bbc55c-kd6tm -- nats --server=nats://nats.nats.svc.cluster.local:4222 bench foobar --pub 1 --sub 5 --msgs=1000000 --js --maxbytes 20GB --purge --replicas=2 --no-progress --pubbatch=100 --pull
11:25:56 JetStream durable pull consumer mode, subscriber(s) will explicitly acknowledge the consumption of messages
11:25:56 Starting JetStream benchmark [subject=foobar, multisubject=false, multisubjectmax=100000, js=true, msgs=1,000,000, msgsize=128 B, pubs=1, subs=5, stream=benchstream, maxbytes=20 GiB, storage=file, syncpub=false, pubbatch=100, jstimeout=30s, pull=true, consumerbatch=100, push=false, consumername=natscli-bench, replicas=2, purge=true, pubsleep=0s, subsleep=0s, dedup=false, dedupwindow=2m0s]

NATS Pub/Sub stats: 95,057 msgs/sec ~ 11.60 MB/sec
 Pub stats: 47,747 msgs/sec ~ 5.83 MB/sec
 Sub stats: 47,528 msgs/sec ~ 5.80 MB/sec
  [1] 15,410 msgs/sec ~ 1.88 MB/sec (200000 msgs)
  [2] 12,056 msgs/sec ~ 1.47 MB/sec (200000 msgs)
  [3] 11,976 msgs/sec ~ 1.46 MB/sec (200000 msgs)
  [4] 9,556 msgs/sec ~ 1.17 MB/sec (200000 msgs)
  [5] 9,530 msgs/sec ~ 1.16 MB/sec (200000 msgs)
  min 9,530 | avg 11,705 | max 15,410 | stddev 2,157 msgs

Monitoring

• Inbuilt monitoring exposed as Prometheus metrics, details here.

Cluster upgrades

• NATS provides well-documented paths around upgrading clusters, as long as we provision multi-node clusters and can perform a gradual rolling-restart.

• In case of running NATS via Statefulsets on Kubernetes, this can be handled automatically by the underlying Kubernetes statefulset controller considering we use .spec.updateStrategy to be RollingUpdate which ensures an incremental rollout of the updates one pod at a time.

• In case of running NATS on bare VMs, an operator will have to perform such a rolling upgrade of the cluster either manually or via tooling depending on how the cluster is setup.

Failure Scenarios

• In the event of total service unavailability of NATS, we might experience loss of data especially in scenarios where NATS is where we land incoming data first.

• In the event of loss of one or more nodes in a given NATS cluster, clients can still continue to push new events as long as other stream-replicas are configured and can assume new leadership for affected streams. For clients that cannot withstand loss of messages, stream replication is highly advised which ensures any affected streams will recover as soon as any cluster-deterioration is remedied.

• While stream replication ensures data redundancy for ingested data, asynchronous writes might still lead to loss of data. To minimize any loss of data, clients should prefer synchronous writes to ensure all ingested data is durably replicated before their writes get acknowledged, with the caveat that synchronous writes will reduce overall write-performance.

Note Further details of how NATS publishers or consumers must be designed is out of scope for this blueprint. All user-facing documentation for building NATS applications will be developed separately.

• All ingested data is persisted durably via NATS JetStream. In the event of unrecoverable messages however, we can rely on an explicit disaster recovery setup to recover data, which includes:
  • Automatic recovery in case of intact quorum nodes for replicated streams, or
  • Manual recovery from periodic stream backups.

Note To ensure all persisted data is recoverable, we’ll need to integrate such a disaster recovery procedure for NATS into our general GitLab DR plans & tooling, taking into consideration all GitLab deployment types.

Note, in the specific case of Siphon, all data buffered within NATS and due to be exported to ClickHouse also remains available in Postgres. In the scenario where Siphon fails to connect to NATS or there is data loss on NATS, Siphon can perform a full-resync to ensure data consistency across Postgres & ClickHouse again. For other use-cases where this is not possible, we’ll have to depend on recovering lost data automatically or manually from backups as stated above.

• We do not expect auth failures while using a centralized model with users/accounts setup within NATS at deployment-time but the introduction of an external auth callout service can add further failure domains to the system. As a dependency, we’ll need to guarantee reliability SLOs on our implementation of an auth-server to be equal or higher than those for the NATS service itself. The development of such an abstraction is out-of-scope for this iteration of the blueprint.

Additional Context

Why not Kafka?

Given our needs to queue/buffer data durably, Apache Kafka comes as an obvious first choice. However, given the operational & distribution complexity around running Kafka especially as we shift focus towards running GitLab as cloud-native deployments, it becomes less favorable for our purposes. Following are some of our past discussions around the challenges Kafka brings:

• Support for Kafka across deployment-environments is non-existent.
• Kafka can be cost-prohibitive regardless of scale.
• Kafka can be operationally intensive.
• Non-trivial effort to replace current usage.

Key considerations when comparing Kafka with NATS

View page source - Edit this page - please contribute.