GitLab CI Events
Status | Authors | Coach | DRIs | Owning Stage | Created |
---|---|---|---|---|---|
proposed |
furkanayhan
|
grzesiek
|
fabiopitino
jreporter
cheryl.li
|
devops verify | 2023-03-15 |
Summary
In order to unlock innovation and build more value, GitLab is expected to be the center of automation related to DevSecOps processes. We want to transform GitLab into a programming environment, that will make it possible for engineers to model various workflows on top of CI/CD pipelines. Today, users must create custom automation around webhooks or scheduled pipelines to build required workflows.
In order to make this automation easier for our users, we want to build a powerful CI/CD eventing system, that will make it possible to run pipelines whenever something happens inside or outside of GitLab.
A typical use-case is to run a CI/CD job whenever someone creates an issue, posts a comment, changes a merge request status from “draft” to “ready for review” or adds a new member to a group.
To build that new technology, we should:
- Emit many hierarchical events from within GitLab in a more advanced way than we do it today.
- Make it affordable to run this automation, that will react to GitLab events, at scale.
- Provide a set of conventions and libraries to make writing the automation easier.
Goals
While “GitLab Events Platform” aims to build new abstractions around emitting events in GitLab, “GitLab CI Events” blueprint is about making it possible to:
- Define a way in which users will configure when an event emitted will result in a CI pipeline being run.
- Describe technology required to match subscriptions with events at GitLab.com scale and beyond.
- Describe technology we could use to reduce the cost of running automation jobs significantly.
Proposal
Decisions
Requirements
Any accepted proposal should take in consideration the following requirements and characteristics:
- Defining events should be done in separate files.
- If we define all events in a single file, then the single file gets too complicated and hard to
maintain for users. Then, users need to separate their configs with the
include
keyword again and we end up with the same solution. - The structure of the pipelines, the personas and the jobs will be different depending on the events being subscribed to and the goals of the subscription.
- If we define all events in a single file, then the single file gets too complicated and hard to
maintain for users. Then, users need to separate their configs with the
- A single subscription configuration file should define a single pipeline that is created when an event is triggered.
- The pipeline config can include other files with the
include
keyword. - The pipeline can have many jobs and trigger child pipelines or multi-project pipelines.
- The pipeline config can include other files with the
- The events and handling syntax should use the existing CI config syntax where it is pragmatic to do so.
- It’ll be easier for users to adapt. It’ll require less work to implement.
- The event subscription and emiting events should be performant, scalable, and non blocking.
- Reading from the database is usually faster than reading from files.
- A CI event can potentially have many subscriptions. This also includes evaluating the right YAML files to create pipelines.
- The main business logic (e.g. creating an issue) should not be affected by any subscriptions to the given CI event (e.g. issue created).
- The CI events design should be implemented in a maintainable and extensible way.
- If there is a
issues/create
event, then any new event (merge_request/created
) can be added without much effort. - We expect that many events will be added. It should be trivial for developers to register domain events (e.g. ‘issue closed’) as GitLab-defined CI events.
- Also, we should consider the opportunity of supporting user-defined CI events long term (e.g. ‘order shipped’).
- If there is a
Options
For now, we have technical 5 proposals;
- Proposal 1: Using the
.gitlab-ci.yml
file Based on; - Proposal 2: Using the
rules
keyword Highly inefficient way. - Proposal 3: Using the
.gitlab/ci/events
folder Involves file reading for every event. - Proposal 4: Creating events via a CI config file Separate configuration files for defininig events.
- Proposal 5: Combined proposal Combination of all of the proposals listed above.
GitLab CI Events Proposal 1: Using the .gitlab-ci.yml file
GitLab CI Events Proposal 2: Using the rules keyword
GitLab CI Events Proposal 3: Using the .gitlab/ci/events folder
GitLab CI Events Proposal 4: Defining subscriptions in a dedicated configuration file
GitLab CI Events Proposal 5: Combined proposal
d3a6fdb2
)