Verify:Pipeline Authoring Group

Useful Links

Product

• Product Vision
• Pipeline Authoring Category direction
• number of unique users who trigger ci_pipelines (Performance indicator) - Internal link
• CI/CD Development Documentation
• CI/CD Components Documentation
• CI/CD Catalog Documentation

Team Handles

Team Resources

• Team Resources
• Workflow board: ~group::pipeline authoring

Videos

• GitLab Unfiltered: Pipeline Execution group (CI Related)
• CI Backend Architectural Walkthrough - May 2020
• Frontend CI product / codebase overview - June 2020
• CI/CD Catalog Demo

Core domain

Products

Features

• Pipeline creation
• YAML syntax
• CI/CD configuration lint tool
• CI/CD Variables
• Additional features can be found here

Technical Roadmap

Remainder of FY25

These areas are our high-level engineering driven goals for the remainder of FY25. Though they are ambitious and subject to change, it gives insight into where our focus will be in these areas.

Performance & Cost Reduction

Pipeline Creation Speed

NOTE: Pipeline creation feedback issue. We welcome your thoughts on your experience with the pipeline creation process.

Goals:

• Understand the components of the pipeline creation service to improve speed.
  • This graph captures data around the pipeline creation performance.
• Top 3 identified issues to optimize pipeline creation performance.
  • Bulk insert Sidekiq jobs when creating a pipeline
  • Expand CI variables lazily and selectively
  • Evaluate rules once for parallel jobs

Efficiency

Goals:

• Transition from entry classes to JSON schema for data structure validation. - Issue

Visibility

CI Catalog instrumentation

Goals:

• Top 3 identified issues to add more visibility for usage in CI/CD Catalog with adding instrumentation.
  • Capture GraphQL call times
  • Implement instrumentation to track performance of Ci::Catalog::Listing queries
  • Capture page load times

FY26 top of mind

• Support for GraphQL subscriptions - epic
• Pipeline performance improvements - epic
• CI Variables improvements - epic
• MR Pipeline Tab migration to GraphQL - epic

Exciting things and accomplishments

This section will list the top three most recent, exciting accomplishments from the team.

• Recently, we released the CI/CD Catalog to GA.
• We added usage statistics and a sort option for popularity in the CI/CD Catalog index page

Team Members

Stable Counterparts

To find our stable counterparts, look at the Pipeline Authoring product category listing.

Capturing User Feedback

We highly value user feedback! Please use the issue below to capture feedback and insights for our newest feature, CI/CD Catalog:

• CI/CD Catalog Feedback

Community Contributions

Pipeline authoring at GitLab involves a deep understanding of our CI/CD system and its intricate interactions with various components. This complexity can present challenges for those unfamiliar with the codebase and underlying architecture.

We encourage community contributors to work on issues with the ~“seeking community contribution” label. These issues are specifically selected to provide a clear and manageable path for external contributions.

Due to the inherent complexity of pipeline authoring, contributions to issues not marked for community contribution may require significant domain knowledge and familiarity with GitLab’s internals. While we appreciate all contributions, Merge Requests for these issues may require more extensive review and may not be merged if they don’t align with our architectural vision or best practices.

Guidance for Reviewers

When reviewing Merge Requests from community contributors, please consider the following:

• Issue Selection: Verify if the contribution addresses an issue labeled ~“seeking community contribution”. If not, assess the complexity and required domain knowledge.
• Mentorship: If the contribution shows promise but needs guidance, provide constructive feedback and consider offering mentorship to help the contributor succeed.
• Clear Communication: Explain the rationale behind any required changes or decisions clearly and respectfully, ensuring the contributor understands the context and reasoning.

If the issue picked up by the community does not have the ~‘seeking community contribution’ label and/or is too complex please consider using the following template:

"Thank you for your interest in contributing to GitLab! We appreciate you taking the initiative to work on this. This particular area involves complexities that may require deeper domain knowledge of GitLab's CI/CD system. To ensure a smoother review process and increase the likelihood of your contribution being merged, we recommend focusing on issues with the ~'seeking community contribution' label. These are specifically curated for external contributors. A curated list of issues is available here: https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=popularity&state=opened&label_name%5B%5D=group%3A%3Apipeline%20authoring&label_name%5B%5D=Seeking%20community%20contributions&first_page_size=100"

Group Meetings

For every meeting, it is expected that it contains a list of agenda items to discuss along with the meeting notes when the meeting takes place. This will make it easier for people to catch up if they are unable to attend.

Please note that sync meeting schedules are flexible and can be moved to accomodate required participants. For an up-to-date schedule of all team meetings, please consult the Group’s Calendar.

The table below briefly outlines the objectives and key details of regular team meetings:

Dashboards

Cross-functional prioritisation

UX, Product Manager and Engineering Manager meet weekly to discuss cross-functional prioritisation in addition to any other topics that require the quad to collaborate on. Additionally, the quad also reviews the dashboard which shows the % of MRs that are bugs vs maintenance vs features to ensure the team’s efforts are properly aligned to the prioritisation.

Design Collaboration

We hold a bi-weekly design sync meeting open to all team members where we discuss any design-related topic.

How We Work

Planning

Issues are refined and weighted prior to assigning them to a milestone. We use candidate:: scoped labels to help with planning work in future milestones. This label allows us to filter on the issues we are planning, allowing Product, Engineering, and UX to refine issues async that have workflow::design and workflow::ready for development labels applied. Weighting also helps with capacity planning with respect to how issues are scheduled in future milestones.

We create a planning issue as part of our milestone planning process and the workflow board is the single source of truth (SSOT) for current and upcoming work. Product is the DRI in prioritizing work, with input from Engineering, UX, and Technical Writers. The planning issue is used to discuss questions and team capacity. Prior to the beginning of each milestone, issues identified in the planning issue will be assigned to that milestone and engineers can assign prioritized issues to themselves from the top of the workflow::ready for development column in the workflow board.

Finding issues that need refinement

Issues that needs to be refined are listed in the planning issue associated with the upcoming milestone(s)

How Engineering Refines Issues

side note: we prefer using Refining over Grooming

The purpose of refining an issue is to ensure the problem statement is clear enough to provide a rough effort sizing estimate; the intention is not to provide solution validation during refinement.

Checklist for Refining Issues

1. Does the issue have a problem statement in the description?
2. Does the issue have the expected behaviour described well enough for anyone to understand?
3. Does the issue explicitly define who the stakeholders are (e.g. BE, FE, PM, UX and/or Tech Writer)?
4. Does the issue have a proposal in the description? If so:
   1. Does the proposal address the problem statement?
   2. Are there any unintended side effects of the implementation?
5. Does the issue have proper labeling matching the job to be done? (e.g. bug, feature, performance)
6. If the issue is a type::bug, is it clear how to reproduce the behavior and can a sample CI configuration file be provided?

Any one on the team can contribute to answering the questions in this checklist, but the final decisions are up to the PM and EMs.

Use of Sub-tasks

We use sub-tasks in issues to help with refinement. The goal is that we have one SSOT issue that contains threaded discussions around all aspects of the feature which include backend, frontend, UX, Product, and documentation.

Steps for Refining and Weighting Issues

Engineers will:

1. Go through the checklist above for refining issues they assign to themselves prior to each milestone.
2. Add a weight based on the definitions.
3. Update the ~workflow:: label to the appropriate status, for example:
   • ~“workflow::design” if further design refinement is needed, and let the designer know.
   • ~“workflow::ready for development” when refinement is complete and a weight has been applied, signaling that it’s ready for implementation and the issue can be scheduled accordingly.
   • ~“workflow::planning breakdown” if more investigation or research is needed, the status does not move, and the PM and EMs should be pinged.
4. Unassign themselves from the issue when they are done refining and weighting the issue.

Weighting Issues

The weights we use are:

The above table is contextual. Domain knowledge, experience level, and time at GitLab can impact an engineer’s perspective on whether an issue requires extra investigation or surprises.

Weights are not set in stone. We do our best to get it right during refinement, but we want to be transparent and accurate. If an issue is taking more effort than is reflected in the existing weight, the DRI on the issue is encouraged to change the weight. We want accurate documentation of the level of effort that was required.

Anything weighted as a 5 or larger should be broken down. These should not be ready for development.

Syntax deprecation process

Our CI syntax keeps evolving. We cannot support all keywords indefinitely, so deprecating and removing keywords is inevitable.

GitLab does not have a versioning system for CI/CD configuration. Therefore, it is critical to over-communicate our deprecation purposes to our users and take the necessary precautions to reduce the impact on their projects. Deprecating a keyword is risky because it will break all pipelines using it, and in some cases, users are not aware of the keyword they use in their pipeline. The deprecation process described below is similar to the deprecating and removing features process, with additional steps to reduce the risks which involved with removing a CI/CD keyword.

1. Deprecation notice - Syntax removal introduces a breaking change, as outlined in our deprecation process, we must notify the community and customers, which means including a deprecation notice in the monthly release post.

2. Track keyword usage - Tracking keyword usage should begin as early as possible. It is a mandatory step that helps estimate the user impact, timing, and needed effort. The more users use the keyword, the more time it takes to remove it (It took more than four years to move from remove to deprecation for ’type’ keyword).

3. In-app warning - Provide our users with an in-app notification that we plan to remove a keyword they use in their pipeline. Our customers will get a notification in each run of the pipeline that uses the deprecated keyword. The warning will be printed:

   • At run time on the pipeline page and logs.
   • In the pipeline editor, while authoring a pipeline.

   This step is optional if the keyword usage is relatively low (Recommend minimal reach of ~5% impacted users).

4. Keyword removal - The keyword will be removed from our code and schema and should happen in a major version. Once removed, using the keyword will result in a lint error.

Workflow

We use the Pipeline Authoring Workflow issue board to track what we work on in the current milestone.

We follow the product development flow to ensure that the problems we’re solving are well understood and the solutions are well defined and validated before the implementation.

Our UX counterpart is the DRI for the SSOT designs, and they have the right to push any enhancement discussions in MRs into follow-up efforts in created and linked issues.

We use workflow labels to efficiently communicate an issue’s state. Using these labels enables collaboration across teams and communicates an issue’s current state. Issues that have been refined but are not scheduled (are in Backlog) should have the workflow::scheduling label applied

The DRIs throughout each phase of the workflow are responsible for keeping the workflow labels up-to-date.

Async Issue Updates

In order to optimize async collaboration we use issue updates to share progress completed on a specific issue or epic.

Weekly updates on progress and status will be added to each issue by its assignee. A weekly update may be skipped if there was no progress. The update should be provided in the issue rather than in the related merge requests, as those do not provide a view of the overall progress. This applies to issues with the labels workflow::in dev, workflow::in review, or workflow::design.

The status comment can include what percentage complete the work is, the confidence of the person that their estimate is correct and, a brief note on what was done. It’s perfectly acceptable to have multiple updates if more than one DRI is working on the issue.

As a part of the async update it’s important to verify that the issue and related MRs workflow labels are correctly set.

Example

## Async status update

- Complete: 80%
- Confidence: 90%
- Notes: expecting to go into review tomorrow

Aligning on feature development

The Engineering DRI works with the Product Designer throughout the workflow:in dev phase to uncover possible problems with the solution early enough that exhibit unexpected behaviour to what was originally agreed upon. If there are changes to be added that weren’t agreed upon in the initial issue - a followup issue should be made and the Engineering DRI should work with the Product Manager to schedule that issue in a following iteration. This allows us to focus on cleanup over signoff, iterate quickly on issues with a low level of shame, and still make sure we accomplish what we’ve agreed upon. We should be careful not to hold off on completing these followup issues so that we don’t build up a significant amount of Deferred UX issues.

If we find that solutions are consistently not matching the agreed upon design, we will hold a retrospective with the DRI, designer, and product manager to discuss where the gaps in communication are so that we can improve. It may be necessary to begin requiring a UX approval for merge requests on certain issues to help the Engineering DRI meet the requirements.

Technical Debt

• We track our team’s technical debt using the following the Technical Debt Prioritization board, where we track issues with upcoming candidate labels and compare against current milestone.
• In an effort to promote more transparency across the stage where collaboration may be needed, starting in August 2024, we’ve introduced a Verify Pipeline teams Technical Debt board that shows the top of mind priorities in the Pipeline teams.

Retrospectives

The team uses a monthly async retrospective process as a way to celebrate success and look for ways to improve. The process for these retrospectives aligns with the automated retrospective process used by many teams at GitLab.

A new retrospective issue is created on the 27th of each month, and remains open until the 26th of the following month. Team members are encouraged to add comments to that issue throughout the month as things arise as a way to capture topics as they happen.

On the 16th of each month a summary of the milestone will be added to the issue and the team will be notified to add any additional comments to the issue.

As comments are added to the issue, team members are encourage to upvote any comments they feel are important to callout, and to add any additional discussion points as comments to the original thread.

Around the 26th of the month, or after the discussions have wrapped up the backend engineering manager will summarize the retrospective and create issues for any follow up action items that need to be addressed. They will also redact any personal information, customer names, or any other notes the team would like to keep private. Once that has been completed the issue will be made non-confidential and closed.

View the current Pipeline Authoring retrospective.

Team Communication

    <h2>Requires the group-slug argument</h2>
    <h2>Requires the group-slug argument</h2>

The team is globally distributed and separated by many timezones. This presents some challenges in how we communicate since our work days only overlap by a couple hours. We have decided as a team to embrace asynchronous communication because scheduling meetings is difficult to coordinate. We meet as a team one day per week, on Wednesdays for a team meeting and our team’s Engineering Managers schedule regular 1-on-1s.

• We are pro-active about asking questions in advance with the understanding that the turnaround on receiving an answer is usually 1-2 working days, subject to one’s workload.
• Our engineering plan and issue boards need to be updated regularly to communicate what stage our work is in.
• Any meetings that we hold need a preset agenda and are recorded for folks who are unable to attend.
• Having a positive work/life balance, despite these challenges, must be a high priority and handled intentionally so no one is routinely starting early or staying late to attend meetings.

How to work with us

Slack

Daily standup updates are posted to [g_pipeline-authoring](https://gitlab.slack.com/archives/CPCJ8CCCX', group_label: ‘group::pipeline authoring’, group_issues_url: ‘https://gitlab.com/groups/gitlab-org/-/issues?label_name%5B%5D=group%3A%3Apipeline+authoring&scope=all', gitlab_subgroup: ‘pipeline-authoring-group).

Feel free to ask us questions directly in this Slack channel and someone will likely get back to you in 1-2 working days. We will use following emojis to respond to the posted question accordingly:

• - :eyes: to indicate that one of us has seen it
• – :white_check_mark: to indicate that the question has been answered

The verify stage has a separate Slack channel under #s_verify, which encompasses the other groups of Verify.

GitLab Issues and MRs

Most spontaneous team communiation happens in issues and MRs. Our issues have a group label of #. You can also tag a team member with @mention in the issue if you have someone specific to ask.

If you need to call the attention of the entire group, you can tag @gitlab-com/<%= gitlab_subgroup %>.

Shared Team Calendar

The Pipeline Authoring team has a shared calendar that integrates with the #g_pipeline-authoring Slack channel to inform the team of important dates from Time Off by Deel.

Instructions on how to integrate with the Verify:Pipeline Authoring calendar can be found here.

Developer Onboarding

Refer to the Developer Onboarding in Verify section.

View page source - Edit this page - please contribute.