Global Search - How we work

Scope of this page

This page carries the Global Search group’s own process. The group page stays short: who the team is, what it builds, what it runs, and what it is working on. Anything the team itself needs to look up — how work is scheduled, who reviews what, which automations exist, how the indexers are released, and how production incidents reach us — lives here.

Work

We follow the general workflow and principles defined in Product Development Flow and Engineering Workflow. To bring an issue to our attention, create an issue in the relevant project, add the ~"group::global search" label and any other suitable labels. For an urgent issue, contact the Product Manager or the Engineering Manager named on the group page.

Guidelines the team follows day to day:

• We use asynchronous communication with each other and with other GitLab teams via GitLab, Slack, Google Docs, and similar.
• We encourage all backend engineers in the team to have their changes reviewed by someone else in the group. It is good for knowledge sharing.
• We organize our tasks under epics and issues. The Product Manager and Engineering Manager go through the backlog at the planning phase of each release and put issues into the next one or two milestones. Issues on the milestone board are sorted by priority, highest first.
• We apply the Deliverable label to issues we intend to close in a milestone, before the milestone starts. Issues added during a milestone should not have it. We review these issues in the middle of the milestone, usually the first week of each month, and remove the label from issues that are not likely to make the release.
• We apply the Stretch label to issues we intend to start during a milestone but are not committing to close.
• We work with the UX team on features needing design input by applying a UX workflow label and assigning the UX counterpart. We use workflow::problem validation and workflow::solution validation for user research and workflow::design for UI design and prototyping; workflow::ready for development indicates development can start. For minor UX changes we contact our UX counterpart or the Product Design Manager directly.
• We work with the Developer Experience team on issues needing a testing perspective by creating an RFH.
• We work with the Technical Writing team on issues needing documentation change by applying the documentation label and assigning our counterpart. The documentation change normally happens together with the code change.
• We work with our stable counterpart in the Security team on issues needing a security perspective, using team planning issues for communication.
• We collaborate with the Support Engineering team on issues directly, and invite our counterpart to the team meeting every month.
• When team members are ready for their next task they pick an issue from the milestone board and assign it to themselves, prioritizing Deliverable issues. The issue owner is responsible for the solution, and can break the issue into smaller iterative sub-issues.
• Before going out of office for an extended time, assign items still in review to the Engineering Manager, who reassigns as needed. Whoever reviews work from someone out of office is welcome to complete the requested changes if they are comfortable with the remainder.
• We review and prioritize bugs every week. Severity approximates the Risk Matrix for potential risk and frequency; priority is based on total impact over time. The weekly pass is: review all new bugs for content, priority, severity and milestone; review bugs missing priority or severity; prioritize bugs for the current milestone, targeting 10% of scheduled work; and schedule the rest against capacity, severity, priority and relationship to scheduled work.

MR reviews

• The MR author decides whether the initial or maintainer review should be done by a Global Search team member, and indicates that in a comment or by assigning reviewers.
• Draft status means the MR is not ready to merge, but the author may still assign a reviewer. Unless the review is urgent, wait for the pipeline to pass before assigning one.
• We use Conventional Comments in review comments.
• The author resolves only the threads they have fully addressed; anything else is resolved by the reviewer. On an MR with many threads it helps the reviewer to revisit open threads to pick up where the discussion was left.

Reviewer and maintainer types

Beyond the usual backend and frontend maintainer roles, the team owns two shared components with their own maintainer groups. Both are enforced through .gitlab/CODEOWNERS in gitlab-org/gitlab, so an MR touching the paths below requires approval from that group whoever the author is.

The distinction between the required and optional sections matters: a change to a core framework abstraction blocks on a framework maintainer, while a new entity implementation only requests one. See the advanced search migration review guidelines before reviewing in the migrations group.

Automation

Each automation’s process is documented in the repository it runs from. The links below are entry points, not a duplicate of the process.

Marking advanced search migrations obsolete

Migrations older than one milestone before the last required stop are marked obsolete automatically. Two GitLab Housekeeper keeps do the work, both living in gitlab-org/gitlab:

1. Keeps::MarkOldAdvancedSearchMigrationsAsObsolete — prepends Search::Elastic::MigrationObsolete, swaps the spec for the deprecated-migration shared example, updates the dictionary file, and assigns a Global Search backend engineer.
2. Keeps::DeleteObsoleteAdvancedSearchMigrations — deletes all but the most recent obsolete migration.

The marking keep runs monthly from the engineering-productivity/team scheduled pipelines, in the automatic-mark-search-migrations-obsolete job defined in that project’s .gitlab-ci.yml. The schedule sets AUTO_MARK_OBSOLETE_SEARCH_MIGRATIONS, which gates both the job and its failure alert to #g_global_search_alerts. The removal keep is run manually.

The MR assignee still has manual steps: checking marked_obsolete_by_url, clearing .rubocop_todo/ references, removing migration_has_finished? backwards-compatibility branches, and backing up deleted migrations to the migration graveyard. These are listed in the advanced search migration style guide.

Indexer releases

Both indexers release themselves weekly from a scheduled pipeline and document the process in-repo:

1. Elasticsearch indexer: doc/process.md in gitlab-org/gitlab-elasticsearch-indexer. A weekly schedule with SCHEDULE_TYPE=weekly_release runs release_create_mr, which uses git-cliff to decide whether conventional commits warrant a bump, opens the release MR, and tags on merge. renovate-gitlab-bot then opens the MR bumping GITLAB_ELASTICSEARCH_INDEXER_VERSION in gitlab-org/gitlab. That document also lists the current maintainers and the trainee maintainer process.
2. Zoekt indexer: DEPLOYMENT_PROCESS.md in gitlab-org/gitlab-zoekt-indexer, covering both the indexer and the Zoekt webserver. Tagging is automated the same way, but propagating a release is a manual chain across the CI image, CNG, the gitlab-zoekt chart and the main GitLab chart. Follow the steps in that file rather than reproducing them here.

Incident escalation

For a production incident needing Elasticsearch or Zoekt expertise, escalate through the Tier 2 On-Call Program. SREs and on-call engineers can reach the team through the global-search-team Slack group. The Global Search Incident Management document is the reference for catching up on development status while resolving an incident, and in an emergency we follow the Incident Management handbook.

Because the team is small and needs specific domain knowledge, we borrow team members with Elasticsearch expertise from other groups to cover on-call escalation when understaffed, especially over holidays. Domain experts are identified by domain_expertise on their team member profile and may be contacted when SRE and Tier 2 on-call engineers cannot resolve a production incident. We do not expect domain experts to work outside their normal working hours.

Onboard a domain expert from another group

• Suggest the team member add elasticsearch as their domain_expertise in their team member profile.
• Add the team member to the global-search-team Slack group, which SREs and other on-call engineers use to contact us in an emergency.
• Create the access request granting them access to the Elasticsearch cluster.
• Schedule a walk-through session covering the current architecture and development status.

Offboard a domain expert

• Remove the team member from the global-search-team Slack group.
• Revoke the team member’s Elasticsearch cluster access.

Breaking changes

Before a major milestone starts we prepare an epic with all the breaking change issues linked. We work to get approvals as usual but keep each MR in draft to prevent it merging before the major milestone. An independent MR can target master; otherwise we use a sequence of MRs whose target branches point at each other, so each one retargets master as the previous merges.

Every MR created before the breaking change milestone carries a warning in its description stating that it must be kept as a draft and cannot be merged until the milestone date.

Bugfix backports

We review bugfix merge requests every week, using the scoped labels backport::required, backport::skip and backport::complete.

• backport::skip is applied to merge requests that need no backport.
• backport::required is applied in the initial review to merge requests that need backporting to a previous release. The DRI follows the patch release process to backport the fix, then applies backport::complete.

Performance testing

We are exploring Rally for performance testing the Elasticsearch cluster. Workload data is determined using Kibana and stored in a Google Sheet (internal).

Boards and links

• Global Search Team Milestone Board
• Global Search Team Workflow Board
• Bug Review Board
• Global Search Team Epics

View page source - Edit this page - please contribute.