Package Stage
Mission and Vision
The Package stage creates a secure environment where both source code and dependencies can live by allowing you to publish, consume, and discover packages across a variety of languages and platforms all in one place. Our vision is to be our customers’ single source of truth for storing and distributing packages and container images across the entire DevOps lifecycle.
Team Structure
The Package stage consists of two groups:
- Package:Container Registry - Backend Go service for OCI-compliant container image storage
- Package:Package Registry - Package management features within GitLab Rails application
Team Members
The following teams make up the Package stage:
Stable Counterparts
| Name |
Role |
 Greg Myers
|
Security Engineer, Application Security, Package (Package Registry, Container Registry), US Public Sector Services, Gitaly Cluster, Analytics (Analytics Instrumentation, Product Analytics), AI Working Group |
 Jackie Porter
|
Director of Product Management, Verify & Package |
 Tim Rizzi
|
Principal Product Manager, Package |
| Format |
DRI |
| npm |
@dmeshcharakou |
| Maven |
@10io |
| PyPI |
@radbatnag |
| NuGet |
@mkhalifa3 |
| Terraform |
@radbatnag |
| Generic |
@dmeshcharakou |
How We Work
Milestone Planning Process
Our milestone planning aligns closely with our product roadmap and engineering commitments. The process balances delivery of strategic initiatives with operational needs like security issues and customer bugs.
Product Manager responsibilities:
- Creates milestone planning issue with clear goals based on roadmap
- Defines key projects and expected outcomes
- Ensures security issues and customer bugs are considered
- Maintains alignment with product strategy and roadmap
Engineering Manager and DRI responsibilities:
- Reviews carryover work from previous milestone
- Selects next set of issues from their projects to meet roadmap timelines
- Evaluates realistic delivery capacity
- Surfaces risks and timeline impacts early
- Updates roadmap dates if needed
Issues are prioritized as:
Package:P1: Committed work for the milestonePackage:P2 + Stretch: Stretch goals if capacity allows
Non-roadmap items that may be planned include:
- Security issues (to meet SLAs)
- Customer bugs
- Minor reliability/performance fixes
Instrumentation is a key requirement for all features to measure impact and effectiveness. This helps us validate adoption and make data-driven decisions.
Issue Management
Async Issue Updates
The purpose of async updates is to communicate progress and allow others to prepare for upcoming work as necessary. In an all-remote culture, we keep the updates asynchronous and put them directly in the issues.
The async update communicates the progress and confidence using an issue comment and the milestone health status. Add a comment in your issue with the title Async Update once per week, or when something notable happens with regard to the issue. It’s preferable to update the issue rather than the related merge requests.
The async update comment should include:
- what percentage complete the work is, in other words, how much work is done to put all the required MRs in review
- the confidence of the person that their estimate is correct
- notes on what was done and/or if review has started
- it could be good to include whether this is a front end or back end update if there are multiple people working on it
Issue Weighting Guidelines
| Weight |
Description |
Confidence Level |
| 1: Trivial |
Well understood, no investigation needed, exact solution known |
≥90% |
| 2: Small |
Well understood, minimal investigation needed, few surprises expected |
≥75% |
| 3: Medium |
Well understood but requires investigation, some surprises expected |
≥60% |
| Larger |
Should be broken down into smaller issues |
≥50% |
An issue with weight 1 should take no more than 2 days to complete.
Current Projects
Container Registry
Package Registry
Technical Guidelines
Breaking Changes Process
Breaking changes can be disruptive to our customers’ workflow and we attempt to avoid them or reduce the impact. When a breaking change is unavoidable, we provide as much visibility and advance notice as possible.
- Announce deprecations, breaking changes, and removals at least 3 milestones before implementation
- Before major version milestone:
- Implement breaking change with a feature flag
- Feature flags are mandatory unless explicitly justified
- In major version milestone:
- Roll out the feature flag
- If stable, open feature flag cleanup MR
- If issues occur:
- Fix before milestone end and re-roll out, or
- Disable flag and defer to next major version
We follow a structured process for releasing new package formats:
-
Experiment Phase
- Feature flag enabled on staging
- Comprehensive testing:
- Small and large package operations
- Performance with hundreds of packages
- All supported authentication methods
- Custom GitLab-specific features
- Document any issues in feature flag rollout issue
-
Beta Phase
- Limited GitLab.com rollout
- Dogfooding with internal teams
- Selected customer testing
- Bug triage and resolution
-
Generally Available
- Full documentation update
- Feature flag removal
- Self-managed release
- Release communications
Alert and CI Flake Management
The team monitors the Slack channel #g_container-registry_alerts for service alerts and CI notifications. Process for handling alerts:
- Everyone is responsible for monitoring alerts during their working hours
- When a new alert appears:
- Add 👀 emoji to signal you’re investigating
- Review alert details (runbook, dashboard, pipeline, Sentry issue)
- Use available resources to evaluate the problem
- Determine if safe to ignore based on:
- Existing issue coverage
- Auto-resolution
- Logs showing resolution
- If not safe to ignore:
- Review #production and #incident-management channels
- Consider reporting an incident
- Share details in #g_container-registry
- Add a comment thread to document your review
- Add ✅ emoji when resolved
Alert Occurrence Template
## Alert Occurrence Update
- **Occurrence Count**: X (previously Y)
- **Date/Time**: [Insert timestamp]
- **Last occurrences**: [Insert slack link]
Key Resources
Product Context
Container Registry History
Originally launched in milestone 8.8, the Container Registry integrated Docker Distribution registry into GitLab. A key challenge was storage management - deleted images weren’t actually removed from storage without downtime-requiring garbage collection.
To address this and enable future features, we:
- Forked Docker Distribution
- Implemented online garbage collection
- Added a metadata database
- Created cleanup policies
This work enables future capabilities like:
- More robust API and UI
- Enterprise features (image signing, protection)
- Improved stability and reliability
The registry adheres to OCI standards for Image and Distribution specifications while maintaining backward compatibility with Docker specifications.
Measuring Results
- Lighthouse Metric: (Count of Packages Published + Count of Containers Published) / Billable Users
- Monthly Active Users (GMAU)
- Error Budget compliance
- Say-Do ratio for delivery commitments
- Operating costs:
- Storage costs
- Data transfer costs
Dashboards
Additional Resources
Testing Strategy
Local Development Testing
-
Unit Tests
- Container Registry (Go):
make test
- Package Registry (Rails):
bundle exec rspec ee/spec/models/packages/
- Frontend:
yarn jest ee/spec/frontend/packages
-
Integration Tests
- Test against different storage backends
- Verify API endpoints
- Check authorization rules
End-to-End Testing
The Package group uses GitLab QA for E2E testing. This is crucial for features that integrate multiple services.
Common Test Scenarios
-
Container Registry
# Publishing a container image
Resource::Container::Image.fabricate_via_api! do |image|
image.name = 'my-image'
image.repository_name = 'my-project'
end
-
Package Registry
Overview
The goal of this page is to create, share and iterate on the Jobs to be Done (JTBD) and their corresponding job statements for the Package stage. Our goal is to utilize the JTBD framework to better understand our buyers’ and users’ needs.
Goals
Utilize JTBD and job statements to:
- Understand our users’ motivations.
- Validate identified use cases and solutions.
- Continuously test and iterate features to ensure we are meeting our customers’ and users’ needs.
- Create a transparent view for our stakeholders into the current and future state of the product.
Challenges
- Most customers do not purchase GitLab solely for the Package stage. So, there will be assumptions in the JTBD that from the buyer’s perspective, the motivations for ‘buying’ Package are the same.
Jobs To Be Done
Reliable software dependencies
When building and deploying software, I need to reliably fetch my project’s software dependencies, so my development and product ecosystem can operate without interruption.
Crystal Poole
Backend Engineer
Hayley Swimelar
Jaime Martínez
João Pereira
Senior Backend Engineer
Senior Backend Engineer
David Fernandez
Dzmitry Meshcharakou
Moaz Khalifa
Rad Batnag