CI Catalog

September to October (Milestone 17.5)

Milestone 17.5 (September 14, 2024 - October 11, 2024)

Goals

• Visibility into where components are used - Epic
  • Create fields to return project list where components were used in a pipeline - #466575 (in-dev)
• Security & Compliance workflow epic for CI Catalog
  • Spike + POC Security policies - publish vs usage ci components - #474093
• Release/Publish 2.0 enhancements
  • Add new publish API endpoint with input params - #442783 (in-review)
  • release-cli to extract and validate metadata - #442785
  • Add an indicator if the release goes to the catalog - #438958
• Index/Details page enhancements
  • Update filtering options in CI/CD Catalog index page (Backend part) - #437643
  • Better visualization when the project description is long - #448385
• Search/Filter enhancements
  • Add the by publishing date option to the sort dropdown in /explore - #440508
  • Add an illustration in the search result page - #466412

August to September (Milestone 17.4)

Milestone 17.4 (August 9, 2024 - September 13, 2024)

Goals

• Visibility into where components are used - Epic
  • Create fields to return project list where components were used in a pipeline - #466575 (in-dev)
• Release/Publish 2.0 enhancements
  • Add new publish API endpoint with input params - #442783 (in-review)
• Inputs enhancements
  • Allow interpolation to use local context data - #438275 (in-dev)
  • POC to create JSON schema SSOT for spec keyword - #467375 (complete)

July to August (Milestone 17.3)

Milestone 17.3 (July 13, 2024 - August 9, 2024)

Goals

• Allow administrator to restrict users from publishing to CI/CD Catalog
  • Add GraphQL mutations and types to policy #465265 (blocked by outcome of POC)
• Inputs enhancements
  • Allow interpolation to use local context data - #438275 (in-dev)
  • POC to create JSON schema SSOT for spec keyword - #467375 (in-dev)

June to July (Milestone 17.2)

Milestone 17.2 (June 15, 2024 - July 12, 2024)

Goals

• Index/Details page enhancements
  • Add illustration in the search result page - #466412 (Deferred to candidate::17.4)
  • Add type and description to InputType for Components tab - #466669 (Complete)
• Release/Publish 2.0 enhancements - span multiple milestones
  • Add new publish API endpoint with input params - #442783 (Continuing into 17.3)
• Admin capabilities in CI/CD Catalog - span multiple milestones
  • Add ci_component_source_policy JSON schema - #465264 (Complete)
• Inputs enhancements
  • Allow interpolation to use local context data - #438275 (To start in 17.3)
  • POC to create JSON schema SSOT for spec keyword - #467375 (Continuing into 17.3)

May to June (Milestone 17.1)

Milestone 17.1 (May 11, 2024 - June 14, 2024)

Goals

• Create API to support future requests for badge additions (Complete)
• How to use components from different Cells (In Progress)
  • From recent conversation, determining if CI Catalog support can be deferred to Cells 1.5 at the moment.
• Post-GA follow-ups/technical debt
  • Exclude pre-release from catalog, latest tag, and shorthand fetching (Complete)
  • Add sorting option for prerelease for semver concern (On Hold)
  • Remove ignore rule on inputs and path for catalog_resource_components (Complete)
  • GA follow-ups from “Create CI component usage record when component is included in pipeline” - 1 (Complete) and 2 (Complete)
• Tableau component usage visualization work - 1 (Complete) and 2 (Complete)

April to May (Milestone 17.0)

All deliverables for CI Catalog GA are now complete.

• Finish remaining Go-To GA efforts
  • Remove beta label for CI/CD Catalog (Complete)
  • Remove beta label for catalog resource toggle (Complete)
  • Release Process Refinements for GA
    • release-cli pre-GA tasks (Complete)
  • Details page for GA
    • Relative URL breaks CI/CD component catalog project reference (Complete)
    • Fix images not rendering on ReadMe tab (Complete)
• Finishing remaining initial loading badges work
  • Set verification_level on publish and fix enum mismatch (Complete)
  • Allow service object to create VerifiedNamespace record (Complete)
  • Create API to support future requests for badge additions
  • NOTE: Initial badge load has been initiated via request

March to April (Milestone 16.11)

Goals

• Go-To GA efforts
  • Inputs for GA (Complete)
    • Change catalog_resource_components.inputs to spec (Complete)
    • CI Interpolation for arrays (Complete)
  • Instrumentation for GA (Complete)
    • Table creation for component tracking usage (Complete)
  • Release Process Refinements for GA
    • release-cli pre-GA tasks (In-Verification)
    • Support Self Managed components (Complete)
  • Details page for GA
    • Relative URL breaks CI/CD component catalog project reference (FE work In-Review / BE work complete)
    • Construct component path from parts (stop fetching it from the database) (Complete)
    • Remove the experimental label in the component tab (Complete)
• CI Catalog UX improvements
  • Add badges for components (Complete)
    • NOTE: Backend issue remains before badging starts showing up in CI Catalog.
  • Reorganize information in the detail (In-Review)

February to March (Milestone 16.10)

Goals

• Go-To GA efforts
  • Support Self Managed component to distribute components for Self managed customers. (In Verification)
  • Semantic versioning (Complete)
  • Inputs Enhancements
    • Boolean and number support (Complete)
    • Text interpolation with arrays (In Verification)
  • release-cli pre-GA tasks (FE Complete /BE In-Dev)
• Telemetry instrumentation for GA
  • Implement Tracking for release execution time (Complete)
  • Table creation for component tracking usage (In-Dev)
• CI/CD Components to GA work
  • ~latest returns latest semantic version (Complete)
  • Migrate Versions to follow SemVer convention (Complete)

January to February (Milestone 16.9)

Updates for current Go-To-GA list

• Enforce semantic versioning
  • POC currently in-progress and in review - continue to 16.10
• Support CI interpolation with arrays
  • Implement text interpolation - (Complete)
  • CI interpolation with arrays - To be continued in 16.10 after its prioritized blocker
• Spikes
  • Spike issue to distributed components for Self managed customers.
  • Spike issue to calculate number of times a component is used.
• Threat Model diagrams in-progress to be provided to security. - waiting on security feedback.

Other milestone goals

• Improve the UX for the CI/CD catalog
  • Make star rating default sorting - continue to 16.10 due to capacity
  • Fix Markdown not rendering in CI/CD Catalog (Complete)
• Helper efforts for components
  • Provide components as helpers to test other components - Waiting on product confirmation on prioritization for 16.10
  • Components toolkit to test GitLab-maintained components (Complete)
• Badges
  • Create catalog_verified_namespaces table (Complete)

December to January (Milestone 16.8)

Goals

• Complete initial template to component migration list.
  • AutoDevOps Build component and Test component is complete but discussion on whether Test should exist due to future deprecation.
• Improve UI in the Catalog details page [1, 2]
• Implement Your resource tab in the index page
• Add components tab to the catalog details page
  • BE/FE collaboration needed before feature flag can be rolled out.
• Move inputs to GA ready by completing text interpolation for arrays and !reference.

November to December (Milestone 16.7)

CI Catalog - Pages & Navigation

• 100% complete, Complete, Delivered in 16.7
• Status: As of 2023-12-08, last MR is merged to make Catalog available in explore navigation permanently.

CI Catalog - Search & Filter

• 100% complete, Complete, Delivered in 16.7
• Status: All Beta work is in production.
• Risks/Blockers: Beta work complete.

CI Catalog - Release Process refinements

• 100% complete, Complete, Delivered in 16.7
• Status: On 2023-12-01, the toggle back and forth is complete is now in production to complete all Beta work.
• Risks/Blockers: Beta work complete.

October to November (Milestone 16.6)

• CI Catalog - Pages & Navigation
  • Move shared components to Free Tier
  • Add route and nav for Global CI/CD Catalog
  • Coordinate with Foundations on adding Global catalog to the Explore navigation
  • Make the README tab the default view for component
  • Add an indicator to the catalog resource project
  • Prepare Ci::Catalog::Listing for global CI Catalog
  • Move GraphQL Catalog code to FOSS
  • Remove fork count from Catalog details page
  • Add a copy-to-clipboard button in the code snippet in the component tab
  • Empty state when there is no metadata for the components details
  • Add global Catalog arguments to GraphQL
  • Add Vue application for Global page ~workflow::in review
  • Make ciCatalogResource accept a fullpath argument ~workflow::in dev
  • Add components field to ciCatalogResource ~workflow::in dev
  • Add the new components tab
  • FF rollout ci_catalog_components_tab
  • Add namespace to scope for Catalog resources query
  • Remove projectPath argument from ciCatalogResources
• CI Catalog - Search & Filter
  • Add GraphQL search filter and sort by created_at to ciCatalogResources
  • Create database indices for CI Catalog
  • Denormalize name and description in Ci::Catalog::Listing
  • Add Search bar ~workflow::in review
  • Update catalog_resource.latest_released_at when version is created/deleted
• CI Catalog - Release Process refinements
  • Add mutation to mark a catalog resource as draft
  • Add path column where to persist full path to component YAML file
  • Update the docs to reflect the recommended method for the release
  • Fix regexp to scan for component files
  • Scanning components on release and collect metadata
  • Update the releases logic in catalog resources to use the Version association ~workflow::in review
  • Create a migration to update state and add metadata to existing catalog resources
• Completion of [Inputs Enhancements]
  • Support options: with inputs defining default: value

Week of October 2, 2023

Goals

• Frontend CI Catalog Details page work
• Scanning components on release and adding release sort

Week of September 25, 2023 (Milestone 16.5)

Team Capacity

• 3 Backend Engineers (Leaminn, Avielle, Laura)
• 1 Frontend Engineer (Frédéric)

Goals

• https://gitlab.com/gitlab-org/gitlab/-/issues/387632 to add support for variables ininputs: syntax so when expand_vars is used, error is raised due to security reasons. ~workflow::in review
• https://gitlab.com/gitlab-org/gitlab/-/issues/408382 to add released sort to CI Catalog.
• https://gitlab.com/gitlab-org/gitlab/-/issues/411438 to support CI interpolation with arrays.
• https://gitlab.com/gitlab-org/gitlab/-/issues/415413 to scan a catalog resource for components on release creation.
• https://gitlab.com/gitlab-org/gitlab/-/issues/424962 to put the current right side column in the Catalog header. - ~workflow::in review
• https://gitlab.com/gitlab-org/gitlab/-/issues/424966 to add the README tab with the current content.

Week of September 18, 2023 (first week of Milestone 16.5)

Team Capacity

• 4 Backend Engineers (Avielle, Laura, Kasia, Leaminn)
• 1 Frontend Engineer (Fred)

Goals

• https://gitlab.com/gitlab-org/gitlab/-/issues/387632 to add support for variables ininputs: syntax so when expand_vars is used, error is raised due to security reasons. ~workflow::in review
• https://gitlab.com/gitlab-org/gitlab/-/issues/415413 to scan a catalog resource for components on release creation.
• https://gitlab.com/gitlab-org/gitlab/-/issues/408382 to add released sort to CI Catalog.
• https://gitlab.com/gitlab-org/gitlab/-/issues/424962 to put the current right side column in the Catalog header - ~workflow::in review
• https://gitlab.com/gitlab-org/gitlab/-/issues/415637 to add an optional description field under input - handled by Community Contributor

Week of September 11, 2023 (last week of Milestone 16.4)

Team Capacity

• 2 Backend Engineers
• %16.4 security priorities are nearly complete so more BE focus is shifting in later %16.4

Goals

• https://gitlab.com/gitlab-org/gitlab/-/issues/415413 to scan a catalog resource for components on release creation.
• Spike follow-up to https://gitlab.com/gitlab-org/gitlab/-/issues/411438

Week of September 4, 2023 (Milestone 16.4)

Team Capacity

• .5 Backend Engineers
  • Due to %16.4 security priorities, the weekly goals list will be shorter due to our focus there.
  • Working on https://gitlab.com/gitlab-org/gitlab/-/issues/423456 for feature addition for needs: parallel: matrix enhancements.

Goals

• https://gitlab.com/gitlab-org/gitlab/-/issues/418996 for marking catalog resource as draft, if final version removed.

Week of August 28, 2023 (Milestone 16.4)

Team Capacity

• 1.5 Backend Engineers
  • Due to %16.4 security priorities, the weekly goals list will be shorter due to our focus there.
  • Working on https://gitlab.com/gitlab-org/gitlab/-/issues/423456 for feature addition for needs: parallel: matrix enhancements.

Goals

• https://gitlab.com/gitlab-org/gitlab/-/issues/411394 for adding instrumentation for number of components are used.
• https://gitlab.com/gitlab-org/gitlab/-/issues/418996 for marking catalog resource as draft, if final version removed.

Week of August 21, 2023 (Milestone 16.4 begins)

Team Capacity

• 3 Backend Engineers

Goals

• https://gitlab.com/gitlab-org/gitlab/-/issues/411394 for adding instrumentation for number of components are used.
• https://gitlab.com/gitlab-org/gitlab/-/issues/415853 for updating CI component fetching for updated directory structure - to be merged this week.
• https://gitlab.com/gitlab-org/gitlab/-/issues/409846 work continues for creating an SSOT for CI config loading
• https://gitlab.com/gitlab-org/gitlab/-/issues/411438 CI interpolation with arrays spike work continues.

Week of August 14, 2023 (Milestone 16.3 ends)

Team Capacity

• 3 Backend Engineers
• 2 Frontend Engineers

Goals

• https://gitlab.com/gitlab-org/gitlab/-/issues/409041 for showing pipeline status for latest version of catalog resource.
• https://gitlab.com/gitlab-org/gitlab/-/issues/415287 for creating catalog_resource_components table to unblock other issues.
• https://gitlab.com/gitlab-org/gitlab/-/issues/412948 for updating permissions for namespace catalog & update resolver
• https://gitlab.com/gitlab-org/gitlab/-/issues/409846 for complete last MR for CI config loading

Week of August 7, 2023 (Milestone 16.3)

Team Capacity

• 1.5 Backend Engineers
• 2 Frontend Engineers

Goals

• https://gitlab.com/gitlab-org/gitlab/-/issues/418785 for moving CI Catalog to be a premium feature.
• https://gitlab.com/gitlab-org/gitlab/-/issues/390458 for input type validation.
• https://gitlab.com/gitlab-org/gitlab/-/issues/409041 related to showing pipeline status for latest version of catalog resource.
• https://gitlab.com/gitlab-org/gitlab/-/issues/415287 for creating catalog_resource_components table to unblock other issues.
• https://gitlab.com/gitlab-org/gitlab/-/issues/415853 for updating CI component fetching for updated directory structure.