Handbook Development

Development environment and processes for maintaining the handbook

The handbook site uses Hugo for static page generation from Markdown.

The handbook uses a base theme called Docsy. A custom theme override is located in the Docsy GitLab Theme project and automatically included in the handbook setup.

Support

For help and support with the development environment, please reach out in the public #handbook Slack channel.

If you encounter a problem or bug, please open an issue or MR in the respective handbook projects.

Edit the handbook in your browser

Follow the editing the handbook documentation.

Architecture Overview

flowchart TD
    A(docsy) --> B
    B(docsy-gitlab) --> H
    B(docsy-gitlab) --> I
    C(www-gitlab-com) --> H
    C(www-gitlab-com) --> I
    D(handbook) --> H
    E(handbook-internal) --> I
    H{build} --> |Deploy| F
    I{build} --> |Deploy| G
    F[handbook.gitlab.com]
    G[internal.gitlab.com]

There are four internal projects and one external project used in the process of building and deploying the internal and external handbooks.

  1. Docsy is the base theme used for both handbook sites.
  2. Docsy GitLab is an extension to the base Docsy theme with GitLab specific customizations and features. Docsy GitLab is included in both handbook sites.
  3. www-gitlab-com is the source repository of data files used by both handbook sites.
  4. handbook is the public handbook site.
  5. internal-handbook is the internal handbook site.

Run the handbook locally for edits

Supported methods:

  1. Docker or compatible runtime (all software preinstalled in a container image)
  2. Local installation (experienced users, more software dependencies, can break)

Requirements

  • Command line: git, wget (for syncing data). Additional requirements for local installation, see below.
  • Docker or compatible runtime (for running the Hugo environment in a container)
    • On macOS: Docker Desktop, Rancher Desktop, etc.
    • On Linux: Docker engine, Podman, etc.

Clone the handbook Git repository

Cloning the repository allows you to manually edit the handbook locally. If you prefer to use the Web IDE, please continue reading the editing the handbook documentation.

We recommend using git to clone the repository and then editing the handbook with a text editor such as Visual Studio Code, Typora, Nova or Sublime to name a few.

Clone the repository:

Public handbook:

git clone git@gitlab.com:gitlab-com/content-sites/handbook.git

Internal handbook:

git clone git@gitlab.com:gitlab-com/content-sites/internal-handbook.git

Docsy GitLab theme:

git clone git@gitlab.com:gitlab-com/content-sites/docsy-gitlab.git

Set up the repository

After cloning the repository, sync the required data files from the data file location (currently the www-gitlab-com repository). Without this step, the handbook cannot be run locally.

Open a terminal, navigate into the cloned handbook repository path, and run the sync-data.sh script.

Example for the public handbook:

cd handbook

./scripts/sync-data.sh

Running Hugo

Hugo needs to be run to generate the static files from the handbook markdown content. There are two ways to run Hugo:

  1. Running the Handbook in Docker
  2. Running the Handbook Locally

Parameters for Hugo

The handbook is huge, and by default, the hugo server command loads everything to memory.

The following options for the hugo command can be helpful for debugging or otherwise running locally:

  • --environment=production: generate a production build (asset minification, checksums, etc)
    • Note: hugo v0.x.x+extended version must be installed, use hugo version to check
    • Note: npm i must be run as a prerequisite to install postcss and autoprefixer dependencies
  • --renderToDisk: slower but requires less memory to run. Useful if you have less than 16GB allocated to docker machine
  • --verbose: enables verbose logging output
  • --templateMetrics and --templateMetricsHints: prints metrics related to how frequently templates are invoked and how much time is being spent evaluating them
  • --printMemoryUsage: periodically prints memory usage while the site is building

Build static files

To render the entire site to disk (and inspect the output in ${PWD}/public), purge the generated files first, and then run Hugo.

make clean

docker run --rm -v $(pwd):$(pwd) -w $(pwd) hugomods/hugo:exts hugo
# or
hugo

Permissions

For some pages to render you may need a personal access token. Generate one and export it as an environment variable prior to running hugo commands:

export HUGOxPARAMSxGITLAB_API_KEY="<token>"
hugo ...

If you need to make changes to the underlying theme you’ll need to make changes in the Gitlab-Docsy module.

Linting content

We use markdownlint-cli2 and Vale to enforce rules in handbook content.

markdownlint

Refer to the markdown guide for style guide information, and guidance on rules.

We use markdownlint-cli2 in our pipelines with a slightly customized set of rules. Before pushing any changes, you should run markdownlint-cli2 and fix any suggested changes to avoid pipeline failures.

To run markdownlint-cli2 using Docker, run:

docker run --rm -v $(pwd):$(pwd) -w $(pwd) davidanson/markdownlint-cli2 content/**/*.md

If you have markdownlint-cli2 installed locally, you can run the command in the relevant repository. This method is recommended when editing the config file (look for the markdownlint file), and testing the changes. To have markdownlint automatically fix some errors, add the fix option:

markdownlint-cli2 --fix "content/**/*.md"

Rules are configured to be close to the GitLab documentation markdownlint, without some of the stricter styling rules. The relevant rules are also noted in the markdown guide

Vale

We use Vale to warn when some rules from the Handbook Markdown Guide. are broken. Vale is not run in pipelines.

To run Vale using Docker, run:

docker run --rm -v $(pwd):$(pwd) -w $(pwd) jdkato/vale content

Docsy GitLab Theme Development

Docsy is used as a base for the internal and public handbooks. Modifications and extensions have been added on top to ensure efficient handbook usage.

All Hugo partials and shortcodes should be added to this repository, instead of duplicating them locally into the handbook projects.

The Docsy GitLab theme is integrated into the public/internal handbook repositories using a Go module in go.mod.

Local Development

To develop on the Docsy GitLab theme locally build the handbook (or internal-handbook) project with the updated local copy of the docsy-gitlab theme following the steps below:

  1. Add the following line to the bottom of the go.mod file in the handbook project:

    replace gitlab.com/gitlab-com/content-sites/docsy-gitlab => /YOUR/LOCAL/PATH/TO/docsy-gitlab

  2. Run the Hugo server:

    hugo server

  3. Load the site locally:

    open http://localhost:1313/

Any changes made to files in local docsy-gitlab project will be rendered in the local instance.

Releases

Changes made to docsy-gitlab will be made available to the handbook and internal-handbook after the version is bumped and deployed to each project. This includes all theme updates with partials, shortcodes, and any changes to /docs. Much of this process has been automated, but a few manual steps are still required. To release a docsy-gitlab change follow these steps:

  1. After the merge request is merged, a pipeline on the main brach will kick off. After the pipeline succeeds, the release job can be triggered manually. For example: https://gitlab.com/gitlab-com/content-sites/docsy-gitlab/-/pipelines/1304430830
  2. The release job will do three things:
    1. Create a release in the docsy-gitlab project. Example: https://gitlab.com/gitlab-com/content-sites/docsy-gitlab/-/releases/v0.3.25
    2. Create a merge request to handbook. Example: https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/6086
    3. Create a merge request to internal-handbook. Example: https://gitlab.com/gitlab-com/content-sites/internal-handbook/-/merge_requests/4771
  3. Assign a handbook backend maintainer or codeowner to each MR for review.
  4. Once the merge request to handbook and internal-handbook have been merged, the theme changes will be deployed to the sites.

For more information on the release process, see the release script.

NOTE: The release process depends on the GITLAB_TOKEN access token to perform some actions. This token needs to be regularly refreshed. An issue has been created to trigger reminders to refresh the token.

Dependencies

Updating dependencies

Note: Before updating dependencies, review the go.mod file in the target project for additional notes and requirements.

Always update dependencies individually to isolate potential bugs and regressions. Never pull latest but pin the dependencies to a specific released version.

hugo mod get server.com/project-name@version

Google Docsy

hugo mod get github.com/google/docsy@v0.9.1

Do not bump Docsy versions without testing them extensively in MR review apps.

Redirects

You can add redirects to the layouts/index.redirects file in the relevant repository. Refer to the GitLab Pages redirect documentation for how these are formatted.

CI/CD Pipelines

Handbook & Internal Handbook

The CI/CD pipelines for the handbook and internal-handbook projects are roughly the same, and should remain that way.

build stage

The build stage performs a few linter and security jobs (handbooklint, hugolint, markdownlint, and secret_detection), and a build job which builds the whole site and saves it as CI artifacts.

deploy stage

The deploy stage uses the artifacts generated in the build stage to deploy the production site for changes to the main branch or review apps which can be deployed for MRs.

notify stage

The notify stage has two jobs that are triggered on specific events:

  1. post_comment_on_failure posts a comment to the MR when the markdownlint or handbooklint jobs fail.
  2. notify_slack_on_build_failures post a message to the #handbook-escalation Slack channel when there is a build failure on the main branch. This type of failure would indicate a broken CI pipeline.

$BUILD_AND_TEST_ONLY mode

The pipelines can be triggered to run in $BUILD_AND_TEST_ONLY mode which means they will run all the test and build jobs, but none of the deploy or notify jobs. A pipeline can be triggered in $BUILD_AND_TEST_ONLY mode by setting the $BUILD_AND_TEST_ONLY CI variable to true. This is used primarily by the downstream pipelines triggered from the www-gitlab-com project.

$GITLAB_TOKEN

Tools or scripts that need to interact with the GitLab API to perform handbook tasks should use the GITLAB_TOKEN CI variable. This is group level access token called Docsy GitLab Release with a developer role and has api and write_repository scopes. This token will expire after a year and will need to be refreshed. Follow the steps below to refresh the token:

  1. Create a new developer group level access token in <https://gitlab.com/groups/gitlab-com/content-sites/-/settings/access_tokens with api> and write_repository scopes. Give it the name GITLAB_TOKEN_YYYY_MM_DD (replace the YYYY_MM_DD portion with the current date).
  2. Update the value of GITLAB_TOKEN in docsy-gitlab, handbook, and internal-handbook with the new token value.
  3. Create a new release of Docsy GitLab Theme and ensure the pipeline works correctly.
  4. Delete the old token.
  5. A Zapier zap is configured with a webhook at the group level that should post to the #handbook-escalation channel when it is about to expire.

GitLab Pages Deployment

The deploy and review apps are configured in the pages job in the .gitlab-ci.yml configuration file.

  • Review app deploys can be run for merge requests by running the manual pages job, or by adding the ~"deploy-review-app-always" label to the MR.

  • Production deploys are only run from the main branch.

  • Conditional CI variables are used to configure the environment for review apps using GitLab Pages Multiple Deployments.

    variables:
  PAGES_PREFIX: 'mr$CI_MERGE_REQUEST_IID'
  ENVIRONMENT: 'mr$CI_MERGE_REQUEST_IID'
  HUGO_BASEURL: "${CI_PAGES_URL}/${PAGES_PREFIX}"

The route map is configured in the .gitlab/route-map.yml.

Docsy GitLab

The CI/CD pipeline for docsy-gitlab is responsible for building, testing, and releasing the Docsy GitLab theme used by both handbook and internal-handbook.

test stage

The test stage performs the markdownlint, and secret_detection jobs.

build stage

The build stage performs runs the hugo command to ensure the theme builds successfully.

release stage

The release stage performs one manual job to create a new release of the theme and merge requests to both handbook and internal-handbook. See the Releases section for more details.

www-gitlab-com

The CI/CD pipeline for www-gitlab-com is responsible for triggering downstream builds of both handbook and internal-handbook in $BUILD_AND_TEST_ONLY mode. This ensures that any changes to dependent data files in www-gitlab-com will build successfully for both handbook and internal-handbook.

Danger review and reviewer roulette

Right now the handbook projects doesn’t use the Reviewer Roulette feature of the danger-review CI/CD component. See issue.

We use the architecture plugin to provide guidance on how to review MRs that contain changes to architecture evolution design documents.

In order to make Danger post message to MRs we need to setup a project token (api scope with Developer role) and a CI/CD variable called DANGER_GITLAB_API_TOKEN to contain this project token (Masked but not Protected).

For more details, follow the Danger documentation and inspect the MR adding the reviewer roulette to the public handbook.

A Zapier zap is configured with a webhook at the group level that should post to the #handbook-escalation channel when it is about to expire.

Code quality report MR commenter

The CI/CD linting jobs generate code quality report artifacts for MR widget integration. If there are errors detected, a custom CI/CD job posts a Markdown table summary as MR comment, linking to helpful handbook editing resources.

The parse-codequality-report.sh script in the handbook and internal-handbook generates the code quality report posted in the MR. Changes to these reports need to be applied to both projects.

The code quality report relies on the GITLAB_TOKEN to perform the reporting actions on any MRs.

CI/CD maintenance tasks

  1. Test and bump the Hugo image versions, aligned with the config/_default/config.yml minimum version.
  2. Test and bump the davidanson/markdownlint-cli2 Docker image tags in the markdownlint job. We are not using the latest tag to avoid production failures pulling different latest versions.

Architecture

The Markdown files get parsed by Hugo, and converted into static HTML files, including all assets (JS, CSS, images, etc.). The static files are uploaded to GitLab Pages.

Templates and Partials

Review the Hugo development documentation to learn more.

Triage bot

A triage bot exists to help triage issues and close stale merge requests. See the triage project readme for more information.

Handbook Maintenance
Documentation on how to carry out maintenance tasks.
View page source - Edit this page - please contribute. Creative Commons License