Use case study: using secrets in a CI job
Objectives
- To map out how users can use their native GitLab secrets in their CI jobs.
- Given OpenBao is a fork of HashiCorp Vault, we want to confirm its compatibility with our Vault integration in Runner.
- At a high level, gain a better understanding of how to structure OpenBao policies and JWT roles to be compatible with a project’s varied permissions per GitLab user role.
Prerequisites
The workflow requires that the templated policies for each combination of capabilities (e.g. read+update
, read+update+create
) are predefined. For example, consider the following templated policy that allows full access to a project’s secrets:
bao policy write project_full_access - <<EOF
path "kv-v2/data/projects/{{identity.entity.aliases.auth_jwt_02163755.metadata.project_id}}/*" {
capabilities = [ "read", "create", "update", "delete", "list" ]
}
EOF
The policies are associated to JWT roles on authorization. The project_full_access
policy is particularly important for the initial project owner role:
bao write auth/jwt/role/project_owner - <<EOF
{
"role_type": "jwt",
"policies": ["project_full_access"],
"token_explicit_max_ttl": 60,
"user_claim": "user_id",
"claim_mappings": {
"project_id": "project_id"
},
"bound_audiences": "secrets.gitlab.com",
"bound_claims_type": "glob",
"bound_claims": {
"user_access_level": "owner"
}
}
EOF
Given OpenBao policies are deny by default, this initial JWT role is necessary to grant project owners full access to read and write secrets.
Initial setup workflow
Details the steps and technical information for when the project’s native secrets are set up for the first time.
- Project owner enables GitLab Secrets manager through the GitLab UI.
- Project owner defines additional permissions on which GitLab user roles can read, write, or create secrets through the GitLab UI.
-
By default, project owners have full access and other roles are denied.
-
For example, if the owner allows read-only access for
developer
role then, through the OpenBao API, the Rails backend definesproject_88_developer
:# The format of the role name is `project_<project-id>_<user-role>` bao write auth/jwt/role/project_88_developer - <<EOF { "role_type": "jwt", "policies": ["project_read_only"], "token_explicit_max_ttl": 60, "user_claim": "user_id", "claim_mappings": { "project_id": "project_id" }, "bound_audiences": "secrets.gitlab.com", "bound_claims_type": "glob", "bound_claims": { "user_access_level": "developer" } } EOF
-
Unlike the
project_owner
generic role, we have to define other non-owner roles tied to the project because projects may have different combinations of permissions per user role.
-
- Project owner defines secrets through the GitLab UI.
-
User defines details such as name, key, and value. Sample input:
- name:
Production Database Password
- key:
DB_PASS
- value:
mydbpass
- name:
-
The secret is stored in OpenBao under
kv-v2/data/projects/88/ci/DB_PASS
, with the JSON data:{ "data": "mydbpass" }
-
The user doesn’t need to enter the secret value in JSON format. The Rails backend transforms the input into JSON object with the
data
key before sending it to OpenBao.
-
- Developer uses the
secrets
keyword in the.gitlab-ci.yml
.-
Sample configuration:
job-with-secrets: secrets: MY_SECRET_ON_OPENBAO: key: DB_PASS # Translates to kv-v2/data/projects/88/DB_PASS, field `data`
-
There is no need to specify
id_tokens:VAULT_ID_TOKEN
asaud
defaults tohttps://secrets.gitlab.com
where OpenBao service is. -
Unlike with HashiCorp Vault, there is no need to define CI/CD variables.
- The
VAULT_SERVER_URL
defaults tohttps://secrets.gitlab.com
where OpenBao service is. - The
VAULT_AUTH_ROLE
defaults toproject_<project_id>_<job_user_role>
to match the JWT role in OpenBao.
- The
-
- The CI job runs and
MY_SECRET_ON_OPENBAO
is available as an environment variable.- OpenBao verifies the integrity of the ID token and validates the
bound_claims
if it matches the custom claims, specially theuser_access_level
which contains the GitLab user role of the user. - Similar to HashiCorp Vault secrets, this is a
file
variable.
- OpenBao verifies the integrity of the ID token and validates the
Technical implementation findings
High-level technical implementation details pertaining to OpenBao and Rails to support the workflow.
- The OpenBao service needs to be properly configured to make it compatible with the workflow.
- Configure JWT authentication to make it work with ID tokens authentication.
- The documentation shows instructions using the
vault
CLI, but it should work similarly forbao
. - The OpenBao API is reachable through
https://secrets.gitlab.com
. - To reference the
project_id
in the templated policy, it was needed to get the value of the JWT auth mount accessor (auth_jwt_02163755
from the result ofbao auth list
). This has to be automated during deployments so that the templated policies remain up-to-date with the correct accessor. The mount accessor value is persisted in storage and keeps its value even when the OpenBao server is restarted and sealed.
- The Rails backend needs the accompanying implementations to support the workflow.
- ActiveRecord model for the secrets. Listing secrets and viewing details in the UI shouldn’t make a request to OpenBao.
- ActiveRecord model for the permissions. Listing permissions in the UI shouldn’t make a request to OpenBao.
- Update ID tokens related implementation to support the use of ID tokens without the need to define
id_tokens
in the CI configuration. - Proper mapping of defaults for
VAULT_SERVER_URL
andVAULT_AUTH_ROLE
.
How to test locally
The policies and roles structure presented here was first tested locally on a GDK setup and OpenBao server running on dev
mode.
Here’s a step-by-step guide on how to test this locally:
-
Make sure GDK is properly set up with runner.
- Tested on a GDK with Docker executor and pointed
gdk.test
to172.16.123.1
but this should also work with a shell executor. - Confirm that you can successfully run a CI pipeline on a test project.
- Tested on a GDK with Docker executor and pointed
-
Create the test project for fetching the secrets from OpenBao later.
- Track its project ID. In this example, the project ID was
53
.
- Track its project ID. In this example, the project ID was
-
Start-up the OpenBao in
dev
mode.bao server -dev -dev-root-token-id="dev-only-token"
- This makes OpenBao reachable at
http://127.0.0.1:8200
. - You might need to run
export BAO_ADDR='http://127.0.0.1:8200'
for thebao
CLI commands below to work.
- This makes OpenBao reachable at
-
Enable kv-v2 secrets engine.
bao secrets enable kv-v2 # By default mounts to `kv-v2/data`
-
Enable jwt authentication.
bao auth enable jwt
-
Configure OpenBao JWT authentication.
bao write auth/jwt/config \ oidc_discovery_url="http://gdk.test:3000" \ bound_issuer="http://gdk.test:3000"
-
To test the policy and role generated for a project owner with the GitLab user role
owner
, create the templated policy and the JWT role for the specificowner
role. The JWT role was based on the GitLab Vault sample server role.-
Take note of the value of the JWT auth mount accessor when you run
bao auth list
:Path Type Accessor Description Version ---- ---- -------- ----------- ------- jwt/ jwt auth_jwt_02163755 n/a n/a token/ token auth_token_90d6d0c1 token based credentials n/a
-
define the templated policy and reference the
project_id
through the metadata of the mounted JWT auth plugin:bao policy write project_full_access - <<EOF # owners have full read-write access to their project's secrets # copy over the `auth_jwt_02163755` mount accessor value path "kv-v2/data/projects/{{identity.entity.aliases.auth_jwt_02163755.metadata.project_id}}/*" { capabilities = [ "read", "create", "update", "delete", "list" ] } EOF
-
define the JWT role and associate the
project_full_access
policy:bao write auth/jwt/role/project_owner - <<EOF { "role_type": "jwt", "policies": ["project_full_access"], "token_explicit_max_ttl": 60, "user_claim": "user_id", "claim_mappings": { "project_id": "project_id" }, "bound_audiences": "secrets.gitlab.com", "bound_claims_type": "glob", "bound_claims": { "user_access_level": "owner" } } EOF
-
-
Create a sample secret that we want to fetch in the CI job.
bao kv put -mount=kv-v2 projects/53/foo val=my-long-passcode
-
On the test project, configure the
.gitlab-ci.yml
to fetch secrets from OpenBao using the existing Vault integration.test_openbao: variables: VAULT_SERVER_URL: http://127.0.0.1:8200 VAULT_AUTH_ROLE: project_owner id_tokens: VAULT_ID_TOKEN: aud: secrets.gitlab.com secrets: SECRET: vault: projects/53/foo/val # translates to secret `kv-v2/data/projects/53/foo`, field `val` token: $VAULT_ID_TOKEN script: - echo "testing..." - cat $SECRET - echo "done."
VAULT_AUTH_ROLE
matches the JWT role we created earlier.aud
matches the role’sbound_audiences
.- The ID token generated in this job is matched by OpenBao using the
bound_claims
, specifically theuser_access_level
which is included in the custom claims of the ID token.
-
Run a pipeline and confirm that in the job trace there’s a masked output of the secret that it fetched from OpenBao.
e47101dc
)