Staging Ref environment

Detailed information about Staging Ref environment

Staging Ref

Staging Ref is a sandbox environment used for pre-production testing of the latest Staging Canary code with full access to the environment and control over data.

Name URL Purpose Deploy Database Terminal access Slack channel
Staging Ref staging-ref.gitlab.com Pre-production testing Frequently (Parallel to gstg-cny) Separate and local All engineers #staging-ref

Purpose

  • Cover testing needs of Test Platform and Development teams in a production-like environment
  • Admin testing access
    • Current Staging (gstg) has customer data which is a blocker for giving more access to Development and Test Platform teams.
  • Testing different paid tiers
  • Democratizing testing and better test data
  • Better access to test accounts and wider permissions
  • Performant sandbox environment for engineers

Environment information

Deployment process

Staging Ref deployment runs parallel to Staging Canary deployment. Deployer triggers a job in Staging-Ref GET Config to update the environment. Notifications about new deployments are sent to the #announcements Slack channel.

Staging Ref pipelines do not block the deployment. If there are any failures with deployment to gstg-ref, please reach out to @release-managers. After successful deployment, Sanity and Full QA pipelines are triggered. Results are posted to #e2e-run-staging-ref and analysed by Test Platform on-call DRIs. Please refer to the Test Platform Department pipeline triage rotation schedule to identify the current DRI.

@startuml staging-ref
left to right direction

card "**Deployment**" as deploy #667ab3

card "gstg-ref" as gstg_ref #ffee9a {
 together {
  card "**Deployer**" as deployer #6a9be7
  card "**Staging-Ref GET Config**" as stg_ref_get #FF8C00
 }
}
card "**QA**" as gstg_ref_qa #ffa7db

card "**gstg-cny**" as gstg_cny #ffee9a
card "**gstg**" as gstg #ffd59a
card "**gprd-cny**" as gprd_cny #ffd500
card "**gprd**" as gprd #7966b3
card "**QA**" as gstg_cny_qa #ffa7db
card "**QA**" as gstg_qa #ffa7db
card "**QA**" as gprd_cny_qa #ffa7db

deploy -[#554488]-> gstg_ref
deployer -[#554488]-> stg_ref_get
gstg_ref -[#554488]--> gstg_ref_qa

deploy -[#554488]-> gstg_cny
gstg_cny -[#554488]-> gstg_cny_qa
gstg_cny_qa -[#554488]-> gprd_cny
gprd_cny -[#554488]-> gprd_cny_qa
gprd_cny_qa -[#554488]-> gstg
gstg -[#554488]-> gstg_qa
gstg_qa -[#554488]-> gprd

@enduml

How to use Staging Ref

Staging Ref is a safe playground for engineers who want to test latest Staging(gstg-cny) code. Staging Ref has several advantages that allow it to be a full-fledged sandbox environment:

  • Staging Ref deployments do not block the deployment process and can be tweaked or updated by any GitLab engineer. Hence GitLab engineers have wide permissions and full control over the environment.
  • Environment follows 3k hybrid architecture, so it is more performant than existing Staging(gstg) and could be used for load testing if needed.

To sign in to the environment, navigate to staging-ref.gitlab.com and use your GitLab Google account in ‘Sign in with Google’ option.

After signing in you can proceed using the environment as required. If destructive changes were done to the environment or it ended up in a bad state after testing, create a request to rebuild the environment. Please reach out to the #staging-ref Slack channel or raise an issue in Staging-Ref GET Config. The process is automated with Staging-Ref GET Config and will take about an hour to finish.

Enable Feature Flags

ChatOps commands can be used to enable or disable Feature Flags on Staging Ref. You can run this command in the #staging-ref Slack channel and notifications will be sent to #e2e-run-staging-ref after a flag is enabled/disabled.

Admin access

To promote your user to Admin, please sign in as Admin using the Staging Ref credentials from 1Password Engineering vault. Then navigate to the Admin Area’s Users page and edit your user’s Access Level.

Note that Staging Ref environment is shared across all engineers. If you plan to perform changes to GitLab Admin settings, use the #staging-ref Slack channel to communicate changes broadly.

SSH access and Rails console

If you have gcloud or kubectl set up locally, then follow Connect from your local terminal. If not, then you can opt to connect via your browser.

Connect via your browser
  1. Get access to the GCP project gitlab-staging-ref
  2. Visit gitlab-toolbox workload in the gitlab-staging-ref project
  3. In the Managed pods section, click on the Name of a Running pod. For example, the Name looks like gitlab-toolbox-5955db475c-ng2xr.
  4. Click the Kubectl dropdown near the top
  5. Hover over Exec to reveal a sub menu
  6. Click toolbox
  7. A Cloud Shell should start up
  8. Edit the command kubectl exec -it gitlab-toolbox-5955db475c-ng2xr -- bash (the toolbox will have a different suffix) to execute the bash command with the interactive and TTY options.
  9. At this point, you should be logged in to the toolbox pod. For Rails console, run gitlab-rails console.
  10. See Kubernetes cheat sheet for more
Connect from your local terminal
  1. Navigate to the staging-ref cluster or to the staging-ref geo cluster
  2. Click Connect
  3. Copy the command and run it locally to get kubeconfig
  4. Follow Kubernetes cheat sheet
  5. Also see additional developer tools

Request access to GCP project and environment

If you need access to Staging Ref components in the GCP project(gitlab-staging-ref), please reach out in the #staging-ref Slack channel. Test Platform Engineering Managers can add you to gcp-staging-ref-sg@gitlab.com Google group.

As another option you can create an issue in the access-request project. Requests for access to server environments requires the approval of your manager and an Infrastructure manager.

Note that GitLab configuration changes will be overwritten by a new deployment to the environment. Environment updates can be locked if needed by a request to @release-managers in the #staging-ref Slack channel.

A simplified process to request SSH access to Staging Ref virtual machines and the GKE cluster is being worked on in issue#343938.

Trigger QA pipelines

Sanity or Full QA pipeline may be triggered on demand in staging-ref project. Please reach out to Test Platform on-call DRIs if there are any questions.

Monitoring

Monitoring implementation was done in (epic#594). Documentation can be found in the runbooks.

Dashboards for Staging Ref can be found in Grafana under the staging-ref folder. There are other existing dashboards which may also show Staging Ref information if you select environment=gstg-ref.

The Geo secondary site is running Grafana at https://geo.staging-ref.gitlab.com/-/grafana. Credentials can be found in EU site monitoring section in Staging Ref credentials in 1Password Engineering vault.

If you need a specific dashboard or an existing dashboard does not work please reach out to #staging-ref channel.

Upgrade paid plans

By default, all users and groups are on the Free plan. To upgrade a paid plan use Admin account and do the following:

  1. Navigate to Admin area.
  2. Select Users or Groups section depending on what entity you would like to upgrade.
  3. Search for user or group by name and click “Edit”.
  4. Select the required paid plan in “Plan”.
  5. Click “Save changes”.

Watch this demo to see an example when a group was promoted to Premium plan.

Pre-existing test accounts

Staging Ref environment has pre-existing accounts that can be used for testing. For example, Admin accounts on different paid plans, Auditor user, QA users. All credentials are stored in Staging Ref credentials in 1Password Engineering vault.

Working with a SAML SSO enabled group

An Okta application has been setup to act as SAML idP for https://staging-ref.gitlab.com/groups/saml-sso-group. Two users with names gitlab-qa-saml-sso-user1 and gitlab-qa-saml-sso-user2 have been created and added to Okta and assigned the application. These users are also available in staging-ref environment.

Please note that all credentials and values for fields mentioned below are saved in 1Password Engineering Vault in “Staging Ref credentials” under “User credentials for saml-sso-group Group”.

For using SAML SSO, you will need to:

  1. As an admin, create the group at https://staging-ref.gitlab.com/groups/saml-sso-group if it does not already exist.

  2. Upgrade the pricing plan of this group to “Premium”.

  3. Visit https://staging-ref.gitlab.com/groups/saml-sso-group/-/saml and:

    • Check “Enforce SSO-only authentication for web activity for this group”
    • Update the value of “Identity provider single sign-on URL” to the value saved in 1Password
    • Update the value of “Certificate fingerprint” to the value saved in 1Password

  4. Save the changes.

  5. Sign out.

The first time you visit https://staging-ref.gitlab.com/groups/saml-sso-group and try to log in, you will be asked to sign in to GitLab with an existing account to link the SAML identity. Use username gitlab-qa-saml-sso-user1 or gitlab-qa-saml-sso-user2 to sign in. The credentials are in 1Password.

Future iterations and known limitations

Staging Ref environment has some known limitations that will be worked on:

Other outstanding work for Staging Ref is tracked in GitLab issue tracker.

Feedback

If you need some additional custom configuration for Staging Ref to be explored or you have other feedback and ideas for improvements, please reach out to #eng-allocation-new-staging Slack channel or add a comment to the feedback issue#350744.

Last modified June 21, 2024: Rename qa-staging-ref to e2e-run-staging-ref (f9de3e4e)
View page source - Edit this page - please contribute. Creative Commons License