GitLab Dedicated Overview

GitLab Dedicated Support overview.

Overview

GitLab Dedicated, from a support perspective, works as a combination of SaaS and Self-Managed. Customers have full Admin access to the instance, but no access to the infrastructure, nor to the backend configurations. This workflow captures the differences, and details of providing support for GitLab Dedicated.

If you’d like to work on GitLab Dedicated tickets, consider creating an issue using the template in Support Training, and read the overview.

Below is a list of other GitLab Dedicated Support workflow pages. This is list is a temporary measure to workaround the lack of until workflow categories are reintroduced

Here are links to other pages about GitLab Dedicated around GitLab:

Handling GitLab Dedicated tickets should be approached the same way as other tickets: use the docs, the handbook and the various issue trackers to help address the customer request. Anything that is related to our provisioning of the tenant or how we manage the infrastructure requires a different approach:

  • check the docs and the handbook
  • point the customer to the GitLab Trust Center at trust.gitlab.com for inquiries related to compliance
  • point the customer to their CSM for any questions not addressed by the GitLab Trust Center

Test and reproduction GitLab Dedicated instance

GitLab Support has access to a GitLab Dedicated instance for testing and problem reproduction purposes. This instance can be accessed at the following URLs:

To receive an invite, ask Armin, Brie, Daphne or Wei-Meng.

Conducting a test

When running a test on the GitLab Support Dedicated instance,

  • consider whether your test can be conducted in an instance deployed via the Sandbox Cloud Realm
  • communicate about it on the Slack channel #spt_pod_dedicated
  • revert your changes when you are done

As the test instance is shared within the GItLab Support team, post a message at the beginning of a test with an estimate duration if the test is likely to impact the performance of the instance. Upon test completion, revert your changes and use the emoji :done: to show the test is completed and the instance has been restored to the previous state.

The Switchboard console can be accessed at https://console.gitlab-private.org/tenants/40. Follow these instructions to request access. You may use this example access request as a starting point.

This instance is deployed to the Test environment.

Administrative access to a Dedicated instance

The GitLab Dedicated team does not have administrative access to the Admin Area in the GitLab application on Dedicated instances and neither does the GitLab Support team. Select individuals in the customer organization do have access to the Admin Area. Any support requests that require a GitLab instance administrator to make a change in the Admin Area, for example resetting 2FA, has to be performed by the appropriate team within the customer organization.

Sharing internal logs, data & graphs

We should not visually or physically share internal logs, data and graphs with GitLab Dedicated customers by default. Examples of things we should not share include, but are not limited to, screenshots of graphs, copied log entries, and raw log dumps.

For avoidance of doubt:

  1. GitLab Dedicated comes with a Sevice Level Availability SLO, which if not met results in financial penalties for GitLab.
  2. Capacity is limited for the GitLab Dedicated engineering teams as of October 2023. The teams want to spend the majority of their time on engineering tasks and avoid spending time answering non-critical customer questions.

Sharing internal logs, data and graphs without adequate context and explanation may cause customers to misinterpret the provided information, creating more work for all teams involved and, in the worst case, cause unnecessary damage to GitLab’s relationship with the customer.

If you assess that sharing such internal logs, data and graphs with the customer would create results for the customer and for GitLab, consult with a Director of Support. Be aware that a formal process for this is still being defined, and that there will be delays as approvals are currently ad hoc.

Working with logs

Working with logs has been moved

Working with Grafana

Working with Grafana has been moved

View instance metadata

Use the Switchboard app. More information can be found in the Switchboard workflow.

Feature Flags are not supported

In GitLab Dedicated, feature flags are not supported, meaning we do not able enable/disable a feature flag for a Dedicated instance. When customers request feature flags to be modified in the GitLab Rails console, the GitLab Support team should:

  • create or find an issue in the appropriate issue tracker about making this feature generally available (without a feature flag).
  • notify the appropriate Product Manager in the issue with a comment that followed the feedback template.
  • notify the customer that we don’t keep tickets open even if the underlying issue isn’t resolved.
  • check whether the customer has any questions about the next steps.

Support team members with questions can check in the #spt_pod_dedicated Slack channel for additional guidance.

Configuration changes

GitLab Dedicated uses the Cloud Native Hybrid reference architecture. Instance implementation and changes are done via the instrumentor project.

If it’s an emergency, escalate the emergency and contact GitLab Dedicated infrastructure team on Slack, using channel #g_dedicated-team.

When any changes are required besides those listed below, raise an issue in the GitLab Dedicated issue tracker using a Request for Help template. Be sure that the support::request-for-help label is added.

  1. In the ticket, ask the customer to provide the required information. In this case, it’s an IAM principal.

    • The IAM principal must be an IAM role principal or IAM user principal.
    • The IAM user principal has the following format: arn:aws:iam::<Customer_AWS_Account_ID>:user/user-name. The IAM role principal has the following format: arn:aws:iam::<Customer_AWS_Account_ID>:role/role-name. Keep the format of these two in mind to avoid prolonging the ticket if an unexpected format is provided.

  2. Open a new PrivateLink Request issue and confirm that the support::request-for-help label is added.

  3. Add the IAM principal to the issue. The Environment Automation team will provide a Service Endpoint Name.

  4. The customer will follow the steps in our documentation.

Outbound PriveLink allows any traffic from the GitLab Dedicated instance, or hosted runners for GitLab Dedicated to not be exposed to the public internet. While we advise for the onboarding to be completed, the customer can start these steps before.

  1. Open a new PrivateLink Request issue and confirm that the support::request-for-help label is added.

    • As a comment in the issue, request two Availability Zone IDs (AZ IDs) that can be used by the customer. For hosted runners, we only need a primary Availability Zone ID (AZ IDs).

  2. Provide the IAM role Principal to the customer. It has the following format: arn:aws:iam::<AWS_Account_ID>:role/reverse_private_link@<tenant_id>. You will find this information in the Reverse Private Link IAM Principal row of the Tenant Details section in Switchboard. Alternatively, read the instructions in the issue created for information on how to find the <AWS_Account_ID> and <tenant_id>.

  3. Provide the two AZ IDs from the issue to the customer. An example AZ ID is: use-az1 or usw-az4. Note: These are not AWS Zone IDs.

    • Provide the two AZ IDs early in the ticket to avoid prolonging the ticket. The AZ IDs must be in the same region as the customer’s tenant instance. The customer can then make the decision of which specific zones that can be used. AZ IDs are shared between different zones in a region but cannot be used outside of the region. For example, AZ IDs in us-west-1* cannot be used in us-west-2*. Some of the zones in each reach share AZ IDs with other zones in the same region but you must work with the customer to find the overlap.

  4. Ask the customer to provide the required information. In this case, it’s a Service Endpoint Name, a list of AZ IDs they will be using (should match provided AZ IDs), and Domain Name (with one of two options).

    • The Service Endpoint Name uses a reverse domain name notation and has the following format: com.amazonaws.vpce.<region>.<vpce-svc-identifier>

  5. Fill in the issue with the information provided by the customer and follow next steps in the issue.

IP Allowlist Request

In most cases, customers should use Switchboard to update the IP allowlist for their GitLab Dedicated instance. If this is not possible:

  1. Ask the customer to provided the required information in the ticket. In this case, it’s a comma-separated list of IP addresses.
  2. Open a Request for Help issue and confirm that the support::request-for-help) in the GitLab Dedicated issue tracker.
SCIM / OIDC with IP Allowlist request

Customers who use the IP allowlist may request to enable the SCIM or OIDC endpoints to the internet. This is a simple on/off toggle but must be performed by the Environment Automation team:

  1. Open a Request for Help issue in the GitLab Dedicated issue tracker. (Confirm that the support::request-for-help label has been applied)

SAML Request

  1. Ask the customer to provided the required information in the ticket. In this case, it’s a SAML configuration block or can be a list of information provided by a customer.
  2. Open a new SAML Config Request issue and confirm that the support::request-for-help label is added.
  3. Add the customer provided information and match it with the required formatting.

Application Logs Request

  1. In the ticket, ask the customer to provide the required information. In this case, it’s an IAM principal.

  2. Open a Request for Help issue in the GitLab Dedicated issue tracker.

  3. Provide the IAM principal to the Environment Automation team.

  4. Provide the name of the S3 bucket to the customer.

Filing issues

In cases where Customer Support needs to interact with GitLab Dedicated engineers to gather information or similarly debug a problem at tenant’s request (when Grafana or OpenSearch does not suffice), raise an issue in the GitLab Dedicated issue tracker using the Request for Help template.

During the course of the investigation, you may realize that you need to escalate a Request for Help (RFH) issue to another team. You should follow the existing process to formally request help from another group in the GitLab Development Team. When doing this:

  • Summarize the investigation thus far and make sure the ask for the team you are escalating to is clear
  • Apply the ~workflow-infra::Escalated label to the RFH issue in the GitLab Dedicated issue tracker

Handling GitLab Dedicated emergencies

GitLab Dedicated customer emergencies follow the same definitions and exceptions as regular customer emergencies.

If the customer is reporting an availability or performance issue:

  1. Check the GitLab Dedicated incident management tracker for relevant open incidents.
  2. If there is an open incident:
    • Inform the customer that the GitLab Dedicated infrastructure team is actively investigating.
    • Get in touch with the Dedicated SRE on-call and determine if the customer needs to be involved with troubleshooting.
    • Assist the customer and the Dedicated SRE as necessary.
  3. If there isn’t an open incident, escalate the emergency.

Escalating an Emergency issue

If involving the Dedicated team becomes necessary:

  1. Note initial findings in a new-incident issue.
  2. Follow the Dedicated on-call runbook.

Engaging the GitLab Dedicated CMOC

If the nature of the emergency reaches the point where we only need to provide async status updates to the customer, consider engaging the GitLab Dedicated Communications Manager on Call to take over.

Troubleshooting tips

Tagging logs while running tests

Customers can add a custom identifier, such as the ticket ID, to the user-agent field when testing. This makes it easier to filter logs related to the test.

For example:

curl -k -vvv -A"GitLabSupport012345" "https://tenant.gitlab-dedicated.com/users/sign_in"
Last modified November 27, 2024: [GitLab Dedicated] Add info about when to use trust.gitlab.com -> CSM flow to resolve customer inquiry (21318d53)
View page source - Edit this page - please contribute. Creative Commons License