Troubleshooting cloud licensing

Scenario

Customer was opted into cloud licensing because of a bug and their instance activated cloud licensing, an entity change happened and the new subscription was not on cloud licensing

Troubleshooting workflow

1. Determine whether the customer should be on cloud licensing
2. If no, proceed with:
   1. Removing the current cloud license by asking the customer to run these commands via rails console: l = License.find 123 and then l.destroy. You need to provide the license number for the first command. You can find this number in the URL of the cloud license in the customersdot license menu (example: https://customers.gitlab.com/admin/license/123123123)
   2. Once the license has been removed, the customer should have the ability to upload a standard license file on their instance.
3. If yes, proceed with:
   1. Switching the new subscription to cloud license features
   2. Removing the current cloud license by asking the customer to run these commands via rails console: l = License.find 123 and then l.destroy. You need to provide the license number for the first command. You can find this number in the URL of the cloud license in the customersdot license menu (example: https://customers.gitlab.com/admin/license/123123123)
   3. Once the license has been removed, the customer should be able to input the new activation code that was generated and emailed to them when you switched the subscription to cloud features.
   4. See example ticket where customer did want to be on cloud licensing

How to switch a subscription to cloud license features

1. Go to the customer’s Zuora Subscriptions tab in customersdot (URL example: https://customers.gitlab.com/admin/customer/123123/zuora_subscriptions)
2. Tick the relevant feature boxes. DO NOT tick the Cloud Licensing checkbox. Click the Update button.
3. Click on the cloud icon next to the update button to generate an activation code for the subscription and email it to the customer.
4. Check that a code has been generated by viewing the Cloud Activations tab. Ensure that the code was generated for the correct subscription.
5. Impersonate the customer’s portal account to ensure that the activation code is available for the subscription.
6. Once the customer has used the activation code on their instance, a cloud license should have been generated (check the licenses section) and the Cloud Licensing checkbox from step 2 should now be checked.

Troubleshooting Network Connectivity

Cloud licensing requires a connection to customers.gitlab.com over port 443 (HTTPS), and this connection must remain available throughout the duration of using a cloud license. The GitLab server will typically check in once per day, as well as once during activation, and any time a manual sync is performed. While general networking is typically outside the scope of what we can support, there are a number of things we can easily test for to help users diagnose any potential network or HTTPS issues blocking the connections.

Check DNS

The connection will rely on the hostname customers.gitlab.com, meaning a DNS lookup will be performed. The IP addresses of customers.gitlab.com may change periodically, so it’s important to verify what they are right now:

For example, using dig or host commands:

dig customers.gitlab.com

host customers.gitlab.com

Keep in mind, IPv6 may occasionally be a factor in network troubleshooting. If a server is configured to prefer IPv6 over v4, then be sure that their server is able to resolve IPv6 (AAAA) records:

dig AAAA customers.gitlab.com
host -t AAAA customers.gitlab.com

Basic IP and TCP connectivity

In most cases, a ping will suffice to verify that a connection is available over an IP address, be it IPv4 or v6. After obtaining the current IP addresses:

ping 104.18.21.224
ping -6 2606:4700::6812:14e0  #ping6 may be available too, but is an alias for ping -6

Pinging the hostname (customers.gitlab.com) is fine too, but directly at an IP ensures both IPs are available. A simple one liner can be used to ping both IPs:

dig +short customers.gitlab.com | xargs -I% ping -c1 -W5 %

Note that ping is not technically a TCP or UDP connection, but rather an ICMP, which may occasionally be blocked for various networking reasons. A slightly more advance technique can be used to test TCP connections, using either nc (preferred) or telnet:

$ nc -zv customers.gitlab.com 443

Connection to customers.gitlab.com 443 port [tcp/https] succeeded!

$ echo "." | telnet customers.gitlab.com 443

Trying 104.18.20.224...
Connected to customers.gitlab.com.
Escape character is '^]'.
Connection closed by foreign host.

Verifying HTTP/S connections

The simplest method to check this is by using curl. Specifically, sending a HEAD request to https://customers.gitlab.com, which will both verify TCP connectivity and ensure a secure TLS connection is available:

curl -I https://customers.gitlab.com/

A successful reply will generally look like a 302 (Found), meaning the connection was succesful, and customers.gitlab.com replied with a redirection. More concisely:

$ curl -IL -q https://customers.gitlab.com/ | grep -i location

location: https://customers.gitlab.com/customers/sign_in

Implies success.

Troubleshooting SSL/TLS

In most circumstances, if system curl is working, then there won’t be a need to check further TLS settings. However, because GitLab does offer several options for configuring various TLS and SSL certificate settings, including custom certificate authorities, it may be necessary to perform some more advanced SSL/TLS troubleshooting.

For example, if the customer server is making use of custom certificate authorities (CA), such as when SSL packet inspection is employed, they will need to add that root CA certificate to /etc/gitlab/trusted-certs on the server, then reconfigure GitLab.

• Refer to our documentation on how SSL works in GitLab and troubleshooting SSL in GitLab for more in-depth information. In particular, relying on the packaged version of openssl to check and verify SSL connectivity:

echo | /opt/gitlab/embedded/bin/openssl s_client -connect customers.gitlab.com:443

It may be useful to run a SSL Server Test and then provide the page to customers. It’s beyond the scope of our support to assist with most of the output on this page, but providing it may prove useful to customer network admins for additional troubleshooting on their end.

Obtain DevTools > Network HAR file during activation

A user’s browser’s DevTools can be particularly useful in diagnosing cloud license connectivity failures, especially since the GitLab internal API (graphql) response can be viewed.

1. Open the DevTools (usually ctrl+shift+i) and navigate to the Network tab
2. (re)Load the page at /admin/subscription
3. Attempt to activate the cloud license code
4. In DevTools > Network, locate any graphql resources and view the Response pane. There may be errors recorded here, feel free to share.

• Note that multiple graphql resources may be present, and not all will be related to the cloud licensing activation process.

Since there will be a lot of information presented in the DevTools, feel free to suggest that the customer generate a network HAR file and attach it to the ticket for closer inspection by us.

Caution: Advise the user to sign out of the GitLab session they recorded to invalidate their session credentials. See sec.Okta.com/harfiles for context.

Simulating a cloud license SeatLink attempt from within the Rails application

System tools and binaries such as ping and curl are helpful, but keep in mind that GitLab is a highly complex application that bundles many of its own tools and binaries. GitLab the application may handle outbound connections slightly differently than system-supplied binaries will. The best way to verify this is to check that the connection is possible directly from rails console. Login to the server and enter rails console with sudo gitlab-rails console and run:

URI_PATH = '/api/v1/seat_links'
base_uri = EE::SUBSCRIPTIONS_URL # For GitLab version lower than 16.0.0
base_uri = ::Gitlab::Routing.url_helpers.subscription_portal_url # For GitLab version 16.0.0 and above

head_resp = Gitlab::HTTP.head(base_uri)

pp head_resp

That command will return output very similar to what a typical curl -I would return, but it relies on internal ruby classes to route the request in the same way GitLab would, therefore exposing any potential issues that are happening specifically within the GitLab application.

Custom proxy settings

At present, cloud licensing does not officially support network proxies, deep packet inspection, etc. But if the customer is aware of custom network proxy configurations on their end, they may be able to configure GitLab to ignore them via the no_proxy environment variable. More information is available in our documentation on Setting custom environment variables

Activating a license with the GraphQL API

In cases where the license activation functions are unavailable (for example, due to a 500 error on the billing page), the GraphQL API can be used to activate a cloud license directly using the Mutation.gitlabSubscriptionActivate endpoint. With this method, a subscription can be activated even if there is already an activated cloud license present on the instance. This method is useful to avoid the downtime caused by removing the existing license via the UI before activating another key:

1. Have the customer navigate to https://<their-self-managed-gitlab-site.com>/-/graphql-explorer
2. Run the following mutation by replacing <activation code> with the actual 24-character cloud activation code:

mutation {
  gitlabSubscriptionActivate(
    input: {
      activationCode: "<activation code>"
    }
  ) {
    errors
    license {
      id
      activatedAt
      startsAt
      usersInLicenseCount
      lastSync
    }
  }
}

The mutation will return the following if there are no errors:

{
  "data": {
    "gitlabSubscriptionActivate": {
      "errors": [],
      "license": {
        "id": "gid://gitlab/License/12345",
        "activatedAt": "2023-07-28",
        "startsAt": "2023-07-04",
        "usersInLicenseCount": 10,
        "lastSync": "2023-07-28T18:34:14Z"
      }
    }
  }
}

General Tips

1. You can resend an activation code via email by clicking on the envelope button on the customer’s Cloud Activations tab.

View page source - Edit this page - please contribute.