End-to-End Test Failure Issue Debugging Guide

Troubleshooting Failure Issues (Video 3 minutes)

Most Common Fixes

• Element not found? → Check if UI changed in recent MRs
• Timing out? → Look for spinners in screenshot, check performance, check for page errors
• 401 Unauthorized? → Token expiration issue
• Only in staging-canary/staging environment? → Check #staging channel for environmental issues and recent feature flag toggles

Debugging the Failure

1. Check the screenshot and exception for any obvious errors, examples:

   • ElementNotFound → UI element missing/changed
   • TimeoutError → Unexpected behavior or slow loading
   • AssertionError → Unexpected data or behavior
   • WaitExceededError → Look for spinners still loading in screenshot
   • 401 Unauthorized → Check expiring tokens
   • Server errors displayed in the UI → environmental issues or test set-up issues

2. Check GitLab instance under test - Use the found: labels (note: failure can be across multiple instances)

   • found:master → Ephemeral environment, failed in scheduled pipeline against master branch
     • Open the latest failed job from the Reports section
     • Check where the test is failing (GDK, CNG, Omnibus)
     • View the job name - this indicates the test configuration

     Note: Merge Requests will be blocked when tests are failing against GDK and CNG. These failures tend to be flaky failures as the test would have usually failed in a previous merge request. Tests against Omnibus are optional and allowed to fail.

   • found:<environment> → Failed in a live environment (debugging guide)
     • Open job to see if failure was a smoke job, or check metadata of test to see if test is a smoke test
     • If test is failing in a single environment, check environment status (#staging, #production)

     Note: :smoke test failures in staging-canary will block deployments.

3. Check failure frequency and timing

   • Observe when the failure issue created to identify the first occurrence
   • Observe the frequency of occurrences in the Reports section
   • Failure patterns:
     • Multiple recent consistent failures are more likely to be a real issue, needing immediate action
     • Sporadic failures could mean test flakiness OR application instability (race conditions, timing issues)
     • Use the first occurrence time to check commits/deployments immediately prior to the issue occurring

4. View the test file for recent changes

   • Click the File URL link in the failure issue metadata and review recent commits to the test file

5. Try to reproduce locally against your GDK, example:

      cd qa
      bundle install
      WEBDRIVER_HEADLESS=false GITLAB_QA_ADMIN_ACCESS_TOKEN=<admin PAT> QA_LOG_LEVEL=DEBUG QA_GITLAB_URL=http://gdk.test bundle exec rspec qa/specs/features/browser_ui/3_create/repository/add_file_template_spec.rb
   

6. If failure is from a live environment and passing against GDK, try against live environment or manually verify the functionality works in live environment.

   • https://docs.gitlab.com/development/testing_guide/end_to_end/feature_flag_testing/#running-e2e-tests-against-staging

7. Check application logs for signs of failure

   • Check job artifacts for master failures
   • Check https://nonprod-log.gitlab.net for staging failures
   • Check https://log.gprd.gitlab.net for production failures

8. Check subsequent test runs

   • Click Test case link in this issue
   • Check labels in test case issue for latest status of test case → If the test has subsequently passed the test or environment may be flaky

9. Check recent feature flag toggles (if failure is in a live environment)

Triage Actions

Apply appropriate label per classification guide

Note: Failure issues will be auto-closed after 30 days of no updates.

*Flakiness can be caused by the test OR the application itself being unreliable under certain conditions

Quarantining tests

Quarantine is a temporary measure for:

• Stale/broken tests (Feature works for users)
• Known acceptable issue causing :smoke test failures or excessive noise

1. Use Fast Quarantine for an urgent quarantine
2. Follow up with a long-term quarantine
3. Tag this issue with ~quarantine and ~automation:prevent-auto-close

For detailed quarantine workflows, see the Test Quarantine Process

Need further assistance?

Product teams own triaging and maintenance of their own tests. For guidance, reach out to Developer Experience in the #s_developer_experience Slack channel.

View page source - Edit this page - please contribute.