Creating and managing a research participant panel

A research panel is a frequently used means for recruiting participants when conducting research consisting of studying the same group of opted-in participants through various methods and stages that are developed as part of a research effort. This page walks through a self-service process to create and manage panels.

When should I use a research panel?

Recruiting participants for research with niche topics can take a lot of time and effort to find. Frequency, subject-matter, scope, and scale are all areas of consideration as to when a panel may be needed. If you have a need for a larger group surrounding a certain niche criteria focusing on an issue (that will contain multiple research studies) over a longer period of time, this would be a time to consider using a panel. For example, larger enterprise users or users that are on Ultimate that have experience using a SAST tool and a role that involves using security tools or practices.

It is also important to consider the various pros and cons when utilizing a panel:

Pros:

• Opportunity to follow the same participants over multiple research initiatives to assist with tracking trends and changes throughout time and different product areas.
• Better chance of gaining in-depth knowledge and an understanding of your participants through their consistent involvement with the research.
• Ability to recruit participants on an expedited timeline, especially when you have a difficult or niche user profile.

Cons:

• Participant fatigue if users are contacted too frequently about participating in studies, which may also lead to having to refresh panel participants.
• Bias may begin to occur if the same people are surveyed multiple times, possibly causing us to over-index on feedback from a few users, instead of representing as many users as possible. Participants may also begin to recognize your research patterns and respond in a less natural manner.
• Periodically having to update the panel to ensure the appropriate participants are still included and there is an appropriate mix of individuals.

Initially, answering the following questions will help determine whether a panel is the right solution for your project.

• Why do you need a panel?
  • It’s best use a panel for users who are typically hard to recruit, either because there aren’t many of them (for example, Enterprise users), because they have attributes that are specialized and make them harder to find (for example, extensive experience with Kubernetes), or because they have been harder for us to recruit in the past (for example, self-managed users).
• How long will this panel need to be sustained?
  • For panels to be the most effective, setting a duration is crucial to ensure you are not over-indexing on feedback from a few users which will bias your results. Research Operations recommends a panel to last no longer than 6 months to help ensure that you are not over-indexing on panel participants.
• How large will the panel need to be?
  • Start with an idea of how large your panel needs to be. This will depend on the goals of the panel and the frequency of participant sessions. As a data point for general guidance, previous panel sizes created by UX Researchers ranged from 30-500 panelists.
• Will you be able to frequently communicate with the panelists to keep them engaged? A panel is a commitment. You will need to ensure you have the capacity to maintain communication and engage with the panelists on the schedule provided to them upon registration.

Suggested communication cadence:

• Upon selection: When panelists are selected to participate in the panel, let them know what to expect.
• Monthly: This could include updates about the research effort or upcoming research sessions.
• Conclusion: When the panel is no longer needed.

How do I create and conduct research using a panel?

Establish a DRI for the panel

Before creating a panel, you will need to establish a Panel DRI. This person is responsible for steps within the creation of the panel, communicating with participants, and panel management and tracking.

Creating a panel

To create a panel, follow the steps below. If you have any questions, please contact a UX Research Operations Coordinator to assist you.

1. Determine panelist criteria
   • Review this link for information on how to assist in selecting criteria to identify participants.
   • Who does this: Research project DRI
2. Set a timeline for how long you plan to maintain and use the panel
   • The longer a panel runs, the higher the likelihood of overusing the same participants increases.
   • Who does this: Research project DRI
3. Find your participants by creating a research recruitment request
   • To populate your panel, open a research recruitment request and indicate that the request is for a research panel.
   • You can find participants yourself and skip this step.
   • Who does this: Panel DRI and Research Operations
4. Decide who will be in your panel and use the Panel Tracking & Reimbursement Template to compile participants
   • After respondents complete the screener, you will need to review responses and determine who qualifies for the panel based on your criteria.
   • Add the panelists and their information in the Panelist Information sheet in the Panel Tracking & Reimbursement Template. To do so, download the results from Qualtrics and transfer to the Panelist Information sheet.
   • Who does this: Panel DRI
5. Communicate with participants regarding their inclusion in the panel, who their point of contact will be for the research effort, and frequency and length of participation. Refer to Communication guidelines for participant outreach.
   • Who does this: Panel DRI
   • See Participant email communication examples for templates.

Conducting research with your panel

After you have established your panel, follow these steps to utilize panel participants.

1. Identify participants you want to reach out to in your panel
   • Sometimes you may want to use a mixture of panelists and other recruitment methods for a study.
   • Check in the Panel Tracking & Reimbursement template to see how many times they have participated in previous studies. If they have already participated in (4) 60-minute sessions during the calendar year, remove them from the panel.
   • Who does this: Panel DRI
2. Schedule your sessions
   • Schedule your sessions using Calendly. Here is an email template you can use when scheduling sessions with panelists.
   • Who does this: Panel DRI
3. Compensate your participants by submitting a incentives request
   • To compensate panelists as they participate in studies, follow our existing process for incentives by submitting an incentives request.
   • Who does this: Panel DRI

As you use your panel, you will need to continually manage and track panelist information and engagement. Refer to the Panel management and tracking section.

Guidelines when communicating with panelists

One goal of a panel is to keep participants engaged and not to incur fatigue from over communication to where they no longer want to participate. Below are some guidelines on how to communicate with your panel:

• Outline communication guidelines initially within the recruiting screener, such as how often they’ll be contacted, who will contact them, how they’ll be contacted, and how team members will coordinate contact. To stay within your guidelines, set up reminders on your Google calendar.
• Consistency is key. If you communicate to participants that they will receive a total of three communications a month, send no less than and no more than three. Detouring from the set expectations could potentially make the panelist think that the research is not organized or serious, causing disengagement.
• Track how often we’re communicating with panelists in the Panel Tracking & Reimbursement Template. Do not reach out more than once a month to panelists.
• If participants contact the designated point of contact, you must reply in a 24-hour time window if it is during the week (otherwise, the next business day will suffice). It is a top priority to be responsive and consistently engaged.
  • When the designated point of contact goes on PTO, ensure there is someone to cover communication with panelists.

Panel management and tracking

Once you have surveyed and compiled the list of participants that you will include in your research effort, you will be required to track all panelist activity related to communications, participation, and incentives.

To manage your panel, use the Panel Tracking & Reimbursement Template. This spreadsheet will encompass all information surrounding each panelist, their activities, and the reimbursements they receive while the panel is active.

How to use the spreadsheet:

There are three individual sheets in the tracker spreadsheet.

• The Panelist Information sheet contains all panel participants and their general survey information from the initial screener. You will use this sheet to store information on panelists and to source participants. The first line of the template is filled in as an example.
  • The columns include:
    • First name
    • Last name
    • Email
    • Sign-up source (i.e, Qualtrics, CSMs)
    • Country
    • Outreach total - Automatically updated column that shows the total number of outreaches that the participant has received.
    • Session total - Automatically updated column that shows the total number of sessions the participant has been in. The cell highlights as orange when a panelist has participated in 3 sessions and red if the participant has been in 4 sessions.
    • Participant rating - Provide a general rating describing their participation.
    • Notes - Provide additional information on how the participant did in the session and reasoning why or about why not to use them again, along with other items worth mentioning that could be pertinent to the research.
    • Job title or persona
    • Screener response/demographics - These columns are unlimited, and you may tailor them to your specific screener. Consider adding the attributes you care about most in the first columns, such as company size, industry, and so on. Track additional columns to ensure you have a diverse panel. These may vary from panel to panel or across stage groups.
• The Panelist Reimbursement sheet will be used in coordination with Research Operations for reimbursements and to track the number of sessions a panelist participates in. The first line is filled in as an example.
  • The columns include:
    • First name
    • Last Name
    • Email Address
    • Project DRI
    • Research Issue
    • Session Length
    • Amount
    • Country
    • Paid - to be completed post session
    • Participation Date - A new line will be created for a participant for each different date they participated.
• The Outreach sheet is used to track every panelist outreach. You will need to paste in panelist information each time you send outreach communication. It will automatically update summary information in the Panelist Information sheet.
  • The columns include:
    • Email
    • Date
    • Outreach type
    • Notes

To avoid exceeding a total $600 gratuity per year per participant, a panelist may only participate in four 60-minute sessions per fiscal year. Once a panelist has reached four 60-minute sessions per calendar-year limit, they are no longer eligible to participate in studies. Remove the panelist from the panel and communicate to them that they are no longer eligible.

To keep the Panel Tracking & Reimbursement Template in a secure location, you will need to store your sheet in the Research Panels folder in the UX Research Google Shared Drive. Share settings should be restricted to the team members that need the required access.

Managing personally identifiable information (PII)

When managing panels, you are responsible for protecting Personally Identifiable Information (PII). Typically, this is done by Research Operations, but when teams are self-creating their own panels, they are responsible for managing the participants and their associated PII. Review best practices for data that is classified as ORANGE, and familiarize yourself with this overview of PII.

Always follow these best practices for handling PII within panels:

• Only collect the minimal amount of information you need for a person. Data you will likely collect:
  • First name
  • Last name
  • Email
  • Sign-up source (i.e, Qualtrics, CSMs)
  • Country
  • Job title or persona
  • Screener response/demographics

Do not collect data just to have it available for a non-specific purpose. For example, do not collect “time in a role” if role tenure is not relevant to the research effort.

• If someone is working from a public workspace, screens should not show panel information.
• Report any compromised data (see GitLab’s data classification) by creating a security incident.
• Remember to follow GitLab’s privacy policy.
• Participants may request
  • To be removed from the panel: Remove them, and then reach out to a Research Operations team member to ensure they are removed from other active panels and other communications.
  • To see their personal data: They can submit a Personal data request to see which lists they are subscribed to. Reach out to a Research Operations team member to help with this request.

What is an example of a panel in action at GitLab?

Here is an example of a panel used by Verify and Package. The team created this issue to draft a common screener for a panel that they could use across multiple studies to expedite recruitment efforts. In this scenario, the goal was to pool resources for user profiles that are hard to find.

Researchers outlined the following aspects to build upon this issue:

• Target personas
• Types of studies to use the screener
• Study formats
• Product stages

Tips and tricks

• You can mix and match recruiting sources. For example, you may recruit some participants from the panel and some from other sources, so that you don’t over sample from your panel. Do not use the same person more than four times in a year for a study.
• Build rapport with your panelists. Since you may talk to them more than once, you can build a connection with them.
• Consider sending out research participation requests in waves, rather than sending to your entire list. For example, if you have a panel of 1,000 users, send research participation requests in waves of 100, so that you don’t exhaust the entire panel all at once.
• When in doubt, ask. If you have questions or doubts on any aspect of starting or managing a panel, ping the #uxr_reops channel for advice.