How to write an effective screener

Screener goals

The questions in a screener have two conflicting goals:

• They must elicit specific information about users.
• They should also avoid revealing specific information about the study.

Step 1: Define the traits of the participant(s) you are looking for

When thinking about the users of your product, you will likely come up with a number of categories of ’typical’ users. The more specific you can make your descriptions of your users, the better you will be at identifying people who match your user profile. It is important to note that job titles alone don’t fully reflect a person’s tasks, duties, or behaviors. The example below shows how you can turn user traits into screener questions.

User trait	Screener question
Takes long vacations	What is the average duration of your personal trips? 1-2 days 3-6 days 7-13 days 14-20 days 21+ days
Takes multiple vacations each year	How many personal trips have you taken in the past year? 1-2 3-6 7-10 12+ none
Takes vacations with other people	Who do you travel with on the majority of your personal trips? I travel solo With friends With family
Makes all the travel arrangements	For personal trips, who is responsible for making the travel arrangements (flight, hotel, etc.)? I am solely responsible I share the duties with someone Someone else makes the arrangements, but I make the decisions Someone else is solely responsible
Prefers to stay at high-end resorts	For personal trips, what type of accommodations do you stay at the most? High-end resort (Auburge, Fairmont, etc.) Budget hotels (Clarion, Comfort In, Motel 6, etc..) Mid-price hotels (Tru, Windgate, Hampton Inn, etc.) High end hotels (St. Regis, Autograph Collection, etc.) Airbnb rentals Vrbo rentals Other (text)

Step 2: Structure your screener questions

When thinking about your screener questions, try to organize them like a funnel to narrow down participants, starting with broad questions and moving to more specific questions.

Step 3: Build the screener

Here are some best practices when making a screener from scratch:

• Create a screener draft in Google Docs to easily collaborate with others before implementing in a survey platform.
• When writing up a screener draft, use brackets for internal notes, so the researcher and other stakeholders can indicate whether an answer should terminate/reject a respondent. Brackets can also be used to indicate when a question should be skipped.
• If you need to use advanced logic to display questions and/or route respondents down a certain path based on their previous answers, Qualtrics is a great tool.

Final drafts of screeners at GitLab are created in Qualtrics. If you don’t have access to Qualtrics, request it. If you are using a different recruiting method or an unmoderated study platform (such as UserTesting.com), you will create the screener in those platforms.

Screener tips

When crafting questions, keep the following in mind:

• Ask open-ended questions about behavior, similar to the questions you ask in an interview. Yes/No questions lead participants to assume what you’re looking for.

Incorrect	Correct
Do you take two or more vacations per year? Yes [Reject] No	How many personal vacations do you take per year? 0 [Reject] 1 [Skip to question 4] 2 [Reject] 3 [Reject] 4+
Are you traveling with a partner, friend, or family member? No, I’m traveling by myself [Reject] Yes, I’m traveling with someone else	For this upcoming trip, who are you traveling with? By myself [Reject] With my spouse/partner With my family (spouse/partner AND kids/external family) With a friend With a group of friends I’m not sure [Reject]

• Provide similar options to choose from to mitigate guessing the purpose of the study.
  • To screen participants for their natural behavior, provide multiple options to mask what you’re looking for.

Incorrect	Correct
Where did you purchase your upcoming flight from? Priceline [Skip to question 5] Expedia [Reject] Other [Reject]	Where did you purchase your upcoming flight from? AlaskaAir.com Delta.com [Reject] United.com [Reject] Priceline.com [Skip to question 5] Spirit.com [Reject] Expedia.com [Reject] Other [Reject]

• Use plausible distractor answers to conceal the subject of the study.
• Avoid stand-out screener options, which indicate behavior that you’re looking for.
  • Participants can look for answers that stand out from the other answer options.
  • Using multi-select can allow for multi-guess without consequence.
• Multiple correct answers are OK and can provide information about your participants you’d normally get from an interview.

Incorrect	Correct
If you needed to get to a meeting on the other side of downtown, about 2 miles away, which of the following would you consider doing to get to your meeting? Walk Hang glide Swim Crawl	If you needed to get to a meeting on the other side of downtown, about 2 miles away, which of the following would you consider doing to get to your meeting? Walk Rent a bike Rent a scooter Take a Lyft

• If you are looking for a specific behavior, you can ask for verification of that behavior by including a statement in the screener such as “As part of the session, you will be asked to share your screen and walk through a previous hotel reservation.” Give them the option to agree or disagree to share their information.
  • This can guarantee your participant has completed the behavior you are looking for.
  • This is helpful if you need to verify actions or usage to bucket participants into segments.

Example GitLab Screener

• Screening Question Repository (internal access only)