Planning: Core Questions

To build your evaluation strategy, you and your team should answer some core questions before you continue. This ensures everyone is on the same page and has a clear goal in mind.

1. Research Questions: What is the goal of the evaluation?

What do you want to find out? What are the research questions that you want to answer? Write down the questions that you want to answer with your evaluation.

Depending on your research objectiv, there are different ways how you can formulate your research questions [1]:

Research Objectives	Research Questions Formulations Examples
Describing and Exploring	What are the characteristics of X? How has X changed over time? What are the causes of X? How has X dealt with Y?
Explaining and Testing	What is the relationship between X and Y? What is the role of X in Y? What is the impact of X on Y? How does X influence Y?
Evaluating and Acting	What are the advantages and disadvantages of X? How effective is X? How can X be improved?

Thereby, good research questions fulfill certain criteria [1, 2]:

focussed on a single topic
not based on value judgements
answerable within practical constraints
cannot be answered with yes or no
cannot be answered with easily-found facts

Here are some examples of good and bad research questions [1,2]:

Bad Research Question	Good Research Question
How does social media affect people’s behavior?	What effect does the daily use of YouTube have on the attention span of children aged under 16?
Has there been an increase in childhood obesity in the US in the past 10 years?	How have school intervention programs and parental education levels affected the rate of childhood obesity among 1st to 6th-grade students?
Is X or Y a better policy?	How effective are X and Y policies at reducing rates of Z?

HCI Autonomous Driving Example

Research Question:

What effect does the colour of light as a feedback modality have on the passenger’s user experience and perceived feeling of safety?

What are you evaluating?

When you have formulated one or more research questions, you know what you want to find out. The next step is knowing what you have to test to answer your research questions.
Make sure to know exactly what you want and need to evaluate. Are you evaluating one or more specific aspects of your system, or are you interested in the bigger picture?
Take each research question and add what exactly you will evaluate in your concept/prototype to answer it.

HCI Autonomous Driving Example

Research Question:

What effect does the colour of light as a feedback modality have on the passenger’s user experience and perceived feeling of safety?

What to evaluate:

Test different colours of light for different driving situations.

Which user testing methods will you use? And what type of data do you want to collect?

Once you have defined your research questions and what aspects of your concept/prototype you want to evaluate it is time to decide which user testing methods to use.
Choosing the proper user testing methods will allow you to collect the data you need to answer your research questions.
There are many testing methods that you can consider. The ones that we recommend are:
- A/B Testing
- Interviews
- Observation
- Questionnaires
- Think-Aloud Testing
Look at the Evaluation Strategy cards that begin with "Evaluation:" to find testing methods we recommend.
Write down all testing methods you will use and which research question they will answer.

HCI Autonomous Driving Example

Research Question:

What effect does the colour of light as a feedback modality have on the passenger’s user experience and perceived feeling of safety?

What to evaluate:

Test different colours of light for different driving situations.

Type methods to use and data to collect:

UEQ-S + think-aloud + post-interview → qualitative and quantitative data → answering research question 1

Next Card

Sources

[1] - Shona McCombes. “Writing Strong Research Questions | Criteria & Examples” https://www.scribbr.com/research-process/research-questions/ [Accessed December 2022]. 10 2022.

[2] - Imed Bouchrika. “How to Write a Research Question: Types, Steps, and Examples” https://research.com/research/how-to-write-a-research-question#types [Accessed December 2022]. 10 2022.

[3] - Imed Bouchrika. “How to Write a Research Question: Types, Steps, and Examples” https://research.com/research/how-to-write-a-research-question#types [Accessed December 2022]. 10 2022.