How to Design a Research Questionnaire That Produces Useful Data

A questionnaire can look professional and still produce poor data.

You can have attractive formatting, dozens of questions, and hundreds of responses but discover during analysis that the questions do not actually measure what your study needed.

The problem often begins when students start writing questionnaire items before deciding what evidence each item is supposed to produce.

A stronger process works in the opposite direction:

Research question → objectives → concepts or variables → evidence needed → questionnaire items → response options → analysis

Every question should earn its place.

If you cannot explain how a questionnaire item contributes to answering your research question or objectives, ask whether you need it.

Questionnaire vs survey

The terms questionnaire and survey are sometimes used interchangeably, but they are not necessarily identical.

A questionnaire is the instrument containing the questions or items participants answer.

A survey can refer more broadly to the research process that uses a questionnaire to collect information from a sample or population.

For example, your study may involve:

• defining a target population;

• selecting a sample;

• administering a questionnaire;

• collecting responses; and

• analysing the resulting data.

The questionnaire is therefore one component of the broader survey design.

This distinction matters because good questionnaire wording cannot compensate for weak sampling, poor recruitment, or an inappropriate research design.

Start with the research question—not the questionnaire

Do not begin with:

“What questions can I ask my participants?”

Begin with:

“What evidence do I need to answer my research question?”

Suppose your research question is:

What relationship exists between students' perceived quality of formative feedback and their academic engagement?

You now need evidence about at least two concepts:

• perceived quality of formative feedback; and

• academic engagement.

Your questionnaire should generate evidence relevant to those concepts.

An item asking:

Which social-media platform do you use most often?

may be interesting, but unless social-media use contributes to the study, it probably does not belong in the questionnaire.

Questionnaire design begins with relevance.

Turn your objectives into evidence requirements

Consider this objective:

To examine undergraduate students' perceptions of the usefulness of formative feedback.

Before writing items, ask:

What would I need to know to address this objective?

Possible dimensions might include whether feedback:

• is understandable;

• arrives in time to be useful;

• identifies strengths;

• identifies areas for improvement; and

• helps students understand how to improve future work.

Those dimensions can then guide questionnaire development.

This is much stronger than writing twenty general questions about “feedback” and hoping some of them become useful later.

Your research aims and objectives should therefore help determine what belongs in the instrument.

Decide what each concept means

Questionnaires often measure concepts that cannot be captured adequately by one obvious question.

Consider:

Academic engagement

What does that mean in your study?

Does it refer to:

• class participation?

• time spent studying?

• persistence with difficult tasks?

• attention during learning?

• emotional involvement?

• several related dimensions?

You need to define the concept before deciding how to measure it.

Otherwise, two researchers could both claim to measure “engagement” while collecting completely different evidence.

This is also where your conceptual and methodological reasoning matters. Your instrument should operationalise the concepts the study actually proposes to investigate.

One concept may require several items

Suppose you want to measure students' perceptions of feedback usefulness.

One item might ask:

The feedback I receive helps me understand how to improve future assignments.

But one item may not adequately represent the whole construct.

You might also need appropriately designed items concerning:

• clarity;

• timeliness;

• specificity;

• usefulness for future work; and

• guidance on improvement.

That does not mean more questions are automatically better.

The goal is adequate measurement—not questionnaire length.

If you are using an established multi-item scale, follow the evidence and scoring guidance for that instrument rather than casually adding or deleting items.

Open-ended vs closed-ended questions

Questionnaire items can broadly be open-ended or closed-ended.

Closed-ended questions

Closed-ended questions provide predetermined response options.

For example:

How often did you attend scheduled tutorials during the past four weeks?

• Never

• Once

• 2–3 times

• 4–5 times

• More than 5 times

Closed-ended questions can be efficient for respondents and easier to code and analyse when the response categories appropriately represent the possible answers.

Open-ended questions

Open-ended questions allow respondents to answer in their own words.

For example:

What is the most useful change your lecturer could make to the feedback you receive?

Open-ended items can capture experiences or responses you did not anticipate.

But they also require more effort from respondents and usually more work to analyse.

Do not include an open text box after every closed question simply because your survey software allows it.

Use open-ended questions when the information they can provide is genuinely useful to your research.

Research-methods guidance similarly distinguishes open-ended items, which permit unrestricted responses, from closed-ended items with predefined response options; wording and response formats can themselves influence the data produced.

Ask one thing at a time

One of the most common questionnaire problems is the double-barrelled question.

Consider:

How satisfied are you with the clarity and speed of lecturer feedback?

What should a student answer if the feedback is very clear but consistently late?

One response cannot represent both judgments.

Split the question:

How satisfied are you with the clarity of lecturer feedback?

and:

How satisfied are you with how quickly lecturer feedback is provided?

This produces interpretable evidence for each issue.

Questionnaire-design guidance recommends separating conceptually different issues because combining them into one item makes responses ambiguous.

A useful test is:

Could a reasonable respondent want to give two different answers to this question?

If yes, you may have a double-barrelled item.

Avoid leading questions

A leading question encourages respondents toward a particular answer.

For example:

How much has the university's excellent new online-learning platform improved your learning experience?

The wording already tells participants that the platform is “excellent” and implies improvement.

A more neutral version might ask:

How, if at all, has the new online-learning platform affected your learning experience?

Or, if a closed-ended measure is needed:

Overall, how would you rate your experience using the new online-learning platform?

Question wording should not reveal which answer the researcher prefers. Leading wording can bias responses rather than measure participants' views independently.

Avoid loaded questions and assumptions

A question can also be problematic because it assumes something that may not be true.

For example:

Why do you struggle to manage your academic workload?

This assumes the respondent struggles.

A student who does not struggle has no accurate way to answer.

You could first establish whether the experience applies:

During the current semester, how difficult have you found managing your academic workload?

Then, where appropriate, use follow-up logic for respondents who report difficulty.

Likewise:

How often do you use AI tools to complete assignments?

assumes the respondent uses AI tools.

A preceding filter question may be necessary.

Make questions specific

Vague questions produce vague evidence.

Consider:

Do you study regularly?

What does “regularly” mean?

Every day?

Several times a week?

Only during examination periods?

A more specific question could ask:

During the past seven days, on how many days did you spend at least 30 minutes studying outside scheduled classes?

The appropriate wording depends on what the study needs, but the respondent should understand what period, behaviour, or judgment they are being asked about.

Research-methods guidance emphasises that effective questionnaire items should be brief, relevant, unambiguous, specific, and objective.

Choose recall periods deliberately

Questions about behaviour often depend on memory.

Compare:

How often do you miss lectures?

with:

During the past four teaching weeks, how many scheduled lectures did you miss?

The second gives respondents a defined recall period.

But shorter is not automatically better.

Your recall period should fit:

• how frequently the behaviour occurs;

• how memorable it is;

• what period matters to the research question; and

• whether respondents can reasonably recall it.

Asking participants to remember every minor behaviour over the previous year may produce poor-quality answers.

Use plain language

Your participants should not need specialist knowledge merely to understand your questionnaire.

Avoid unnecessary jargon, abbreviations, complicated sentence structures, and academic vocabulary that the target population may interpret inconsistently.

For example:

To what extent do pedagogical modalities facilitate your metacognitive engagement?

may sound academic but could be useless if respondents do not understand it consistently.

The objective is not to impress participants.

It is to measure their responses accurately.

Design response options as carefully as the questions

A well-written question can still fail if the response options are poor.

Consider:

How many hours per week do you work in paid employment?

• 0–10

• 10–20

• 20–30

• More than 30

Where does someone working exactly 10 hours belong?

The categories overlap.

A better structure might be:

• 0 hours

• 1–10 hours

• 11–20 hours

• 21–30 hours

• More than 30 hours

Closed-ended categories should generally be mutually exclusive so that one response does not fit multiple categories, and sufficiently exhaustive so that legitimate answers are not forced into inaccurate options.

Where appropriate, options such as Other, Don't know, Not applicable, or Prefer not to say may be necessary—but they should have a methodological reason rather than being added mechanically.

Likert-type items

Students frequently use Likert-type response formats to measure attitudes, perceptions, or agreement.

For example:

The feedback I receive clearly explains how I can improve my work.

• Strongly disagree

• Disagree

• Neither agree nor disagree

• Agree

• Strongly agree

The item presents one statement and an ordered response scale.

But simply adding:

Strongly disagree → Strongly agree

to a collection of statements does not automatically create a valid measurement instrument.

You still need to consider:

• what construct the items represent;

• whether each item measures one clear idea;

• whether the response scale fits the item;

• whether multiple items are intended to form a scale;

• how responses will be scored; and

• what evidence supports the measure's interpretation.

Keep response scales consistent where appropriate

Changing response formats unnecessarily can increase cognitive burden and mistakes.

For example, one section might use:

1 = Strongly disagree, 5 = Strongly agree

while the next unexpectedly reverses it:

1 = Strongly agree, 5 = Strongly disagree

Unless there is a defensible reason, such inconsistency can create avoidable errors.

Consistency does not mean every question needs the same response scale.

Frequency, satisfaction, agreement, and factual behaviour may require different formats.

The principle is to avoid unnecessary complexity.

Plan the analysis before collecting data

Do not wait until the questionnaire closes to ask:

“What am I going to do with these responses?”

For every important item or group of items, know what kind of evidence it is intended to produce and how that evidence relates to the research question.

Suppose an objective is:

To examine the relationship between students' perceived feedback usefulness and academic engagement.

If feedback usefulness is measured using several questionnaire items, you need to think beforehand about:

• whether those items form a scale;

• how they will be scored;

• whether any items require reverse coding;

• how missing responses will be handled;

• how academic engagement will be measured; and

• what analysis can address the proposed relationship.

This is especially important when the questionnaire is intended to test a research hypothesis.

A questionnaire should produce data your planned analysis can actually use.

Be careful when adapting established questionnaires

Sometimes an existing questionnaire or measurement scale already assesses the concept you need.

Using an established instrument can be valuable because previous research may provide evidence about its measurement properties.

But “validated questionnaire” does not mean:

This instrument is valid everywhere, for everyone, forever.

Evidence supporting an instrument's interpretation may depend on:

• population;

• language;

• cultural context;

• administration method;

• construct;

• scoring procedure; and

• intended use.

If you substantially rewrite items, remove questions, change response scales, translate the instrument, or use it with a very different population, previous evidence may not transfer automatically.

Also check whether the instrument is copyrighted, licensed, or requires permission before reproducing or modifying it.

Do not alter an established scale simply because some items seem inconvenient.

Validity is not a checkbox

Students sometimes write:

“The questionnaire was valid because my supervisor checked it.”

Expert review may be useful, but validity is broader than one person's approval.

Validity concerns whether the evidence and interpretations produced by the instrument are adequately supported for the purpose in which they are being used.

Depending on the study, relevant evidence may concern:

• whether items adequately cover the construct;

• whether respondents interpret items as intended;

• relationships among items;

• relationships with other measures;

• expected group differences; and

• consequences or limitations of interpretation.

You do not need to perform every possible validation procedure in every student project.

But you should avoid treating validity as a simple yes/no property attached permanently to a questionnaire.

Reliability also needs interpretation

Reliability concerns consistency or precision of measurement.

For multi-item scales, students frequently report internal-consistency statistics such as Cronbach's alpha.

But a high coefficient does not automatically prove that:

• the scale is valid;

• all items measure one construct;

• the questionnaire is well designed; or

• the instrument is appropriate for your population.

Reliability evidence should be interpreted alongside the construct, scale structure, number of items, and intended use.

Do not chase a particular coefficient by deleting items mechanically until the number becomes attractive.

Measurement decisions should remain conceptually defensible.

Pretest the questionnaire

Even carefully written questions can behave differently when actual respondents see them.

Before full data collection, test the questionnaire with people who resemble the intended participants where feasible.

A pretest can reveal problems such as:

• confusing wording;

• ambiguous response options;

• missing answer categories;

• unclear instructions;

• questions interpreted differently from what you intended;

• excessive completion time;

• broken skip logic;

• technical problems on mobile devices; and

• uncomfortable or intrusive wording.

Questionnaire-development guidance recommends pretesting because it can expose wording, interpretation and administration problems before the main study.

Do not ask only:

“Was the questionnaire okay?”

Ask participants which questions were difficult, what they thought particular questions meant, and whether any answer they wanted to give was unavailable.

Pilot study vs questionnaire pretest

These terms are sometimes used loosely, but they can represent different activities.

A questionnaire pretest focuses particularly on whether the instrument works as intended.

A pilot study may rehearse a broader portion of the research process, including:

• recruitment;

• consent;

• questionnaire administration;

• data management;

• response rates;

• timing;

• preliminary analysis procedures; and

• logistical feasibility.

Your institution may use these terms differently, so follow its guidance.

The important point is to test the parts of the research process that could fail before committing to full data collection.

Put questions in a sensible order

Question order can affect respondent experience and potentially responses.

A questionnaire often works better when it begins with questions that are:

• relevant;

• reasonably easy to answer; and

• not unnecessarily sensitive.

Related questions can be grouped into logical sections.

Clear transitions can help when the topic changes.

More demanding or sensitive questions may be better placed after respondents understand the questionnaire and have progressed through it—although the appropriate order depends on the study.

Avoid jumping randomly between unrelated topics.

The questionnaire should feel like one coherent instrument rather than several lists pasted together.

Use skip logic carefully

Not every question applies to every participant.

Suppose you ask:

Have you used the university counselling service during the past 12 months?

Participants answering No should probably not be required to rate the quality of counselling they received.

Online questionnaire tools can use branching or skip logic to show relevant follow-up questions.

But test that logic carefully.

A routing error can:

• hide necessary questions;

• show irrelevant questions;

• create missing data; or

• accidentally exclude participants from entire sections.

Always test every important route through the questionnaire before launch.

Handle demographic questions thoughtfully

Demographic variables can be useful for:

• describing the sample;

• assessing representation;

• examining relevant subgroup differences; or

• controlling for theoretically justified factors.

But do not collect demographic information merely because questionnaires usually contain a demographics section.

Ask:

What will I do with this information?

If you have no research, analytical, ethical, or reporting reason to collect a characteristic, you may not need it.

This is particularly important when information could increase identifiability.

Data minimisation is good research practice as well as a privacy consideration.

Sensitive questions need extra care

Questions concerning topics such as:

• health;

• income;

• trauma;

• discrimination;

• sexual behaviour;

• illegal activity;

• substance use; or

• other private experiences

may require additional ethical and methodological care.

Depending on the study, consider:

• whether the question is necessary;

• whether respondents can skip it;

• whether a Prefer not to say option is appropriate;

• how privacy will be protected;

• whether wording could cause unnecessary distress;

• what support information may be needed; and

• what your ethics approval permits.

Do not collect sensitive information simply because it might be interesting later.

Keep the questionnaire only as long as necessary

Long questionnaires can increase respondent burden.

But “short” is not automatically good either.

A five-question instrument that fails to measure the study's central concepts adequately is not superior to a longer instrument that has a clear purpose.

The better principle is:

Include enough well-designed items to generate the evidence your study needs—and remove questions that do not contribute.

For every item, ask:

What decision, variable, objective, or analysis requires this question?

If there is no convincing answer, consider removing it.

A worked questionnaire example

Suppose your study asks:

What relationship exists between students' perceptions of formative-feedback quality and their academic engagement?

Step 1: Identify the evidence needed

You need evidence concerning:

• perceptions of feedback quality; and

• academic engagement.

Step 2: Define the concepts

Feedback quality might involve:

• clarity;

• timeliness;

• specificity; and

• usefulness for improvement.

Academic engagement might involve appropriately defined behavioural, cognitive, or other dimensions relevant to your study.

Step 3: Decide how the concepts will be measured

You might use an established instrument where suitable, or develop justified items based on literature and the study's conceptual definitions.

Step 4: Draft focused items

Instead of:

My lecturer gives clear and timely feedback that motivates me and improves my grades.

split the ideas.

For example:

The feedback I receive clearly identifies how I can improve my work.

and:

I receive feedback early enough to use it in later assignments.

Each item now addresses a more specific idea.

Step 5: Choose appropriate response options

If the items are statements about students' perceptions, a consistent agreement scale might be appropriate if justified by the measurement design.

Step 6: Map items back to the study

Create a simple design table:

If an item cannot be mapped to a study requirement, reconsider why it exists.

Step 7: Pretest

Ask suitable participants to complete the questionnaire and investigate:

• interpretation;

• missing options;

• completion time;

• usability;

• confusing items; and

• technical problems.

Step 8: Revise before launch

Make changes based on evidence from the pretest—not simply because you prefer different wording afterward.

If substantive changes affect an established measure, consider what that means for its measurement properties and comparability.

Common questionnaire-design mistakes

Writing questions before defining what needs to be measured

This produces interesting questions rather than useful evidence.

Asking two things in one item

Double-barrelled questions make responses difficult to interpret.

Leading respondents

Questions should measure participants' views rather than communicate the researcher's preferred answer.

Using vague time periods

Words such as often, regularly, or recently may be interpreted differently unless that ambiguity is intentional.

Overlapping response categories

A respondent should not fit two categories when only one response is permitted.

Missing legitimate response options

Closed-ended categories need to accommodate the plausible answers relevant to the study.

Using complicated academic language

Respondents need to understand the questions consistently.

Collecting unnecessary demographic information

Every additional piece of personal information should have a purpose.

Modifying established scales casually

Changing wording or scoring may change what the instrument measures.

Ignoring mobile usability

An online questionnaire that looks excellent on a laptop may be frustrating or unusable on a phone.

Skipping pretesting

The first time real participants interpret the questionnaire should not ideally be during your main data collection.

Designing the analysis after collecting responses

If you do not know how an item contributes to the analysis, you may discover too late that the questionnaire cannot answer the research question.

A questionnaire-design checklist

Before launching your questionnaire, ask:

• Does every important section connect to a research question or objective?

• Have I clearly defined the concepts or variables I need to measure?

• Does every item have a methodological purpose?

• Does each question ask one thing at a time?

• Is the wording neutral rather than leading or loaded?

• Are questions specific enough for respondents to interpret consistently?

• Are recall periods appropriate?

• Are response categories mutually exclusive where required?

• Are the response options sufficiently complete?

• Are scale directions and labels consistent?

• Have I planned how important responses will be coded and analysed?

• Have I justified any established instrument or adaptation I use?

• Have I avoided unnecessary personal or sensitive questions?

• Does skip logic work correctly?

• Is the questionnaire usable on the devices participants are likely to use?

• Has it been pretested appropriately?

• Can the final questionnaire actually produce the evidence needed to answer the study?

If you cannot explain what an item contributes, do not keep it simply because the questionnaire looks more substantial with another question.

Final takeaway

A good research questionnaire is not a collection of questions that sound academic.

It is a measurement and data-collection instrument designed around the evidence your study needs.

Start with the research question and objectives.

Define the concepts or variables.

Decide what evidence would represent them.

Choose or develop items capable of producing that evidence.

Design response options with the same care as the questions themselves.

Plan the analysis before collecting data.

Then pretest the instrument with suitable participants and correct problems before the main study begins.

The most useful question during questionnaire design is therefore not:

“What else can I ask?”

It is:

“What evidence will this item produce, and how will that evidence help answer my research question?”

When every important item has a clear answer to that question, your questionnaire is far more likely to produce data you can actually use.