How to Ensure Reliability and Validity in Your Research

Which types of reliability are relevant?

Not every type of reliability applies to every thesis.

Choose the form that fits your instrument and research design.

Test-retest reliability

Test-retest reliability examines whether the same instrument produces similar results when used again under similar conditions.

This is relevant when the concept should remain reasonably stable over time.

It may be less useful when:

• participants are expected to change;
• an intervention takes place;
• the first measurement influences the second;
• the research period is very short.

Inter-rater reliability

Inter-rater reliability examines whether different researchers or assessors reach similar conclusions.

This can be relevant when:

• several researchers code interviews;
• observers classify behaviour;
• assessors score open answers;
• documents are categorised using a coding scheme.

You can strengthen it by:

• defining categories clearly;
• training assessors;
• coding part of the data independently;
• comparing differences;
• refining the coding instructions.

Internal consistency

Internal consistency examines whether several items designed to measure the same concept produce consistent results.

This is often relevant for questionnaires with multi-item scales.

For example, a scale for employee engagement may contain several statements about energy, dedication and involvement. The items should fit together while still covering the construct adequately.

A high consistency score does not automatically prove validity. Very similar questions may produce consistent answers while measuring only a narrow part of the concept.

Parallel-forms reliability

Parallel-forms reliability compares two equivalent versions of the same test or instrument.

Both versions should measure the same concept at a similar level.

This type is less common in many bachelor’s and master’s theses. Only discuss it when you actually use equivalent versions.

How to strengthen reliability and validity step by step

Step 1: Check your programme requirements

Start with:

• the graduation manual;
• the assessment criteria;
• the methodology guidelines;
• instructions from your thesis supervisor;
• examples provided by your programme.

Check which terms your programme uses.

It may ask for:

• reliability and validity;
• trustworthiness;
• credibility and dependability;
• generalisability;
• research quality;
• methodological justification.

Use the concepts that fit both your research design and programme.

Step 2: Link quality criteria to the research design

Do not create a standard paragraph that could fit any thesis.

Ask:

• Which errors could affect this study?
• Which forms of bias are possible?
• What could make the results inconsistent?
• What could make the conclusions inaccurate?
• Which quality criteria fit this method?
• Which measures have I actually taken?

A survey, interview study and experiment require different quality measures.

Step 3: Define and operationalise your concepts

Start with clear theoretical definitions.

For each central concept:

1. define what it means;
2. identify relevant dimensions;
3. decide how each dimension will be observed or measured;
4. connect the indicators to survey questions, interview topics or other data.

For example, “student satisfaction” is too broad until you define which aspects are included.

These might be:

• supervision;
• communication;
• workload;
• learning resources;
• assessment;
• sense of belonging.

A clear operationalisation strengthens the link between theory and measurement.

Step 4: Select a suitable sample

The sample influences both the quality and scope of your conclusions.

Explain:

• who belongs to the target population;
• which inclusion and exclusion criteria you used;
• how participants were selected;
• why the sampling method fits the research question;
• how many people were invited;
• how many participated;
• which relevant groups may be underrepresented;
• how non-response could affect the findings.

Do not call a sample representative without evidence.

For qualitative research, focus on whether participants have relevant knowledge and whether different perspectives are included.

For quantitative research, consider sample size, selection, response and comparability with the population.

Step 5: Develop and test your research instrument

Explain how the survey, interview guide, observation form or coding scheme was developed.

You can strengthen the instrument by:

• basing questions on literature;
• using existing validated scales;
• asking experts to review the content;
• conducting a pilot;
• checking whether questions are clear;
• removing leading or double-barrelled questions;
• using consistent response scales;
• documenting changes.

A validated instrument may strengthen your study. It is not automatically valid for every context.

Changing wording, language, target group or response options can affect the original validity.

Step 6: Standardise data collection

Use a consistent procedure where appropriate.

For a survey, this may involve:

• giving all respondents the same instructions;
• using the same question order;
• keeping the survey open for a defined period;
• preventing duplicate responses;
• applying consistent inclusion rules.

For interviews, this may involve:

• using the same topic guide;
• explaining the study in a consistent way;
• asking comparable core questions;
• recording interviews with permission;
• documenting follow-up questions;
• using a consistent transcription approach.

Consistency does not mean that every qualitative interview must be identical. Semi-structured interviews allow relevant follow-up questions.

Step 7: Analyse the data transparently

Explain how you moved from raw data to findings.

For quantitative research, describe:

• data cleaning;
• missing values;
• coding;
• scale construction;
• descriptive statistics;
• statistical tests;
• assumptions;
• software;
• significance levels where relevant.

For qualitative research, describe:

• transcription;
• initial coding;
• category development;
• theme development;
• theory-driven or data-driven choices;
• use of software;
• checks by another researcher;
• how quotations were selected.

Transparency helps the reader assess how consistent and credible the analysis is.

Step 8: Report limitations honestly

No research is completely free from limitations.

Explain:

• which risks remained;
• why they could not be removed;
• how they may have influenced the findings;
• which conclusions remain justified;
• which claims should be made cautiously;
• what future research could improve.

Do not write only:

The research was reliable and valid.

A stronger explanation is: All interviews followed the same topic guide and were recorded and transcribed. This improved consistency. However, participants were recruited through one organisation, which limits the transferability of the findings to other settings.

Research quality in qualitative research

Qualitative research does not always aim to produce identical results in a repeated study.

Participants, contexts and interactions may differ.

Depending on your programme, you may discuss:

Credibility

Do the findings represent the participants’ experiences or the investigated context convincingly?

You may strengthen credibility through:

• prolonged engagement;
• relevant participant selection;
• member checking;
• triangulation;
• peer review;
• searching for contrasting cases.

Dependability

Is the research process logical, documented and transparent?

You may strengthen dependability through:

• a clear interview procedure;
• a documented coding process;
• an audit trail;
• consistent research decisions;
• peer review of codes or themes.

Transferability

Have you described the participants and context clearly enough for readers to assess whether the findings may apply elsewhere?

Transferability does not require statistical generalisation.

Confirmability

Can the reader see how the findings follow from the data rather than only from the researcher’s expectations?

You may strengthen confirmability through:

• transparent coding;
• supporting quotations;
• reflexive notes;
• peer discussion;
• documenting changes and decisions.

Use the terminology required by your programme. Some programmes still ask students to discuss reliability and validity for qualitative research. You can then explain how the chosen measures support consistency, credibility and transparency.