You have chosen your research topic.

Your research questions appear to be clear. You have chosen a quantitative, qualitative or mixed-methodology. You are aware of your participants and you have made a decision on how to gather the data.

Then are your research design valid?

This question can render research methodology much more complex than it actually is.

The research design is not just any research design which applies questionnaires, interviews, statistical software or a big sample. It is a design in such a way that you can gather relevant evidence and come up with reasonable conclusions that can be trusted as per your study.

Simply put, ask yourself:

Does my research design actually allow me to answer the question I am asking?

It is best to begin there.

The validity of research has various dimensions. Internal validity is an issue of whether the design, conduct and analysis can be used to make credible conclusions within the study whereas external validity is an issue of whether the findings can be credibly generalized outside the context of the study.

We had better see what this means without making it a book on research methods.

First, What Do We Mean by Research Design?

A research design is the general outline that will tie your research question to the evidence you gather and the conclusions you will derive in the end.

It has an impact on decisions like who will be involved, how the participants will be chosen, what information will be gathered, which tools will be taken, and how the data will be analyzed.

That is the reason why research design is larger than merely stating:

This is a quantitative study.

Or:

This is a research based on interviews.

The statements are characterizations of personal methodological decisions. The combination of those choices is explained by a research design.

To take an example, suppose that you are interested in exploring the relationship between job performance and employee training.

You require a design that can measure the pertinent concepts, gather the right evidence by a suitable population and analyse the connection appropriately.

Even a good academic writing will not help resolve the real problem unless the design is able to do those things.

According to the US National Institute of Health, the study design, sampling, data collection and analysis decisions, all can affect the quality of a study and the capacity to warrant proper conclusions.

The Quick Answer: What Makes a Design Valid?

In case of a clear correspondence between: a research design is stronger.

Research Question → Research Design → Participants → Measurement → Data Collection → Analysis → Conclusion

The stages must be sensible in terms of each other.

Suppose your research question is:

“What is the relationship between social media use and academic performance among university students?”

What you require is data that is capable of reasonably measuring social media use and academic performance, a suitable sample of university students, and a methodological approach that can be used to analyze the hypothesis relationship.

Imagine now that you are gathering interviews of five company managers.

The interviews as such may be perfectly conducted but do not answer the research question.

Alignment thus is the start of validity.

Internal Validity: Can You Trust What Happened Inside the Study?

Internal validity questions whether the conclusions made in the course of a study are credible instead of being elucidated by bias and other influences.

This is especially relevant when researchers come up with causal assertions.

Suppose that a researcher presents a new method of teaching and students begin to have better test scores.

One may be tempted to find:

“The new teaching method improved performance.”

But wait.

Were the students tutored further within the time frame?

Was the second test easier?

Weak students drop out of the study?

Was there another significant change that occurred in the university?

These other explanations are important.

Internal validity is related to the issue of whether the study design and practice can enable the researcher to draw a plausible conclusion as opposed to falsely attributing some result to a faulty cause. That confidence can be undermined by bias due to selection, measurement, attrition, and other aspects of the research process.

To elaborate,Scribbr’s guide to internal validity that presents easy-to-understand examples of typical threats to students.

Selection Bias Can Quietly Damage Your Study

Who comes into your study is important.

Suppose you are interested in researching job satisfaction in a whole organisation but only gather the answers of employees who attend a voluntary management-development programme.

Such workers may be different to the rest of the workforce.

Maybe they are more ambitious, more active or more contented with the organisation.

When you believe their answers to accurately reflect the entire population, then you can reach conclusions that are misleading.

This does not imply that all research projects need random sampling.

Qualitative studies, such as those, often employ purposive selection since researchers require participants with experience with the matter in question.

The question is whether the sampling strategy suits the purpose of the research and have you become aware of its limitations.

Claim not more than your sample provides you with the opportunity to claim.

Measurement Matters More Than Students Sometimes Realise

Imagine that your dissertation is the study of employee engagement.

What will you do to tell whether somebody is engaged or not?

Employee engagement cannot be put on a weighing scale.

It is a construct, and that is, you must have proper indicators or measurement items to depict it.

Now suppose that the focus of your questionnaire is the questions of whether the employees like their salary or not.

You may be measuring something, but are you measuring employee engagement?

Measurement and construct validity comes in there.

Construct validity is a question of whether the operationalization of a concept reflects the concept that the researcher wants to study. This is different to internal and external validity since a research can be done meticulously and still measure the wrong thing.

This is the reason why questionnaire questions cannot be made up by just inventing them since they sound good.

In case of need, researchers can adopt or modify the already tested measurement scales and justify the sources of those measures.

Reliability and Validity Are Not the Same Thing

The two terms are used interchangeably; however, the meaning of these two is not the same.

Reliability is mainly about consistency. Validity is about whether the measurement or conclusion is appropriate and accurate for its intended purpose.

Consider a weighing scale that constantly increases the weight by five kilograms.

You take three steps on it and get virtually the same wrong answer.

The test is equivalent.

Yet it is not so.

The above example is just a simple one that can be used to explain why something can prove to be reliable even though it may not prove to be valid at all.

Both may have to be taken into account in research instruments. A measure must give consistently reliable results, and one that should be what it is purported to measure.

External Validity: Do Your Findings Travel?

Internal validity poses the question of whether the conclusions of the study are sensible.

Another question posed by external validity is:

Can these findings reasonably apply elsewhere?

Assume that you are doing a study of 120 undergraduate students in a single university.

Are you automatically stating that the results are representative of all the university students in the country?

Probably not.

Do you say they represent working adults?

Even less likely.

External validity deals with how the results can be used to generalize to other individuals, people, contexts or times.

This is why the properties of your sample are important.

The results of one particular group can still be useful. The error is going far beyond the research evidence in making claims.

An excellent researcher has clear study boundaries.

Bigger Samples Do Not Automatically Fix Validity

Students sometimes believe:

“If I get more respondents, my research will be more valid.”

Not necessarily.

Sample size is important especially in statistical analysis, but size alone is not a good design.

Consider gathering 2,000 answers of totally inappropriate respondents.

You have just a very large dataset that is responding to the incorrect research question.

A smaller yet properly chosen sample can be more valuable evidence at times.

Other significant questions will be who are the participants, how they were chosen, whether they represent the target population and whether the sample is appropriate to the kind of conclusions you may want to make.

Do not pursue many because it seems so.

Your Data Collection Method Must Fit the Question

The other significant aspect of validity is the selection of a method of data collection that will be able to give you the necessary information.

In case your research question is:

“How do first-generation university students experience academic pressure?”

A questionnaire with five yes/no questions might not provide the participants with the chance to elaborate on those experiences.

Semi-structured interviews might possibly present more information.

In this case, your research question is:

“Is perceived usefulness associated with intention to adopt mobile banking?”

A structured quantitative questionnaire can be far more appropriate in case these constructs are well defined and measured in the appropriate way.

Both questionnaire and interview are not necessarily more valid.

Fit is dependent on validity.

The method should be motivated by the question.

Your Analysis Must Fit the Data Too

Validity is not halted at the end of data collection.

You may have good data, and may make feeble conclusions, when you analyse it wrong.

In quantitative research, the research question, variables, level and assumptions of measurement used in the statistical method must be equal.

Write not:

Data analysis was done by use of SPSS.

SPSS is software.

What was it that you did?

Have you computed the descriptive statistics?

Test reliability?

Examine correlations?

Use regression?

Compare groups?

The method of analysis is important.

On the same note, qualitative researchers ought to describe the methods of analysis used on interviews, documents or observations. In case thematic analysis applies, the methodology must articulate how the researchers will proceed to the raw data to codes, patterns and themes.

Valid design ties the question, data and analysis.

Statistical Significance Does Not Automatically Mean “Valid”

A p-value of less than a specific threshold does not necessarily demonstrate the validity of a whole research design.

Statistical conclusion validity deals with the issue of whether the conclusions of statistical tests have accurately reflected the relationship or differences in data. Such conclusions can be influenced by factors, including lack of sufficient statistical power, improper analysis or breaking significant assumptions.

That is why statistical analysis is to be planned but not chosen after viewing the results.

Where feasible, seek what analysis is required to answer each research question or test each hypothesis.

Validity Is Not Only a Quantitative Issue

There are cases when students can only relate the term validity with questionnaires and statistics.

One also requires a clear way of research quality in qualitative research even though the qualitative traditions might apply various concepts and terminologies.

Qualitative researchers can refer to credibility, dependability, confirmability and transferability, as an example.

The precise criteria is based on the methodology and research tradition.

The most significant principle is the same: the readers need to see how the research was done and why its interpretation may be trusted.

This may include a description of the process of participant selection, recording of analytical choices, proper coding processes, alternative interpretations or demonstrating how the conclusions were based on the data.

Do not impose quantitative validity tests upon qualitative research merely because you have found the tests in some other dissertation.

Apply quality criteria that are relevant to your design.

Watch Out for the “Perfect Design” Myth

No zero limitations research design.

Even good studies entail decisions and compromises.

Experiments that are highly controlled can enhance confidence regarding the causal relationship, although tightly controlled conditions can occasionally complicate the question of whether a similar result is going to be observed in actual real-life scenarios. Literature on research thus identifies a possible conflict between internal and external validity.

That is no reason to have the research bad.

It implies that researchers should be aware of what their design will be able and unable to establish.

The dissertation cannot be a good one that claims that nothing is limiting it.

It determines significant constraints and does not draw overreaching conclusions.

This is an indicator of better research thought.

A Simple Way to Test Your Own Research Design

Write your focal research question on the top of a blank page.

then go by your design.

Ask:

What evidence do I need to answer this question?

Then:

Does my data collection method actually produce that evidence?

Next:

Are the people or materials I am studying appropriate for the question?

Then:

Am I measuring the concepts I claim to be measuring?

And finally:

Does my analysis allow me to reach the conclusion I want to make?

Should there be a loose linkage anywhere in that chain, inquire into it.

This is a basic practice that can help a great deal more than a second page of research-method definitions.

Be Careful With Cause-and-Effect Language

This is especially significant in the student research.

The discovery of a relationship between two variables does not necessarily imply causality between the two variables.

Imagine that you analyze and find out that employee motivation is related to job performance.

Can you even at this moment conclude:

“Employee motivation causes higher performance”?

Not necessarily.

There might be other factors that affect both variables and the design might be not able to determine the causality.

The language you use ought to be what the research design is able to back up.

The relationship, association, difference, influence, prediction and cause do not necessarily have the same methodological meaning.

A good researcher is aware of what the design enables him or her to say.

Validity Starts Before You Collect the Data

Among the most critical errors is the attempt to consider the concept of validity as the topic to be discussed at the end of the methodology chapter.

At that point, there can be certain issues that are hard to rectify.

The research question, research design, selection of the participants, development or selection of instruments and the analysis should be regarded as validity.

As an example, finding out after you have collected some data that your questionnaire has not been properly represented to one of your key constructs is a far greater challenge than finding out that problem during questionnaire development.

Research planning matters.

And good design is prophylactic.

How to Write About Validity Without Sounding Like a Textbook

You do not need three pages to describe all the types of validity in your methodology chapter.

State the applicable concept in a concise manner, provide methodological literature that supports it and then demonstrate its applicability to your research.

Instead of writing:

Validity in research is very important. There are a number of forms of validity…

move towards your study:

The questionnaire measures were modified to the already existing scales that were used to measure the study variables to enhance construct validity.

Then say which scales, whence came they and how did they come to be changed.

Usually application is of more value than pages of generic definitions.

The Final Validity Check

Prior to finalizing your methodology, consider the study as a single system as opposed to various headings.

Your design should be in line with your research questions.

Your subjects must represent the population or phenomenon that you would like to study.

The desired concepts should be measured by your instruments.

The avoidable bias should be minimized in your data collection process.

The data should be appropriate in your analysis.

What you conclude on ought to remain within what the evidence can substantiate.

And your limitations ought to be met and not concealed.

When these elements are interrelated in a logical sense, then you are developing a far more robust research design.

Final Takeaway: Validity Is About Trusting the Logic of Your Research

Then how is a research design any good?

It is not one test, one statistic or one paragraph in Chapter 3.

Validity is based on the quality and reasonableness of the entire research process.

A study must pose a suitable question, glean pertinent evidence, quantify its concepts in a suitable manner, reduce the significant sources of error, apply apt analysis, and draw conclusions that can be supported by the design.

Internal validity is a question that requires answers to the question of whether the conclusions in the study are credible. External validity deals with the extent to which those results can be extrapolated outside of the research. Construct and measurement validity aid the researcher to reflect on whether he/she measures what he/she purports to measure or not.

The most crucial one, do not say that your design is valid just because you have adopted a recognized method.

A questionnaire may be very poorly designed.

Inappropriate participants can be recruited in an interview study.

A big data has the power to quantify the wrong thing.

Weak data can be analyzed using sophisticated statistical programs.

The point is that all the components of the research design should be compatible with each other in that they can address the research question credibly.

When you can answer why your design is suitable to your question, why your participants are suitable, why your measures are capturing the intended concepts, why your analysis is suitable to the data, and why your conclusions are not beyond the evidence, you will be much nearer to proving that you have an appropriate research design.