What should be considered in the research design?

Sarah Wright, PhD, MBA, Bridget C. O'Brien, PhD, Laura Nimmon, PhD, Marcus Law, MD, MBA, MEd, and Maria Mylopoulos, PhD

Disclaimer

Corresponding author: Sarah Wright, PhD, MBA, Toronto East General Hospital, 825 Coxwell Avenue, Toronto, ON M4C 3E7 Canada, [email protected]

Copyright Accreditation Council for Graduate Medical Education

Associated Data

Wright_Appendices1-2.doc (28K)

GUID: 1B5FBE97-3723-47B9-9407-06E55BEB9029

Editor's Note: The online version of this article contains references and resources for further reading and the authors' professional information.

The Challenge

“I'd really like to do a survey” or “Let's conduct some interviews” might sound like reasonable starting points for a research project. However, it is crucial that researchers examine their philosophical assumptions and those underpinning their research questions before selecting data collection methods. Philosophical assumptions relate to ontology, or the nature of reality, and epistemology, the nature of knowledge. Alignment of the researcher's worldview (ie, ontology and epistemology) with methodology (research approach) and methods (specific data collection, analysis, and interpretation tools) is key to quality research design. This Rip Out will explain philosophical differences between quantitative and qualitative research designs and how they affect definitions of rigorous research.

What Is Known

Worldviews offer different beliefs about what can be known and how it can be known, thereby shaping the types of research questions that are asked, the research approach taken, and ultimately, the data collection and analytic methods used. Ontology refers to the question of “What can we know?” Ontological viewpoints can be placed on a continuum: researchers at one end believe that an observable reality exists independent of our knowledge of it, while at the other end, researchers believe that reality is subjective and constructed, with no universal “truth” to be discovered.1,2 Epistemology refers to the question of “How can we know?”3 Epistemological positions also can be placed on a continuum, influenced by the researcher's ontological viewpoint. For example, the positivist worldview is based on belief in an objective reality and a truth to be discovered. Therefore, knowledge is produced through objective measurements and the quantitative relationships between variables.4 This might include measuring the difference in examination scores between groups of learners who have been exposed to 2 different teaching formats, in order to determine whether a particular teaching format influenced the resulting examination scores.

In contrast, subjectivists (also referred to as constructionists or constructivists) are at the opposite end of the continuum, and believe there are multiple or situated realities that are constructed in particular social, cultural, institutional, and historical contexts. According to this view, knowledge is created through the exploration of beliefs, perceptions, and experiences of the world, often captured and interpreted through observation, interviews, and focus groups. A researcher with this worldview might be interested in exploring the perceptions of students exposed to the 2 teaching formats, to better understand how learning is experienced in the 2 settings. It is crucial that there is alignment between ontology (what can we know?), epistemology (how can we know it?), methodology (what approach should be used?), and data collection and analysis methods (what specific tools should be used?).5

Key Differences in Qualitative and Quantitative Approaches

Use of Theory

Quantitative approaches generally test theory, while qualitative approaches either use theory as a lens that shapes the research design or generate new theories inductively from their data.4

Use of Logic

Quantitative approaches often involve deductive logic, starting off with general arguments of theories and concepts that result in data points.4 Qualitative approaches often use inductive logic or both inductive and deductive logic, start with the data, and build up to a description, theory, or explanatory model.4

Purpose of Results

Quantitative approaches attempt to generalize findings; qualitative approaches pay specific attention to particular individuals, groups, contexts, or cultures to provide a deep understanding of a phenomenon in local context.4

Establishing Rigor

Quantitative researchers must collect evidence of validity and reliability. Some qualitative researchers also aim to establish validity and reliability. They seek to be as objective as possible through techniques, including cross-checking and cross-validating sources during observations.6 Other qualitative researchers have developed specific frameworks, terminology, and criteria on which qualitative research should be evaluated.6,7 For example, the use of credibility, transferability, dependability, and confirmability as criteria for rigor seek to establish the accuracy, trustworthiness, and believability of the research, rather than its validity and reliability.8 Thus, the framework of rigor you choose will depend on your chosen methodology (see “Choosing a Qualitative Research Approach” Rip Out).

View of Objectivity

A goal of quantitative research is to maintain objectivity, in other words, to reduce the influence of the researcher on data collection as much as possible. Some qualitative researchers also attempt to reduce their own influence on the research. However, others suggest that these approaches subscribe to positivistic ideals, which are inappropriate for qualitative research,6,9,10 as researchers should not seek to eliminate the effects of their influence on the study but to understand them through reflexivity.11 Reflexivity is an acknowledgement that, to make sense of the social world, a researcher will inevitably draw on his or her own values, norms, and concepts, which prevent a totally objective view of the social world.12

Sampling Strategies

Quantitative research favors using large, randomly generated samples, especially if the intent of the research is to generalize to other populations.6 Instead, qualitative research often focuses on participants who are likely to provide rich information about the study questions, known as purposive sampling.6

How You Can Start TODAY

1. Consider how you can best address your research problem and what philosophical assumptions you are making.

2. Consider your ontological and epistemological stance by asking yourself: What can I know about the phenomenon of interest? How can I know what I want to know? What approach should I use and why? Answers to these questions might be relatively fixed but should be flexible enough to guide methodological choices that best suit different research problems under study.5

3. Select an appropriate sampling strategy. Purposive sampling is often used in qualitative research, with a goal of finding information-rich cases, not to generalize.6

4. Be reflexive: Examine the ways in which your history, education, experiences, and worldviews have affected the research questions you have selected and your data collection methods, analyses, and writing.13

How You Can Start TODAY—An Example

Let's assume that you want to know about resident learning on a particular clinical rotation. Your initial thought is to use end-of-rotation assessment scores as a way to measure learning. However, these assessments cannot tell you how or why residents are learning. While you cannot know for sure that residents are learning, consider what you can know—resident perceptions of their learning experiences on this rotation.

Next, you consider how to go about collecting this data—you could ask residents about their experiences in interviews or watch them in their natural settings. Since you would like to develop a theory of resident learning in clinical settings, you decide to use grounded theory as a methodology, as you believe asking residents about their experience using in-depth interviews is the best way for you to elicit the information you are seeking. You should also do more research on grounded theory by consulting related resources, and you will discover that grounded theory requires theoretical sampling.14,15 You also decide to use the end-of-rotation assessment scores to help select your sample.

Since you want to know how and why students learn, you decide to sample extreme cases of students who have performed well and poorly on the end-of-rotation assessments. You think about how your background influences your standpoint about the research question: Were you ever a resident? How did you score on your end-of-rotation assessments? Did you feel this was an accurate representation of your learning? Are you a clinical faculty member now? Did your rotations prepare you well for this role? How does your history shape the way you view the problem? Seek to challenge, elaborate, and refine your assumptions throughout the research.

As you proceed with the interviews, they trigger further questions, and you then decide to conduct interviews with faculty members to get a more complete picture of the process of learning in this particular resident clinical rotation.

What You Can Do LONG TERM

1. Familiarize yourself with published guides on conducting and evaluating qualitative research.5,16–18 There is no one-size-fits-all formula for qualitative research. However, there are techniques for conducting your research in a way that stays true to the traditions of qualitative research.

2. Consider the reporting style of your results. For some research approaches, it would be inappropriate to quantify results through frequency or numerical counts.19 In this case, instead of saying “5 respondents reported X,” you might consider “respondents who reported X described Y.”

3. Review the conventions and writing styles of articles published with a methodological approach similar to the one you are considering. If appropriate, consider using a reflexive writing style to demonstrate understanding of your own role in shaping the research.6