What is the name of the bias when participants in an experiment respond differently than they otherwise would just because they are in the experiment?

Non-response bias

Non-response bias can occur when subjects who refuse to take part in a study, or who drop out before the study can be completed, are systematically different from those who participate. In simple descriptive epidemiology, for example, the prevalence of depression in a community may be underestimated if those with depression are less likely to participate in the cross-sectional survey than those without depression. An association between lack of social support and depression may be overestimated either if those with good social support are less likely to take part if they are depressed or if those with poor social support are less likely to take part if they are not depressed. Again, note that when an association between an exposure and a disease is being estimated, bias will only occur if the error operates differentially with respect to both.

Non-response bias can be minimized by minimizing non-response. Non-response becomes a critical issue when response rates fall below 70%, but significant non-response bias can still occur even at these levels of participation. The likelihood of non-response bias having occurred can be assessed (although not quantified) by comparing the characteristics of responders and non-responders. Usually, some basic sociodemographic information such as age and sex is available from the register or database from which the subjects have been recruited. Similarity of responders and non-responders in terms at least of these basic characteristics is reassuring but does not exclude the possibility that bias has occurred.

6 Sources of Error

Beyond statistical sampling error and nonresponse bias, there are other sources of error in surveys that are not easily quantified. The responses to survey questions and measurement of opinions and behavior can be affected by how questions are worded, what response categories are offered (e.g., whether a middle category or a don't know or no opinion response is offered or allowed), and whether questions should have fixed or forced choices, or whether questions should be asked and responses recorded as open-ended questions in which respondents can offer any answer they choose. Further, there may be ‘context effects’ that are produced by the order in which questions are asked, so that the way respondents answer questions may be influenced by what they had thought about and responded to in previous questions (Schuman and Presser 1981, Asher 1998).

A major source of error can occur depending on how research problems are formulated or specified, as when researchers assume respondents have familiarity with the object about which they are asked to offer an opinion or response. Care needs to be exercised in drawing inferences about actual behavior from survey measures of opinion, and self-reports of future—or even past—behavior. Reported voting in an election after the fact may not match actual behavior based upon data on ‘validated’ votes. Difficulties may also arise in how poll results are reported, such as when journalists or researchers themselves do not report enough information about their survey data and methods to allow their audiences or readers to evaluate their results (Cantril 1991, Traugott and Lavrakis 2000)

Motivation for Analyzing Non-Response Bias

To illustrate and underscore the importance of analyzing non-response bias, consider the following scenario. A researcher working for a marketing firm wishes to estimate the average age of New Yorkers who own telephones. In order to do this, the researcher attempts to conduct a phone survey of 1000 individuals drawn from the population of phone-owning New Yorkers by dialing randomly chosen residential phone numbers. However, after 1000 attempts, the researcher is in possession of only 746 valid responses because 254 individuals never answered the phone and therefore could not be reached. At this point, the researcher averages the ages of the 746 respondents with valid responses and considers whether this average is likely to be too high or too low. Does one expect the 254 non-responders to be roughly the same age as respondents who answered their phones?

After thinking it over, the researcher concludes that the average age of the 746 responders is a biased estimate because the surveys were conducted during business hours when workers (as compared to older retirees) were less likely to be at home. If working age respondents are underrepresented, then the average among the 746 valid age responses is biased upward. In this case, the difference between the biased average and the true but unobserved average age among all telephone owners is precisely non-response bias.

Social scientists often attempt to make inferences about a population by drawing a random sample and studying relationships among the measurements contained in the sample. When individuals from a special subset of the population are systematically omitted from a particular sample, however, the sample cannot be said to be random in the sense that every member of the population is equally likely to be included in the sample. It is important to acknowledge that any patterns uncovered in analyzing a nonrandom sample do not provide valid grounds for generalizing about a population in the same way that patterns present in a random sample do. The mismatch between the average characteristics of respondents in a nonrandom sample and the average characteristics of the population can lead to serious problems in understanding the causes of social phenomena and may lead to misdirected policy action. Therefore, considerable attention has been given to the problem of non-response bias, both at the stages of data collection and data analysis.

Nonresponse Bias

The methodological concerns do not end once a sample of potential respondents has been contacted. Error or nonresponse bias may also be introduced because some members of the selected sample are unable or unwilling to complete the survey. The extent of bias depends on both the incidence of nonresponse and on how nonrespondents differ from respondents on variables of interest. The effect of nonresponse is to confound the behavioral parameters of interest with parameters that determine response. Nonresponse bias is not unique to Internet surveys but the potential problem is quite severe for Web-based surveys that have low response rates and nonrandom recruitment procedures.

Web-survey nonresponse might be aggravated because potential respondents encounter technological difficulties. Internet respondents need to have basic literacy skills, know how to surf the Web, be able to use the mouse to select response options from menus, and know how to type answers in the fields provided. Furthermore, technological hurdles, such as browser incompatibility and slow Internet connections, will influence whether a potential respondent completes a survey. Since Internet access tends to be correlated with demographic characteristics such as income and age, Internet survey data will provide biased results if these demographics affect the variables of interest.

Several methods exist to account for selection bias in survey samples but these corrections are complicated by the fact that Web-based surveys provide very little information about nonrespondents. Techniques such as propensity weighting or other simple weighting schemes may be useful in improving the representativeness of Internet survey samples. Simple weighting schemes may be useful in minimizing these biases and errors if there is a strong relationship between the weighting variable and the data in the survey. Supplementing Web surveys with telephone surveys can be used to develop appropriate weighting schemes.

While nonresponse bias is a significant concern for Internet surveys, recent research makes apparent the fact that traditional methodologies, like RDD telephone surveys, may also be problematic. Alvarez et al. report data from a telephone survey in which they began with 13,095 residential telephone numbers to obtain 1500 complete interviews. Of these, 3792 phone numbers were bad in some way, 5479 produced no answer or complete interview, and 1469 produced a valid contact but the survey interview was refused. As few telephone survey studies report statistics like these, it is impossible to characterize the extent to which contemporary telephone survey techniques produce representative samples. The Alvarez et al. evidence suggests that RDD techniques do not necessarily provide truly random samples. Obtaining random samples from large populations may be difficult over the Internet but telephone surveys are not a panacea.

3 Complete-case, Available-case, and Weighting Analysis

A common and simple method is complete-case (CC) analysis, also known as listwise deletion, where incomplete cases are discarded and standard analysis methods applied to the complete cases. In many statistical packages this is the default analysis. Valid (but often suboptimal) inferences are obtained when the missing data are MCAR, since then the complete cases are a random subsample of the original sample with respect to all variables. However, complete-case analysis can result in the loss of a substantial amount of information available in the incomplete cases, particularly if the number of variables is large.

A serious problem with dropping incomplete cases is that the complete cases are often easily seen to be a biased sample, that is, the missing data are not MCAR. The size of the resulting bias depends on the degree of deviation from MCAR, the amount of missing data, and the specifics of the analysis. In sample surveys this motivates strenuous attempts to limit unit nonresponse though multiple follow-ups, and surveys with high rates of unit nonresponse (say 30 percent or more) are often considered unreliable for making inferences to the whole population.

A modification of CC analysis, commonly used to handle unit nonresponse in surveys, is to weight respondents by the inverse of an estimate of the probability of response. A simple approach is to form adjustment cells (or subclasses) based on background variables measured for respondents and nonrespondents; for unit nonresponse adjustment, these are often based on geographical areas or groupings of similar areas based on aggregate socioeconomic data. All nonrespondents are given zero weight and the nonresponse weight for all respondents in an adjustment cell is then the inverse of the response rate in that cell. This method removes the component of nonresponse bias attributable to differential nonresponse rates across the adjustment cells, and eliminates bias if within each adjustment cell respondents can be regarded as a random subsample of the original sample within that cell (i.e., the data are MAR given indicators for the adjustment cells).

With more extensive background information, a useful alternative approach is response propensity stratification, where (a) the indicator for unit nonresponse is regressed on the background variables, using the combined data for respondents and nonrespondents and a method such as logistic regression appropriate for a binary outcome; (b) a predicted response probability is computed for each respondent based on the regression in (a); and (c) adjustment cells are formed based on a categorized version of the predicted response probability. Theory (Rosenbaum and Rubin 1983) suggests that this is an effective method for removing nonresponse bias attributable to the background variables when unit nonresponse is MAR.

Although weighting methods can be useful for reducing nonresponse bias, they do have serious limitations. First, information in the incomplete cases is still discarded, so the method is inefficient. Weighted estimates can have unacceptably high variance, as when outlying values of a variable are given large weights. Second, variance estimation for weighted estimates with estimated weights is problematic. See Estimation: Point and Interval. Explicit formulas are available for simple estimators such as means under simple random sampling (Oh and Scheuren 1983), but methods are not well developed for more complex problems, and often ignore the component of variability arising from estimating the weight from the data.

Available-case (AC) analysis (Little and Rubin 1987 Sect. 3.3) is a straightforward attempt to exploit the incomplete information by using all the cases available to estimate each individual parameter. For example, suppose the objective is to estimate the correlation matrix of a set of continuous variables Y1,…,Yp. See Multivariate Analysis: Overview. Complete-case analysis uses the set of complete cases to estimate all the correlations; AC analysis uses all the cases with both Yj and Yk observed to estimate the correlation of Yj and Yk, 1≤j, k≤p. Since the sample base of available cases for measuring each correlation includes at least the set of complete cases, the AC method appears to make better use of available information. The sample base changes from correlation to correlation, however, creating potential problems when the missing data are not MCAR or variables are highly correlated. In the presence of high correlations, there is no guarantee that the AC correlation matrix is even positive definite. Haitovsky's (1968) simulations concerning regression with highly correlated continuous data found AC markedly inferior to CC. On the other hand, Kim and Curry (1977) found AC superior to CC in simulations based on weakly correlated data. Simulation studies comparing AC regression estimates with maximum likelihood (ML) under normality (Sect. 6) suggest that ML is superior even when underlying normality assumptions are moderately violated (Little 1988a). Although AC estimates are easy to compute, standard errors are more complex. The method cannot be generally recommended, even under the restrictive MCAR.

2 Problems with Panel Surveys

By attempting repeated interviews with the same sample, panel surveys have problems not found in single or repeated cross-sectional designs, the most important of which is panel nonresponse (initial-wave respondents may not respond in later waves). An additional potential problem with panel surveys is panel conditioning, where responses in a given interviewing round may be conditioned by participation in prior rounds of interviews.

Methods developed to cope with nonresponse bias include minimizing nonresponse in panel surveys and developing statistical adjustments for existing nonresponse. Existing panel surveys typically devote sizeable resources to maintaining high response rates, and sometimes are quite successful. For example, the National Longitudinal Survey of Youth conducted interviews in 1991 with 89 percent of the respondents in its initial, 1979, interview (MaCurdy et al. 1998). Losses in the British National Survey of Health and Development amounted to only 12 percent after 26 years (Atkins et al. 1981).

Incentive payments, respondent reports, persuasion letters, using administrative data for tracing, and collecting extensive contact information (e.g., on friends and relatives not living in the household who would know of address and telephone number changes) help minimize these problems (Freedman et al. 1980, Clarridge et al. 1978, Call et al. 1982).

As with any survey, sizeable nonresponse in a panel survey gives rise to concerns about nonresponse bias. The situation with the first wave of a panel survey corresponds to that with a cross-sectional survey in that very limited information is available on the nonrespondents. The situation with later wave nonresponse in a panel survey is, however, different: in this case a good deal of information is available about later wave nonrespondents from their responses on earlier waves. The earlier wave information can be used to investigate the possibility of nonresponse bias and to develop imputation and weighting nonresponse adjustments that attempt to reduce the bias (Kalton 1986, Lepkowski 1989).

With regard to conditioning, there is ample evidence from several surveys that initial responses in a panel survey differ substantially from those given in subsequent waves (Bailar 1975, 1979, Ghangurde 1982). In the case of the US Current Population Survey, estimates of unemployment from households entering the sample for the first time are almost 10 percent larger than the average over all eight monthly reporting periods. It is not clear whether there is more response bias in the initial or subsequent waves, because the repeated contact with respondents has ambiguous effects on the quality of the data. The crucial question, as yet unanswered for most phenomena reported in surveys, is whether it is merely the reporting of behavior or the behavior itself that is affected by panel membership.

It may be that data collected in subsequent panel waves is less biased, because repeated contact increases the probability that respondents understand the purposes of the study and are thus increasingly motivated to make the effort necessary to give more accurate answers. On the other hand, there is evidence from a validation study (Traugott and Katosh 1979) that extended participation in a panel study on election behavior not only increased the accuracy of responses on voting behavior but may indeed have increased the amount of voting, so that the behavior of the panel was no longer representative of the behavior of the population at large.

It seems unlikely that panel participation has pervasive behavioral effects, especially when changes in the behavior under investigation require more effort than making a trip to the polls. For example, economic behavior such as work effort, saving, commuting, and home ownership are all unlikely to be affected by responses to occasional interviews. Responses to attitudinal questions may be affected by panel membership if participation stimulates interest in the subject matter of the survey.

The limited membership in a rotating panel acts to reduce the problems of panel conditioning and panel loss in comparison with a nonrotating panel survey, and the continual introduction of new samples helps to maintain an up-to-date sample of a changing population. Rotating panels are used primarily for the estimation of cross-sectional parameters, objective (a), for the estimation of average values of population parameters across a period of time, objective (b), and for measuring net changes, objective (c). A rotating panel survey will generally provide more precise estimates of point of time and, especially, of change parameters than a repeated survey of the same size. Moreover, a rotating panel survey will sometimes have a cost advantage over a repeated survey. This will occur when it is cheaper to conduct a reinterview than an initial interview, as for instance is the case in the US Current Population Survey where initial interviews must be conducted by personal visit whereas reinterviews on some waves may be conducted by telephone (US Bureau of the Census 1978).

The ability of a rotating panel survey to measure components of individual change, objective (c), and to aggregate data for individuals across time, objective (d), is clearly restricted. Since rotating panels are not intended to serve these objectives, they can be designed to avoid the heavy expense of following movers that occurs with nonrotating panel surveys. Thus, for instance, the Current Population Survey employs dwellings, not households or persons, as the sampled units, so that there is no need to follow households or persons moving between panel waves.

In a split panel survey, the panel survey component can be used to measure components of individual change, objective (c), and to aggregate data for individuals over time, objective (d). Its permanent overlap aids in the estimation of net change, objective (b), between any two waves whereas the overlap in a rotating panel survey aids only in the estimation of net change between certain prespecified waves.

Both rotating and split panel survey designs provide samples of new entrants to the population and the capacity to use their panel survey components to check on biases from panel conditioning and respondent losses.

Introduction

Issues of measurement quality are among the most critical in scientific research because the analysis and interpretation of empirical results depend intimately on the ability to accurately and consistently measure the phenomena of interest. This may be more difficult in social and behavioral sciences, in which the targets of measurement are often not well specified; even when they are, the variables of interest are often impossible to observe directly. For example, concepts such as social status, personality, intelligence, attitudes, values, psychological or emotional states, deviance, or functional status may be difficult to measure precisely because they reflect difficult to define variables and are not directly observable. Even social indicators that are more often thought to directly assess concepts of interest (e.g., education level or race) are not free of conceptual specification errors that lead to imprecision. The inability to define concepts precisely in a conceptually valid way produces errors of measurement, but measurement problems are also critically related to the nature of the communication and cognitive processes involved in gathering data.

Sometimes, the term reliability is used very generally to refer to the overall stability or dependability of research results, including the absence of population specification errors, sampling error, nonresponse bias, as well as various forms of measurement errors. Here, the term is used in its more narrow psychometric meaning, focusing specifically on the absence of measurement errors. Even then, there are at least two different conceptions of error—random and nonrandom (or systematic) errors of measurement—that have consequences for research findings. Within the psychometric tradition, the concept of reliability refers to the absence of random error. This conceptualization of error may be far too narrow for many research purposes, where reliability is better understood as the more general absence of measurement error. However, it is possible to address the question of reliability separately from the more general issue of measurement error, and later the relationship between random and nonrandom components of error is discussed.

Errors of measurement occur in virtually all measurement, regardless of content, and the factors contributing to differences in unreliability of measurement are worthy of scrutiny. It is well-known that statistical analyses ignoring unreliability of measures generally provide biased estimates of the magnitude and statistical significance of the tests of mean differences and associations among variables. Although the resulting biases tend to underestimate mean differences and the strength of relationships making tests of hypotheses more conservative, they also increase the probability of type II errors and the consequent rejection of correct, scientifically valuable hypotheses about the effects of variables of interest.

This article discusses the major approaches to estimating measurement reliability. There are two traditions for assessing reliability: (i) the classical test theory or psychometric tradition for continuous latent variables and (ii) the recent approach developed for categoric latent variables. From the point of view of either tradition, reliability estimation requires repeated measures across multiple levels of the variable. This article focuses mainly on how repeated measures are used in social research to estimate the reliability of measurement for continuous latent variables.

Bias

Sources of bias that may occur in clinical studies include selection, measurement, unacceptability, confounding, recall, referral, volunteer, withdrawal, attention, investigator, and verification, among others. To complicate matters further, the same type of bias may be known by different names or be a subset of some other bias (see later discussion). Many are self-explanatory. A few of the more important types of bias are discussed here.72 Selection bias is the distortion of study effects resulting from the sampling of subjects and includes volunteer bias, nonresponse bias, and bias resulting from loss to follow-up. Another subtype of selection bias is referred to as detection bias.

Measurement bias (also information bias) is distortion of the study effect resulting from inaccurate determination of the study variables (either exposure or disease). Measurement bias may be divided into nondifferential and differential misclassification. Nondifferential information bias can occur if the exposure is not accurately assessed. This type of bias may occur in occupational research if job titles are used as a surrogate for exposure status. Another form of nondifferential measurement bias is unacceptableness bias, in which the exposure may be underreported by patients if it is unacceptable behavior. This is likely to have an impact on all subjects, not just subjects with the disease of interest. Differential misclassification bias includes recall bias, in which the recall of information about exposure is influenced by whether the person has the disease (i.e., cases may have more accurate memory of events leading to disease than controls who have no disease). Interview bias can occur if the circumstances under which different groups of subjects are interviewed are incompatible. These circumstances include time from exposure to interview, setting of the interview, person doing the interview, manner in which questions are asked (prompting), and whether the subject has knowledge of the research hypothesis. Case-control studies are particularly vulnerable to information bias.

Confounding bias is a distortion of the study effect that results from mixing of the exposure associated with the disease with the effects of one or more extraneous variables. An extraneous variable that wholly or partially accounts for the apparent effect of the exposure or that masks an underlying true association is called a confounder. Examples of confounding are (1) an apparent association between an exposure and a disease that may be due to another variable, and (2) an apparent lack of association between exposure and disease that results from failure to control for the effect of some other factor. Brunner and colleagues73 present an example of confounding bias in pediatric rheumatology. These investigators attempted to identify risk factors for damage in childhood-onset SLE. An association was found between damage and disease duration, indicating a possible (and logical) cause-effect relationship between the two. When the data were corrected for the confounder disease activity over time, disease duration disappeared as a predictor of damage.

To assess the possibility of confounding, the standard technique of stratifying the data by the potential confounder may be used. One looks for an association between the exposure (as a possible causal factor) and the disease; then one compares the subjects who have the confounder with subjects who do not to see whether an association exists. Another common method is to use Mantel-Haenszel procedures to calculate an overall relative risk in which the results from each stratum are weighted by the sample size of the stratum.74 Only established risk factors for the disease should be investigated as potential confounders. In brief, these can be dealt with in the design of the study (i.e., by matching) or by stratification or multivariate analysis (see later discussion).

Results

There were no between-survey (SFD vs. VS) participant demographic differences. The completion rate was 45%; data weighting mitigated nonresponse bias. For SFD (n=792), LCA revealed four groups, all sharing concerns regarding respecting patient wishes and minimizing suffering. The four groups were otherwise distinguished by unique concerns that their members highlighted: an older adult remaining severely disabled (34%), family consensus (26%), doubt regarding prognostic accuracy (21%), and long-term care costs (19%). For VS (n=796), LCA revealed five groups, four of five having similar concern profiles to the SFD groups. The largest group (29%) expressed the most prognostic doubt. An additional (16%) prioritized religious concerns.

Limitations of the Included Studies

Many studies relied upon voluntary participation from convenience samples, both of which potentially introduced biases and limit the ability to extrapolate data. Most cross-sectional studies were at high risk of nonresponse bias and were particularly susceptible to self-selection and recall biases. These nonrandomized, uncontrolled designs have potentially resulted in statistically overestimated effects.

Great variation exists in the definition of the anatomic boundaries of the neck.3 Some studies required participants to complete questionnaires about their neck pain. Some used validated questionnaires, such as the Nordic Musculoskeletal Questionnaire, which contained a diagram identifying the anatomic boundaries of the neck. Others included a diagram identifying the anatomic boundaries of the neck or a physical examination by a researcher. These were not validated prior to the study but attempted to standardize the definition of the neck.9,12,20,22,25,26 One study relied upon a modified Nordic Musculoskeletal Questionnaire, but the authors did not include details of the modifications or effect upon reliability or validity. It could not be determined whether the anatomic boundaries of the neck were defined in this study.34 Other studies used nonvalidated questionnaires, validated questionnaires such as the NDI or Maastricht Upper Extremity Questionnaire, or nonvalidated modified versions of these, none of which identified the anatomic boundaries of the neck.10,11,14-16,19,27,31,37 Another study did not identify the method used to determine neck symptoms.21 The lack of adequate case definitions potentially introduced bias that may have detrimentally affected the external validity of these studies.

Many authors attempted to limit confounding biases, but these were mostly age, sex, weight, height, and other anthropometrical variables. The common nonoccupational use of computers, tablets, smartphones, and video games in prolonged or nonoptimal postures has the potential to confound both dependent and independent variables. Likewise, nonoccupational psychosocial variables have the potential to confound occupational psychosocial variables. Studies that did not examine or control for psychosocial variables may have been affected by the confounding effect of psychosocial variables on neck pain or posture. It is impossible to determine whether these confounders affected the results of the included studies if they were not statistically considered or controlled.