Today's article is about different types of errors that can occur in surveys and thus have a negative impact on the quality and validity of the survey results. Ideally, sources of error should be reduced as much as possible in order to reflect "reality" as best as possible in the survey results.
A major challenge is that most types of error in surveys, unlike sampling error, remain below the “visibility limit.” The visibility limit describes whether a type of error can be calculated by statistical methods. Thus, for errors below the visibility limit, it is difficult to identify or even measure a potential bias in the results.
But what types of errors are there, and how can they be prevented or reduced? These are the questions we are trying to answer today.
Basically, there are three main error categories.
- Error due to selection of the respondent
- Error due to incorrect information
- Error in the execution of the survey
Each of these categories includes different types of errors and were first described by Herbert F. Weisberg.
Fig. 1 Overview of the different types of errors (Source: Weisberg 2005)
Error due to selection of the respondent
Sampling error
This is the "random error", i.e. the error that arises because only a (random) sample is surveyed instead of the entire population.
Since this error can be calculated by statistical methods, as long as the selection of the sample is random, it is usually represented by specifying a margin of error (e.g. +/- 4%). The sampling error is, as, the only type of error above the visibility limit and thus represents only the "tip of the iceberg" among the types of error. This error can be reduced by increasing the sample size.
Incorrect sampling basis
An incorrect sampling basis consists in the fact that directories of persons, households, companies, etc., which are used as a basis for the survey, do not adequately cover the population. This means that the population and the sample do not match sufficiently, which can lead to significant bias in the results. In general, a sampling population is said to be incorrect if certain parts of the population have either too low a probability (underrepresentation) or too high a probability (overrepresentation) of being included in the sample (see Fig. 2).
This type of error, belongs to the so-called systematic errors, which means that they cannot be calculated and also cannot be reduced if the sample is increased. Thus, this error is below the visibility limit. To minimize this error, it is important to ensure that the directories used sufficiently cover the population.
Fig.2 Coverage of the population by an incorrect sampling base (Source: Groves et al. 2009)
Low response rate
This type of error indicates that respondents cannot be reached by telephone, for example, or refuse to participate in the survey. In the case of high interview dropouts, systematic biases can occur due to low sample exhaustion, which is illustrated in Fig. 3. This type of error is also below the visibility limit. Interview nonresponse can be reduced by, for example, providing financial incentives to respondents. Thus, respondents have an active incentive to participate in the interview.
Fig. 3 Exploitation of a sample (Source: Parasuraman 1986) Sample basis = All elements that could potentially come into the sample, Drawn sample = elements that are part of the sample and have been contacted, Realised sample = Elements that actually participate in the survey
Error due to incorrect information
Missing information in interviews (item non-response)
This type of error mainly involves respondents giving a "don't know" or " no answer" response, especially in closed-ended questions, which allows respondents to evade giving their answer. The absence of information from the respondent can lead to the result not taking into account important information and thus distorting the significance of the result. Missing information in interviews cannot be calculated statistically and is therefore also below the visibility limit. Possible solutions for reducing this error are, for example, "follow-up" by the interviewer in personal or telephone interviews, but also financial incentives for complete information, especially in online surveys, can be useful.
Measurement error among respondents (response bias)
This error occurs when respondents answer questions, for example, superficially or inattentively, dishonestly by giving socially desirable answers, incorrectly due to memory deficiencies, or incorrectly due to other influences. Question wording can also significantly influence this error.
These errors belong to the systematic errors and are normally below the visibility limit. However, with the help of ReDem® it is possible to make these biases visible and measurable. In order to reduce this error, special attention should be paid to neutral question wording when creating the questionnaire. In addition, by cleaning with ReDem®, the error part, which depends on the respondents, can be reduced (see Figure 4).
Fig. 4 ReDem® within the error types
We at ReDem® have specialized in the reduction of as this error often cannot be influenced or prevented by market research institutes. With ReDem® it is therefore possible to lift this error above the " visibility limit " and to reduce it to a large extent.
Error in the execution of the survey
Error during data preparation
These are possible sources of error that may be present in the data set (e.g. incorrect coding of the questionnaire). These rather technical errors can strongly distort the result and should therefore always be checked.
As long as these errors are not eliminated, they are below the visibility limit.
There are three ways in which these errors can be identified and eliminated:
- Check if the code plan matches the existing values.
- Ensure that there is logical consistency of values, such as age and educational attainment.
- Determine if there are outliers that deviate extremely from the other range of values.
Errors due to the way of communication
In this context, types of errors are considered that can occur due to different forms of communication (method) in surveys such as online, telephone, written or face-to-face surveys. In addition, it is relevant whether a survey is conducted with or without an interviewer.
For example, conducting an online study whose target group is older people could lead to significant bias, as older people tend to be more difficult to reach online. Here, a different method, such as a telephone survey, would be more suitable.
This error can neither be measured nor calculated and is therefore below the limit of visibility. Each method has its own advantages and disadvantages. Therefore, it is important to make sure that the question wording and questionnaire structure fit the selected survey method. In addition, it should be checked whether the selected communication method ensures sufficient coverage of the sample.
Errors due to characteristics of the institute
Because errors caused by the specifics of an institute are a systematic type of error, they are often difficult to detect and are below the visibility limit. These errors may arise due to internal practices, habits, and processes that are prevalent in a particular institute and thus may affect the accuracy and objectivity of the research results.
In order to minimize this type of error as much as possible, it is important that institutes regularly review their internal procedures and practices and adapt them if necessary. This can be achieved, for example, through regular training and continuing education for employees, the introduction of quality management systems, or the implementation of guidelines for good scientific practice.
