Can response variance effectively identify careless respondents to multi-item, unidimensional scales?

Abstract

Because careless responding to questionnaire items can be quite common, and even low rates of careless responding can substantially impact some analyzes, researchers have been advised to remove careless respondents before data analysis. However, identifying these respondents is a non-trivial task. In the context of multi-item, unidimensional scales, it has been suggested that response variance may hold information about careless responding. This notion was tested in the current research with two studies. Data collected from respondents indicates that people responding carelessly display more variance in their responses than people responding with more effort. Given this finding, a procedure which uses measures of response variance and cluster analysis to identify careless respondents was developed. The effectiveness of the procedure with different specifications was tested with simulated data and validated with data from actual respondents. On the basis of the results, we advocate using the procedure with specifications that conservatively identify careless respondents. Such an approach will identify the most extreme careless respondents, while maximizing the retention of careful, honest respondents. We discuss the advantages the developed procedure has over existing procedures for identifying careless respondents. © 2015 Macmillan Publishers Ltd.