Regression-based response probing for assessing the validity of survey questions

We describe a procedure for assessing the validity of survey questions. Response probes are administered which ask respondents to say in their own words what came to mind when answering the question. The verbatim responses are coded to a frame which captures the conceptual content of the responses and are then included as predictors in a regression model, where the question that was used to elicit the verbatim responses is specified as the outcome. Controls are included so that the estimated coefficients of the model can be used to interpret whether and to what extent the different cognitive frames identified in the verbatim data align with responses to the question. The technique can be extended to include split-ballot designs, where variants of the target question are randomised across groups. We illustrate the procedure using two example questions: interpersonal trust and fear of crime.