Using Hypothetical Vacancies in Factorial Surveys to Study Employers' Hiring Decisions – A Valid Approach?

Factorial survey experiments are increasingly employed by scholars interested in understanding the general mechanisms underlying employers' hiring decisions in relation to specific applicant characteristics. Usually, a sample of human resource professionals is asked to rate the hiring chances of hypothetical applicants for a hypothetical job. However, using hypothetical job descriptions for the evaluation of applicants in factorial surveys may reduce the internal and external validity of the results. For example, employers might apply different evaluation standards when assessing the quality of applicant profiles for a hypothetical job (put less/more weight on certain characteristics) because it is difficult to put themselves in the actual hiring situation – affecting the internal validity.
In this paper, we contextualize prior factorial survey experiments by examining whether there is a difference in employers' hiring intentions when confronted with real versus hypothetical hiring problems. Despite the growing number of factorial surveys and the potential implications for the validity of these data, this question has been widely neglected so far. We employ a factorial survey experiment among recruiters in different occupational sectors in Luxembourg. Recruiters evaluate the hiring chances of several profiles of hypothetical applicants with varying characteristics either referring to a real vacancy in their company or to a hypothetical (but similar) job type.
Preliminary findings suggest no differences in employers hiring decisions based on the type of evaluation used in the factorial survey. The results partly contradict previous findings from pretest data which showed significant differences between the average hiring chances in the two groups. By examining the internal validity of presenting hypothetical vacancies, this study contributes to methodological research on factorial surveys as well as to the literature studying employers' hiring decisions.