Putting D-efficiency under the microscope: Impacts of design resolution on aliasing and sample size in factorial surveys
Christian-Albrechts-Universität zu Kiel, Deutschland
In empirical survey studies, finding a sufficient number of people who are willing to respond can be challenging – especially in countries where the response rate is low (Engel & Schmidt 2019). For factorial survey experiments, drawing a vignette-sample (“fraction”) from a vignette-universe can reduce the minimum number of respondents needed for studies.
Vignette-samples can be drawn by means of quota designs, including D-efficient designs. In marketing research, resolution III D-efficient designs (also termed “orthogonal arrays”) are mostly used (Kuhfeld 2003). In the social sciences, on the other hand, one should also consider possible two-way interactions that might have an effect. Theoretically, resolution V designs are thus ideal. Due to reasons of practicability, however, resolution IV designs have usually been applied in empirical social research and are considered to be sufficient when it is clear up front which two-way interactions are likely to have an effect.
Two research questions, oriented towards further examining the resolutions of D-efficient designs, guide this presentation: First, in resolution IV designs, are those two-way interactions that are not orthogonalized truly not aliased with any main effects? Second, how does resolution – III, IV (with 1, 3 or 5 two-way interactions orthogonalized) and V – affect the minimum size of the vignette-sample that is necessary to still achieve an adequate level of D-efficiency?
In order to examine these questions, I use SAS-macros written by Kuhfeld (2003) for computing D-efficient samples, pre-construction assessment and post-construction evaluation. This should contribute to a deeper understanding of (factorial survey) experimental setups based on D-efficiency, which have ramifications for the analysis. Additionally, it suggests taking a second look at Kuhfeld’s (e.g. 2003) assumption that higher resolutions will always necessitate designs with larger vignette-samples (and thus larger sets or more respondents).
Engel, U., & Schmidt, B. O. (2019). Unit- und Item-Nonresponse. In N. Baur & J. Blasius (Eds.), Handbuch Methoden der empirischen Sozialforschung (pp. 385-404). Springer VS, Wiesbaden.
Kuhfeld, W. F. (2003). Marketing research methods in SAS: Experimental design, choice, conjoint, and graphical techniques. Cary, NC: SAS Institute Inc.
The Factorial Survey Again: A replication and extension of Shamon, Dülmer, and Giza’s (2019)
University of Gothenburg, Sweden
The survey experiment has, in recent years, become a staple tool for causal inferences in the social scientist’s toolbox. Particularly in vogue, are the factorial vignette and conjoint survey experiments. As perhaps a reason for the factorial design’s increasing popularity, these designs allow researchers to estimate the causal effects of many components of a treatment simultaneously and allow researchers to test several hypotheses in one single administration. Hence, the factorial design is incredibly cost-effective.
However, for these designs to produce valid results, the research subjects’ information intake is central. The participants need to both process and understand a wide range of attributes presented on their screen. If the participants perceive the survey experiment as too demanding or too complicated, they are likely to turn to satisficing strategies to reduce the cognitive burden of completing the questionnaire.
Although researchers have begun to evaluate the potential issues of factorial conjoint and vignette experiments, we know relatively little of how design-complexity affects the answering behavior of participants. Furthermore, we know particularly little about how the presentation format affects the cognitive load for participants. Gaining insights into how participants react to different presentation formats and how reactions vary among different groups may help researchers to design survey experiments that minimize the risk of satisficing behavior, and thus increases the validity of their results.
In this paper, we build on a recently published study by Shamon, Dülmer, and Giza (2019) and test if the presentation format (conjoint or vignette) impacts the answering behavior of participants and how they perceive different presentation formats. The paper compares the impact of four different presentation formats: a single vignette format, a single conjoint format, a paired vignette format, and a paired conjoint format. Therefore, the present study is both a replication of Shamon and his colleagues (2019) work on single factorial formats as well as extending their work by adding the paired factorial design.
The factorial survey – Different presentation formats, different results?
1Forschungszentrum Jülich, Deutschland; 2Universität zu Köln
The factorial survey is an experimental design in which the researcher constructs varying descriptions of situations or individual persons (vignettes) which will be judged by respondents with regard to a particular aspect. Vignettes can be presented in text format as short stories or in a tabular format where the presented information is reduced to dimensions and its levels. The consequences of different presentation formats of vignettes on answer behavior have sparsely been investigated. We build on a cognitive scheme for the question-answer-process in factorial surveys (proposed by Shamon 2014, Shamon et al. 2019) and test respondents‘ reactions towards differently presented vignettes. The factorial survey was conducted online on the basis of a quota sample in Germany.
Social desirability in factorial survey experiments - A comparison of vignette-based and item-based measurements of social norms in consuming internet pornography
1Hochschule Zittau/Görlitz - University of Applied Sciences, Deutschland; 2KU Eichstätt-Ingolstadt, Deutschland
When examining attitudes and norms on sensitive topics, factual survey experiments are often considered to be advantageous over item-based measurements because respondents’ tendency to give socially desirable answers might be reduced. It is assumed that sensitive dimensions are less obvious for respondents when using indirect measurements with multi-dimensional vignettes as compared to direct item-based measurements. Accordingly, social and personal injunctive norms should be less activated and the social desirability bias should be reduced. However, there are only a few methodological studies that address this assumption.
In our study, we use the sensitive topic of “Pornography consumption on the Internet” and compare vignette-based measurements with item-based measurements on the conditions under which pornography consumption appears to be acceptable. In a randomized within-subjects design, situational conditions of pornography consumption were systematically varied in vignettes and it was measured in terms of a personal and a social injunctive norm to what extent respondents consider consumption acceptable in these situations. The vignette dimensions are consumer’s gender, relationship status, reason for consumption, usage, a handicap and the type of pornographic content. The full factorial (n = 144) was used to keep level balance and orthogonality. In addition, the subjective acceptance of pornography consumption under the same situational conditions was measured using single items. The study was carried out with a homogeneous student sample (n = 105).
The comparison reveals hardly substantial differences between item-based and vignette-based measurements so that similar conclusions can be drawn. This also applies to sensitive dimensions such as gender or a handicap of the consumer. Taking the considerably larger research effort into account, the assumed advantage of factorial survey experiments over item-based measurements in investigations on sensitive topics is critically discussed in the light of the results.
The application of factorial surveys to study recruiters' hiring intentions: Comparing designs based on hypothetical and real vacancies?
1Universität Luxemburg, Luxemburg; 2Universität Bern, Schweiz
Factorial survey experiments have been widely used to study recruiters’ hiring intentions. Typically, respondents rate several fictitious and experimentally varied descriptions of applicants for a hypothetical job. However, rating tasks based on descriptions of hypothetical vacancies may lack “psychological realism” for respondents, as these descriptions may not correspond well to real-life hiring problems. It has been proposed that sampling real-world vacancies and the recruiters responsible for filling them might be one way of overcoming this problem. In this case, only the descriptions of applicants are fictitious, but respondents are asked about a familiar real-life hiring problem.
We test whether a design based on real vacancies triggers different responses compared to a design based on hypothetical vacancies in factorial surveys. Despite the growing number of factorial surveys and the potential implications for the validity of their results, answering this question has been widely neglected so far.
We conducted a factorial survey experiment in Luxembourg, in which respondents evaluated hypothetical applicants referring either to a currently vacant position in their company or to a hypothetical job. Due to data restrictions, we were not able to conduct a split ballot experiment (i.e., randomly assign respondents to one of the two conditions). Therefore, we employed a two-step approach to gather our data. We first collected real vacancies together with the contact information of recruiters published on different websites in Luxembourg. In a second step, we sampled recruiters from publicly available lists (e.g., yellow pages) of companies and businesses within the same occupations. Respondents sampled based on the former approach were asked to rate hypothetical applicants based on the real vacancy in their company. Respondents sampled based on the latter approach were asked to rate applicants based on a hypothetical (but similar) job. In both cases, the hypothetical applicant profiles varied in the levels of three characteristics: migrant background, gender, and unemployment.
Overall, we found little evidence for differences in responses depending on whether the design is based on real or hypothetical vacancies. However, using real vacancies might prove beneficial depending on the research interest.
Value of further training opportunities
Universität Bern, Schweiz
Building on prior research on the valuation of job attributes this contribution focuses on the aspirations and subsequent transitions among Swiss vocational education and training (VET) graduates. The implementation of a choice experiment within a long running panel study on educational and vocational trajectories enables the comparison of experimental stated choice data to survey items on stated importance and information on actual trajectories (revealed preferences). By focusing in particular on the valuation of further training opportunities, I evaluate the consistency of these preference measures.
All aboard? Lines of conflict in times of the pandemic
Professur für Methoden der empirischen Sozialforschung, Institut für Soziologie, TU Dresden, Deutschland
Die Corona-Pandemie sowie die Maßnahmen zu deren Eindämmung haben zu massiven Einschränkungen des öffentlichen Lebens und persönlicher Bürgerrechte geführt. Hierbei unterscheiden sich sowohl die zu erwartenden Konsequenzen des Virus selbst sowie die Lasten der erlassenen Regierungsverfügungen zwischen gesellschaftlichen Gruppen stark. So ist zum einen das Sterberisiko ungleich über Alters- und Risikogruppen verteilt, während die Kosten der Ausgangsbeschränkungen insbesondere Jüngere, Menschen mit minderjährigen Kindern, prekär und selbstständig Beschäftigte treffen. Vor diesem Hintergrund untersuchen wir, inwiefern die Akzeptanz staatlicher Maßnahmen sowie die Punitivität beim Verstoß gegen Corona-Auflagen zwischen verschiedenen gesellschaftlichen Subgruppen variieren. Wir folgen dabei Perspektiven, die neben einem potentiellen Generationenkonflikt bei der Bewertung von Corona-Auflagen den sozialen Status ins Zentrum setzen, inwieweit individuelle Ziele für die Bereitstellung von Kollektivgütern aufgegeben werden bzw. Einschränkungen in Kauf genommen. Darüber hinaus untersuchen wir den Einfluss von weiteren sozialpsychologischen Faktoren, die mit diesen Formen pro-sozialen Verhaltens in der Vergangenheit in Verbindung gebracht wurden, z.B. moralischen Überzeugungen, „Patience“ oder Autoritarismus. Zur Überprüfung unserer Hypothesen nutzen wir Daten aus einem Web-Survey an einer deutschen Universität (wissenschaftliches und nicht-wissenschaftliches Personal, Studierende), die im Frühjahr 2020 während der Phase der Ausgangsbeschränkungen durchgeführt wurde. Die Ergebnisse sollen zeigen, welche Faktoren die Akzeptanz der Maßnahmen beeinflussen und wie diese mit moralischer Kompetenz in unterschiedlich betroffenen Gruppen zusammen hängen.