SMARTRIQS: A Simple Method Allowing Real-Time Respondent Interaction in Qualtrics Surveys

SMARTRIQS: A Simple Method Allowing Real-Time Respondent Interaction in Qualtrics Surveys

Journal of Behavioral and Experimental Finance 22 (2019) 161–169 Contents lists available at ScienceDirect Journal of Behavioral and Experimental Fi...

1MB Sizes 0 Downloads 547 Views

Journal of Behavioral and Experimental Finance 22 (2019) 161–169

Contents lists available at ScienceDirect

Journal of Behavioral and Experimental Finance journal homepage:

SMARTRIQS: A Simple Method Allowing Real-Time Respondent Interaction in Qualtrics Surveys Andras Molnar Department of Social and Decision Sciences, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, PA 15213, USA

highlights • • • • •

Allows real-time interaction, including chat, between Qualtrics survey respondents. Customizable group size, roles, conditions, number of stages, randomization, bots. Easy to use: does not require any programming skills or installing any software. Integrated: supports all Qualtrics question types and saves all data in Qualtrics. Free and open-source, features debugging, demo experiments, and survey templates.



Article history: Received 29 November 2018 Received in revised form 6 March 2019 Accepted 11 March 2019 Available online 13 March 2019 JEL classification: C83 C88 C92 Keywords: Online experiment Qualtrics Real-time interaction Research method Software Survey design

a b s t r a c t SMARTRIQS is an open-source solution that allows researchers to design and implement interactive online experiments using the popular survey platform Qualtrics. Unlike other existing platforms that allow real-time interaction, SMARTRIQS does not require any programming skills or installing any software. SMARTRIQS is fully integrated into Qualtrics: researchers create and edit their interactive experiments in the standard survey editor after importing the generic SMARTRIQS survey template. Moreover, all data resulting from real-time interaction are saved in Qualtrics. The potential applications of SMARTRIQS in experimental economics and finance are very versatile, including but not limited to experiments featuring cooperation, coordination, competition, allocation, investment, auction, and other market interactions. Researchers working in related fields (organizational research, consumer research, decision science, social psychology, political science, etc.) might also benefit from using SMARTRIQS, as it offers a convenient and simple method of studying real-time social interaction, including communication, collaborative work, collective decision-making, and voting behavior. © 2019 Elsevier B.V. All rights reserved.

1. Introduction There is a growing trend towards online experimentation in the social sciences, especially with the advent of online crowdsourcing services such as Amazon Mechanical Turk (MTurk) and Prolific1 (Bohannon, 2016). Researchers are increasingly turning to online experiments instead of running conventional lab studies, and not only because of the lower costs and convenience of online data collection. Firstly, crowd-sourced services offer access to significantly larger populations of participants, allowing researchers to have larger samples in their experiments, which is essential for high-powered hypothesis testing. Small samples and low statistical power are very common in economics: Ioannidis et al. (2017) found that the vast majority of economic E-mail address: [email protected]

1 Before 2015: ‘Prolific Academic’, 2214-6350/© 2019 Elsevier B.V. All rights reserved.

studies, up to 90% in some areas, are under-powered, drastically exaggerating reported effects. Secondly, these subject pools are more diverse and representative than the predominantly undergraduate student pools used in lab studies (Buhrmester et al., 2011; Paolacci et al., 2010; Peer et al., 2017). Having more diverse samples increases the external validity of one’s results and allows researchers to study the behavior of groups who are less represented among college students. Finally, being able to recruit from the same pool of participants using the same recruitment methods allows researchers to run replications more easily, which contributes to the reliability and transparency of social sciences.2

2 The reproducibility of scientific results has become a central issue in social sciences (Collaboration*, 2015), although, economics seems to be affected by the ‘replication crisis’ to a lesser extent, see e.g. Berry et al. (2017).


A. Molnar / Journal of Behavioral and Experimental Finance 22 (2019) 161–169

However, there are some limitations to online experimentation: firstly, online studies can suffer from significantly higher attrition rates than those found in lab experiments (Zhou and Fishbach, 2016). High attrition rates do not only lead to wasted resources (funds and participants) but can also threaten the internal validity of experiments, especially when the attrition is asymmetric across experimental treatments. Therefore, researchers wishing to run online experiments have to put extra effort into keeping attrition rates low. Secondly, while conventional lab subject pools have a relatively high turnover rate, and researchers can limit participation to naïve subjects (i.e., who have not been exposed to a conceptually identical experiment), nonnaïveté is relatively common in online studies, especially when using crowdsourced services. This issue is discussed in detail by Chandler et al. (2014) who also describe methods that can minimize nonnaïveté among participants. Finally, while it is relatively easy to implement real-time interaction between participants in the lab (e.g., via z-Tree, Fischbacher, 2007), it is more challenging and laborious to do so online. This limited ability to have real-time interaction prevents a lot of experimental economists from conducting online experiments, depriving them from the potential benefits of online experimentation. To address this limitation, researchers have created several platforms that allow real-time interaction between participants, for example: oTree (Chen et al., 2016), SoPHIE (Hendriks, 2012), LIONESS Lab (Giamattei et al., 2019), or TurkServer (Mao et al., 2012). However, designing custom (i.e., not template) experiments using these platforms requires at least minimal programming skills. Since each platform has a unique architecture and relies on different programming languages, researchers must make a decision whether to invest in learning such skills, and if so, which platform to use. This poses a big dilemma for junior researchers and graduate students in particular, who must learn these technical skills themselves, since usually they have very limited ability to out-source programming to hired professionals or research assistants. Unless a researcher is confident that he or she will run several interactive experiments using the same platform, such an investment is clearly not the best use of their valuable time. At the same time, professional survey platforms such as Qualtrics (2019) or SurveyMonkey (2019) have become very popular among researchers who design non-interactive online experiments. Survey software have several advantages over platforms specifically designed for experiments: they require no programming skills, are completely web-based, and offer a more intuitive and streamlined interface. These features make it easier to create, edit, and manage studies using survey software than using more conventional experimental platforms. Survey platforms also comply with the newest industry standards, with a strong emphasis on user experience and visual design. However, survey software have a major limitation: they do not allow real-time interaction between respondents. As pointed out by Arechar et al. (2018): ‘‘Studies of social behavior often use survey software [...] and emulate interactions through post hoc matching. Although this approach can be powerful, it does not permit the study of repeated, ‘hot’ interactions where live feedback between participants is essential’’. (p. 100) This limitation immediately disqualifies survey software for running interactive economic experiments (or any other interactive study without deception and post-hoc matching). Because of this, experimental economists have to either rely on experimental platforms (which require programming skills) or run their experiments in the lab, forgoing the potential benefits of online experimentation. And while it seems like a straightforward step

to develop solutions that allow real-time interaction between respondents in popular survey platforms, to my knowledge, there is no existing solution that allows this. In this paper I propose a simple method that allows real-time respondent interaction in Qualtrics surveys: SMARTRIQS (https: // This solution offers researchers the ability to design interactive experiments entirely in Qualtrics. The most important characteristic that distinguishes SMARTRIQS from other experimental platforms is its simplicity: it allows researchers to run a wide range of interactive studies online, that can be easily customized, without having to learn any programming language or to install any software. 2. Overview of SMARTRIQS 2.1. Objectives SMARTRIQS was developed with the following five objectives in mind: 1. It should allow real-time interaction and real-time communication between Qualtrics survey respondents. 2. Using it should not require any programming skills or installing any software. 3. It should be fully integrated into Qualtrics, including any data that results from respondent interaction (e.g., responses, decisions, chat logs, etc.). 4. It should offer a lot of flexibility in experimental design, including but not limited to group size, roles, conditions, number of stages, etc. 5. The implementation should be as simple and as intuitive as possible. In principle, achieving the first goal is relatively easy: find a way to establish communication between the survey respondents (the ‘clients’) and a server that receives, processes, and sends back data to clients. This core feature is present in every experimental platform (e.g., oTree, SoPHIE, LIONESS Lab, etc.). The rest of the objectives, however, are more challenging, and require an approach that is specific to survey software. To achieve these objectives, SMARTRIQS relies on three basic features of Qualtrics: the built-in ‘Survey Flow’ editor, the ability to insert custom JavaScript in surveys, and the option to import, export, and reference surveys. 2.2. Architecture SMARTRIQS has two components: a client-side and a serverside (see Fig. 1). Clients are individual Qualtrics respondents, whose data and responses are transmitted to, or retrieved from, the server. Communication between clients and the server is established via JavaScript and asynchronous HTTP (AJAX) requests. The server then executes PHP scripts to access, process, and save data. The main advantage of SMARTRIQS is that researchers do not have to edit any of these scripts (neither JavaScript nor PHP) when designing new experiments: all editing is done in the Qualtrics survey editor. Furthermore, there is a default SMARTRIQS server that researchers can use for running experiments, so they do not have to do anything on the server-side. To use this default server, researchers must read and accept the Data Submission Policy Agreement (see Appendix). The client-side is implemented entirely in Qualtrics and utilizes two main components: embedded data and survey blocks (these terms should be familiar to Qualtrics users). The JavaScript responsible for sending and retrieving data is included in generic survey blocks. These generic blocks were created specifically for SMARTRIQS and can be easily imported to the researcher’s

A. Molnar / Journal of Behavioral and Experimental Finance 22 (2019) 161–169


Fig. 1. The architecture of SMARTRIQS. Participants’ data and responses are transmitted to the SMARTRIQS server via JavaScript. The PHP scripts running on the server save and process data, and send back results to clients via JavaScript. The JavaScript is embedded in generic survey blocks that can be easily imported to a Qualtrics account. Researchers do not have to modify any of the scripts (neither JavaScript nor PHP) when designing new studies or editing existing ones: all editing is done in the Qualtrics survey editor.

Qualtrics account. There are five generic blocks, the first three of which are required in all SMARTRIQS experiments: MATCH, SEND, GET, CHAT, and COMPLETE. These names are self-explanatory: the MATCH block is responsible for matching participants; the SEND block submits data to the server; the GET block retrieves data from the server; the CHAT block allows real-time communication between participants; and the COMPLETE block concludes the experiment. The advantage of these generic blocks is that researchers can use them in any number of experiments, without ever having to modify them. Furthermore, these blocks are referenced, which means that researchers cannot edit them.3 Instead of directly editing these blocks – and the scripts therein – researchers customize their experiments by modifying embedded data and adding or rearranging blocks in the Survey Flow editor. The embedded data serve as input for the generic survey blocks (see Fig. 2). In addition to these generic blocks, researchers can add any number and any type of other blocks to their surveys, which offers almost unlimited flexibility in study design. Moreover, researchers can add the SEND and GET blocks multiple times to the Survey Flow, thereby creating repeated interaction between participants. A more detailed explanation of the architecture and customization process is available at 2.3. Features of SMARTRIQS 2.3.1. Easy to use Users of SMARTRIQS do not have to have any background in programming. Since editing is done entirely in the powerful and intuitive Qualtrics survey editor, creating and modifying interactive experiments is as simple as designing standard surveys: the only additional steps are defining embedded data, a common task in Qualtrics surveys, and importing the generic survey blocks. Researchers who have a minimal experience with Qualtrics will find these steps extremely easy. Even researchers who have never used Qualtrics before, can learn the basics required for using SMARTRIQS in less than 30 min.4

3 This prevents researchers from accidentally removing or replacing essential scripts. Researchers wishing to edit these scripts (for example, in order to add new features to SMARTRIQS), can download the source code and create survey blocks manually. 4 For example, Dare McNamara has two excellent video tutorials: Beginner Qualtrics Training (10 minutes) and Advanced Qualtrics Training (17 minutes).

2.3.2. Integrated SMARTRIQS is fully integrated into Qualtrics: all data, including chat logs, are directly saved in Qualtrics in a user-friendly format, so researchers do not have to use external software or services to parse or merge data. Furthermore, SMARTRIQS can handle any response format in Qualtrics: numeric responses (e.g., input fields, sliders, matrices), forced choice options, open-ended text, scores, etc. 2.3.3. Lightweight SMARTRIQS is extremely lightweight: the scripts (including all client-side and server-side scripts), the survey templates, and the generic survey blocks, combined, take less than 0.5 MB of space, less than the .pdf version of this paper. More importantly, SMARTRIQS does not require researchers to install any software and does not rely on any external JavaScript libraries.5 Even if SMARTRIQS is deployed on a custom server instead of the default one (see Section 2.3.4), there is no installation required: researchers simply have to copy the PHP files to that server and add a new embedded data to the Survey Flow, that specifies the address of the custom server. 2.3.4. Flexible Researchers can use SMARTRIQS in two different ways: (1) they can use the default SMARTRIQS server (recommended) or (2) set up their own SMARTRIQS server. The default server is a virtual host, hosted at Amazon Web Services. Any researcher can use the default server for free after creating a ‘researcher ID’ at https:// The main advantages of using the default server are that researchers do not have to do anything server-side and can focus on the client-side (designing surveys), and that the back-end scripts on the default server are always up-to-date, meaning that bug fixes and new features are added automatically. Researchers can also choose to deploy SMARTRIQS on their own servers. Setting up a server usually entails some costs and involves some programming (there is a detailed guide available at The only real advantage of having a custom server is that it allows the collection and storage of sensitive data (e.g., personal identifiers), which is prohibited when using the default SMARTRIQS server (see the Data Submission Policy Agreement in the Appendix). 5 Using external libraries and advanced functions might lead to compatibility issues, see for example the case of QRTEngine (Barnhoorn et al., 2015), a solution that allowed measuring reaction times in Qualtrics surveys, but was discontinued in 2016, following an update to the Qualtrics Survey Engine that rendered the QRTEngine incompatible with Qualtrics. Since SMARTRIQS relies on ‘native’ JavaScript and AJAX (HTTP) requests, such incompatibility issues are very unlikely.


A. Molnar / Journal of Behavioral and Experimental Finance 22 (2019) 161–169

Fig. 2. The two main components of SMARTRIQS in the Qualtrics Survey Flow editor: embedded data (on top) and survey blocks (on bottom). Left: matching participants in a two-player dictator game. Before inserting the MATCH block the researcher defines a few basic parameters as embedded data (e.g., study name, players per group, roles, etc.). Middle: the dictator’s transfer is sent to the SMARTRIQS server. First, the researcher defines which response to send (‘sendData’) and the current stage (‘sendStage’), then inserts the SEND block. Right: the receiver retrieves the dictator’s transfer. First, the researcher defines whose response to retrieve (‘getData’) from which stage (‘getStage’) and the name of the variable that the retrieved value will be saved in Qualtrics (‘saveData’), then inserts the GET block.

2.3.5. Free and open-source SMARTRIQS is free6 and is licensed under an extended version of the MIT license. Researchers using SMARTRIQS are asked to cite this paper in their academic or other publications. Furthermore, researchers agree that they do not submit any sensitive information (e.g., personal identifiers, medical or criminal records, etc.) when using the default SMARTRIQS server. The source code for the client-side JavaScript and the server-side PHP scripts can be downloaded for free at smartriqs. To foster collaborative research, and if there is sufficient demand, a public library of user-created survey templates will be added to this repository. 2.3.6. Debugging and error logs SMARTRIQS scripts have built-in validation checks for input parameters and variables. Whenever an input variable (e.g., study name, group size) is missing or invalid, an error message detailing the nature of the error is displayed and participation is terminated. To debug and avoid such errors, researchers are strongly recommended to always test their experiments before data collection, by either opening the survey in the Qualtrics Preview mode or opening the study URL in multiple windows or on multiple devices. Participants also receive warning messages if they or other members of their group ‘time out’ (take too much time in a stage or quit the experiment). These warnings are logged and saved directly in Qualtrics as embedded data, allowing researchers to identify groups that had issues with timeout or dropout. 2.4. Features of experiments designed with SMARTRIQS 2.4.1. Group interaction SMARTRIQS allows researches to match participants in groups of 2–8 people. There is no limitation on the number of groups in a study and any number of groups can participate in an experiment at the same time. Matching is fixed, re-matching is not supported in SMARTRIQS (see Section 3.3). Researchers can easily customize who interacts with whom. Each participant can interact with any number of participants within their group (1–7), and each group member can interact with a different number of other members (e.g., one participant – the ‘principal’ – can interact with everyone in the group, but other group members – the ‘agents’ – can only interact with the principal). 6 There is no additional fee on top of the Qualtrics subscription fee.

2.4.2. One-shot or repeated responses, simultaneous or turn-taking interaction Each participant can make several decisions, either simultaneously or sequentially (taking turns). The number of decisions can differ across participants within a group (e.g., one participant makes a decision while the other person is passively observing). The researcher has full control over when and what to transmit between participants (e.g., it is possible that one participant receives the same decision sooner than others). Although there is no hard limit on the number of rounds that can be implemented in SMARTRIQS (or decisions per participant), it is recommended to keep these numbers relatively low to avoid high attrition rates that can threaten internal validity.7 2.4.3. Almost instantaneous matching The matching algorithm of SMARTRIQS was designed with the aim of minimizing waiting times. When treatment and role randomization takes place after matching, the median waiting time is very short, typically under 30 s (see Section 4). In addition, if recruitment is done properly (see best-practices), participants are matched with others almost instantaneously. For example, when using MTurk, researchers are strongly recommended to recruit participants in several smaller batches (e.g., recruit 30–40 workers every 10–15 min), instead of launching a single large batch. This not only guarantees a steady flow of participants but also makes sure that the study link is always displayed to workers on the first pages of available HITs. Researchers are also suggested to offer higher-than-average hourly wages for participation in interactive studies (as of 2019, at least $10–12/h on MTurk), to ensure that participants have large enough incentives to complete the study, keeping dropout rates as low as possible. 2.4.4. Automatic, random assignment to treatments and roles Participants can be randomly assigned to treatments and roles. Randomization is flexible: researchers can either assign treatments and roles before participants are matched in groups, by 7 For example, in an online experiment conducted by Arechar et al. (2018) that featured 20 rounds of public goods game and lasted about 28 min, only about 50% of groups had all group members finish the study. Although the high attrition rate was not a critical issue in this particular experiment (participants could keep playing in reduced groups, i.e., in 3-player or 2-player groups), such high rates of attrition pose a great challenge to experiments with fixed group sizes.

A. Molnar / Journal of Behavioral and Experimental Finance 22 (2019) 161–169

using the ‘Randomizer’ element in the Qualtrics Survey Flow, or let SMARTRIQS assign treatments and roles during the matching process (recommended). While matching is significantly faster when using the latter method,8 the former method is required if participants have to read treatment- or role-specific instructions or do treatment- or role-specific tasks before they are being matched with others (see 2.4.5. Bots, default responses, and dropout management SMARTRIQS also offers a lot of flexibility in handling inactive participants and dropouts. Researchers can set up ‘maximum waiting time’ limits during matching and later stages, and can select what happens if there are not enough participants available after the waiting time is over, or if participants fail to submit their responses before this limit expires. Researchers can choose from three options: 1. Bots and default responses. This method uses bots and default responses to fill missing roles and missing responses, ensuring that every participant can complete the study, even if there are not enough other participants available within the maximum waiting time. Researchers can specify default responses for bots in the Survey Flow—these can be pre-determined or stochastic, or depend on previous decisions (see To avoid any deception, participants are always notified whether they have been matched with other participants or bots (or some combination thereof), and they also receive notifications whenever a response is a default response (as opposed to a ‘real’ response). 2. Default responses only. The second option does not allow participants to be matched with bots but uses default responses whenever someone becomes inactive or drops out in a later stage. If there are not enough other participants available during matching, the survey is terminated. However, if a participant becomes inactive in a later stage, the study continues and the response of the timed out participant is replaced by the default response. In this case, both the timed out participant and the other group members are notified about the timeout and that the default response has been applied. If the timed out participant becomes active again, they can still submit their responses in subsequent stages and complete the study. SMARTRIQS automatically creates a log file (saved as Qualtrics embedded data) detailing which participant has timed out in which stage, allowing researchers to decide later whether to keep or exclude their data. 3. No bots or default responses. The third option is the most conservative as it does not allow for using either bots or default responses. If there are not enough participants available during matching, or if any of the participants time out, the survey is immediately terminated for everyone in the group. Termination is logged in Qualtrics, and the researcher can specify a custom survey termination message with a partial survey completion code, should the participants receive partial compensation for their participation.


2.4.6. Chat and messaging SMARTRIQS also allows real-time communication between participants. Any number of participants in a group can participate in chat. The chat feature is highly customizable: researchers can set up multiple chat stages within one study, allow chat between certain participants only (e.g., based on role or decision), and add time limit to chat. Furthermore, researchers can customize the visual layout of the chat window (e.g., width, height, time format, instructions, see Chat logs are saved in Qualtrics in a user-friendly format. Fig. 3 shows two examples of the chat feature with different settings. 2.4.7. Mathematical operations The built-in mathematical operations in Qualtrics are rather limited and error-prone, however, most economic experiments require some basic operations. SMARTRIQS features a set of mathematical operations that can be easily inserted in the Survey Flow. These operations are the following: minimum, maximum, second maximum (useful for auctions), average, sum, and rank. The rank operation calculates the rank of participants based on either (1) their score (e.g., competition), or (2) the distance of their response from a target value (i.e., accuracy). Researchers can also choose whether to allow ties in ranks or to break ties randomly. 2.4.8. Progress monitor Researchers can also monitor data collection in real-time using the SMARTRIQS progress monitor ( ogress-monitor) The progress monitor shows essential information about groups (group number, condition, group status and progress) and participants (role, ID, last activity, responses) in a simple and concise way. The progress monitor also allows researchers to download data from the server (this is mostly for testing purposes—as all data is saved directly in Qualtrics as well). 3. Applications and limitations 3.1. Online and lab experiments While large-scale online experimentation is an obvious application of SMARTRIQS, participation is not limited to online platforms such as MTurk or Prolific. Interactive studies designed with SMARTRIQS can be also implemented in conventional experimental labs, classrooms, or in the field. Since SMARTRIQS is fully integrated into Qualtrics, anyone can participate in SMARTRIQS experiments who have a device with Internet access and a browser that supports HTML5 and JavaScript. All popular browsers (Google Chrome, Mozilla Firefox, Microsoft Edge, Safari) and OS (Windows 7–10, iOS, Android) are compatible with SMARTRIQS. In addition to desktop computers, participants can also take the interactive surveys on their portable devices (phones, notebooks, tablets). Furthermore, SMARTRIQS does not require neither the participants nor the experimenter to install or execute any software on the client devices, so running conventional lab experiments with SMARTRIQS is extremely easy: all the researcher has to do is to distribute the survey link. Finally, the researcher can monitor responses in real-time using either the SMARTRIQS progress monitor or the Qualtrics analytic tools, which makes managing payments very convenient. 3.2. Sample applications and demo experiments

8 With this method, a new group is created and participants are matched immediately whenever there are enough participants. By contrast, when treatments and roles are determined in Qualtrics before matching, participants have to wait until there is at least one participant in each role, which can cause longer waiting times, especially when the group size is large or when there are multiple treatments.

Most economic experiments that feature a choice of action or allocation between two or more people (e.g., dictator game, ultimatum game, trust game, prisoner’s dilemma, battle of the sexes) are very easy to implement in SMARTRIQS. The built-in mathematical operations also allow for designing more complex


A. Molnar / Journal of Behavioral and Experimental Finance 22 (2019) 161–169

Fig. 3. Sample chat windows. Left: two-player chat without any time limit and 24-h time format (including seconds). Right: four-player chat with time limit and 12-h time format (excluding seconds).

experiments, such as the public goods game or the p-beauty contest game. SMARTRIQS is also an excellent choice for market experiments (buyer/seller interactions, auctions), as well as for studying organizational behavior (group dynamics, real effort experiments, competitive and collaborative tasks). All of the above experiments can be supplemented with realtime chat, which allows researchers to study various aspects of communication and its effect on decision-making. Experiments with real-time communication might investigate persuasion, negotiation, manipulation, private and public signaling, information disclosure, etc. Researchers studying consumer behavior or political science can also use the group chat function to conduct cost-effective online focus group studies. Readers who wish to learn more about the potential applications of SMARTRIQS can find additional content (demo experiments and survey templates) at 3.3. Limitations of SMARTRIQS 3.3.1. Maximum 8 participants per group SMARTRIQS does not support experiments with a group size larger than 8 people per group. While it is technically possible to edit the server-side PHP scripts to allow larger groups, this is not recommended. The issue of high attrition rates becomes exponentially worse for large groups. For example, if the individual attrition rate in an online experiment is only 5% (which can be considered very low for an online study), only about 60% of groups of 10 participants and less than 50% of groups of 15 people are expected to finish the experiment without anyone dropping out. The vast majority of economic experiments, however, are not affected by this limitation of 8 people per group at all, since they typically feature smaller groups. 3.3.2. No re-matching Currently, SMARTRIQS does not support any true re-matching: the composition of groups remain the same after matching. The reason for omitting re-matching is two-fold. Firstly, since the matching algorithm of SMARTRIQS was designed to minimize waiting times, there is no ‘waiting lobby’, unlike in other experimental platforms. However, without a lobby or some other script that forces all participants to wait, truly random re-matching cannot be guaranteed (as people who would be matched in one round and complete the round at the same time would be more likely to be matched again). Secondly, re-matching could potentially lead to higher attrition rates: while a single dropout affects only one group when groups are fixed, the same dropout can affect multiple groups and trigger a cascade of dropouts when groups are re-matched. However, it is possible to implement a limited version of rematching: participants can be recruited in larger groups, then

they can interact with each other in smaller groups in every round. For example, participants can be recruited in groups of 8 and then interact in pairs. A quasi-random reassignment can be achieved by generating random values for each participant in each round, use the SMARTRIQS rank operation to obtain the rank of each participant, and then apply a pre-defined matching rule, e.g. match rank 1 with rank 2, rank 3 with rank 4, etc. 3.3.3. No video or audio chat SMARTRIQS does not support the direct transmission of multimedia files. This implies that SMARTRIQS does not feature any video or audio chat. Transmitting, storing, and processing media files can be computationally demanding, especially when implemented on a large scale, which could cause delays in client–server communication. Furthermore, most Institutional Review Boards consider photos, videos, and audio recordings of participants as ‘personal identifiers’ and have much stricter regulations regarding the collection and storage of such data. If researchers want participants to be able to ‘transmit’ nonpersonal video or audio to each other, the recommended method is to add all of those stimuli directly to the Qualtrics survey, and use ‘Display Logic’, conditioned on participant responses (e.g., a numeric variable). For example, senders can be shown several videos, then choose a video to send to their partner. Their choice is saved as a numeric variable—each number corresponds to a different video. Then, this variable is sent to the SMARTRIQS server (instead of the video itself), and the same variable is retrieved by the receiver. Finally, the corresponding video is displayed to the receiver, using conditional Display Logic. The implementation of this example can be found at (‘‘Multimedia sender’’). 4. Empirical results and participant experience SMARTRIQS has been already tested in multiple experiments with over 2400 participants at the Center for Behavioral and Decision Research (Carnegie Mellon University, CMU), the Decision and Economic Sciences Laboratory (Rutgers University), the Wharton School of Business (University of Pennsylvania), and the David Eccles School of Business (University of Utah). Table 1 summarizes the most relevant details of these studies. All participants were recruited via Amazon Mechanical Turk, using fairly strict eligibility criteria (U.S. participants only, approval rating of at least 98%, and at least 1000 completed HITs). Participants were matched in pairs, except for the study conducted at Rutgers University, in which they were assigned to groups of four. Participants in the four studies conducted at CMU participated in one of the following experiments: word association task (CMU-1), math test and allocation (CMU-2), chat about political views (CMU-3), allocation task with chat and punishment (CMU-4). The data of these four studies is available at this OSF data repository.

A. Molnar / Journal of Behavioral and Experimental Finance 22 (2019) 161–169


Table 1 Studies designed and conducted with SMARTRIQS. Study


Participants Started

CMU-1 CMU-2 CMU-3 CMU-4 UPenn/Utah Rutgers

No No Yes Yes Yes No


Attrition rate











134 192 184 370 889 680

128 178 174 332 870 604

128 176 167 327 779 589

4.5% 8.3% 9.2% 11.6% 12.4% 13.4%

0.0% 1.1% 4.0% 1.5% 10.5% 2.5%

4.6 12.9 11.1 5.5 9.0 8.7

10.0 3.0 18.0 10.8 14.0 21.2

23.0 10.8 3.0 3.4 n.a. 3.0










Attrition rate among those participants who had been matched with others. Median duration of the complete study (in minutes). c Participants’ median waiting time until being matched with others (in seconds). d Participants’ median waiting time for others’ decisions, per round (in seconds). b

4.1. Attrition rates and waiting times Since high attrition rates pose a major challenge to online experimentation, first I look at the attrition rates across these six studies. Total attrition rates (that is, the proportion of participants who completed the study out of participants who started the study) were between 4.5% and 13.4% (M = 11.6%). However, it is worth to note that the total attrition includes everyone who dropped out, even those who were not assigned to any experimental treatment and who were not matched with others. Therefore, a more relevant measure is the ‘post-match’ attrition rate, that is, the proportion of participants who completed the study out of participants who had been matched with others. These post-match attrition rates were considerably lower in most studies, between 0% and 10.5% (M = 5.2%). This indicates that once matched, participants were less likely to drop out, and that the vast majority of groups managed to complete the studies without any issue. However, since each of the studies took less than 15 min to complete on average, it is possible that longer studies would entail significantly higher attrition rates. Secondly, it is also important to understand how participants themselves experienced these interactive studies. One unavoidable feature of online experimentation is the waiting periods: since participants are not recruited for a particular time slot and do not start the study at the exact same time (as in lab studies), participants have to wait until they are matched with others, or until others make decisions. Long waiting times can not only increase attrition but also make participants feel frustrated which can affect their decisions, thus hurt the internal validity of the study. In the six studies conducted with SMARTRIQS, the median match wait times ranged between 3 s9 and 21.2 s (M = 14.7 s), and the vast majority of participants, 79% were matched with others within one minute. Similarly, participants did not have to wait a lot for others. Median wait times per round ranged between 3 s and 23 s (M = 3.8 s), and 88% of participants had to wait less than one minute per round for their partner(s). Fig. 4 shows the full distributions of match times and wait times. These results indicate that both match and wait times were typically low, and for most participants barely noticeable. 4.2. Participants’ subjective experience Although the above results already provide suggestive evidence that participants’ experience was good (low attrition rates, 9 Note that 3 s was the minimum waiting time in these studies. By default, SMARTRIQS has a minimum waiting time of 3 s so that the loading screen and animation can be displayed.

Fig. 4. Cumulative distribution functions of match time and wait time (per round) across the four CMU studies (N = 880). Note that the scale along the X -axis is non-linear.

quick matching and minimal waiting), a more appropriate approach is to look at participants’ actual feelings and perceptions. To investigate this, I included a very short questionnaire at the end of the interactive word association study (CMU-1). In this questionnaire I asked participants the following five questions about their experience: 1. Please rate the following: Overall experience with this study. 2. Please rate the following: The speed of the matching algorithm. 3. Please rate the following: My partner’s response times. 4. Have you participated in other studies on MTurk before that featured real-time online interaction? (yes/no) 5. Please let us know if you have any comments about this study. (open − ended) Participants provided their responses to items 1–3 on a fivepoint scale (1: very bad ... 5: very good). Participants who responded ‘yes’ to item 4, were presented a new question and were asked to indicate whether they agree or disagree with the following five statements: (−3: strongly disagree ... 3: strongly agree) 6. This study was smoother than other studies. 7. Response times in this study were quicker than in other studies. 8. I had to wait less than in other studies. 9. I would prefer to participate in future studies like this study. 10. This study was a more positive experience than other studies.


A. Molnar / Journal of Behavioral and Experimental Finance 22 (2019) 161–169

Participants were not informed in advance about this postexperimental survey, and they did not receive any compensation for answering these questions: taking this survey was optional. The complete data, along with participants’ open-ended comments can be found at this OSF data repository. All participants who completed the word association task (CMU-1, N = 134), including 6 participants who had been matched with bots, completed the post-experimental survey. participants’ overall rating of the interaction was very positive (M = 4.43, 95% CI = [4.29, 4.57], Mdn = 5). Participants also rated the speed of the matching algorithm as ‘good’ (M = 4.07, 95% CI = [3.89, 4.26], Mdn = 4), as well as the waiting time for their partner (M = 3.99, 95% CI = [3.80, 4.19], Mdn = 4). Sixty-nine participants (52%) indicated that they had participated in interactive experiments before. Most of these participants rather agreed that: (1) this study was smoother than other studies (M = 1.35, 95% CI = [1.08, 1.51], Mdn = 1), (2) they experienced quicker response times than in other interactive studies (M = 0.67, 95% CI = [0.27, 1.06], Mdn = 1), and (3) they had to wait less than in other interactive studies (M = 0.57, 95% CI = [0.14, 0.99], Mdn = 1). Most participants also agreed that they would prefer to participate in similar interactive studies in the future (M = 2.00, 95% CI = [1.74, 2.26], Mdn = 2) and that the study was a more positive experience than other interactive studies (M = 1.70, 95% CI = [1.43, 1.96], Mdn = 2). Only one participant (1%) indicated that this study was a less positive experience than other interactive studies and that they would not prefer to participate in similar studies. While it is possible that some of the positive ratings were driven by the type of the experimental task, as opposed to the interaction, these results provide additional suggestive evidence that MTurk participants had a positive experience in studies designed and conducted with SMARTRIQS. 5. Conclusion SMARTRIQS offers a novel and simple method of online experimentation. Yet, it gives researchers a lot of flexibility in study design and allows them to fully utilize the features of a powerful survey software, without having to learn any technical skills. SMARTRIQS is the ideal solution for researchers who want to run interactive experiments online but do not have the time or the resources to do it by using conventional experimental platforms. Furthermore, SMARTRIQS is very convenient for researchers who wish to add real-time interaction or chat to their non-interactive surveys. SMARTRIQS is also a great choice for researchers who want to create easily customizable studies with industry standard visual design and user experience. I believe that many researchers will consider SMARTRIQS a useful tool and a valuable public good. I am expecting that not only experimental economists will make good use of SMARTRIQS but also researchers who work in related fields such as organizational research, consumer research, decision science, social psychology, or political science, studying real-time social interactions, social norms and preferences, collective decision-making, and voting behavior. Finally, since SMARTRIQS is free and opensource, it paves the way for others to design even more complex interactive studies in Qualtrics, and will hopefully inspire them to create similar solutions to other survey platforms. Acknowledgments I thank Ankita Sastry for her help in developing SMARTRIQS and Peggy He and Denise Lin for their assistance in testing SMARTRIQS. I also thank Russell Golman, Nik Gurney, Stephanie Permut, and two anonymous reviewers for their valuable suggestions

and comments on earlier versions of this paper. Furthermore, I thank Mary Rigdon and Eric VanEpps for providing me aggregate statistics about their studies. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Declarations of interest None. Appendix. Data policy Any data collected and processed while using the default SMARTRIQS server is stored on a secure, SSL-encrypted virtual server, hosted at Amazon Web Services. Only the developer of SMARTRIQS has access to this server. Since all data is saved in Qualtrics, data on the SMARTRIQS server is stored only temporarily and is deleted on a regular basis. The following data are transmitted to and stored temporarily on the server:

• Study-specific parameters: researcher ID, study ID, group size, number of stages, roles, conditions, waiting time parameters. • Participant-specific parameters (randomly assigned): treatment, role, participant ID (by default, this is the same as the survey ID in Qualtrics). • Participant data and responses. Note: not all data recorded in Qualtrics are stored on the SMARTRIQS server, only those which are submitted via the SEND and CHAT blocks. • Time stamps (e.g., ‘last activity’ of participant). Data submission policy agreement Researchers who are using the default SMARTRIQS server ( and its subdomains) agree that they DO NOT SUBMIT any of the following personal identifiers of participants to the default SMARTRIQS server via the SEND and CHAT blocks: 1. Name; 2. Birth date (except year); 3. Physical address and any geographic information (except for country/state), including street address, city, county, precinct, zip code, and their equivalent geocodes; 4. Internet Protocol (IP) addresses; 5. Phone/fax numbers, email addresses, social media profiles, personal websites; 6. Identifying numbers: Social Security numbers, Vehicle identifiers and serial numbers, Device identifiers and serial numbers, Medical record numbers, Health plan beneficiary numbers, Account numbers; 7. Any other unique identifying number, characteristic, or code that can identify the participant. Note: the restriction above applies only to submitting such information to the default SMARTRIQS server. Researchers are allowed to collect personal identifiers in Qualtrics without submitting them to the default SMARTRIQS server via the SEND or CHAT blocks, assuming that they have obtained IRB approval. If researchers deploy SMARTRIQS to their own server, they are allowed to transmit and store personal information on their server, assuming that they have obtained IRB approval.

A. Molnar / Journal of Behavioral and Experimental Finance 22 (2019) 161–169

References Arechar, A.A., Gächter, S., Molleman, L., 2018. Conducting interactive experiments online. Exp. Econom. 21 (1), 99–131. s10683-017-9527-2. Barnhoorn, J.S., Haasnoot, E., Bocanegra, B.R., van Steenbergen, H., 2015. QRTEngine: An easy solution for running online reaction time experiments using Qualtrics. Behav. Res. Methods 47 (4), 918–929. 3758/s13428-014-0530-7. Berry, J., Coffman, L.C., Hanley, D., Gihleb, R., Wilson, A.J., 2017. Assessing the rate of replication in economics. Amer. Econ. Rev. 107 (5), 27–31. http://dx. Bohannon, J., 2016. Mechanical Turk upends social sciences. Science 352 (6291), 1263–1264. Buhrmester, M., Kwang, T., Gosling, S.D., 2011. Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?. Perspect. Psychological Sci. 6 (1), 3–5. Chandler, J., Mueller, P., Paolacci, G., 2014. Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behav. Res. Methods 46 (1), 112–130. s13428-013-0365-7. Chen, D.L., Schonger, M., Wickens, C., 2016. oTRee—an open-source platform for laboratory, online, and field experiments. J. Behav. Exp. Finance 9, 88–97.


Collaboration*, O.S., 2015. Estimating the reproducibility of psychological science. Science 349 (6251), Fischbacher, U., 2007. z-Tree: Zurich toolbox for ready-made economic experiments. Exp. Econom. 10 (2), 171–178. s10683-006-9159-4. Giamattei, M., Molleman, L., Yahosseini, K., Gächter, S., 2019. LIONESS Lab—a free web-based platform for conducting interactive experiments online. Hendriks, A., 2012. SoPHIE—Software Platform for Human Interaction Experiments. Ioannidis, J.P., Stanley, T.D., Doucouliagos, H., 2017. The power of bias in economics research. Econom. J. 127 (605), F236–F265. 1111/ecoj.12461. Mao, A., Chen, Y., Parkes, D.C., Procaccia, A.D., 2012. TurkServer: enabling synchronous and longitudinal online experiments. Human Comput. (March), 33–39. Paolacci, G., Chandler, J., Ipeirotis, P., 2010. Running experiments on Amazon Mechanical Turk. Judgment Decis. Mak. 5 (5), 411–419. Peer, E., Brandimarte, L., Samat, S., Acquisti, A., 2017. Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. J. Exp. Soc. Psychol. 70 (January), 153–163. Qualtrics, 2019. Provo, Utah, USA, Retrieved from SurveyMonkey, 2019. San Mateo, California, USA: SurveyMonkey Inc. Retrieved from Zhou, H., Fishbach, A., 2016. The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions.. J. Personality Soc. Psychol. 111 (4), 493–504. 1037/pspa0000056.