What Does It Mean if Your Turkprime Study Is Under Review

Abstract

In contempo years, Mechanical Turk (MTurk) has revolutionized social science by providing a way to collect behavioral information with unprecedented speed and efficiency. Nonetheless, MTurk was not intended to be a research tool, and many common enquiry tasks are difficult and time-consuming to implement as a issue. TurkPrime was designed as a inquiry platform that integrates with MTurk and supports tasks that are common to the social and behavioral sciences. Like MTurk, TurkPrime is an Internet-based platform that runs on whatever browser and does non require any downloads or installation. Tasks that can be implemented with TurkPrime include: excluding participants on the basis of previous participation, longitudinal studies, making changes to a study while it is running, automating the approving process, increasing the speed of data collection, sending bulk eastward-mails and bonuses, enhancing communication with participants, monitoring dropout and engagement rates, providing enhanced sampling options, and many others. This article describes how TurkPrime saves time and resource, improves information quality, and allows researchers to blueprint and implement studies that were previously very difficult or impossible to carry out on MTurk. TurkPrime is designed as a research tool whose aim is to amend the quality of the crowdsourcing data collection process. Various features have been and go on to be implemented on the basis of feedback from the research customs. TurkPrime is a complimentary research platform.

Access to participants is of cardinal importance to researchers in the social and behavioral sciences. Until the belatedly 1990s, almost behavioral researchers had to rely on undergraduate subject field pools every bit their master source of research participants. Academy discipline pools have a number of limitations, including lack of representativeness (Henrich, Heine, & Norenzayan, 2010), and can oft exist labor-intensive and fourth dimension-consuming to use. Most importantly, such subject pools are non available to faculty in many modest schools (run into Kraut et al., 2004). Starting in the late 1990s, the Internet became more commonly used as a source of participant recruitment (Kraut et al., 2004). Recruiting participants over the Net provided a number of advantages over the use of the traditional subject field puddle (Gosling & Johnson, 2010): Researchers were able to recruit vastly more participants in much shorter time periods (Nosek, Banaji, & Greenwald, 2002), and the data were typically more representative and less labor-intensive to learn (Kraut et al., 2004). Accessing big numbers of participants from the Internet, commonly referred to every bit crowdsourcing, as well has substantial limitations. For example, finding participants, incentivizing participation, and preventing multiple participation are only some of the mutual barriers encountered on most crowdsourcing platforms.

One crowdsourcing platform that overcomes many of these limitations is Amazon Mechanical Turk (MTurk). MTurk has in recent years revolutionized behavioral research (Buhrmester, Kwang, & Gosling, 2011), and increasingly is beingness used equally a participant recruitment tool across a broad spectrum of disciplines in the social sciences. MTurk is an online platform where researchers can post studies, instantaneously making them available to thousands of participants effectually the world. Hundreds of participants will typically have completed a study simply hours after it is launched, drastically increasing the speed of the information acquisition procedure relative to traditional methods. Information acquired on MTurk have been found to be valid beyond numerous tasks and countries (Litman, Robinson, & Rosenzweig, 2015; Shapiro, Chandler, & Mueller, 2013; Sprouse, 2011), despite the relatively low price that participants are paid for completing tasks, making the MTurk platform a fast method of acquiring cheap and reliable data (Buhrmester et al., 2011).

Despite the benefits that MTurk offers social scientific discipline researchers, information technology also has its limitations. MTurk was not created with behavioral research in mind, and many features that are fundamental for the research process are not readily bachelor through the MTurk graphical user interface (GUI). To overcome the current limitations of MTurk, multiple tools accept been created to enable researches to achieve specific tasks (Gureckis et al., 2015; Peer, Paolacci, Chandler, & Mueller, 2012; http://gideongoldin.github.io/TurkGate/). For example, Peer et al. (2012) created a Qualtrics tool through which information technology is possible to exclude participants from a study on the ground of participation in previous studies. However, currently no single, integrated platform provides researchers with a comprehensive set of research-specific flexible tools that are delivered through a simple GUI. For this reason, we developed the TurkPrime Data Conquering Platform for the Social Sciences (TurkPrime). TurkPrime is a website on which users tin create MTurk studies (HITs) and through which MTurk HITs can be controlled with substantially greater flexibility than the MTurk platform currently allows.

TurkPrime (see Fig. 1) is a website that utilizes the MTurk application programming interface (API), as well as other programmatic tools, to provide MTurk requesters with a GUI environment with a high level of control over HITs. The TurkPrime GUI environment provides improved functionality over MTurk in six full general areas: control over who participates in the study, flexible control over running HITs, more than flexible communication and payment mechanisms, tools for longitudinal and panel studies, tools to increase sample representativeness, and enhanced written report menstruation indicators.

Fig. 1

Screenshot of the TurkPrime abode folio

Full size prototype

In the present article, we show how the TurkPrime environment allows users to overcome the current limitations of the MTurk GUI. Because detailed tutorials including videos for using TurkPrime are available on the TurkPrime website, in this commodity we focus on discussing the methodological advantages that TurkPrime offers, and also provide a conceptual overview of the TurkPrime platform.

Mechanical Turk basics

Because of MTurk's popularity, a number of MTurk guides are currently bachelor (Berinsky, Huber, & Lenz, 2012; Chandler & Mueller, 2013; Paolacci, Chandler, & Ipeirotis, 2010; Mason & Suri, 2012). Here, we will non provide a comprehensive overview of MTurk, only merely introduce those terms and concepts that are relevant for the TurkPrime features that are discussed throughout this article.

Workers, requesters, and HITs

The MTurk platform was created every bit a worldwide microtask marketplace. MTurk facilitates monetary transactions between people who need various jobs done (requesters) and people who are interested in performing those jobs in return for payment (workers). A chore put up past a requester on MTurk is chosen a homo intelligence task (Hitting). Requesters tin create HITs on the MTurk platform or use MTurk to announce an external Hit. For case, MTurk provides a platform for creating survey questions. A requester may use that built-in MTurk functionality, or instead tin can create a survey on an external platform such as Qualtrics and mail service a link to that survey every bit part of their HIT. We refer to HITs that are completed via an external platform as external HITs, and tasks that utilize the MTurk platform every bit internal HITs.

MTurk makes HITs visible to workers past listing them in the All HITs window, where workers can select a Striking from the numerous ones available. Workers tin organize HITs in various means. By default, the All HITs window organizes HITs by their creation date, such that the newest HITs are presented at the acme of the list. HITs tin can as well be sorted by toll, title, and HIT length.

Worker ID

The MTurk platform allows for 2-mode interactions between workers and requesters. For example, requesters can send email invitations to workers and consequence bonuses based on performance. Requesters tin also consequence qualifications to workers. A qualification is a status indicator that reflects whether or non a worker is eligible to take a specific HIT with a requester. Past using Worker IDs, MTurk prevents workers from participating multiple times in the same study. This is one of the critical features that makes data collection on MTurk different from regular online information platforms, where at that place is typically fiddling control over a participant's ability to complete a written report multiple times.

When signing up for an business relationship, each worker is assigned a unique Worker ID. This Worker ID cannot be changed and remains constant as long as the worker uses that account. The Worker ID is key for the overall MTurk functionality for multiple reasons. For instance, past using a worker's ID, a requester may invite a specific worker to take a follow-upward HIT or transport a monetary bonus to that worker.

Approving and rejecting HITs

MTurk keeps track of workers' Hit histories and work quality by means of a number of indicators. Requesters can utilise these indicators equally choice criteria when specifying the eligibility of workers for taking a Hit. When workers complete a study, their Worker IDs automatically appear in a list of completed requester's HITs. A requester can corroborate the Hitting, at which betoken the payment for that Hit is made automatically by MTurk from the requester's funded MTurk account to the account of the worker. A requester can as well turn down a HIT if information technology was not performed adequately, for whatever reason. Overall, the requester has a lot of command over the worker and has deciding power over whether or not a HIT is rejected. A rejected Hitting results in a worker not being paid. Additionally, the numbers of canonical and rejected HITs remain part of a worker'southward permanent record. The ratio of approvals to rejections makes up a worker's approval rating, which can exist used by a requester at the beginning of a study to select workers. Selecting workers with high approval ratings has been shown to better data quality (Peer, Vosgerau, & Acquisti, 2014).

A worker's approving rating is updated every time the worker finishes a study and is either rejected or approved past a requester. The reciprocal relationship, whereby workers would rate requesters, is not provided by MTurk. However, various websites on which workers leave comments virtually specific HITs and requesters, such equally Turkopticon (https://turkopticon.ucsd.edu/) and TurkerNation (http://turkernation.com/), have been created, in part to provide workers with information most the quality of requesters and HITs. Workers normally provide data on these websites about whether a HIT pays adequately, how long a requester takes to pay for a Hit, and whether a requester rejects HITs unfairly. Requesters should continue track of these websites, considering information provided at that place can affect workers' willingness to accept a specific requester'southward HIT. Additionally, requesters should manage their reputations on these sites, because the information provided in that location is not always accurate, and tin potentially negatively affect participation rate, and thus may compromise the recruitment process.

Monitoring worker quality

In add-on to a worker'due south approval rating, MTurk also keeps track of the number of HITs the worker has completed and the number of HITs returned. When a requester creates a HIT, it becomes visible to the MTurk worker customs on the All HITs folio, which lists the titles of all available HITs. A worker may click on the proper name of a Hitting that they are interested in—for example, "transcribe information"—which opens that Hit in preview style, in which a more detailed description of the Hitting becomes visible. For an internal Striking, workers will see a preview of the task, and for an external Hitting, they volition come across a link to the external site, and typically a more detailed description of the job.

All of the MTurk functionality described thus far can exist navigated through the MTurk GUI and does not require any programming. In addition to the GUI, MTurk provides an API with which much more flexibility can be achieved over HITs. However, use of the API requires a noesis of programming and a substantial time commitment. The result is that nearly requesters practise not benefit from the full set of resources that MTurk offers. The standard GUI interface imposes disquisitional restrictions on what requesters can do with their HITs. For example, they cannot easily prevent workers from participating in a study on the footing of their participation in previous studies, fix longitudinal studies, automatically corroborate worker assignments upon completion of a study, communicate with multiple workers at once, grant bonuses to more than ane worker at a fourth dimension, change the price or description of a study after it has been launched, or control the rate at which data are collected. TurkPrime was created as a way to overcome these and other limitations, for the purpose of giving requesters more control over their studies.

Using TurkPrime

Initial TurkPrime setup

To use TurkPrime, a requester must (a) have an existing MTurk account, (b) create a TurkPrime account, and (c) link the TurkPrime business relationship to their MTurk account. The TurkPrime website (TurkPrime.com) has a Setup Mechanical Turk tab, which provides the directions for associating a TurkPrime business relationship with an MTurk account. This brusk procedure involves providing TurkPrime with a requester's MTurk account access keys, which are found on their MTurk security credentials page. One time the account is set up, requesters can launch, stop, and brand changes to their MTurk HITs directly from TurkPrime.

The TurkPrime surroundings

TurkPrime has three primary windows. The Design Survey page (see Fig. 2) is where studies are set up and prepared for launch. The Dashboard (encounter Fig. 3) provides a view of all running and completed studies. It additionally contains an interface through which a report tin be controlled, and indicators that provide information near the study'due south progress. An additional Workers Groups (see Fig. four) folio is discussed below, in the Longitudinal and Console Studies department. As on MTurk, TurkPrime provides a Sandbox environs in which HITs can be simulated without any actual data being collected.

Fig. 2

Screenshot of the Pattern Survey folio

Full size image

Fig. 3

Screenshot of the Dashboard extended view

Full size image

Fig. 4

Screenshot of the Worker Groups window

Full size epitome

Design Survey page

TurkPrime studies are launched from the Design Survey folio, which can exist accessed from TurkPrime's habitation page. The Pattern Survey page is designed to look like the Pattern HITs page on MTurk. Experienced MTurk users volition notice that setting up a report on TurkPrime is very similar to doing so on MTurk. For instance, requesters volition notice fields for naming a study, indicating the cost, and selecting the quality of workers. The MTurk Design Survey folio also contains fields that are not plant on MTurk, such as excluding or including participants from prior studies (see Fig. 4).

Control over who participates in a study

TurkPrime provides a number of ways to limit and grant access to a study for specific workers. In upshot, TurkPrime automates the process of assigning qualifications. TurkPrime makes it possible to assign qualifications to participants from multiple studies simply by selecting those studies from a dropdown window or pasting sets of Worker IDs in the Include and Exclude windows.

Excluding workers

Allowing participants to complete a report multiple times can lead to a reduction in consequence sizes (Chandler, Paolacci, Peer, Mueller, & Ratliff, 2015). It is thus important to exist able to exclude participants from a HIT on the basis of prior participation in a similar HIT. On TurkPrime, participants can be excluded from a HIT in two ways. The get-go is to enter their Worker IDs into an Exclude box when launching a HIT. One can enter whatsoever Worker ID in this box, even if the Worker IDs are from a HIT that was not run on TurkPrime, provided that these workers take an active human relationship with a requester. The second manner to exclude participants is at a study level. When creating a new Hit (e.thou., HIT iv), a requester may select a whole Hit (Hit ane) or HITs (Hit 1, Hit 2, Hitting iii) for exclusion from HIT 4, using the dropdown menu in the Exclude Workers Who Completed These Surveys window. This option requires that the HITs were run on TurkPrime. It is also possible to apply these windows in combination to exclude both whole studies and individual workers from other studies.

A third style to exclude participants is to create worker groups. Worker Groups is a highly flexible tool with which groups tin exist created for exclusion and reused beyond multiple studies (see Fig. 4). For example, the IDs of all workers who participated in a series of studies tin be added to a grouping. These Worker IDs can come from studies that were run on TurkPrime or that were run direct from the requester'due south MTurk account. One time the group is created, it becomes visible on the Pattern Survey page and can exist selected for exclusion. Worker groups is also useful for creating customized exclusion groups that are based on workers' responses. For example, workers who do not pass attention manipulation checks can be added to a permanent exclude group that a requester may add together to all studies. Other uses of worker groups discussed below include creating groups of experts for stimulus preparation, the creation of panels, and longitudinal studies.

Including workers

The Include option assigns qualifications to workers by means of which those workers, and only those workers, become eligible to accept a specific Hit. The Design Survey page includes two controls through which workers can exist included in a study. When setting up a report on the Design Survey folio, under the Avant-garde tab, all of a requester's previously launched HITs appear in the Allow Workers Who Completed These Surveys field. A requester can select multiple studies whose workers they wish to include. When including multiple studies, the "All" choice includes workers who take taken all of the selected studies, and the "Whatever" choice includes workers who take taken any of the selected studies. The Include Workers tool allows requesters to run longitudinal studies that are described in more particular beneath.

Enhanced control over HITs

Editing a HIT after launch

TurkPrime gives requesters full control over a study after information technology has been launched. Afterward launching a study, a requester can have multiple actions to control HITs that are however running and HITs that have already been completed (see Fig. three). The Edit button takes the requester dorsum to the Design page, where changes can exist made to any of the fields, including the payment field, written report clarification, qualifications, and any of the other fields. A requester can also pause a written report and proceed at a after time.

Being able to alter a report while it is running has multiple research-related uses. For instance, it is oftentimes difficult to gauge the length of a study and, thus, how much to pay participants. With the Edit feature, a pilot study can be run to assess a study'due south completion time. On the basis of this information, the written report description and price can be altered prior to launching the full study. In outcome, this allows researchers to more hands behave pilot studies, on the ground of which a Hit's parameters can be tuned prior to launching the full study.

Restart

Additional control over a running Hitting is provided past the Restart button, which is found on the Dashboard. The purpose of the Restart characteristic is to enable requesters to collect big datasets by periodically making the Hitting more visible to workers. Considering thousands of HITs are available on MTurk at a given betoken in time, it is important to take MTurk'southward approach to organizing HITs into consideration, then equally to maximize the likelihood of workers seeing a HIT. For instance, trivial features of a chore such as its proper name tin affect participation rates. HITs that commencement with the letter A or the letter Z are more likely to exist taken than HITs that start with the alphabetic character South, because those are the tasks that come offset when sorting by HITs' titles (Chilton, Horton, Miller, & Azenkot, 2010). Studies accept besides shown that older HITs are less likely to exist viewed by workers, and that participation rates driblet off very quickly for older tasks. Specifically, participation in tasks that are more than 24 h former drops off by 70% and continues to do so every consecutive solar day (Chilton et al., 2010). The Restart feature finer creates a brand new Hitting with characteristics identical to those of the original, and automatically prevents multiple participation by assigning qualification to workers who have taken the HIT already. The issue is that the HIT is bumped up to the top of the All HITs window, which has an immediate impact on the visibility of the HIT and substantially increases participation rates.

Flexible payment mechanisms

The receipt of timely payment is a central business concern for MTurk workers. Delays in payment tin can event in negative reviews on Turker Nation and Turkopticon and tin can have negative effects on participation rates as a consequence. Conversely, fast payment is usually mentioned past workers in positive reviews of HITs. The current MTurk payment mechanisms, however, can result in lengthy delays and can be time-consuming for the requester. These delays arise for a number of reasons. Requesters have to approve each worker'due south submitted Striking in guild for the worker to receive payment. Once a worker completes a HIT, they must wait for the requester to review and corroborate it. For the vast majority of HITs, the review process entails verifying what is referred to every bit the "hugger-mugger code" that is appended to the end of a study. To verify that workers have really completed the full Striking, requesters routinely suspend a brusque alphanumeric code at the end of their study or survey. The workers are then asked to enter this lawmaking in an MTurk response field prior to submitting their HIT. Prior to approving the HIT, requesters review the code to make sure that it is right.

Reviewing hole-and-corner codes tin can exist time-consuming, especially for studies with large sample sizes, and tin result in payment delays. To overcome this problem, TurkPrime includes two optional automated verification mechanisms. Both mechanisms check the underground code entered past the worker against a prespecified secret code and automatically approve a submitted HIT on the footing of whether the codes match. This results in automatic and firsthand payment delivery to the workers and saves the requesters time spent having to do this manually. The two verification mechanisms are called fixed and dynamic secret codes.

Stock-still secret code

A fixed secret lawmaking is a single lawmaking entered by the requester on the Blueprint Survey page that is the same for all workers. This lawmaking is manually added by the requester to a study, usually at the end of a survey. Workers are asked to enter this lawmaking on Mechanical Turk subsequently they have completed the study. Equally we mentioned above, MTurk provides the capacity to create internal HITs. The surreptitious code entered by workers on MTurk becomes part of this internal HIT. TurkPrime then compares the lawmaking entered by the worker confronting the undercover code that was entered by the requester on the Design Survey folio. A submitted assignment is automatically approved when at that place is a lucifer. In the upshot of a mismatch, the assignment enters a Pending status. Pending assignments require manual review by the requester. Because static hugger-mugger codes are the same for all workers, in that location is a possibility that such codes may exist shared by workers amid themselves. For this reason, it is recommended that dynamic secret codes be used when possible.

Dynamic hush-hush lawmaking

A dynamic secret code works through integration with Qualtrics. The Qualtrics Survey Period functionality can be used to brandish a unique code that is shown to the worker and is as well automatically passed to TurkPrime. As with the stock-still secret lawmaking, the dynamic secret code is entered by workers on MTurk when they complete a report. When a worker enters the dynamic secret code, it is automatically checked by TurkPrime, and workers are automatically approved and paid. Workers are not automatically rejected. In the event that a requester rejects a worker in error, the Dashboard contains a Contrary Rejections button through which rejections of individual workers or all workers can be immediately reversed.

Batch bonus

An additional characteristic for enhancing payments is the batch bonus. Bonuses are ordinarily used by researchers as a way to incentivize participation, which has been establish to accept a substantial effect on data quality and the creativity of workers (Buccafusco, Burns, Fromer, & Sprigman, 2014). The MTurk API, nonetheless, only allows for 1 bonus to be given to one worker at a fourth dimension. This results in payment delays and tin can exist fourth dimension-consuming to implement. The TurkPrime Batch Bonus window is available on the Dashboard, through which bonuses can be assigned to all or some workers at once.

Tools for longitudinal and panel studies

Longitudinal studies are HITs that are open up to workers who accept taken previous HITs with a requester. Setting up longitudinal studies requires (1) A Hit that is open only to specific workers on the footing of participation in previous studies, (2) matching participants across the phases of a written report, and (iii) notifying participants that a Hit is bachelor. Equally we described previously, a HIT that is open only to specific workers tin be set up by using the Include control.

Embedding study-specific fields

Because longitudinal studies have multiple phases, there are typically multiple data files that store the results from the dissimilar phases. Worker IDs permit information to exist matched across phases. TurkPrime facilitates the process of matching Worker IDs across multiple data files past automatically passing each worker's ID to Qualtrics via an embedded query cord in the survey'southward URL. Requesters can so automatically insert the Worker ID into data files. In Qualtrics this is easily accomplished by embedding the Worker ID field into the survey period. Equally of now, this characteristic is just bachelor for Qualtrics studies.

Worker notification

Opening up a Striking to specific workers differs from a typical HIT in a critical fashion. Workers who are qualified to take the Hit may not exist in front of a computer at the fourth dimension that a study is launched, and thus may never know that the HIT is available. For this reason, TurkPrime enables a batch e-mail feature for longitudinal studies with which eastward-mails tin can exist sent to all eligible participants, informing them that the HIT is available, how much it pays, and any other information that a requester may wish to include to incentivize the workers to participate in the study. The e-post automatically includes the HIT link.

Worker groups

TurkPrime has a divide Worker Groups window that enables the creation of panels, which can be considered to be a special example of longitudinal studies. Panel groups are collections of workers who share common characteristics. For case, these tin can exist political groups such as Republicans or Democrats. Groups of Worker IDs can be entered into the Groups window and given a name. Once a group is created, it becomes visible in the Groups dropdown card on the Pattern Survey page. A study tin and so exist created that is open up only to the participants of these groups. Worker grouping studies can exist particularly constructive when requesters have accumulated a large number of workers that can then exist utilized every bit a requester's ain field of study puddle (see Von Emden, 2015). For example, a requester tin create fourth dimension-series studies with equal numbers of Republican and Democratic participants, and follow them over time to examine their opinions and event-related opinion changes.

To summarize, the Include controls, worker groups, batch emails, and embedded Worker IDs constitute a set up of tools that enable researchers to conduct longitudinal and time-series studies on Mechanical Turk.

Tools for increasing sample representativeness

The speed with which data tin can exist acquired on MTurk is 1 of the major benefits of this platform. Nevertheless, very fast acquisition of data also has the potential for unintended negative consequences, since collecting data quickly has the potential to create bias. For case, if a study is launched at a specific fourth dimension on a weekday, such as Monday morning, the sample will exist biased to workers who are at home on a solar day when nigh people are working (Rosenzweig, Litman, & Robinson, 2015). To first this potential source of bias, TurkPrime has a microbatch feature that allows requesters to break up HITs into small segments and to specify the fourth dimension interval at which the segments will be launched. For example, a study may be timed to launch over 12-h periods across all 7 days of the week.

Additionally, a time release feature allows requesters to specify the time that the study will be launched. With this feature, a report can be prepared at one time and launched at a later time. This is specially useful for international requesters who launch their studies at a time that corresponds to nighttime in the U.s.a..

Enhanced study menstruation indicators

The extended Dashboard view provides a number of diagnostic tools with which a study can exist monitored. These include bounce rate, completion rate, and median completion time.

Completion charge per unit

Dropout rate tin can be an of import indicator of data quality. A high dropout rate may mean the presence of a choice bias that may influence the representativeness of the results. If a Hitting takes much longer to consummate than is indicated in the report description, workers will be more likely to return or abandon information technology, influencing the dropout charge per unit. A requester may desire to alter the pay rate if the completion charge per unit is low. A very high dropout rate may also mean that something is wrong with the study, such as a broken link that prevents participants from completing the Striking. Information technology is typically skilful exercise to written report the completion charge per unit (Eysenbach, 2004); notwithstanding, this information is non available on MTurk. Because completion rate can exist an important indicator of a written report's quality, TurkPrime makes completion rate available on the Dashboard, so that researchers can monitor the completion rate in real time.

Bounce rate

Bounce charge per unit is similar to completion rate, and is important for the aforementioned reasons. What makes bounce charge per unit different is that it besides counts participants who did not accept the Striking, only merely previewed it. Nosotros define the bounciness rate as 100% – [Accepted Assignments] / [Previewed Assignments]. Previewed Assignments is set up to be the full of unique IP addresses that previewed this consignment. Since almost all workers have unique IP addresses, this is a reasonable estimate of the total number of previews.

Completion time

The 3rd indicator bachelor on the extended Dashboard view is the median completion fourth dimension. We accept consistently observed that many workers are outliers in terms of completion times. This may be due to the fundamental lack of experimenter control over the study setting, which makes it easy for participants to take breaks in the middle of a study. These outliers affect the mean completion fourth dimension. Because completion time is critical for setting appropriate compensation rates, we made the median fourth dimension available on the Dashboard. Nosotros have consistently observed that median completion times are significantly lower than the means.

Related platforms

Mechanical Turk is becoming increasingly important for data acquisition across the spectrum of behavioral research fields. At the aforementioned time, although the demand for increased versatility in the range of online enquiry designs increases, the MTurk GUI has remained mostly unchanged. For this reason, a variety of research platforms have emerged to facilitate the use of MTurk for behavioral research. TurkPrime is 1 such platform, just other platforms are besides bachelor (Gureckis et al., 2015; Peer et al., 2012; http://gideongoldin.github.io/TurkGate). These platforms vary widely in the range of tools that they offer and the GUIs that they provide. One platform that offers a wide range of tools is psiTurk. Like TurkPrime, psiTurk offers the abilities to exclude participants, offering bonuses, and brand automatic payments, among many other features. psiTurk is open-source platform for designing flexible online studies (see Gureckis et al., 2015). Ane primal deviation betwixt TurkPrime and psiTurk is that psiTurk runs on a UNIX-based command line interface. As such, psiTurk only runs on UNIX-based systems, such as Linux and Mac OS. TurkPrime, on the other paw, has a point-and-click interface. TurkPrime runs on any browser from any operating system. In addition, TurkPrime provides multiple features that, to the all-time of our knowledge, are not currently offered on other platforms. These features include automatic checks of hush-hush codes, including unique underground codes generated by Qualtrics; Microbatch functionality for improved sample representativeness; worker groups; the power to easily restart a Hitting; longitudinal study command, including embedding query string parameters to automatically insert a Worker ID into a information file; and full ability to modify any aspect of a HIT afterwards information technology is launched.

Usage information

Adoption by the community

As of the writing of this article, TurkPrime has 1726 registered users who have run 9750 unique MTurk HITs. In all, over 60,000 unique workers have completed HITs on TurkPrime, and 207,000 assignments have been completed on TurkPrime within the last 30 days.

Data protection

TurkPrime uses ship layer security encryption (also known every bit HTTPS) for all transmitted information. All information access is blocked except for explicitly white-listed IP addresses, in addition to being secured with user passwords. Furthermore, MTurk data, including Access Central ID and Clandestine Access Key, are encrypted with AES-256 encryption, the standard adopted past the National Institute of Standards and Technology.

Conclusion

To summarize, TurkPrime was created as an extension of Amazon Mechanical Turk to optimize MTurk functionality for the needs of researchers. MTurk is a powerful crowdsourcing environs that provides substantial advantages over other online subject recruitment platforms. Among these are fast and easy access to thousands of research participants, a payment mechanism to incentivize participation, a way to foreclose multiple participation by the same individual, and a high level of confidentiality. In recent years, MTurk has go the most popular crowdsourcing research platform, with close to 1,000 peer-reviewed papers published from its data per year.

The vast power of the MTurk API, however, has been considerably underutilized by most researchers. TurkPrime offers a programming-free user interface that enables researchers to harness the full ability of MTurk. TurkPrime enables researchers to employ research designs, including longitudinal, fourth dimension-series, and panel studies, that would otherwise be difficult and time-consuming, and that would crave substantial API programming to implement. TurkPrime enhances the ease with which requesters tin can communicate with workers, including sending batch e-post notifications and bonus payments; provides a flexible way to pause up studies over time; gives users complete control over launched HITs; automates the approval/rejection procedure; provides Qualtrics integration; and allows for the creation of custom worker groups for inclusion and exclusion across studies. Worker groups have the boosted benefit that requesters can notify workers who are not monitoring MTurk at the time of study launch. This makes it possible for requesters who have accumulated large worker pools to reach more workers than they would exist able to by launching a report on MTurk alone.

TurkPrime was created every bit a dynamic environment that is responsive to the needs of researchers. Every bit such, new features have been recommended by the inquiry community and are being added in an ongoing endeavour to enhance the quality and usability of online enquiry.

References

• Berinsky, A. J., Huber, M. A., & Lenz, G. South. (2012). Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis, twenty, 351–368. doi:ten.1093/pan/mpr057

  Commodity  Google Scholar

• Buccafusco, C., Burns, Z. C., Fromer, J. C., & Sprigman, C. J. (2014). Experimental tests of intellectual holding laws' inventiveness thresholds. Texas Police Review, 93, 1921–1980.

  Google Scholar

• Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of cheap, yet high-quality, data? Perspectives on Psychological Scientific discipline, 6, 3–v. doi:ten.1177/1745691610393980

  Article  PubMed  Google Scholar

• Chandler, J., & Mueller, P. (2013). Methodological concerns and advanced uses of crowdsourcing in psychological research. Manuscript submitted for publication.

• Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26, 1131–1139. doi:10.1177/0956797615585115

  Article  PubMed  Google Scholar

• Chilton, 50. B., Horton, J. J., Miller, R. C., & Azenkot, S. (2010). Chore search in a human ciphering market. In Proceedings of the ACM SIGKDD Workshop on Homo Ciphering (pp. 1–9). New York, NY: ACM Press.

  Affiliate  Google Scholar

• Eysenbach, M. (2004). Improving the quality of Spider web surveys: The Checklist for Reporting Results of Cyberspace E-Surveys (CHERRIES). Journal of Medical Internet Inquiry, 6, e34.

  Article  PubMed  PubMed Central  Google Scholar

• Gosling, S. D., & Johnson, J. A. (2010). Avant-garde methods for conducting online behavioral research. Washington, DC: American Psychological Clan.

  Volume  Google Scholar

• Gureckis, T. M., Martin, J., McDonnell, J., Rich, A. S., Markant D., Coenen, A., . . . Chan, P. (2015). psiTurk: An open up-source framework for conducting replicable behavioral experiments online. Beliefs Research Methods. Advance online publication. doi:10.3758/s13428-015-0642-eight

• Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the earth? Behavioral and Encephalon Sciences, 33, 61–83. doi:ten.1017/S0140525X0999152X. disc. 83–135.

  Article  PubMed  Google Scholar

• Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of Board of Scientific Affairs' Advisory Grouping on the Conduct of Research on the Internet. American Psychologist, 59, 105–117. doi:10.1037/0003-066x.59.2.105

  Article  PubMed  Google Scholar

• Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship betwixt motivation, monetary compensation, and data quality among United states of america- and Bharat-based workers on Mechanical Turk. Behavior Research Methods, 47, 519–528. doi:10.3758/s13428-014-0483-10

  Commodity  PubMed  Google Scholar

• Mason, Due west., & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical Turk. Behavior Enquiry Methods, 44, i–23. doi:10.3758/s13428-011-0124-6

  Article  PubMed  Google Scholar

• Nosek, B. A., Banaji, One thousand. R., & Greenwald, A. G. (2002). Harvesting implicit group attitudes and behavior from a sit-in spider web site. Group Dynamics: Theory, Enquiry, and Practice, half-dozen, 101–115. doi:10.1037/1089-2699.vi.1.101

  Article  Google Scholar

• Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Conclusion Making, 5, 411–419.

  Google Scholar

• Peer, Eastward., Paolacci, 1000., Chandler, J., & Mueller, P. (2012). Selectively recruiting participants from Amazon Mechanical Turk using Qualtrics. Unpublished newspaper. Retrieved from ssrn.com/abstract=2100631

• Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation equally a sufficient status for data quality on Amazon Mechanical Turk. Behavior Inquiry Methods, 46, 1023–1031. doi:10.3758/s13428-013-0434-y

  Commodity  PubMed  Google Scholar

• Rosenzweig, C., Litman, Fifty., & Robinson, J. (2015, May). Getting representative data on MTurk? Symposium presented at the meeting of the Clan for Psychological Scientific discipline, New York, NY.

• Shapiro, D. Northward., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220. doi:10.1177/2167702612469015

  Commodity  Google Scholar

• Sprouse, J. (2011). A validation of Amazon Mechanical Turk for the drove of acceptability judgments in linguistic theory. Behavior Enquiry Methods, 43, 155–167. doi:ten.3758/s13428-010-0039-7

  Article  PubMed  Google Scholar

• Von Emden, R. (2015, July 15). Exploratory TurkPrime study: Windfall bias—Young versus erstwhile [Web log mail]. Retrieved from world wide web.pavlov.io/2015/07/15/the-effects-of-age-on-the-windfall-bias/

Download references

Writer information

Affiliations

1. Lander College, Flushing, New York, USA

   Leib Litman, Jonathan Robinson & Tzvi Abberbock

2. Department of Psychology, Lander College, 75-31 150th Street, Flushing, New York, 11367, USA

   Leib Litman

Respective author

Correspondence to Leib Litman.

Additional data

Leib Litman and Jonathan Robinson share first authorship of this article.

Rights and permissions

Open Access This commodity is distributed under the terms of the Artistic Eatables Attribution 4.0 International License (http://creativecommons.org/licenses/past/iv.0/), which permits unrestricted employ, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were fabricated.

Reprints and Permissions

About this commodity

Cite this article

Litman, Fifty., Robinson, J. & Abberbock, T. TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behav Res 49, 433–442 (2017). https://doi.org/10.3758/s13428-016-0727-z

Download citation

• Published: 12 April 2016

• Issue Engagement: April 2017

• DOI : https://doi.org/ten.3758/s13428-016-0727-z

Keywords

• Mechanical Turk
• Crowdsourcing
• Online research

reamsagnight.blogspot.com

Source: https://link.springer.com/article/10.3758/s13428-016-0727-z

Share this post