You are here

SAGE online ordering services and account tools will be unavailable due to system maintenance beginning Saturday, December 14 at 1:00am GMT until Sunday, December 15 at 8:00pm GMT. If you need immediate assistance, please contact SAGE Reception on +44 (0) 20 7324 8500 or info@sagepub.co.uk and they will assist in forwarding your query to the correct SAGE Department. Thank you for your patience and we apologise for the inconvenience.

Conducting Online Research on Amazon Mechanical Turk and Beyond
Share

Conducting Online Research on Amazon Mechanical Turk and Beyond



June 2020 | 260 pages | SAGE Publications, Inc

Conducting Online Research on Amazon Mechanical Turk and Beyondwritten by Leib Litman and Jonathan Robinson, provides researchers with information on essential aspects of this important research platform. Drawing on their experiences creating and running TurkPrime, an online platform designed specifically to connect applied social science researchers with Amazon Mechanical Turk, the authors hone in on common researcher pitfalls and key information researchers need to know before beginning their studies. First, this book is a step-by-step technical manual of how to use MTurk, starting from the beginning and focusing in on more challenging aspects of research like longitudinal designs and obtaining representative samples. Screenshots of the platform walk readers through each part the research process, from setting up an account to evaluating data quality. Second, the book is a profile of MTurk systems and cultures, drawing from the authors’ own research into the platform and data from TurkPrime. Studies from disciplines like psychology, business, political science, computer science, sociology, communications and others have used MTurk for participant recruitment. The aggregated data on hundreds of thousands of participants and tens of millions of tasks in this text provides the clearest snapshot to date on participant characteristics in MTurk. Finally, the book is a contribution to the dialogue on the development of MTurk, detailing issues of ethics, bias, effectiveness of research, and comparisons to other online methods from a deeper perspective. Rarely-addressed technical issues like APIs, pay rates, and sampling methodologies offer concrete courses of action for researchers. This book is applicable to novice and experienced researchers alike, and is an excellent source of information for researchers wanting to know more about how to best use Amazon Mechanical Turk.

 
Chapter 1: Introduction
A Scientific Revolution in the Making

 
A Brief History of Online Research in the Social and Behavioral Sciences: From HTML 2.0 to Mechanical Turk

 
The use of online samples in applied behavioral research

 
Amazon Mechanical Turk

 
Leib Litman, Cheskie Rosenzweig, Jonathan Robinson
Chapter 2: The Mechanical Turk Ecosystem
How Quality is Maintained

 
Reputation Mechanism

 
Selectively Recruiting Specific Workers

 
Protections for Workers

 
Communicating with Workers

 
A Worker’s Perspective

 
Worker Communities

 
 
Chapter 3: Conducting a Study on Mechanical Turk
Sample Project

 
Setting up a Requester Account on Mechanical Turk

 
Creating a HIT

 
The ‘Design Layout’ tab

 
Monitoring Progress on the Requester’s Dashboard

 
When a Worker Runs Out of Time

 
Sample Study Results

 
Conducting Follow-up Studies Using Requester-Issued Qualifications

 
Appendix A: Checklist for best practices of setting up a Mechanical Turk HIT

 
 
Chapter 4: API and Third Party Apps
Third Party API-based Platforms

 
Common uses for API scripts and third party API-based apps

 
TurkPrime

 
Jesse Chandler, Gabriele Paolacci, David Hauser
Chapter 5: Data quality Issues on Mturk
Defining and Measuring Data Quality

 
Measuring Individual Participant Data Quality

 
Causes of and Cures for Poor Data Quality

 
Concluding Thoughts

 
 
Chapter 6: Who are the Mechanical Turk Workers?
Sources of data

 
Location of Workers in the US

 
Demographics of Mechanical Turk

 
 
Chapter 7: Sampling Mechanical Turk Workers: Problems and Solutions
Sampling on Mechanical Turk

 
Sources of sampling bias

 
The Problem of Superworkers

 
Time-of-day Effects

 
Pay Rate

 
Dropout

 
Sampling Best Practices

 
 
Chapter 8: Data Representativeness of Mechanical Turk samples
Representativeness, surveys, and survey sampling

 
The methodology of survey sampling

 
Mechanical Turk as a sampling frame

 
The fit-for-purpose framework

 
Chapter Overview:

 
 
Chapter 9: Conducting Longitudinal Research on Amazon Mechanical Turk
Why Longitudinal Research?

 
Retention, Longitudinal Research, and MTurk

 
Case Studies

 
Best practices for longitudinal research

 
 
Chapter 10: Beyond Mechanical Turk: Using Online Market Research Platforms
Limitations of MTurk

 
Online probability-based panels

 
Online market research platforms

 
Overall comparisons between Mechanical Turk and market research platforms

 
 
Chapter 11: Conducting ethical online research: A data-driven approach
Historical background

 
Risk of harm in online research

 
Research on sensitive topics

 
A deeper dive into controversial and complex issues

 
Economics of Mechanical Turk: Considerations for setting wages

 
Setting wages: Considerations of ethics and methodology

 
Considerations for rejecting, blocking and disqualifying workers.

 
Practical advice for requester/worker interactions

 
Anonymity

 
 
Appendix A
Michael Hall, Neil Lewis, Jesse Chandler
Chapter 9: Conducting Longitudinal Research on Amazon Mechanical Turk
Leib Litman, Jonathan Robinson, Cheskie Rosenzweig
Chapter 6: Who are the Mechanical Turk workers?
Leib Litman, Jonathan Robinson, Cheskie Rosenzweig
Chapte 10: Beyond Mechanical Turk: Using online market research platforms

“This is the best practical guide I have seen written in any methodologically written book!” 

Michaela Porubanova
Farmingdale College

“Great overview of literature developing in several fields; it's hugely valuable to have all of this in one place.” 

Kevin Munger
New York University

“They cover the whole gamut of history/background, practical concerns and scholarly concerns.” 

See-yeon Hwang
Sam Houston State University

"I am looking forward to being able to hand this to my students and know they'll be ready to get started on MTurk and really understand the whole system (i.e., not just the mechanics of getting data, but really "getting" the culture and so on). I would probably also find myself citing this book as it discusses many of the issues I personally research.” 

Alice M. Brawley
Gettysburg College

Preview this book

For instructors

This book is not available as an inspection copy. For more information contact your local sales representative.

Purchasing options

Please select a format:

ISBN: 9781506391137
£35.99