Mechanical Turk, Research Ethics, and Research Assistants

You may also like...

5 Responses

  1. David Zaring says:

    Huh – my sense, based on chats with experimenters around my university, is that Mechanical Turk is closing on becoming 50% a venue for social research. Not sure how they got the relevant HSR approvals, but I think they’ve done it in spades. So you’ll have company if you do it.

  2. John Horton says:

    There’s been quite a bit of discussion about these issues
    in various places.

    On the ethics/legal aspects of crowdsourcing, there is
    Alek Felsteiner’s paper:

    As well as his post on the Crowdflower blog focusing on regulation (as well as some back-and -forth in the comments):

    There was a debate at HCOMP 2010 on fairness of crowdsourcing (stimulated by a paper by Six Silberman et al.):

    Here’s my own little paper investigating what Turkers seem to think of their employers:

    Regarding experiments, I also have a paper similar to the one you linked to, re: conducting experiments online:

    Anecdotally, there seems to be lots of variance in what different universities require to gain IRB approval.

  3. Do you see MTurk as different from other online or internet-based data collection methods (like Mahzarin’s implicit bias website or others)? Over the last 15 years or so such internet-based methods have become common in a number of social science disciplines. As a result there’s been much discussion on both ethics and practice–how to ensure informed consent, how to ensure privacy, how to ensure data quality, whether respondents differ from in-person respondents, etc. (Here’s one of many examples:

    More broadly, though, I don’t think there is (or should be) a question that such respondents are research subjects within an IRB’s purview. Not all researchers know or think to get IRB approval, though, which is a separate issue….

  4. Broken Turk says:

    Mechanical Turk is a phenomenal example of the power of the internet to leverage all of the world’s intelligence and human processing surpluses. Massive untapped resources. However, Mechanical Turk’s technological potential has advanced faster than its systems for managing users (workers), and a look through the forums where Turkers talk shop reveals an embarrassing litany of frustrations, sudden blocks and bans, uncertainties, and customer services incidents that Amazon would never stand for in its flagship site. I’ve documented the most poignant of these criticisms at a blog called Broken Turk (

  5. FuckYouTurker says:

    The science you idiots are generating after fucking over your participants with horseshit pay in order to further your careers is a joke. You put up 15 pages of questionnaires wanting authentic responses after shitting in the face of you participants by paying them 5 or 10 cents. You pay garbage, you get garbage, and your results or laughable horseshit. I’d throw any finding from turk subjects in the trash. Why should YOU further your career at the expense of bad science and exploitation. Fuck you people.