Authors

Public Pad

Latest text of pad Vjky1itZKd
Saved Nov 18, 2009
For those who haven't hear, Mechanical Turk is a "marketplace that enables computer programs to co-ordinate the use of human intelligence to perform tasks which computers are unable to do" (via Wikipedia). You can think of it as an online wanted ad page where the jobs look something like "I'll give you 2ยข if you tell me if there is a dog in this picture: [picture that may or may not contain a dog]." The basic idea is that while computers are really good at a lot of things (modular arithmetic, matrix operations) they are also really terrible at a bunch of things people happen to be good at (pattern recognition, anything involving social context). You can read more about it on the wikipedia page(http://en.wikipedia.org/wiki/Amazon_Mechanical_Turk ) or at the main page(https://www.mturk.com ).
 
I've been using said service to describe some pictures I'm using in my current project and I have to say that I'm very happy with the service. It would take me hours to do what the service was able to do in just a few minutes and I can focus on the guts of the project instead of the bitch work (no offense mechturk workers!). However, I quickly found that you can't always trust the answers provided.
 
I'm not sure if it's because some people are trying to take advantage of the system by using bots to answer questions (which is obviously against the whole idea of the service to begin with) or some of the workers are just lazy and hope you wont notice that you're paying them to do nothing. But it comes down to really terrible answers. Luckily I have a few ways you can prevent these terrible answers.
 
 
Job properties
 
The first thing you're going to want to do is set your qualifications high. I would suggest higher than 80% (meaning only workers with an approval rating of 80% or higher can answer your questions). Also set a low assignment duration -- something like 15 minutes. The assignment duration is the amount of time the worker has to answer your question once he/she has started it.
 
I got these tips from one of the people that worked on some of my jobs. Setting these qualifications will make it more difficult for bots to answer your jobs. Apparently some of these bots grab 400+ jobs before they start working through them. This blocks legitimate workers from taking those jobs (and giving you good results). However, with a 15 minute duration if the bot can't get to your job in less than 15 minutes then the job is released and the bot is no longer able to work on it.
 
 
Automated rejections
 
If you're releasing 300+ jobs at a time you're going to want to automate your rejections. Before releasing the jobs make sure you have some ways of automatically rejecting ridiculous answers. You also want to stress the criteria for rejection in the instructions for the job so that you have something to point at in the rejection comments.
 
Stressing the criteria for rejection also ensures less false-positives. I've rejected a few submissions that just missed the criteria and felt pretty terrible about it.
 
 
Multiple assignments
 
Mech Turk gives you the option of allowing multiple submissions for any job. Use this feature to get information from many different people. I figured that at least 1/3 of the submissions were legitimate and so have started allowing 3 submissions for each job. Also, for every rejected submission I just add another assignment (which allows another submission) and hope that the new one will yield better results.
 
 
Decoys
 
For those of you with a budget too small for 3 assignments per job try setting up a decoy question for each job. This is just a question that you already know the answer to and so can automatically reject any submissions that miss this question. I've stopped using this method because it feels a little mean-spirited and could lead to increased false-positives, but generally it worked.
 
 
Silly requests
 
One way to make sure tha the instructions are carefully read is to include a silly request with each answer. For free response answers it can be something like "make sure the last word in your response is 'watermelon'." I got this idea from Van Halen(http://www.snopes.com/music/artists/vanhalen.asp ). Again this could increase false-positives, but it will make sure that the workers are reading carefully.
 
 
Check spelling
 
Plan on checking the spelling of words in answers because people tend to make plenty of mistakes. In my last batch of jobs I had a total of 632 submissions from workers, the sum of all words in the answers for all the submissions was 6,233 and 2,182 of these were misspelled. That means that 35% of the words used to answer my questions did not exist. Again, be warned: check your workers' spelling. Here's a simple script(http://snippets.dzone.com/posts/show/7457 ) written in Ruby to get you started.
 
 
Ask for feedback
 
I would suggest that you put in a non-required question asking workers for feedback. And don't just throw that answer away, take a look at all of them and take it into account when pushing out the next batch of jobs. You never know what kinds of problems your instructions have until you have to act on those instruction with no knowledge of the purpose of the job.
 
 
Conclusion
 
In the end you want to be as fair to the legitimate workers as possible. This means reducing false-positives, asking good questions, and listening to their feedback. Also when starting out I'd suggest you set a long auto-approve time (I started with 3 days) to give you time to adequately review and save the submissions. Anyway I hope this helps!