Hi guys, I'm a PhD student (Requester: Fabiana Thiel), and this is my first time using mturk. I read every possible topic on being a good requester i could lay my hands on. I'm paying fairly (6-7/hour) and that was hard to get talking about academic funding. I reply every single email. I try to approve the very next day every HIT. When someone fails the attention check, I point it out, including which was the answer the person gave me instead of the supposed answer. I'm very clear on the instructions. I unrejected a case when i first put up the hit and there was a mistake as soon as the worker pointed it out, no questioning. And now this.. I have a worker who missed a VERY CLEAR attention point, at the beggining of the study, she emailed me saying she was running out of time, which is a lie, since she completed her study in 18 minutes (1 hour HIT). Everybody knows I can't use this data in academic research since quality must be high, and now she rated me at TO as someone who uses data for free. I have very little reviews in TO since this is my first study. How can I avoid this kind of worker? Can this review hurt my HIT by making people stay away from it?
Hi sir, Nothing to worry about a bad review. If 100 turkers do ur hits n only one gave u a bad review then nothin to worry. No one will take that review a serious one. I am 1.5 year old in mturk n done ariund 50000 hits. I never considered reviews which are less than 5 turkers.
We check how they rate the other Hits they have done. Do they give them a poor rating also? Is this what they do? If so then I, at least, ignore those.
Yes, it's someone who appears to give bad reviews all the time. But i talked to him and explained again why i needed to reject his work and he think is fair to reverse bad review, thank you all for the help. Being new at this is not easy! Fabiana Thiel
Yeah don't give in to those that didn't do their work properly. Stick to your guns. She missed the attention question, it's her fault. You shouldn't have to pay for someone who isn't reading the survey thoroughly. You have an option to go to the reviews and state your side of the story. I've seen requesters reply back to reviews. That makes it more credible. Like the others say, we judge the bad reviews and depends on what it is, we ignore it mostly and if they have alot of good reviews, it's not a problem.
Honestly rejections suck but if you do surveys a lot you know most of them have AC's in them.. if you miss one it's really no one's fault but your own. I know especially with University surveys funding is hard to come by so I certainly wouldn't blame you for not paying for something that wasn't done properly. Just my .02
You could put more restrictions in place, like only 10,000 HITs approved - but at that point you're cutting out too much of the "workforce". Survey takers and writers have a much lower HIT count since they can't plow through a HIT a minute. One idea You can have a survey that pays a low rate (10 to 25 cents) and reward bonuses for people who pass all of the attention checks. The logic is there, but the process would still be flawed...survey takers KNOW what to look for in attention checks. Another idea When a worker passes all of the attention checks, you could assign them a qualification. Following surveys could require the qualification. That will also cut down your worker pool significantly. Overall, I wouldn't change what you're doing. As others have said, it isn't just the review people look at - but also the review history. Savvy turkers know it isn't reasonable to just take everything at face value. Don't let workers hold you hostage by threatening a bad TO review.
You might want to set a minimum for accepted hits (if you haven't already) along with a 99% or higher acceptance rate I think that helps requesters (many times---but not every time) I've been turking since October, and I have 2200 accepted hits, and a 99.8% acceptance rate, so if you set it for 2500 hits, you will most likely get people who have at least several months of turking under their belt and are serious about their work. There are exceptions to this of course, no big group of workers will be perfect. But I also don't think that a poor review will hurt the # of people who will do your hits. You can also set your hits up only for people with the Master's Qualification. They have tens of thousands of hits completed and years of experience. Most people on MTurk don't have that Master's Qualification though, so you are shrinking your pool when you set this up. Many workers without a Master's Qualification will do just as good of a job with your hit. Good luck with your work, it is neat to hear from the Requester's side of things. A good number of mturkers take their work very seriously.
Best advice IMO. I've read complaints where the requester responds and it's nice to see both sides of the story. People that have been doing surveys for awhile will see right through the nonsense.