I run surveys on Mturk and I have two questions: 1. I use some standard attention filter questions to catch people who randomly respond to questions. The point is that if someone is not actually answering questions they shouldn't be paid. Since some filters are harder than others, and mistakes do happen, instead of rejecting work based on attention filters, I ask people to return the HIT when a filter question is failed. I received a nasty-gram recently saying that returning HITs is almost as bad as a rejection, and that it negatively affects their ability to do HITs in the future. I am not aware of any qualification that limits based on return rates, or of any way for a requester to access the return rate of a worker. Amazon told me that return rates are displayed to workers for informational purposes only, but I wanted to double check with the community. Does returning HITs negatively impact workers besides the unpaid time spent on the HIT? 2. I maintain an outside database of workers so that I can do custom qualifications without using the API. However, to see if a worker qualifies for my HIT, I need to look them up using their workerId. To my knowledge this is only accessable after a HIT is accepted. Is this true, or is there a way to access the workerId programmatically before the HIT is accepted. Second, if not, and if I wanted to avoid having people accept the HIT, find out they do not qualify and ask to return the HIT, is there a better way to get their workerId than asking a worker to type it in?
Hello. Maybe I am not the best person to answer this but since there is no replay... Returning the HIT is definitely nothing compared to rejecting a HIT. I never saw any HIT's asking for "abandonment rate" as a qualification. Or I did and it was something ridiculously low (almost impossible to have). You can return HIT's all day as I noticed without any consequence which I cannot say for rejection. Rejection ruins worker's account completely and it can get it banned.
" Does returning HITs negatively impact workers besides the unpaid time spent on the HIT?" Nope. I took surveys for a while and it was nothing more than an extreme annoyance to find out I don't qualify for a HIT and be asked to return it. I was never told to return it if I failed quality checks - that would probably burn more (since I pay attention), but to have taken the time to do the survey and then be told to return it -- that's where you're going to get nasty-grams. People won't believe that you're not using their data. They will think this is your way of getting answers for free. Doesn't mean it's true, but perception is reality. There isn't a way to find someone's ID before they've accepted a HIT. You can set a qualifier for your HIT. You can have people complete a test, a questionnaire, a task, an essay - probably anything you'd like. I'm not a requester but I do understand your dilemma with wanting to keep the quality control high for your surveys without pissing people off in the process.
The alternative to asking them to return the HIT is to reject it. I figure they like returning better than rejecting. I clearly state that the survey contains filters in the opening, and most filters are dreadfully easy (e.g. Select answer 3.). The qualification tests/essays etc. can only be created through the programming API, and I don't have the time to learn some .NET programming language to figure out how to use it. Though I do see the annoyance of accepting a HIT to find out you don't qualify. That's what Im trying to improve right now. I think Im going to start having people enter their ID before accepting to check if they already qualify. If they don't they can take a test to qualify themselves. If they qualify they can accept the HIT. It will then double check if they qualify (i.e. check to make sure they entered the correct id number etc.) and if they need to take the test again cause they copy and pasted wrong they can waste their time if they wish since it was their own fault.
Or you could have a qualification "test" where there's a link to a survey off-site, just like the ones you do now. I'm sure there are options. Just suggesting - not recommending anything. I wouldn't want to deal with the masses when it comes to making money, that's for sure.
I would like to share my views.... not sure they will work. The attention question... The survey must not proceed until the worker correctly does what is intended. A message should flash on the screen that he needs proper attention to proceed further. If that is repeated for more than three times, you can ask the worker to return the hit mentioning the reason. But one thing you people must take care, not all..... though: Mention a justifiable time in which it can be completed or better donot mention. Some hits mention quick / 5 mins survey, but in no way that could be completed in 5 mins. In a case I took a survey mentioned 10mins, duration 60 mins, I could complete the same only after 45mins. I did that with mu usual speed but.....for 15cents. too bad. Please don't forget to mention the page numbering (1 of n pages).. Some requester mention the number of questions that will be asked.... that is also okay. Make your survey simple and avoid to0 much of repetitive questions which makes the worker to feel bored and to miss the attention question.
1. I have never seen a HIT that uses the return rate as a qualifier. Since rejections do count against us, I would rather have a return than a rejection. In fact, I don't know of any turkers who would choose a rejection over a return. 2. I agree with Whimsy's suggestion. You could put something like "Qualified workers only" somewhere in the title. When the HIT is viewed, at the top you could put the link to your database/check outside of mturk. That way, no one can blame you if they accept the HIT without checking to see if they are qualified first.
I agree here. I would much rather have a return than a rejection. Me personally I do not have a problem with accepting a hit to see if I qualify. No big deal I just return it if I do not meet survey requirements. The only time I have seen any hits requesting certain return rate was 20%. I can not imagine being able to work my return rate up that high. Take care and good luck with your project.
I have seen some surveys where the list of Amazon Turk IDs was actually listed in the hit description and you could check yourself. I've seen others where there is an external link to a file that has all the IDs in them and you can check yourself. The alternative is to make sure that you tell us in the HIT description that there will be a qualifying question and you may be asked to RETURN the HIT. Full knowledge and they shouldn't moan. I appreciate you not wrecking our work chances by even asking. Thank you!
Another idea: Post your HIT for a somwhat nominal amount. Pay everyone but give bonuses to the people who pass all the quality checks.
ive thought of that whimsy, and this is how i would ideally like to do it (its how most panel survey companies work). However, in the mturk web interface you have to manually give bonuses individually. It takes like 30 sec to a min for each 1. If I post a survey for 500 people it would take an entire day to bonus people. If they can come up with a way to do it all at once like the acceptance/rejection process goes, that is definitely my go-to method.
I'm positive there is a way around doing this by hand but I don't know how to direct you. The work I do for CastingWords and previously SpeechInk - a majority of the money is made through bonus payments. Transcription is submitted Grade is given Payment is a base amount plus a bonus that's determined by the base pay and a grade. Between the different job types, that could be hundreds a day - but I receive mine once the transcription has been graded. If your quality checks had a value of zero (pass) or 1(fail) Each value added. Zero is complete pass. If you've got a friend that's got a couple hours to write you some code, it could be very beneficial. I'm just trying to let you know that I believe there's a way to automate the giving of bonuses. 'Scuse any typos. Writing from my nook while gaming bots take overmy computer
the automated way is done using the .NET API. I know some web and server programming, nothing close to a .NET language. If I ever befriend someone who can help me I'll look into it.
When you say Web and server programming, what tools were you using? The Mechanical Turk developers documentation says that there are bindings for four programming languages available: C#, Java, Perl and Ruby, as well as a suite of command line tools. The C# stuff is really just meant for the .NET Common Language Runtime, so I'll bet you could translate the .NET stuff there to any programming language that has a .NET implementation (the one I'm currently thinking about is IronPython). The command line tools, I suspect you could chain together in a Windows batch or Linux shell script with some other tools to accomplish what you want. I've not read the developer and API docs closely nor completely, but I'm just suggesting it's possible that you can leverage your existing skillset to accomplish what you want to do.
If requesters started to use a Return rate to qualify people, I think that would freak a lot of people out. For those requesters that have "all work done, please return hit" - Less would do those for fear of hurting the rating. Same goes for ones that use "majority rules". I can be really good at reading bad handwriting, fuzzy numbers, making out words from half cut off, or store names when you see "Krog" and Know what store it is but get rejected for "cant see full store name". - You have to return these for fear of rejection, even though you could provide very good information. (Not talking bad about anyone) - But having English as a first lanaguage does help figure out a lot more than if it was a second language. (This is why I stear clear of 95% of all Spanish and German hits, I may miss something!!)
Ridiculously Long-Winded Response to Question I did some more reading and as best as I can tell, there is no way for you to get a worker's ID before they accept a HIT from you. I think that your best solution will wind up including elements from both MTurk's qualifications API and validation against your database when workers accept HITs. As long as a worker has submitted a HIT to you in the past, you can grant a qualification to that worker without having them request it first. See the AssignQualification operation in the API reference - http://docs.amazonwebservices.com/A...piReference_AssignQualificationOperation.html What I envision is, you build a list of workers from your database, then create a custom qualification for your specific HIT. Assign that qualification to everyone on your list, then use the NotifyWorkers API call to let them know that you have a HIT specifically for them and for which they're pre-qualified. (You should also clearly and prominently mention that they're only allowed to work on the HIT once and that multiple submissions will result in rejections.) Once one of them works on your hit and submits it, call the RevokeQualification API function with that worker's ID so they can't take it again. There are other safeguards you could put in place. If you're serving up the pages for the HITs from your server, you could check the incoming HTTP GET request for the worker's ID and instantly compare it to the list of workers that have submitted the HIT, then send them a message that they aren't allowed to work on the HIT more than once and tell them they must return the HIT (this is a variation of what I think CrowdSource and some others do). I know this is what you're trying to avoid, but MTurk doesn't seem to allow you to create a HIT limiting a worker to only completing it once. (Evidently, it's a discontinued function.) See also these threads on the AWS Discussion forums: Prevent workers from doing survey more than once ? at https://forums.aws.amazon.com/thread.jspa?threadID=63786 and Avoid a worker doing more than one assignment per HITtype at https://forums.aws.amazon.com/thread.jspa?messageID=169416. The latter discusses the now-deprecated SetWorkerAcceptLimit function, which would have made exactly what you're trying to do a snap, instead of having to jump through all these hoops.
once I man up, find the time, and learn a language that can use the API, Im sure this will all be a breeze.
More Long-Windedness A little more information. If you create HITs through the MTurk HIT design system, the only qualifications it seems you can validate against are Location, Hit Approval Rate %, Adult Content Qualification, and Number of HITs approved. However, the API for the QualificationRequirementStructure includes the following qualifications: PercentAssignmentsSubmitted - an integer representing the ratio of the number of hits a worker has submitted (S) to those they have accepted (A)- INT(S/A) PercentAssignmentAbandoned - an integer representing the ratio of the number of hits a worker has abandoned (i.e. allowed the time limit to expire) (B) to those they have accepted (A) INT(B/A) PercentAssignmentsReturned - an integer representing the ratio of the number of hits a worker has returned (R) to those they have accepted (A)- INT(R/A) PercentAssignmentsApproved - an integer representing the ratio of the number of hits a worker has had approved (V) to those they have submitted (S). This directly corresponds to the Hit Approval Rate in the Mturk HIT design system. - INT(V/S) PercentAssignmentsRejected - an integer representing the ratio of the number of hits a worker has had rejected (X) to those they have submitted (S) - INT(X/S) If more requesters start using the API versus the HIT design system to create their HITs, then I can see where return rate will start to become an issue. To my mind, the only ones that should matter are PercentAssignmentAbandoned (the worker can't finish his work fast enough, or takes on more than he can handle) and PercentAssignmentsApproved/PercentAssignmentsRejected (the worker can't do the work correctly). However, being a worker, I am biased.