Greetings. I'm an academic researcher in Japan where I study about second language fluency development. I'm new to MTurk, though one small batch I did last year to get my feet wet got a mention here. Now I'm getting ready to release a much larger set of hits for my research and I'd like to get some feedback on the survey design before requesting it officially. I've set up a draft of the hit page here on my university web site. The form is disabled, and except for the instructions, the audio files are disabled. However, you should be able to get an idea of what I'm asking. I'd appreciate your thoughts on any of the following questions. 1. Is the pay attractive enough? I gather from skimming various posts that 3$ for a 30-minute hit is about the minimum fair amount. However, after the first hit, I think people can finish hits in other batches (7 total) in about 20 mins. So, if they do a hit in all seven batches, they can get 21$ + 5$ bonus for total 2.5 hrs work -- about 10$ an hour. Is my thinking on target? 2. Does the hit seem too hard? After reading several posts, I get the impression that workers really prefer to know all the details in advance. So, I've leaned toward lots of explanation about the task, how to earn a bonus, what conditions might lead to rejection, etc. But I'm a bit afraid that since I'm an unknown right now (no TO, yet), workers will be hesitant to accept. What do you think? 3. I'm thinking a good strategy is to start by getting a small number of masters' level workers (>=95% accept rate, in US/Canada) to do the hits, check the results, and then open it up wider to get a large number of hits from lots of workers. For my purposes, it's ideal to have a small number of workers do all seven batches rather than than have a large number of individuals doing only 1-2 hits each. Are there many master workers who would be interested in this hit? Thank you very much for your time and for helping out a novice requester. I really appreciate your feedback. Sincerely, rlrose
Your compensation is reasonable and your task will be completed quickly at this rate. Your instructions are excellent and comprehensive. The examples will be very useful. Workers want to give you the best possible product/results, so complete and thorough instructions as great. Given the scope of your task, I don't think workers will hesitate based on the long instructions. Being transparent about your approval/rejection policy and reaching out here on the forum will go a long way towards establishing trust with the worker community. For your test batch below, consider adding "If you enjoyed my task, please leave a rating at https://turkopticon.ucsd.edu/". Hopefully this will garner some TO ratings in time for the main batch. The masters qualification is a marketing ploy by Amazon designed to separate you from your money. The masters qualification is not necessarily indicative (no offence to some of the great masters workers out there) of experienced workers, as seen here and here. In a recent study, workers with 10,000 approved HITs and a <=98% percent substantially outperfomed those with the masters qualification. I would recommend using these settings (<10,000, <=98%) for the best results, as this attracts workers who have experience and demonstrate accuracy. Good luck with your project and I look foward to working with you.
I like the idea of your HIT. The pay is fair and reasonable. I love the detailed instructions and examples. I'll echo what the poster said above me, Master's don't necessarily mean good workers. Amazon seems to assign that qual at random. Correct me if I'm wrong but the better indicator of good workers is; Number of Approved Hits and Approval Percentage Rate.
Thanks for your replies, swiftkick and LiquidMist. I appreciate the thoughtful comments. I hadn't realized that Master's might just be a marketing scheme, so your advice has helped me to re-think what pool of workers I should target. One quick clarification, though: Based on the study you linked, I think you mean >10,000 and >=98% (i.e., completed HITs > 10,000 and acceptance rate >=98%), right? Again, thanks for your time. I should post the official HITs within a day or two and will update this thread when I do. Sincerely, rlrose
I'd be very interested in working on these hits. The only thing that would prevent me from doing so would be the "Location is US/CA" stipulation. I'm from Great Britain. Is there any way you could open them up to include us too? And maybe other commonwealth countries that have English as their native language? If that's at all possible, I look forward to the hits!
Thanks, iceblink. Yes, I'm thinking about that. Of course, I mean no offense by limiting it to US/CA: People from other countries should be equally capable of evaluating fluency. My plan right now, though, is to open up a small number of hits to a somewhat narrower group and check to see that there isn't a task design problem. For this purpose, a rather more homogenous group of speakers is better (yes, I know; US/CA isn't all that homogenous, but more so than no restriction). Then, as a second stage, open up a large number of hits to a broader group. I hope then you'll be able to work on them. Sincerely, rlrose
Thanks, heyscotttt. I'm finalizing them now. I'll put another notice up here when they go live. Sincerely, rlrose
Thank you very much to all who viewed this thread and especially to those who took time to reply. It was very interesting and helpful to me as a new requester. This is just to let you know that my HITs will go live shortly. Search for "Listen to nonnative English speech and rate fluency (group x)", where x=1 to 7. At the moment, there are only 10 HITs in each group, but after those are done and I review them, I will offer many more (perhaps 40 per group?). So, even if you don't earn the bonus in the first round, you can pick up where you left off in the second round. Thank you in advance for your work! Sincerely, rlrose
Thanks to all of you who have done my HITs so far. They seem to be disappearing quickly. I have one question, though: Although my instructions clearly say that workers should listen to each recording in full (at risk of rejection), the data I'm getting suggests that several people are listening to some of them for only a few seconds, then either stopping or skipping ahead to later in the recording. There are 38 30-second clips meaning it should take a minimum of 19 minutes, yet some people finish in 15 minutes or less. Now, I'm not totally confident about this: I need to check the results more closely, but this seems to be the case. Can anyone shed some insight on why this might be? Thanks a lot, rlrose
Workers will always try to do the work as fast as possible. To us, time is money. I have been starting the next recording at about 28 seconds of the preceding one because there is often a lag at the beginning. At best that shaves a minute or two off of each hit. Also, some workers will risk a rejection on what they perceive as a high paying HIT but disregarding the instructions.
Question...is that really your voice on the AC's? You sound like you could be a reader for audiobooks, if that's really you
One last question. Did you post all 7? I only ever saw 6 and wasn't able to get your number 1. I always hate to miss out on any bonus opportunity, LOL
Thanks bobbie01, That helps me understand a lot. I regret, though, that the HITs that workers finished in far less than 19 minutes are basically useless for me. I really don't like to do it, but I think I'm going to have to reject several HITs at that level. I won't block anyone though, so if someone wants to do it over, they may; especially if they're close to the bonus. For the main round of HITs that I'll release later, I will definitely highlight/boldface/emphasize even more that workers have to listen in full. Sincerely, Ralph PS. Yup, that's me on the AC's. Actually, I don't think my voice is so good. That's why I put the music there --- to distract people. As for me, I like the sound of that music more so than my voice... PPS. I did post 7 groups of HITs, but they all got snatched up quickly. There may be a handful more shortly (i.e., the above-mentioned rejections), so keep an eye out. Anyway, if you miss those, there will be a much larger second round (perhaps 40 HITs per group) after a week or so.
Ralph, I caught 4 of the 7 yesterday and found them interesting and I enjoyed doing them. If you are getting a lot of bad data a suggestion you could do would be to make it a closed qualification. You could issues out qualifications to people who did the work up to the standard you were looking for then in future hits add that qualification as a requirement for the hits.
Ralph, I don't think anyone likes the sound of their own voice but you have an easy voice to listen to. I listen to a ton of audiobooks so I know what a bad voice sounds like, LOL I also loved that music and appreciated that you put what it is on the HIT.
Thanks, heyscotttt. Glad you could do the HITs. I appreciate it. Yes, I have thought about that using a qualification. However, that might not work for my current project since the round of HITs I just finished to too small. I mean, if I issue a qualification to those who did it well and then release the next large round, there won't be enough people to do the work. However, I expect to do more of these kinds of fluency evaluations (or similar speech evaluation) tasks in the future so I'll probably give a qualification to the good workers from the current jobs and then rely on those people for future projects. Sincerely, rlrose
Thanks, bobbie01. The music was released by the original artist under a Creative Commons Attribution-ShareAlike license, so I felt obliged to give proper attribution in context. Sincerely, rlrose
For those who have been following this thread, I'd like to let you know that I'm getting ready to release a much larger set of HITs in an hour or so. Workers who have already done some of the seven groups can do the remaining groups in order to get the bonus. There are a much larger number of HITs this time (40 per group), so it should be a little easier to get. I have sent an e-mail to those who did HITs before to tell you which groups you've already done. So, please check your mail before accepting any of these new HITs. Thanks, rlrose