TurkOpticon - a site that allows workers to rate requesters. We can look up requesters by their ID number and read reviews, leave our own reviews, etc.
It's short for Turkopticon. It's a website/forum where we grade requesters based on pay, communication, fairness, approval time, etc. It's a script downloaded to Amazon mechanical turk and you are able to see the score before accepting the Hit. The rating is like 1 out of 5, 5 being the best. As workers we rate and write our reviews based on our experience. The rating is right there, so you can decide quickly if you want to do work for a particular requester, or log in to Turkopticon and read the reviews.
There are definitely more potential rejections, around 350 to be exact, but there are 3 control values that I threw out, bringing it down to 160 or so. If the potential rejection is from a worker who has previously done work for me and has a 95% or better approval rating, then I am overlooking non-exact matches...unless they are way, way off.
Thanks for pointing this out. I deleted the post and banned the user who I suspect is the person I referenced above.
Andy -- yup, you got the right one. Thank you for taking care of that for us. Everyone else -- I just finished reviewing the name HITs and wound up rejecting 15. Many of the ones I rejected had control answers that were left completely blank with no comments, and the others were just too far off the control value to accept.
Thank you for all of the communicating you're doing with us. It's a very refreshing change. I got 2 rejections out of the ones I did and I thank you for going through them so fast. Are you still planning on putting more of these HITs up soon?
I just did a few more with the telephone #'s. It strikes me that most of them were incomplete #'s - not sure if you might want to make a field for "Incomplete audio" - there Were digits spoken and it wasnt a poor quality audio so I wasn't sure what to do. Since a partial # is probably of no use what so ever, I left the field blank but did of course reference it in the comments section at the bottom.
Phone numbers are a bit tricky for us to find programmatically. We try to look for a sequence of digits near certain keywords but these keywords aren't always spoken, so we also have to just look for a certain number of digits -- this is where we pick up random numbers that are not in the context of a phone number. We are trying to cast as wide a net as possible with our algorithm without catching too much, but as you can tell it is not perfect. I have many more HITs to upload in the next day or so, but I first have to make quite a few changes. I will post something in this forum when they are ready -- would love to have many of you with whom I have been in contact with continue to do these HITs for me. --Franklin
Realizing that names were tricky and the subject of much discussion on this forum, I was as lenient as I could possibly be, especially when it came to reviewing work from those who have previously worked for me. I hated to reject the ones that I did, but I just couldn't overlook some of the variations from the control value.
All of my name HITs were approved. I really appreciate your leniency and understand that you need 100% accuracy. The hardest part for me was the ones where a thick accent says the name, and no clarification is made for spelling. I think I am late on this, but I suggest leaving more audio for us to extract the information from.
No worries at all they were certainly justifiable and you have been very helpful to all of us with communication. I am definitely looking forward to the new batch and I'm sure many others are as well!
I worked on these hits as well the other day. Thankfully all of mine were approved. I'm not a requester so I don't know for sure but I thought there was some way that you could do something like this: - Have each audio transcribed by two different people - There is some way to compare the two to see if they are both the same answer - Auto approve if the answers are identical - Flag for manual review if they are different If I can find where I read about doing something like that I will post a link.
Do you mean have the audio transcribed by 2 different Workers? If so, I ran a test about a month ago and allowed 3 different workers to work on the same HIT, and it went very well...results were more clear-cut to approve, too. However, it triples our cost. If you can find the link that would be great. Thanks!
How does everyone feel about having to take a qualification test before being allowed to work on any HITs? I am going to start a separate post for this topic.
*raises hand* I'm in ( PS..post it early or late..Im out from 8-7pm cst and I don't want to miss it hehe!, or make a lot so I can grab one when I get in)