Hi, all. I have been thinking about creating a qualification test that Workers would be required to take before being allowed to work on any of my HITs. How would you feel about that? One main concern I have by doing this is that I would lose the benefit of having a scalable workforce, at least in the beginning until more Workers complete the test. The other idea I have to improve the quality of the results is to limit my work to those who have 5000 or more approved HITs...but how many of you are out there? Discuss. --Franklin
I wouldn't mind a qual test at all. These HITs are my cup of tea so I have no problem testing for them. As far as being over 5k.. there are many, many of us above that number. Although I do know we also have quite a few great workers on the forum here that haven't reached that number yet. In the end it's entirely up to you and there are a good number of us on the forum that are passed the 5k mark.
In order to achieve the results you're looking for, a qualification makes sense. In the beginning, there will be a certain amount of trade-off, similar to any speed vs accuracy situation, until a sufficient number of workers pass the qual. I'm only at about 2500 hits approved, so unfortunately for me, I wouldn't be able to do your work for a while if you want 5000+, but that's your decision. You could always institute that as a starting point and re-evaluate depending on results.
I'm new to posting here, but not new to mTurk. A qualification test always seems to be a good idea. That way, if the work isn't up to par, the qualification can be pulled. I have done your HITs, and they were all approved, and I really enjoyed them. I really look forward to doing more.
A qualification test with all the 'challenges' one can encounter doing your work will save you a lot of time dealing with rejections.
Qual test is a good idea. About 5K approval thing, it is up to you. Although I would not be able to process for a while because I am at little over 2.5K.
I have no problem at all with taking a qualification test. I also think that a qualification test would be more beneficial than limiting work to those who have over 5000 approved hits. There are many people on mturk who don’t have the English skills needed to complete transcription work, but a large percentage of those people will have over 5k approved hits, as they will have worked on data entry hits and image based hits.
I would be fine with either. Maybe you can post two separate HITs: ones for the qual test and ones with the specific qual and see which one gets you better results?
qual test is good... i'm not so sure about the 5k part.. those numbers do not necessarily mean they are good workers.. you can have 5k and still a pretty low approval rating.. I would say do something in the middle.. do a qual test, require 2.5k approved hits (that shows they are not brand new at this) and a 98% approval rating (to show they do quality work).
QualTest Author:Franklin Harrell Qualified Users: 0 Is this your qualification test? But this is just a qual request. It does not require me to complete a test. Anyways, I've requested it.
A qualification test is a very good idea but plz don't limit your work to only those who have 5000 or more HITs. I am just over 2.7k and all the HITs I did for you were approved, so it doesn't mean that only the 5000+ workers are good workers. Also plz don't post your qualificaton test as a HIT, just make it as a qualification test.
I think 2.5k is a good number, a way to meet in the middle.. many who reach 1k do so with crap jobs and may not be experienced enough yet fr this work.. 5k is a bit high because it takes awhile to reach.. 2.5k is a good compromise.. if those workers are giving poor quality then raise it to 5k and see if the results will be better
Great input as usual, everyone. I would rather implement a qualification test and keep the approved HITs requirement where it is currently set. Now to figure out how to do it. Believe it or not, some of these things aren't as easy as one would think. The MT interface is rigid and very limited in what it allows Requesters to do, which requires us to develop much of this offline and that requires significant programming experience. I will figure it out eventually, being the nerd that I am, and if not then it's a good thing I work with a bunch of uber-nerds!
Now I am confused. Don't post the qualification test as a HIT, just make it a qualification test? Meaning it would show up on the Qualifications tab when you login to mturk.com? I don't know how to do that.