Qualification test

Discussion in 'Requesters' started by fharrell, Oct 12, 2012.

  1. fharrell

    fharrell User

    Joined:
    Oct 10, 2012
    Messages:
    56
    Likes Received:
    0
    Hi, all. I have been thinking about creating a qualification test that Workers would be required to take before being allowed to work on any of my HITs. How would you feel about that? One main concern I have by doing this is that I would lose the benefit of having a scalable workforce, at least in the beginning until more Workers complete the test. The other idea I have to improve the quality of the results is to limit my work to those who have 5000 or more approved HITs...but how many of you are out there? Discuss.

    --Franklin
     
  2. Turkalicious

    Joined:
    Jun 4, 2012
    Messages:
    1,642
    Likes Received:
    0
    I wouldn't mind a qual test at all. These HITs are my cup of tea so I have no problem testing for them. As far as being over 5k.. there are many, many of us above that number. Although I do know we also have quite a few great workers on the forum here that haven't reached that number yet. In the end it's entirely up to you and there are a good number of us on the forum that are passed the 5k mark.
     
  3. Faiythe

    Faiythe User

    Joined:
    Aug 25, 2012
    Messages:
    393
    Likes Received:
    0
    I'm at 2.8K approved :( so I would be out. Tho I still think you should do a Qual!
     
  4. Kris

    Kris User

    Joined:
    Sep 21, 2012
    Messages:
    801
    Likes Received:
    0
    In order to achieve the results you're looking for, a qualification makes sense. In the beginning, there will be a certain amount of trade-off, similar to any speed vs accuracy situation, until a sufficient number of workers pass the qual. I'm only at about 2500 hits approved, so unfortunately for me, I wouldn't be able to do your work for a while if you want 5000+, but that's your decision. You could always institute that as a starting point and re-evaluate depending on results.
     
  5. Mom2ManyMasterOfNone

    Joined:
    Aug 8, 2012
    Messages:
    196
    Likes Received:
    0
    I'm new to posting here, but not new to mTurk. A qualification test always seems to be a good idea. That way, if the work isn't up to par, the qualification can be pulled. I have done your HITs, and they were all approved, and I really enjoyed them. I really look forward to doing more.
     
  6. granit

    granit User

    Joined:
    Sep 5, 2012
    Messages:
    3,043
    Likes Received:
    0
    A qualification test with all the 'challenges' one can encounter doing your work will save you a lot of time dealing with rejections.
     
  7. chase

    chase User

    Joined:
    Aug 19, 2012
    Messages:
    62
    Likes Received:
    0
    Qual test is a good idea. About 5K approval thing, it is up to you. Although I would not be able to process for a while because I am at little over 2.5K.
     
  8. tonic1987

    tonic1987 Active Member

    Joined:
    Mar 20, 2012
    Messages:
    26
    Likes Received:
    0
    I have no problem at all with taking a qualification test. I also think that a qualification test would be more beneficial than limiting work to those who have over 5000 approved hits. There are many people on mturk who don’t have the English skills needed to complete transcription work, but a large percentage of those people will have over 5k approved hits, as they will have worked on data entry hits and image based hits.
     
  9. bredeanf

    bredeanf User

    Joined:
    Aug 7, 2012
    Messages:
    2,833
    Likes Received:
    0
    I would be fine with either. Maybe you can post two separate HITs: ones for the qual test and ones with the specific qual and see which one gets you better results?
     
  10. footballchick29

    Joined:
    Aug 6, 2012
    Messages:
    2,150
    Likes Received:
    0
    qual test is good... i'm not so sure about the 5k part.. those numbers do not necessarily mean they are good workers.. you can have 5k and still a pretty low approval rating.. I would say do something in the middle.. do a qual test, require 2.5k approved hits (that shows they are not brand new at this) and a 98% approval rating (to show they do quality work).
     
  11. abrahamrohithroy

    Joined:
    May 24, 2012
    Messages:
    886
    Likes Received:
    0
    QualTest
    Author:Franklin Harrell
    Qualified Users: 0

    Is this your qualification test? But this is just a qual request. It does not require me to complete a test. Anyways, I've requested it.
     
  12. sunny4691

    sunny4691 Active Member

    Joined:
    Jul 29, 2012
    Messages:
    40
    Likes Received:
    0
    A qualification test is a very good idea but plz don't limit your work to only those who have 5000 or more HITs. I am just over 2.7k and all the HITs I did for you were approved, so it doesn't mean that only the 5000+ workers are good workers. Also plz don't post your qualificaton test as a HIT, just make it as a qualification test.
     
  13. footballchick29

    Joined:
    Aug 6, 2012
    Messages:
    2,150
    Likes Received:
    0
    I think 2.5k is a good number, a way to meet in the middle.. many who reach 1k do so with crap jobs and may not be experienced enough yet fr this work.. 5k is a bit high because it takes awhile to reach.. 2.5k is a good compromise.. if those workers are giving poor quality then raise it to 5k and see if the results will be better
     
  14. fharrell

    fharrell User

    Joined:
    Oct 10, 2012
    Messages:
    56
    Likes Received:
    0
    Great input as usual, everyone. I would rather implement a qualification test and keep the approved HITs requirement where it is currently set. Now to figure out how to do it. Believe it or not, some of these things aren't as easy as one would think. The MT interface is rigid and very limited in what it allows Requesters to do, which requires us to develop much of this offline and that requires significant programming experience. I will figure it out eventually, being the nerd that I am, and if not then it's a good thing I work with a bunch of uber-nerds!
     
  15. footballchick29

    Joined:
    Aug 6, 2012
    Messages:
    2,150
    Likes Received:
    0
    :) please keep us informed as to when the test/hits will come back out.. glad we could help
     
  16. fharrell

    fharrell User

    Joined:
    Oct 10, 2012
    Messages:
    56
    Likes Received:
    0
    Now I am confused. Don't post the qualification test as a HIT, just make it a qualification test? Meaning it would show up on the Qualifications tab when you login to mturk.com? I don't know how to do that.
     

Share This Page