Lots of new HITs on labeling human pose in the images

Discussion in 'Do My HIT!' started by TurkPoseEstimation, Mar 29, 2013.

  1. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Hi,

    I have submitted quite a few HITs on labeling body poses of people and we will add more HITs with this task in the future. Please take a look:

    The groupId is 2F7PLSP75KLO22CULB8QMQVJRD9ZTA

    Its a rather simple task that everyone can do. The default reward is 2 cents. Those people that are able to accurately complete the task will receive a qualification an an upgrade in payment.

    I am also very interested in feedback regarding the labeling tool used in the task.

    all the best,
    M.
     
  2. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    You can also find the HITs by searching for
    "Label body parts of the person in the image" or "Max Planck"

    Best,
    M.
     
  3. gfsincere

    gfsincere User

    Joined:
    Mar 24, 2013
    Messages:
    168
    Likes Received:
    0
    Wait, so about 2 minutes of work for 2 cents? lol ok.
     
  4. justlookin

    justlookin User

    Joined:
    Jan 20, 2013
    Messages:
    1,122
    Likes Received:
    0
    Don't worry, if you do a great job you get a raise to .03 per HIT. That's about .90 cents an hour.
     
  5. Shego

    Shego User

    Joined:
    Feb 4, 2009
    Messages:
    475
    Likes Received:
    0
    Ninety whole cents? Pretty darn close to irresistible...let's see; If I work on these for 3 hours I can buy a 4-pack of toilet paper. Oh no, wait..I've got enough for that in my couch cushions. Sigh...smh
     
  6. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Fair enogh :)

    The key issue right now is the quality of labeling, the HITs have to be cheap because we have to average the results from multiple workers to get a decent quality. This is of course not fair to workers who try to do their best, but have to compensate for others that are not as careful.

    We can definitely pay more for HITs, but can only do that for people that are careful and reliable.

    I did about one hundred of our HITs to test the interface, and the average time per HIT is about a minute. I guess this requires a bit of routine, which you get after some time.

    In any case, if you decide to invest more time in our HITs I am open to suggestions.
     
  7. Haekon

    Haekon User

    Joined:
    Mar 14, 2013
    Messages:
    206
    Likes Received:
    0
    Maybe I'm just old fashioned, but I think I'd just prefer to pick cotton. The pay is better.
     
  8. admiraljohn

    admiraljohn Mod Squad

    Joined:
    Oct 2, 2012
    Messages:
    2,581
    Likes Received:
    0
    How about we offer him some constructive ideas rather than just offering rude and snippy comments about his pay rate? If people do nothing but complain like this when requesters post here, requesters will eventually stop posting here.

    Generally, TurkPoseEstimation, work is only considered worth doing when you average about 10 cents a minute. Offering a task that takes two minutes and offers ten percent of that may not net you many quality workers.

    You may want to consider increasing the rate for these HITs if you want to attract good workers.
     
  9. raitch

    raitch User

    Joined:
    Mar 12, 2013
    Messages:
    132
    Likes Received:
    0
    You could always set up a qualification test for the workers who are willing and able to do good work for you. You can weed out "bad" workers and make it so only those who meet your requirements can do the task. Then up the pay for those qualified workers and everyone's happy.
     
  10. shyaamist

    shyaamist User

    Joined:
    Mar 27, 2012
    Messages:
    415
    Likes Received:
    0
    At that pay rate, the only way you are going to get quality work is with a custom qual. No way is almost anyone, unless they are really poor/bored/bad etc gonna do this work they way it is!
     
  11. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Thanks for the feedback, I was originally very exited about the possibility to have qualification tests for the workers, but it turned out that the quialification tests do not support the "external question" type that I use to implement interface in this HIT. Take a look here:

    AWSMechTurk/2007-06-21/AWSMechanicalTurkRequester/ApiReference_ExternalQuestionArticle.html

    It says
    I might be mistaken at this point, and will need to dig more into this issue. For now I am thinking of granting qualification based on the current performance of the workers, and then paying more to workers with qualification.
     
  12. VixSin

    VixSin User

    Joined:
    Mar 6, 2013
    Messages:
    275
    Likes Received:
    0
    I get this thinking to a degree, but it is an insult that a requester knew they were paying at max $1.20 an hour and expecting quality. You have to pay for quality. Even if there was an incentive to get a qualification, I wouldn't touch these for less than .15, and that is if I am very bored (this hasn't happened in awhile). That is the reality. Most of us here give quality work, we deserve more for our time than this.

    Qualification test is the way to go. Some requesters do a number of 0.00 hits and give out qualifications based on those results. This may be a better option for you possibly.
     
  13. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Alright, so following the discussion here we have a new policy.

    The tasks for qualified users will be paid 10 cents per HIT. We also have a set of qualification tasks that are paid 3 cents per HIT. In order to obtain a qualification one needs to perform at least 25 qualification HITs correctly.

    All the best and hope you enjoy working on our HITs. We are a research institution working on algorithms for computers to understand contents of images e.g. to make cars automatically recognize pedestrians on the street or cameras to focus on faces of people. We do appreciate your contribution.
     
  14. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
  15. justlookin

    justlookin User

    Joined:
    Jan 20, 2013
    Messages:
    1,122
    Likes Received:
    0
    I think you deserve a lot of credit for taking another look at the pay rate on your HITs and setting up a qualification test.
     
  16. Flora

    Flora Administrator
    Staff Member

    Joined:
    Jul 16, 2012
    Messages:
    47,905
    Likes Received:
    3,130
    Looks interesting. I am doing them now. I am finding that the blue font is hard to read against the black box and in the pictures too.

    I did 25 of them. I enjoyed them once I got the hang of the instructions.
     
    #16 Flora, Mar 30, 2013
    Last edited by a moderator: Mar 30, 2013
  17. leftfield

    leftfield User

    Joined:
    Oct 4, 2012
    Messages:
    968
    Likes Received:
    0
    Thank you for coming here and communicating with us on a personal level. I appreciate the chance to help you in your research.

    Would it be a possibility to get some sort of mid-review? I have read the directions thoroughly but I'm still not sure if I am doing the work to your satisfaction. Could you look at some today and approve or reject?

    I see the auto-approve time on the qualification is 15 days. I would hate to have put forth all this effort for $0.75 and a chance at qualification only to get 25 rejections because I didn't quite understand.
     
  18. shyaamist

    shyaamist User

    Joined:
    Mar 27, 2012
    Messages:
    415
    Likes Received:
    0
    Ok, gave 30 of the quals a try. I know I did the first few wrong because I missed one of the items in the tutorial/examples. Now we wait and see if they approve in a speedy manor. At that time, I will decide if its worth it.
     
  19. Blaize

    Blaize User

    Joined:
    Jan 16, 2013
    Messages:
    147
    Likes Received:
    0
    Well I hope these quals will be up for a while. I'm far too chicken to do 25 hits with the risk of rejection and only 1 TO. :( Did two to see if they approve
     
  20. bobbie01

    bobbie01 User

    Joined:
    Jan 29, 2013
    Messages:
    11,255
    Likes Received:
    4
    Thanks for being gracious and listening to the suggestions! I have done a few also but I want to make sure I am doing them correctly before I do any more.
     
  21. bobbyjoeblue

    Joined:
    Mar 27, 2013
    Messages:
    28
    Likes Received:
    0
    Did 25 of these last night and I rather enjoyed it. They're kind of fun. I hope I did them right. I hope, too, that the penalty for being off on them is just not being granted the qualification instead of rejection. Only have a little over 200 hits done in total so I'm nervous. Fingers crossed...
     
  22. Flora

    Flora Administrator
    Staff Member

    Joined:
    Jul 16, 2012
    Messages:
    47,905
    Likes Received:
    3,130
    I agree with you on not getting the qualification instead of rejections because we all tried and gave it our best. I hope he goes by that method.
     
  23. bobbyjoeblue

    Joined:
    Mar 27, 2013
    Messages:
    28
    Likes Received:
    0
    That would be awesome. In the meantime, better get busy increasing my hit count. Feeling a wee bit naked.
     
  24. delerium

    delerium User

    Joined:
    Dec 20, 2012
    Messages:
    149
    Likes Received:
    0
    I completed 26 HITs for the qualification because I made a mistake on one and submitted it (emailed you about this). I agree with everyone else in that if we are wrong I hope you simply do not assign us the qualification rather than rejecting such a large amount of HITs. I know I tried my best on the HITs I completed including writing in notes and sending you an email about one issue I had. I would hate to receive 26 rejections for the work I completed.
     
  25. Kights

    Kights User

    Joined:
    Mar 18, 2013
    Messages:
    1,198
    Likes Received:
    0
    Have any of you been paid yet?
     
  26. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Hi everyone,

    I have just processed the HITs, thank you very much! We have received many nicely labeled images in the last couple of days. I also granted qualifications to everyone who got 25 HITs accepted.

    I also had a minor setback with a large number of HITs with only 3 (!!!) points annotated, those got rejected.
     
  27. Flora

    Flora Administrator
    Staff Member

    Joined:
    Jul 16, 2012
    Messages:
    47,905
    Likes Received:
    3,130
    Thank you! I got the qualification. I'm excited to do work for you.

    I have a question: There was batch of HITS where you do Video Segmentation (one object). I went in and I couldn't understand it at all. It says 'Fill in the objects that has scribbles'. I get it. So the picture appears. I see it already filled in with the color. I don't see any scribbles. After looking around for 8 minutes I had to return it because I was running out of time. I tried another HIT and the same thing happens. I see a deer already shaded in the color. I don't see any scribbles. This HIT is confusing.

    I will instead do the HIT that I was granted qualification for since I already understand how to do it and have had practice and approved for. Thanks.
     
  28. bobbyjoeblue

    Joined:
    Mar 27, 2013
    Messages:
    28
    Likes Received:
    0
    Thank you for the qualification! I found these pretty pleasant to do and I'm looking forward to doing more. And thanks for the communication too.
     
  29. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    those HITs are actually administered by another person, I will forward your comments to the responsible person to see what can be done
     
  30. swrogers

    swrogers User

    Joined:
    Mar 28, 2013
    Messages:
    4
    Likes Received:
    0
    I did five of them yesterday which have been approved. When will you be posting more so I can get up to the required 25 for the qualification? My mouse is kind of wonky (that's a technical term...) so I'm not able to burn through a lot of them at this point, unfortunately.
     

Share This Page