Lots of new HITs on labeling human pose in the images

Discussion in 'Do My HIT!' started by TurkPoseEstimation, Mar 29, 2013.

  1. justlookin

    justlookin User

    Joined:
    Jan 20, 2013
    Messages:
    1,122
    Likes Received:
    0
    Wow it's really great to have feedback and communication like this. I wish all of the other requestors did this- it would make life so much better here!
     
  2. vkn2007

    vkn2007 User

    Joined:
    Mar 13, 2012
    Messages:
    106
    Likes Received:
    0
    Hi,

    Thank you for granting the qualification. I am happy to work on your HITs.
     
  3. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
  4. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Hi everyone,

    sorry for the long delay. For various reasons it took realy a while to collect and prepare the next batch of data. Anyway, here are the new tasks:

    https://www.mturk.com/mturk/preview?groupId=2EFB3Q39Z0B8QZHXQ55V0TVRKPFVK3

    I hope to be posting more tasks on the regular basis now. As always I happy about feedback. Some of you noted that it was hard to label certain points (e.g. facial landmarks) when images are at low resolution. It is indeed true. When the resolution does not permit, try to place the point the best you can, don't worry if it is a bit unprecise.

    I added several hints that are now displayed above the comments field. Take a look at them, they address some of the common issues with annotations that we had in the previous rounds.

    Best,
     
  5. bobbie01

    bobbie01 User

    Joined:
    Jan 29, 2013
    Messages:
    11,348
    Likes Received:
    62
    Good morning, I tried to do one of your hits but it won't load.

    My bigger concern is your TO. It's hard to work for requesters who won't respond to emails and that's the "word on the street".
     
  6. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Hi,

    I did not work with the machanical turk for about 3 month. When I was actively submiting the tasks I did try to respond to Emails regularly. I will continue to do so in the future. Is it perhaps possible that your email was about the task other than pose estimation. If so let me know. We have several tasks handled by different researchers, and I can try to motivate others to be more active in responding to emails.

    Could you send me the ID of the hit that did not work for you? I am trying to check from my side but the hits seem to display ok.

    Best,
     
  7. justcolleen

    Joined:
    Jan 6, 2013
    Messages:
    5,831
    Likes Received:
    4
    Go to this link:

    http://turkopticon.differenceengines.com/main/search

    Search "Max Planck" and spend a minute or two reading. It's an eye opener.
     
  8. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0

    Yep, some of the reviews do not look good at all. This seems to be a combination of several factors. Some of the complains are about the HITS other than pose estimation so I will point them out to people who are responsible for them. One major negative review is from a person who really did not follow the instuctions (annotated the face points only), I could not help in that case but to reject such work. All pose estimation HITS are now reviewed and only clearly incomplete/incorrect once are rejected.
     
  9. bobbie01

    bobbie01 User

    Joined:
    Jan 29, 2013
    Messages:
    11,348
    Likes Received:
    62
    Thank you very much! I didn't realize that this account is used by multiple requesters. Like I said before, communication is key. If I do poor work I would expect to be rejected but I would also expect the requester to communicate with me about what I did wrong.
     
  10. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Hi,

    I added a qualification task. As before one needs to complete at least 25 qualification HITs correctly to get a qualification.

    https://www.mturk.com/mturk/preview?groupId=2PQMB3Q39Z0D20MF3UVE3STVRT8JU5

    To address the issues from before I will not reject any qualification HITS unless they are obviously incorrect, e.g. only 3 points are annotated on the fully visible person.

    Best
     
  11. Draggy

    Draggy User

    Joined:
    Jun 11, 2013
    Messages:
    533
    Likes Received:
    0
    It would be helpful if there was a way to remove the red box after you know which person you are annotating. Sometimes it is in the way of the eyes and nose which makes it hard to be precise on those.
     
  12. bobbie01

    bobbie01 User

    Joined:
    Jan 29, 2013
    Messages:
    11,348
    Likes Received:
    62
    Hi again,
    I think that communication is still an issue. I sent you an email yesterday morning and still haven't heard back from you. I have your qualification so maybe that answers my question as to whether or not I am doing them correctly but I would feel a lot better if I received an answer to my email. Is there a better way to reach you other than through the Max Planck HIT?

    thanks!
     
  13. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0

    Hi,

    did you leave a comment to the HIT in the annotation interface, or you wrote an Email. I am trying to find your Email, but don't see anything relevant in the last 2 days. May be write me a personal message/notification though the forum system?

    I am in the process of evaluating the qualification task. I should be done tomorrow latest.

    Best,
     
  14. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Just a note:

    for the qualification task someone again submitted a *couple of hundred* incomplete annotations. Just head points, nothing else. May be instructions are not clear enough, but there are pictures of fully annotated people with hands and feet all over the tutorial. What should I do now?

    Ufff...
     
  15. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
  16. TaylorMercedes

    Joined:
    Aug 13, 2013
    Messages:
    5
    Likes Received:
    0
    lmao:smash:
     
  17. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
  18. Draggy

    Draggy User

    Joined:
    Jun 11, 2013
    Messages:
    533
    Likes Received:
    0
    I would say you could ask them to do all of the HITs again for free so you can approve the ones they did incorrectly, but since this is a qualification HIT it doesn't help you much to have them do them again properly.

    Maybe just get into contact with the worker and explain your dilemma to them and find a way for them to do 0.00 HITS to make up for what you will end up paying them. You shouldn't have to pay for work that clearly doesn't follow the directions, but if you could find a way for them to somehow do other work for free I am sure they would appreciate NOT getting a couple of hundred rejections. That would seriously cripple most turkers ability to get decent paying HITS.

    In the future could you make it so that your HITs have a limit per worker so this doesn't happen? Maybe cut people off at 30 so that is the most they could do incorrectly? I know that doesn't solve your issue for right now, but it will make it so this is hopefully your last instance where this happens in the qual test.

    Also I got the qualification and think that the hotkeys make it go way faster, but none of the HITs I did have actually been approved yet. Do you just let them go to auto approve?
     
  19. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0
    Thank you for your comments I will try to see what I can do about these HITs.

    I am done processing the qualification round and you should have your tasks approved by now. I am still not finished with the most recent assignments for workers with qualification, this might a little more time.
     
  20. TurkPoseEstimation

    TurkPoseEstimation Verified Requester

    Joined:
    Mar 29, 2013
    Messages:
    65
    Likes Received:
    0

Share This Page