New requester looking for some feedback

Discussion in 'Do My HIT!' started by alexdatanyze, Apr 7, 2015.

  1. alexdatanyze

    alexdatanyze New Member

    Joined:
    Apr 7, 2015
    Messages:
    4
    Likes Received:
    0
    Hey Mturkers, Alex from Datanyze here. We're a startup looking to create millions of hits and we're starting with a few to test MTurk.

    https://www.mturk.com/mturk/preview?groupId=3A0F9TXOEW84A7N4EFE5C8WMN4IOJN

    Our current first batch is just a test, but if you could, please do them and provide feedback. I'll give you a bonus if you complete a few of the batch too!

    Some questions:
    - Do the instructions make sense?
    - Is the pay fair?
    - Is there anything else you want use to know to improve our batches?

    Thanks in advance!

    Alex of Datanyze
     
    #1 alexdatanyze, Apr 7, 2015
    Last edited by a moderator: Apr 7, 2015
  2. Flora

    Flora Admin
    Staff Member

    Joined:
    Jul 16, 2012
    Messages:
    62,970
    Likes Received:
    13,985
  3. Tjololo

    Tjololo Pony
    Staff Member

    Joined:
    Jun 14, 2013
    Messages:
    13,719
    Likes Received:
    105
    Just took a quick look, and it looks promising! A few things I noticed:

    Pros:
    • The hit format is fantastic. Seriously, looks really good, gives options for if things can't be found, some troubleshooting, really nice.
    • The instructions seem to encompass the options fairly well, and give good explanations of what to look for. I like that you covered the case of multiple addresses, though you might add that if a HQ cannot be determined to just pick an address...Just for the fringe case of a place that doesn't specify.
    • I love that the company link is an actual link. You wouldn't believe how much of a time saver that is XD
    Worries:
    • I didn't actually do a hit yet to test this, but what if someone forgets to enter a company phone number source? Do you have logic in place to make sure all required portions are filled out before submitting?
    • "Masking" isn't described, and users may not understand what a link mask is. You might want to give an example of a redirect and a mask to make sure people know.
    Cons:
    • Your quals are VERY unrestrictive. I don't know if this is because it's just a test, but you will DEFINITELY want to redo them. I'd suggest approved hits > 1000, approval rate > 95%. That's a good intro qual. You could go up to 5000, but....
    • Your pay will probably dissuade many people to do them.

    The pay comment is mostly because you're asking people to literally scour these sites to get these pieces of information. The bare minimum most people will accept is $0.10/min (though I can't stress enough how that's the minimum). At $0.04, you're talking less than 30 seconds per hit, which sometimes (depending on the site and a worker's ISP) can be longer than the page loads.

    My thinking on this hit: Company name is easy, open the page and copy-paste, no big deal. The phone number will probably be either on the home page, about page, contact page, that's 3 page loads to find that. Address will probably be on contact, location, about, or home page. Another 4 loads. I can check several places all at once, so I'm looking at probably 4 pages per site. At maybe 10 seconds per page load, that's 40 seconds of JUST waiting time for the site to load...on average. Let's say 10s each task for finding the info, copying and pasting, we're looking at another 30s. That's a minute and 10 seconds for each task, assuming all the info is found on a simple page. If I have to go to more pages, that's more time. I would look at this hit and just move on.

    My suggestion: Bump the pay up to about $0.15 or $0.20 per hit, depending on how you want them done. At $0.15, this hit is fair in my opinion, considering the work you want to go into it (reading the TOS for a phone number, for example). For $0.20, this hit is well paid. That represents 2 minutes of work for each hit at the minimum wage, but only about 1:45 of work if you get the hang of it and start becoming efficient. That means your work would be well paid, and people would do your hits well.

    Another item to consider: how are you handling bad work? Scamming is one thing, where 99/100 hits are submitted as "n/a", or submitting obviously scammy information, but what about the random mistake? If I were to click the link, get a 404, then submit with n/a's and troubleshooting, then find out it was just because I did something weird, would that hit be rejected or overlooked? Are you going to manually review all of these hits (not necessarily you specifically, you could have a team)? Will you do "spot checks" per user, or have a "review" batch set up so mturkers will review the results? Or will you use crowd-consensus and reject workers who lie outside of thresholds of 'correctness'?

    The reason for that last item is because people want to know you're safe. A well paid hit won't get done if it's known a typo or mistake will get someone rejected. My suggestion here: Reject obvious scammers, give people with questionable responses the benefit of the doubt or contact them at some point. Another option is to do a qualifying batch of say 5000, 10 hits per worker gives you 500 potentially qualified workers, who will be working on your batch. Hell, do 2-3 qual batches. A qual batch wouldn't necessarily get rejected (unless they're obviously scamming), but would only grant the qual to people who met your acceptance criteria. You could pick out 10 sites you KNOW the answer to, say you want 500 workers on the batch, and then you would be able to tell who answered incorrectly on one or all of the sites easily with the CSV.

    These are just suggestions, obviously you can make your own decisions about it. Asking your workforce is step one to being an amazing requester. What you're asking for is pretty common as far as hit-type goes, so you're not doing anything people aren't accustomed to. Hopefully you can figure out the "perfect" balance here and get set up to get your work done quickly, fairly, cheaply, and well! Best of luck!
     
  4. alexdatanyze

    alexdatanyze New Member

    Joined:
    Apr 7, 2015
    Messages:
    4
    Likes Received:
    0
    Hi Tjololo! Wow, thank you for all the feedback! They are much appreciated. Yes, this is just a test and to answer some of your questions and concerns:

    We'll look to adjust the pay accordingly. We might do a combination of issued qualifications for higher pay >$0.15 and allow for newcomers to work on our hits at a lower pay as well. Do you think that's a good idea?

    For handling bad work, we do plan to build scripts to cross check with our current data set and if we don't have any, then validate them with more HITs. For the qual batch, we do plan to build it very soon. I can't stress my gratitude in you pointing this out though. If it's slight mistakes here and there, we have a system that will parse the information to check and, say, if it 90% matches to other data points, then it's fine. We do understand human error, so we will factor that in as well. For now, we'll be pretty lenient since we're still figuring things out, but your questions are very helpful! They keep our minds going.

    Thanks again Tjololo!
     
  5. heyscotttt

    heyscotttt User

    Joined:
    Jan 30, 2014
    Messages:
    3,029
    Likes Received:
    0
    Please do let me know when you post again, I missed the first ones.
     
  6. Josh13

    Josh13 User

    Joined:
    Feb 27, 2015
    Messages:
    495
    Likes Received:
    0
    can you let me know when they post worker [redacted]
     
    #6 Josh13, Apr 8, 2015
    Last edited by a moderator: Apr 8, 2015
  7. balavelu

    balavelu Active Member

    Joined:
    Jun 14, 2013
    Messages:
    25
    Likes Received:
    0
    Hi i had missed the last batch.when will you post more hits again?.
     
  8. alexdatanyze

    alexdatanyze New Member

    Joined:
    Apr 7, 2015
    Messages:
    4
    Likes Received:
    0
    Hi Mturkers!

    https://www.mturk.com/mturk/preview?groupId=3HL3WX654CI3HU3IO5BTY9DZ232QPK

    I added another test batch. This time we'll be giving out qualifications for our private batches that will pay >$0.10. That will be created later. It has a little more strict quals as @Tjololo suggested, but it pays more than the initial test batch. We've added more links so you can get to the information faster.

    Please let me know if there are any improvements you want us to make on our HITs. We're still learning and would appreciate the feedback.

    Thanks again!

    Alex of Datanyze
     
  9. adaaaam

    adaaaam Banned

    Joined:
    Jan 10, 2013
    Messages:
    6,861
    Likes Received:
    0
    What the fuck...? Who do you even turk with that makes this true?
     
  10. Tjololo

    Tjololo Pony
    Staff Member

    Joined:
    Jun 14, 2013
    Messages:
    13,719
    Likes Received:
    105
    It's a well known minimum that a lot of requesters shoot for. It's a bare minimum, which means that not many people will do it but at least it's not insulting.

    If you have a suggestion that contradicts it please make it known. Otherwise please refrain from posting non-contributory posts in a requester thread.
     
  11. adaaaam

    adaaaam Banned

    Joined:
    Jan 10, 2013
    Messages:
    6,861
    Likes Received:
    0
    My suggestion is not encouraging requesters with an insulting .1/min. .2/min should be the bare minimum for all turkers.
     
  12. twirdman

    twirdman User

    Joined:
    Apr 21, 2014
    Messages:
    5,900
    Likes Received:
    1
    I don't know if I'd necessarily say that. I will say .1 is insulting. The vast majority of mturk workers are Americans and as such have the living expenses expected of living in America so there is absolutely no reason anyone should be working for less then minimum wage. Requesters need to stop putting up hits for less then minimum wage. Then when you factor in the fact someone working minimum wage is working the entire time with paid breaks something turkers don't get and we also have to spend time looking for hits I think it should be above minimum wage. I think at minimum around .15/min for 9/hr.
     

Share This Page