Please Join Me On A Crusade To Fix Bad CrowdFlower Test Questions!

Discussion in 'Requesters' started by liladybugz36, Nov 4, 2012.

  1. fairviewpark25

    Joined:
    Oct 14, 2012
    Messages:
    279
    Likes Received:
    0
  2. Shego

    Shego User

    Joined:
    Feb 4, 2009
    Messages:
    475
    Likes Received:
    0
    I personally don't work for crowdflower because I hate getting that "Some of your answers were not what we expected" message. Well if you already know the answers, why are you asking us to do the work?! More importantly, their pay sucks!
    I used to spend a great deal of time writing to requesters whose pay or practices were unfair. Sometimes they responded with thanks and made changes, sometimes they never responded at all and other times they defended themselves and a very few times they were downright rude and offensive. We work for terrible wages for the most part, but we know it going in. I think the most I ever made in one day is $80 and the most in one month was a little over $300. I say if it makes you feel better, go ahead and try to reason with unfair requestors. At least you're trying. I think boycotting them would be more effective and more fun ;)
     
  3. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    If you read this entire posting at all you will see that change is in the works thanks to a little effort and not listening to people like you. (Sorry - not trying to be rude)

    CrowdFlower has a following of workers and mostly attracts new workers that enter mTurk because they lack qualifications and do not reject. The idea of boycott is just not plausible when it comes to this company because people will still continue to work on their hits. They already have their foot planted.

    It's not about having "fun," it's about creating a workforce that is fair for everyone. If you clearly read what I had written above you would notice that one of the things I have discussed with Brian (Crowdflower exec) is CrowdFlower's pay. Although they do have an occasional hit that is on par with other requesters, this is just not always the case (read posting #51).

    Right now CrowdFlower is in the process of "reorganizing" to build a force of skilled workers. If you have noticed, not all hits with pay that look favorable coming from them are available to everybody. That was the start of rewarding noticed workers for high quality work. There are a lot of changes taking place including testing for special high paying specialty hits.

    If you work for them and have submitted any complaints through the help button you are already aware that the response times have drastically improved from a year ago. Last year it took a month for me to receive a response on a problem. It now takes less than 24 hours. This is a progressive company moving in the right direction, but everything takes time when a company revamps a system that they have been working with since inception.

    Your post does not seem like one of anger directed at just CrowdFlower, but one of anger at the whole system! Thankfully I do reason with requester's and do communicate regularly with them and thanks to this I have had many rejections reversed, have worked for requester's outside of mTurk privately for higher pay, have earned higher qualifications, have had hits changed and pay increased in hits and have had the opportunities like this, to work on Beta testing with CrowdFlower to improve their system for workers!

    So yes, I do reason with what you consider "unreasonable" requester's, because really they are not very unreasonable and you can benefit a whole lot by doing so!
     
    #43 liladybugz36, Jan 19, 2013
    Last edited by a moderator: Sep 18, 2013
  4. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    You too with your ebook! WTG!!!!
     
  5. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    Not an actual rejection. One of the test questions where he was right and the test question was wrong and the system rejected his answer.
     
    #45 liladybugz36, Jan 19, 2013
    Last edited by a moderator: Jan 19, 2013
  6. TAC

    TAC User

    Joined:
    Sep 15, 2012
    Messages:
    474
    Likes Received:
    0
    I'll use the wording of test question failure instead of rejection next time, makes sense. BTW, I'm a guy. No big deal.

    Anyway, do you know why their HITs cut off at different amounts? For instance, last month I could do 400 of the Classify Adult Images (Contains Mature Content) and this month I got cut off around 130 with my accuracy about 93 or 94 percent. Also, why does the training mode come back into play so much on the same hit when I've been doing them for awhile?

    Curious minds want to know.
     
  7. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    I'm sorry Mr. Man! Yes, I actually asked Brian this the first question in our conversation! It is because they need multiple worker results to compare to each other to determine the accuracy of a question.

    For instance, if 95 out of 100 workers select answer A they know the answer is A. If B is selected, they know the answer is B. If an unlimited amount of data comes from the same person and they select the wrong answer, then the results are no good.

    The number of hits you can do depends on how many hits there are in the batch and how many questions there are in the hit. Even though there are 1000 hits with 5 questions each for 5000 questions, doesn't mean there are 5000 questions that need answering. There are test questions and many, many of the questions are repeated. Each hit has a different variable of these numbers and depending on that is what a single worker can perform.

    The training mode comes into play so often because the hit is new even if it doesn't seem like it. Even though you have worked on the same hit before, it is a new batch and the system doesn't know you have ever worked on so it triggers the initial test. ;)
     
    #47 liladybugz36, Jan 19, 2013
    Last edited by a moderator: Sep 18, 2013
  8. TAC

    TAC User

    Joined:
    Sep 15, 2012
    Messages:
    474
    Likes Received:
    0
    Ok, that makes sense. I have noticed many times that the same pic will come up, sometimes on the same hit page.
     
  9. HappyFish

    HappyFish User

    Joined:
    Sep 8, 2012
    Messages:
    1,778
    Likes Received:
    0
    QC still doing the same things. I hate to rain on your parade, but it's just as bad as it's ever been.
     
  10. wonkatonka

    wonkatonka User

    Joined:
    Sep 26, 2012
    Messages:
    1,066
    Likes Received:
    0
    Companies like CF generally work in that a company reaches out to them and gives them tasks to do, pays a rather large fee, by which CF posts it on Mturk and gives out little money while reaping profits. It sounds weird even when I type it out.
     
  11. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    They are all rolling in the profits and paying us pennies on the dollar.

    One thing with Crowdflower that people don't realize is that some of the hits posted under the CrowdFlower name are just companies paying to use Crowdflower software and don't actually go through CrowdFlower for the posting on mTurk and post their own pay.

    The better hits that pay higher, the ones that we routinely see and people typically like on mTurk, generally are CrowdFlower hits.

    So there are two types of CrowdFlower hits. An actual CrowdFlower hit where the company pays CrowdFlower to build the hit, pays them a fee and CrowdFlower determines what to price the hit. Then (where the problem hits come in) requesters using their software to build and price their own hit.

    CrowdFlower has been trying to "weed out" the bad requesters that access their software, and are trying to be more transparent so workers can easily identify which hits are actually theirs and which hits are someone elses using their software.

    I think it is easy as a long-term Turker to tell the difference between the two; however, it is the new Turker that gets bit by these few bad software users.
     
    #51 liladybugz36, Jan 20, 2013
    Last edited by a moderator: Sep 18, 2013
  12. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    Have you been reading? It's a process. Read the entire post.
     
  13. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    After speaking some more with Brian from CrowdFlower, I thought that with all of your questions (and me the non-expert) it would be beneficial to have him drop in and fill-in the blanks. If he does stop in, please be respectful. Thanks :)
     
  14. fairviewpark25

    Joined:
    Oct 14, 2012
    Messages:
    279
    Likes Received:
    0
    Thanks that's awesome!
     
  15. 420DNR

    420DNR User

    Joined:
    Jan 22, 2013
    Messages:
    218
    Likes Received:
    0
    In my short time mturking, I felt like nothing more than a survey doer with absolutely no power. lil, thank you for showing me, showing us, the mturk community, that we can make a difference.
     
  16. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    You are very welcome!
     
  17. liladybugz36

    Joined:
    Dec 24, 2011
    Messages:
    224
    Likes Received:
    1
    That is the nicest complement. Thank you so much! I'm speechless!!!
     
  18. 420DNR

    420DNR User

    Joined:
    Jan 22, 2013
    Messages:
    218
    Likes Received:
    0
    I was (falsely)rejected for plagiarized work by an unnamed requester, whom I contacted shortly after reading this. I'm now making triple what I would've because of how good my work was. If I had have taken the hit laying down, as I was going to, this opportunity wouldn't have been available. Remember, they're paying us, they're our clients.
     
    #58 420DNR, Jan 24, 2013
    Last edited by a moderator: Jan 24, 2013
  19. hermitsoul

    hermitsoul Member

    Joined:
    Jan 2, 2013
    Messages:
    6
    Likes Received:
    0
    I'm not seeing all these CF improvements. Too often when I try to do their hits, they have html errors so that a button I select on my end selects a different answer on their end. And adult content filter hits have tiny little pictures that are impossible to make out. Not willing to risk rejections over these things. I sent a message asking about a rejection on the radio button issue and kept checking their forum for weeks. Never got an answer. This was within the last six weeks. Another time I had a question about something that wasn't clear in their instructions, and got a response that they wouldn't answer through the Amazon/email system, I had to go back to their website, remember my id and password there, and post the question in their forum. Considering that that effort had produced absolutely nothing earlier, I gave up. Once a week or so I try a test hit and I have yet to find one that is functional enough or has clear enough instructions, that I feel safe about rejections. So the current status, from my viewpoint, is unimproved. I'm not boycotting them, I'm just avoiding them as unnecessarily risky.
     
  20. TAC

    TAC User

    Joined:
    Sep 15, 2012
    Messages:
    474
    Likes Received:
    0
    This is strange:

    Judge the Relevance and Sentiment of content about PayPal (165569)
    --------
    As an exceptional worker, you have been given access to higher paying work (like this). Please not that we actively monitor these jobs to ensure only those who maintain exceptional quality are allowed. Thanks!

    Click accept:
    You are not qualified to work on this task.
    --------

    Am I missing something?
     

Share This Page