http://www.crowdconf.com/confagenda produced by crowdflower, very interesting conference with a lot of behind the scenes of their hits
I personally don't work for crowdflower because I hate getting that "Some of your answers were not what we expected" message. Well if you already know the answers, why are you asking us to do the work?! More importantly, their pay sucks! I used to spend a great deal of time writing to requesters whose pay or practices were unfair. Sometimes they responded with thanks and made changes, sometimes they never responded at all and other times they defended themselves and a very few times they were downright rude and offensive. We work for terrible wages for the most part, but we know it going in. I think the most I ever made in one day is $80 and the most in one month was a little over $300. I say if it makes you feel better, go ahead and try to reason with unfair requestors. At least you're trying. I think boycotting them would be more effective and more fun
If you read this entire posting at all you will see that change is in the works thanks to a little effort and not listening to people like you. (Sorry - not trying to be rude) CrowdFlower has a following of workers and mostly attracts new workers that enter mTurk because they lack qualifications and do not reject. The idea of boycott is just not plausible when it comes to this company because people will still continue to work on their hits. They already have their foot planted. It's not about having "fun," it's about creating a workforce that is fair for everyone. If you clearly read what I had written above you would notice that one of the things I have discussed with Brian (Crowdflower exec) is CrowdFlower's pay. Although they do have an occasional hit that is on par with other requesters, this is just not always the case (read posting #51). Right now CrowdFlower is in the process of "reorganizing" to build a force of skilled workers. If you have noticed, not all hits with pay that look favorable coming from them are available to everybody. That was the start of rewarding noticed workers for high quality work. There are a lot of changes taking place including testing for special high paying specialty hits. If you work for them and have submitted any complaints through the help button you are already aware that the response times have drastically improved from a year ago. Last year it took a month for me to receive a response on a problem. It now takes less than 24 hours. This is a progressive company moving in the right direction, but everything takes time when a company revamps a system that they have been working with since inception. Your post does not seem like one of anger directed at just CrowdFlower, but one of anger at the whole system! Thankfully I do reason with requester's and do communicate regularly with them and thanks to this I have had many rejections reversed, have worked for requester's outside of mTurk privately for higher pay, have earned higher qualifications, have had hits changed and pay increased in hits and have had the opportunities like this, to work on Beta testing with CrowdFlower to improve their system for workers! So yes, I do reason with what you consider "unreasonable" requester's, because really they are not very unreasonable and you can benefit a whole lot by doing so!
Not an actual rejection. One of the test questions where he was right and the test question was wrong and the system rejected his answer.
I'll use the wording of test question failure instead of rejection next time, makes sense. BTW, I'm a guy. No big deal. Anyway, do you know why their HITs cut off at different amounts? For instance, last month I could do 400 of the Classify Adult Images (Contains Mature Content) and this month I got cut off around 130 with my accuracy about 93 or 94 percent. Also, why does the training mode come back into play so much on the same hit when I've been doing them for awhile? Curious minds want to know.
I'm sorry Mr. Man! Yes, I actually asked Brian this the first question in our conversation! It is because they need multiple worker results to compare to each other to determine the accuracy of a question. For instance, if 95 out of 100 workers select answer A they know the answer is A. If B is selected, they know the answer is B. If an unlimited amount of data comes from the same person and they select the wrong answer, then the results are no good. The number of hits you can do depends on how many hits there are in the batch and how many questions there are in the hit. Even though there are 1000 hits with 5 questions each for 5000 questions, doesn't mean there are 5000 questions that need answering. There are test questions and many, many of the questions are repeated. Each hit has a different variable of these numbers and depending on that is what a single worker can perform. The training mode comes into play so often because the hit is new even if it doesn't seem like it. Even though you have worked on the same hit before, it is a new batch and the system doesn't know you have ever worked on so it triggers the initial test.
Ok, that makes sense. I have noticed many times that the same pic will come up, sometimes on the same hit page.
QC still doing the same things. I hate to rain on your parade, but it's just as bad as it's ever been.
Companies like CF generally work in that a company reaches out to them and gives them tasks to do, pays a rather large fee, by which CF posts it on Mturk and gives out little money while reaping profits. It sounds weird even when I type it out.
They are all rolling in the profits and paying us pennies on the dollar. One thing with Crowdflower that people don't realize is that some of the hits posted under the CrowdFlower name are just companies paying to use Crowdflower software and don't actually go through CrowdFlower for the posting on mTurk and post their own pay. The better hits that pay higher, the ones that we routinely see and people typically like on mTurk, generally are CrowdFlower hits. So there are two types of CrowdFlower hits. An actual CrowdFlower hit where the company pays CrowdFlower to build the hit, pays them a fee and CrowdFlower determines what to price the hit. Then (where the problem hits come in) requesters using their software to build and price their own hit. CrowdFlower has been trying to "weed out" the bad requesters that access their software, and are trying to be more transparent so workers can easily identify which hits are actually theirs and which hits are someone elses using their software. I think it is easy as a long-term Turker to tell the difference between the two; however, it is the new Turker that gets bit by these few bad software users.
After speaking some more with Brian from CrowdFlower, I thought that with all of your questions (and me the non-expert) it would be beneficial to have him drop in and fill-in the blanks. If he does stop in, please be respectful. Thanks
In my short time mturking, I felt like nothing more than a survey doer with absolutely no power. lil, thank you for showing me, showing us, the mturk community, that we can make a difference.
I was (falsely)rejected for plagiarized work by an unnamed requester, whom I contacted shortly after reading this. I'm now making triple what I would've because of how good my work was. If I had have taken the hit laying down, as I was going to, this opportunity wouldn't have been available. Remember, they're paying us, they're our clients.
I'm not seeing all these CF improvements. Too often when I try to do their hits, they have html errors so that a button I select on my end selects a different answer on their end. And adult content filter hits have tiny little pictures that are impossible to make out. Not willing to risk rejections over these things. I sent a message asking about a rejection on the radio button issue and kept checking their forum for weeks. Never got an answer. This was within the last six weeks. Another time I had a question about something that wasn't clear in their instructions, and got a response that they wouldn't answer through the Amazon/email system, I had to go back to their website, remember my id and password there, and post the question in their forum. Considering that that effort had produced absolutely nothing earlier, I gave up. Once a week or so I try a test hit and I have yet to find one that is functional enough or has clear enough instructions, that I feel safe about rejections. So the current status, from my viewpoint, is unimproved. I'm not boycotting them, I'm just avoiding them as unnecessarily risky.
This is strange: Judge the Relevance and Sentiment of content about PayPal (165569) -------- As an exceptional worker, you have been given access to higher paying work (like this). Please not that we actively monitor these jobs to ensure only those who maintain exceptional quality are allowed. Thanks! Click accept: You are not qualified to work on this task. -------- Am I missing something?