Please Join Me On A Crusade To Fix Bad CrowdFlower Test Questions!

I Talked to Crowdflower.....

So, a couple of weeks back I had an hour long conversation with an executive (Brian) from Crowdflower. First off all I would like to say that he was very moved by my enthusiasm and felt compelled by my "crusade".

The conversation was really awesome and we discussed everything I could think of that Crowdflower could do to improve for workers, and draw in the mTurk workforce. We compared different Requester's to Crowdflower, and I explained ways they could optimize their system to meet or exceed the standards of those Requester's, and also told them ways they already met those standards.

The end result was me ending up as a beta tester on newly designed hits and giving my feedback on what I thought about them from the design, to glitches and anything I could think of that I felt would improve the flow and work process for an mTurk worker.

I did this on a variety of hits and will continue to do this in the future. So please never say you can't make a difference because all it takes is one person and one idea to enact change! With motivation and desire, anything is possible! There's even a bonus - I even get premium pay to give my feedback!

 

You rock.

 

On numerous occasions I've been dinged with what I thought was a bad test question. I've utilized the "This is unfair" button, typed in why I thought it was unfair, and a few minutes later (while working on the same task) I get a message that my accuracy has been adjusted due to a test question being re-scored.

This is honestly the only option open... you can email them all day but they don't care; if you stop working a task someone else will hop right in to do it, so there's no motivation for them to change.

**EDIT**: I just read liladybugz36's most recent post... apparently I stand corrected. Nicely done!

 

That's good CF can get some first hand experience.

I had 2 reversed today with my score adjusted up that I stated as unfair. This was from hits I did a couple days ago. Seems like it is progressing in the right direction.

 

Very cool! I'm so glad to hear that. It seems like response times from support is improving drastically! This is so awesome to hear!!!! I'm glad to hear that you put your voice in and motivated a positive response.

 

I just had two more incorrect ones and commented accordingly. Are there any certain words or a way that comments should be written that gets more attention or flagged quicker?

Not sure if it's been answered before, but how do these incorrect things get marked wrong in the first place? Hopefully, it's not majority rules.

 

No, it is not majority rules. The test questions are mostly written by the companies that use Crowdflower to source their work through, and some are written internally by Crowdflower staff.

To get attention more quickly, I always use full capital letters in my title and use the word URGENT in it as well. That will grab the staffs attention more easily and single your post out.

They really do fix these questions when they are flagged so I'm glad you are letting them know... and of course for your percentages!

 

When you say title, are you talking about emailing them? I'm talking about disagreeing, marking unfair, and leaving a comment.

 

No, not emailing them. I'm talking about the headline when you leave a comment.

 

Duplicate post. Ignore this.

 

Question: How much does a 55 inch tv cost?

Answer: j h g e h g j h g j h g j y h g jf h g j h



That's not a sign of a messed up grading system. That's a sign of plain laziness. I sent an email to Crowdflower discussing this situation. I think if he just starts hard blocking everyone who is to lazy too give an effort, then it will cut out 99.9% of the problems. Some people are so lazy just over a computer, it makes me feel ashamed to think about their house hold situation. If you can't click on google, read search results, and type your own answer without cheating, I'd be highly surprised if you even give an effort keeping your house clean or brushing your teeth for that matter. I understand the grading system is complicated at times for those who do try. But realize there has to be a grading system that is so strict intact to ensure lazy people are submitting lazy work.

 

Question: Are aquarium heaters easy to install?

Answer: v b v c v c t b v c b c v b c y v b v u c b


Question: What are some different types of allergy symptoms?

Answer: d f h d f g s d a? f g sh df h d g s d fh f d s f h s d f h s d f gh s d f g d



One after another.

 

Sometimes my allergies do feel just about like that.

 

There are correcting things quicker on the adult hits. Had two more incorrect rejections, disputed it, both reversed in a couple days with my score adjusted accordingly. Had another unfair one yesterday (regarding incorrect drug identification - nugs of marijuana WTH?) and left a strongly worded dispute in all caps per suggestion. I expect it to be reversed as well.

It is improving.

 

That's odd. I've done thousands of each CF and CS and never received a rejection.

 

Obviously TAC is talking about getting downgraded for 'errors' and not actual rejections.

 

This is my face reading that.



wtf is a nug of marijuana? Is that the new unit the kids are using these days? Is the metric system gone? It used to be grams and ounces. I must be getting old or something. Time to trade in my bike for a hoverround.

 

Liladybugz, I am totally impressed. I've been away from the forums for awhile, and I come back in and here you are, making even further progress! Really nice job.

 

You have a nug if you're too poor for grams or ounces.

On topic, everybody's still getting cutting off for terrible QC.