Does anyone have a problem where some workers are not taking their survey seriously? Please advise about how you go about filtering out these types of workers (qualifying questions, etc)?
I do most of my survey very attentively coz surveys are my best part of mturking and i just love doin it
Cheers. I am experienced survey worker , done over 200 surveys just 2 rejected ( one by my fault second by the requester being unfair. ) So far i noticed that in the middle of survey questions i get "Pick Average so we could see that you're reading instructions" It's placed like that so person who doesn't read the survey have zero chance to notice it lol. Also , could i have link to your survey on mturk? Id gladly do it. Good luck.
qualifying questions in middle of survey. Make them original. Pay a fair wage. Be discriptive and honest about how long the survey will take
This. I have done a few surveys and I have gotten around 3 rejections, some unfair, others my fault. I do think qualifying questions like above are good. Although I have stopped doing surveys lately.
You need to add gold questions throughout the course of the survey. This means you place a question in the survey where you ask the survey taker to answer a question in a specific way. If you can keep the question very discreet, it has a tremendous impact on the quality of your results. You can tell by the gold questions whether or not the worker is actually paying attention or simply answering questions at random. If they do not answer the gold question exactly how you want it answered, then you know for certain they are not paying attention and can reject the results. This is a very effective model. Just my .02 cents worth.
I agree! Nothing irks me more than accepting a survey hit that supposedly only "takes a few minutes" and 15 minutes later I'm still working on it. Fair pay is mandatory as well. If you want honest, well thought out answers you have to compensate people generously for their time and attention.
+1 to this To the original poster, ask a simple math question and ask them to type down the answer. Another tip, create a big irrelevant para and somewhere in the middle ask them to click on a particular choice. You will filter rats in this question!
Starting with some basic qualifications such as a minimum approval rating and a minimal # of submitted HITs, you could also put a number of attention checks into your survey that have been mentioned previously. Here's a pretty standard type of attention check: Age Range Responses to this survey often vary by age ranges. Prior research shows younger respondents to lean heavier towards liberal attitudes, while progressively conservative attitudes follow along with advanced years. We're testing to see if the population on MTurk follows these past results, or if a highly computer literate pool of respondents will display differing opinions. We're also interested in degrees of self satisfaction and happiness along differing age ranges and geographic populations. Finally, we're testing to see if you are paying attention. If so, please answer that you are between thirty to forty no matter what your actual age. In a recent Unicef study, they found younger respondents and urban dwellers are among the happiest in the East Asia and Pacific regions, while in Western Europe young girls report feeling happy more than boys, while happiness among urban dwellers seems equal. What is your age range? [ ] 18 to 21 [ ] 22 to 30 [ ] 30 to 40 [ ] 40 to 50 [ ] 50+ <Click next to continue> Also, the honesty about the survey length is a big one. I get very irritated when someone says their survey will take 10 minute, and 18 minutes later I'm still chugging through with no end in sight. I'm less likely to want to help someone with their research or give them my full attention if they aren't upfront and honest.