Received this feedback from an email I sent. Anyone want to contribute any ideas on how this work could be improved? This is a GREAT requester that pays good and has tons of hits. There's no reason that he shouldn't get great work from people that do a great job. I know that a lot of people try to do the best they can but it's pissing me off that some of us have to do the clean up for garbage work and waste time and lose earning potential. ---------------------- Hi TAC, Thanks for your message and we appreciate you offering up your help. We have been noticing some people are doing exactly what you say in returning hits quickly regardless of accuracy and it affects the latter phases of our projects. Do you have recommendations on how we can begin weeding those people out to ensure accuracy and properly reward the turkers who actually do great work like yourself? We would certainly appreciate your help and any insight you can provide. Best, Josh
If they're reading this, I'd like to say a few things. The problem with the instructor HITs lie with a few issues. 1. Sometimes people who do the course HITs will use the incorrect university. A great example would be Illinois University and North Illinois University. The course data will be for one school whereas the name would be for another. I wish the people who did the instructor HITs paid closer attention. 2. Some universities are impossible to find course schedules on, it's because they either don't have any system in place (search engine and or pdf files) or they require a student login to access the system.) 3. Courses which haven't been taught in quite some time show up. An example would be BIO 203 and the source link where it came from is marked for the academic years 2008-2010, as an example. Otherwise they're easy and I thoroughly enjoy working on their HITs.
Maybe he could learn from Redwood HITs. You know, where one person reviews another person's work and flags it if it's wrong. I'm not sure what happens to the person whose work gets flagged wrong, but at least it would be a way to weed badly done HITs out of the batch. Perhaps by including options to indicate that the wrong school was used, the wrong catalog was used, other errors or problems such as private class schedules like wonka mentions. Another option - quals or rating systems. Instead of weeding out the badly done HITs, weed out the bad turkers. I think this would be harder to implement but more worth it in the long run. Limiting batches to >500 or >1000 approved is obviously not cutting it. Developing a specific qual or rating system could help.
Actually, I'd like to see it in the format MedCrowd has. With the informative guide and the checkboxes and whatnot. Good ideas!
I say the first thing is to find a source that has both the course catalog and instructor information first and foremost so there is no mistakes to be made. He should have started with look up the course catalogs first, then indictate if the course catalog includes the instructor's information. I wrote this to him. He ignored that suggestion in the first place. There are a lot of colleges that are still missing on his list.
Wouldn't it be great to be able to do these in bulk. Once you find the semester schedule for the school, it would be easy to whip out the course numbers, names and instructor names. Lump them all by schools, majors, then subjects. Maybe do a separate HIT for the email addresses since those are rarely on the course schedules.
Why would someone returning the hit be a problem? Doesn't it just return to the batch for someone else to do? Anyway, I guess I'm one of these people because I have yet to find a correct answer on any of the hits I've tried, Either through the link provided, or going through Google, or finding the course catalog online and looking there. I've returned every one. I don't even bother trying anymore.
bad info is getting in from the first round of HITs. he needs to either raise the requirements to something like 98% or 99%, with a minimum HIT approval of 10,000 or 20,000. that or he needs to look back at the work thats been done and start issuing custom qualifications to people that know how to do the work. this might include setting up a qualification test.
He did something similar by assigning a masters qualification to the HITs. I've emailed them several times through their contact form, their mturk and individual people like Nick. Aside from 2 single replies by two different people, there has been nothing else. I'm not sure if it's because they're backed up from the New Year or they're still not back or it's a genuine case of not caring about their project. I hope a receive a response about my suggestions. In my line of work, when our customers complain about the quality of software or updates we push out, we strive to make it better. We don't ignore complaints or suggestions. That's just bad business.
I don't know why people are having such problems with these. On the professor ones I thought they were very easy but don't ever use the link. I think raising the accuracy and hit approval would be a good idea. It's not that these are that hard it's the first batch of finding the course where there are some people that just don't give a shit and putting random or wrong courses which is causing the problem. Like last night with the product moderation. Come on there is no way someone actually thinks a epson printer looks like a swimming pool.
I wonder what work they are getting for the professor hits. It's not like the ones that are doing the coarse hits don't have access to do the professor hits as well. Josh has been emailed this link since there are some good ideas. Personally I think a qualifier like MedCrowd is a great idea along with a couple other things such as making the approved accuracy at least 98%. I know that sometimes SH but most people should be able to hit 98% if they do good work.
Didn't he raise it to 98 with the Masters and got nothing? The problem remains with those who do the course HITs. I'm sure no one from this forum does them incorrectly, but there are people who dig up information based on courses that aren't being taught anymore or are offered at similar sounding schools. I feel like a parrot having to repeat myself about these problems.
And unfortunately they do little to remedy the situation. Though it isn't easy. If they could place a big, fat, notice in red colored text for those course HITs, it would make the lives of us instructor finders much easier. For what it's worth, yesterday's term specific instructor HITs were a joy. The only issues were some course names being incorrect. And that was 2 HITs out of 220ish.
The find a course professor e-mail HITs are now at 99% + 5000 HITs. I don't know what the requirements were for the last course number HITs, because I just had one find a course professor for a class that doesn't exist (sigh)