CrowdServer

So, there's a new requester on MTurk called CrowdServer. The HITs they've posted are familiar in nature: "Find the E-mail address of the author of a scientific paper." This strikes me as very similar to the HITs posted by Sebastian Darr, however, this new poster has no Turkopticon rating.

I did 15 of their HITs because I like doing research HITs, but I don't want to expose myself too much if they turn out to be another turd of a requester like Sebastian Darr or Lighting Buff.

They've got over 4500 HITs still available at 6 cents per for the interested and incautious:
https://www.mturk.com/mturk/preview?groupId=20NENHMVHVKC22E06CY4R79KYY8425

 

A Darr by Any Other Name...

Well, of the 15 HITs I submitted to CrowdServer, 5 have been rejected with the message "name and email do not match". This is despite me adding messages to many of the HITs giving both the person's full name and explaining where I found the E-mail address (with a URL to the specific Web page.)

In sum, this requester has a broken approval process and does not read messages attached to their HITs by way of explanation.

Additionally, the rejection message is the same as for Sebastian Darr's hits: "name and email do not match". Even the capitalization is the same.

Added to TurkOpticon with rating 1/5/1/1.

 

What I think it is is that Sebastian Darr and his alter-ego/evil(er) twin, CrowdServer, are severely disinterested in investing the time to properly evaluate the results of his/their HITs and are so afraid that some scammer might make off with a few dollars of his/their money that they're more willing to reject lots of good work than possibly approve some bad work.

Personally, I think that if you're not willing to invest the time to properly verify your results from MTurk and put in sufficient but unobtrusive controls to protect yourself from getting scammed, you probably shouldn't be posting work on MTurk. Keep your money and go hire a temp to do the work for you.

 

Well, I was cheesed off enough about what I perceived to be unfair rejections that I fired off a message to CrowdServer telling them their approval process was broken, and unlike with Sebastian Darr, they actually responded to me.

Here's what he said to me:

thank you very much for your contributions. as you experienced, we can only approve email addresses that are clearly related to the author in question. at the moment we have no means to consider the url sources where the email addresses were found. we are sorry that this results in false rejections.

we will therefore try to include a validating mechanism that takes care of this problem as soon as possible, maybe by next week.​

So, at least they know their approval process is broken, but they're going to persist in using it as-is, which will result in false rejections. I've changed my responsiveness rating for CrowdServer on Turkopticon from a 1 to a 3, because at least I heard something from them, even if it's kind of vague and unsatisfying. However, until they do so, I still cannot recommend working for this requester.

As an aside, does anyone think I should send them an E-mail with suggestions on how to improve their approval process? I've got a few ideas, some of them inspired by gamification and which would rely heavily on bonuses and redundant checking, both of which would increase their costs but would also improve the quality of their results.

 

Thank You For Your Helpful Input.

Thank you for your helpful input. Based on your advice, I drafted a 2500 word proposal to CrowdServer which included detailed technical information, use cases, implementation details, and several diagrams, as well several pages of supporting documentation from numerous other websites.

I am looking forward to hearing from them shortly.

 

I interpreted your remark as sarcasm and therefore I reflexively used my own well-developed sarcasm skills in response. If your advice was sincerely offered in good faith, I apologize.