Announcing: Classify AdImages HITs

Dear Turkers - We seek your help in identifying and classifying advertisements (mostly, US English). To give you a flavor of the tasks, we have put up a small number of HITs in 4 HIT-streams (details below). We are still working on many improvements (e.g., HITs for different languages/countries) - they will come in a few months. Once we (i.e., requesters and workers) are all comfortable with the process, then larger volume of HITs will follow.

Here are the 4 HIT-streams (you can search for them using terms like "adimage" or "adimages" or "classify"):
1) ProductName HITs: The goal here is to identify the name of the advertised product. Generally, it will be helpful to know the US product-space to work on these HITs. However, there are some complications. As such, this HIT-stream has a qualification-test to guide you in naming products correctly.

There will be many such HITs coming frequently. So, will highly recommend that you take/pass the qual-test fpr this HIT-stream and then continue to work on a large number of such HITs for many months. (You can take the qual-test multiple times - the goal is to make sure that you understand the process rather than to pass/fail someone.)

2) CompanyName HITs: The goal here is to identify the name of the company that owns the product. Generally, it will be helpful to know the US corporate world to work on these HITs.

3) ProductCategory HITs: The goal here is to identify the category of the product. Generally, it will be helpful to know the US products to work on these HITs.

4) ProductURL HITs: The goal here is to identify the most-specific URL of the product. Generally, it will be helpful to know the US products and their URLs to work on these HITs.

Each of these HIT-streams has Google/Wiki -searches associated with them - so, you can quickly perform some searches if needed.

Please note that these HITs are undergoing some further testing/improvements. Obviously, there will be some times when our evaluation will differ from yours. We will try to work it out with the workers while we improve the process. Your feedback will also help us make it better. So, please do send us your opinion/complaints/experience/frustration/accolade. We will be happy to receive them (though, please be warned that we may be late in responding to them).

Happy Turking and a Happy New Year!,
AdImages Team

Update 1:

Since we announced our HITs, we have received huge support from Turkers - thanks much! We have also received some well-deserved brickbats from Turkers - our apology for any unsatisfactory experience you might have had. Please allow us to respond-to/clarify some questions/issues:

1) Yes, our company also runs ClassifyIt HITs. We know that team well, and they are a great (and, overworked ;-)) bunch of people. So, they might not be able to respond to every one of your complaints (have you considered that they may be overworked?), but they are good people.

2) Our mutual interest in getting quality work done quickly dictates that we reward high-quality Turkers, and discourage less-than-high-quality Turkers. Please note that a high-quality worker for one task is not necessarily a high-quality worker for another task, and vice versa.

3) Obviously, this brings us to the heart of the matter: how do we determine a response is correct/incorrect, and, consequently, a worker is high-quality or not? The only viable option for us to determine the correctness/incorrectness of Turk-responses is to use an automated system to determine that - otherwise, if we evaluated the responses manually, then we will be doing exactly the work that we are seeking your help on. So, manual evaluation of Turk-responses is not a viable option.

4) One of the better automated Turk-response-evaluation strategies is to use a plurality of Turk-responses to determine that a response is correct. This is the strategy that our system currently uses to determine correct/incorrect responses. So, for example, we ask 3 Turkers to identify an Ad, and if 2 of them say that the Ad is for "Jiffy", then our system picks that as the "correct" response. Obviously, this strategy is less than perfect. But, it works in very high percentage of cases currently, and we plan to improve it to reduce the error-rate further.

5) This currently leaves a small percentage of cases where our system makes incorrect decisions about picking the correct responses. While we continue to work to improve our system, this is going to take a little time before we perfect our algorithm. (Though, some such error - however small - will ALWAYS persist.) So, when you get an incorrect "incorrect-response" message, please don't take it personally: it's the stupid machine ! So, in such cases, please drop us a note, and we will try to work it out between us. (Also, do see the caveat below about how it takes us several days at best to attend to notes.)

And, of course, many sincere apologies in advance for such errors.

6) We are currently working to further improve the system, and the work will continue for a few months. So, expect improvements in the next few months. And, yes, expect some errors in the meantime. But, as we explained above, a very high percentage of your correct responses will get correctly recognized and rewarded.

7) The issue of providing guidance to Turk-workers keeps coming up. Thank you - we hear you loud and clear. We have provided some guidance (and a qualification-test) for our Product-Name HIT stream. Obviously, based on your feedback, it needs to be improved. Going to work on it. Similarly, we are working to provide guidance/qualification-test for other HIT-streams. Hopefully, they will help. If not, drop us a note and we will try to improve them further.

8) We do not plan to hard-block anyone - unless the Turker is really incorrigible. We do plan to soft-block low-quality (on our task) Turkers. Currently, our thresholds are at a reasonable level for our maturity. As both sides (i.e., Turkers & Requester) get to know each other better, we will adjust thresholds accordingly.

9) Please understand that, while we will love to respond to all your queries, we are unable to respond to most queries, and even when we do respond, it's often several days late. Please understand that there are only so many hours in a day, and only a few of us trying to get it all working right. I am sure you will much rather have us running the system right than running around answering same query 10 times, no ;-) ?

Our thanks in advance to all who send us a note; and our sincere apology in advance if we are unable to respond to your note.

10) We are together in this business: We need your help, and you need us. Let's help out each other by providing high-quality responses, by helping weed our low-quality response-providers, and by understanding that neither side is out to "get" the other side: we will both make mistakes and learn from them and improve from them.

Again, please do drop us a note if you have some feedback for us!

Thanks much!,
AdImages Team