Hi everyone, I'm Andrew, and I've been posting the HITs on finding planets via transits over the last few days. I just found this forum and saw that there has been some colorful commentary about my HIT. I'd like to clarify a few things based on what I've read so far and ask for some feedback. Some of you mentioned that you thought my HIT was 'fake' because of you googled me and saw that I was doing research that wasn't about astronomy. That's certainly not true; we are working closely with another crowdsourcing site (which you can probably google without too much effort) that uses the same method to find planets and the research here will eventually make its way back into building more effective ways to crowdsourcing different types of work, including finding actual planets. I also saw the post below, which was quite concerning to me: This post was absolutely right. It's very hard to design a study that mitigates the effects of everyone discussing it in a forum and picking and choosing what they want to work on. I'm not quite sure what to make of this, because the discussion has led to all sorts of fancy descriptions of what I could be doing and it seems that some people thought my work was not serious and that they would just try to take as much $ as possible. I wish I could have come here earlier and communicated with everyone, but I'm worried that would have affected the data even more. Finally, I observed that several people were getting disconnected from the HIT (not choosing to finish and submitting). It's unclear to me exactly what was causing this and if anyone would like to help me by telling me what browser they were using and what led to the problem, I'd be very grateful. I'll be posting more of these at some point, but probably not without resolving these issues first which could lead to many problems with the research. Please respond with any questions or comments below or feel free to email me directly at (my forum username)@gmail.com if you have anything more discreet that you would like to discuss. Thanks, Andrew Thanks, Andrew
Thanks for this thread, Andrew. My own problems--probably self-created--occurred when I misread the part of the HIT about "reloading the dashboard". I saw the 24-hour timer (which might be something worth adjusting, perhaps) and the possibility to start/stop the HIT and made the mistake of putting the HIT aside after the tutorial section and then discovering that it had timed out and ended. I think that the "reloading the dashboard" is a bit confusing, but perhaps that's me. For my own thoughts, I would say that language to the effect of "Once you start this task, it must be completed without pause until you would like to stop or the system ends your session." Maybe something like that. Also, without knowing precisely what it is you're trying to study in this HIT, I would say that for pay a ramped-up system in which early good responses are rewarded at an introductory scale/minute, and continuing good responses afford the person taking the HIT a better $/hour potential might be worth exploring. Even though I wasn't able to complete your HIT, I found the subject fascinating, and I hope to participate in however you collect this data for your project in the future.
Hi Andrew, I did start your HIT and I wanted to finish it but I had to leave to go to work and was not sure that it would still be available to me when I got home. I did email you about this today and when you explained your side of it I decided it was in the best interest of everyone to return the HIT. I really wanted to work on it and liked it but I am also a little bit older and needed more tutorial time than maybe some of the other ones did. I have no doubt that I would maybe not been your typical turker. I just started turking in March, am 55 and female. I do rely on my turk money as part of my income as I have some medical problems that prohibit me from a lot of outside jobs. If I get the opportunity again to work on your HIT, I do intend to with the same intent I go into every HIT and that is to try to follow the directions to do the best job I can for you to provide you with useful data as well as providing another source of income for myself. That is my reason why I returned the HIT today, hopefully some of the others will tell you their stories as well.
I absolutely adore Planet Hunters and I would do them nearly full-time! You could probably cut the compensation in half and I would still sit and do Planet Hunters all day. I've done some on the main site in the past, so I feel familiar with the set-up. I was sad to see that I couldn't come back the next day and start working on it again, but if I can do one day a week, that would be super awesome!
A debrief at the end saying not to discus what the bonus amount you were paid was would have kept people from mentioning it. How were we to know up front that the bonuses were different for everyone and not to be mentioned if not told? The very first post said they were making $10 an hour on it then others who did it did not understand why they said that since they got a tiny bonus instead. By the time we figure it out it's already been discussed. A debrief would have prevented any talk of the bonus since then we would have known up front. People quitting because the pay was bad or continuing to work because it was good and saying they wanted to be paid well for their time doesn't mean they were purposefully giving crap. And there was no way to pick and choose which hit one got either. There are scamming workers out there but they normally aren't the ones posting on forums. Or even thinking it wasn't really a study on stars doesn't mean a person didn't try to do the work correctly and follow instructions as they read them to mean, as a good worker would assume you would check for crap and reject it and not want that. You could have a qualifying test as others have said. This would be fairly easy as you could have something similar to the intro and then do a few stars and enter a completion code into the mturk qual test and give a score once graded.
I love the idea of a qualifying test and quality controls within the system. I really love this work and I would do it full-time, as I said.
Andrew, I have returned 3 and then I simply gave up trying. I would very much like to participate, because honestly this HIT seems interesting. Every time I would accept a hit it gave me the message "someone else is working on this" and I got frustrated. If it helps, I'm using Safari on my iPad, but I don't think it was a browser issue.
When talking about cutting compensation in half remember some did not get this awesome pay. Some didn't even get decent pay.
I was one that quit suddenly, due to my own fault. I stepped away to change the laundry over. I read at the beginning that the hit would time out but it didn't tell me it would be after 60 seconds. To me, 60 seconds rushes me. This hit is a little above my intelligence level and I could easily see myself staring at some of these screens for 60 seconds to try to figure them out. Also during the tutorial, please include an example of one that is close & right and one that is close and not right. A lot of requesters like to use a clear cut example but then there is no reference point for those screens that are difficult to call. I too am someone that could have used more tutorial time. I also agree with every single word Henrik said. Thank you for taking the time to listen. I used chrome on a desktop
I have to say that this is one of the frustrating things I have found so far with mturk, is the numbers game. I am one that goes for a little higher paying hits, I take more time than many so therefore my numbers go up much slower than those who just hammer out the hits to get the numbers up for better hits. This doesn't mean I am a bad worker it just means I won't get there as fast. Also some of these qualifying tests are frustrating to me as well. I understand some of them like for transcription and things but some of them are just plain crazy. Like I said previously, I needed more tutorial time. That doesn't mean that I would do a bad job but if you have a qualifiying test with a short clock, I may get timed out and not be able to even participate. Just my two cents.
Hi Andrew, I'm one of those people who made it all the way to the end but did not submit. I had no technical trouble, I was just still unclear if I was not doing the right marking of planets after the tutorial and I did not want to submit work I was unsure of, so I just returned it.
Hi Andrew, We are glad to have you here with us! Like a few others, I had worked on your HIT, finished the tutorial and done about 10 minutes of work but then took a break. I though this would be okay because of the 24 hour timers and the instructions stating the HIT would save my progress. Sadly when I came back it told me I was done and it wanted me to complete and submit the HIT. Not satisfied with my work in the HIT, I chose to return it instead of submit it. I would have loved to finish it, but I did not want to risk rejection and try to only submit quality work of which I didn't feel this was. I used Firefox for this HIT. If the opportunity to complete this type of HIT comes up again, I would surely take it. Thank you in advance! ~B.
What's wrong with doing 10 Rubins a minute? I was able to do more than that consistently unless some hard ones were mixed in, and I am one of the three people that has the preferred worker qual. Secondly, I never did this task, but I think I speak for a lot of people when I politely ask that nobody ever tell a requester they would continue to do their tasks all day even if they cut compensation in half. That just dumb.
Thanks for the suggestion. I thought that explicitly telling people not to talk to others might encourage them to do it, but I am starting to see how the community works around here and how that might help. Some people effectively earned OVER $35 an hour on this task, which was quite painful for me, but I can understand why someone would prefer to do this task for $17 an hour all day rather than other tasks on here.
I think it would. The rule here is not to discuss surveys (other than time and pay) because we don't want to compromise the results of the study, but if we don't KNOW that its a study rather than a "task" we don't know not to talk about it. We often discuss "tasks" openly to help each other and make sure everyone is doing the work correctly. As far as your HIT goes, I did it yesterday but I submitted at around .45 cent bonus because I was unsure of my work. I really enjoyed doing it, though. I agree with others that I would have preferred more in the tutorial portion. Thanks for coming here and posting with us!
Thanks for posting Andrew - I am a little "older" as well and while I started off thinking I knew what I was doing, I found myself getting deeper and deeper into confusion about what you wanted! This is my limitation, not yours. But I would love to work on your HITs with a little more time to learn.:doh:
Hi Andrew, I really liked working on this hit. I also made it to the very end of the task. I left a comment in the feedback section mentioning that I was cut off. I did about 100+ and it led me to the final screen where i had to submit. However i did not time out, I just got to the final screen after i submitted one of the tasks, so I figured I was done with all the work there was to do in that hit. I was using Firefox as my browser.