Crowdsurf "Approval Pending" Status

FYI to those who asked, Crowdsurf's new scoring system is in place.

You no longer need to email them to request your score. After accepting a HIT, click on the "Stats" tab to see your current score and number of HITs completed for the week.

 

Hey guys, is there only bad audios available to work on, these days?

 

Hello! I am not new to this forum but for some reason my old username and password did not work and I had to create a new account. It may show this is my first post. I have been on mturk for years, but have pretty much the last year exclusively worked Crowdsurf HITs and then for more than a year Crowd Virtualizer. I am so upset, and am so glad that this forum is available to vent, and see if others are having any issues as well.

I have worked easily over 1000 hits for Crowdsurf, receiving may 2 or 3 rejections tops since their inception. I consider myself to be an excellent transcriber. So, why in the world has my Crowdsurf transcription rating taken a complete nosedive? It has taken me over a year and half to get my score to where it is, and I basically have lost 20-25 points in my score in the last 2 days. The first hit I did after the new scoring system it went up 10 points. Then I did some hits two nights ago and I got email after email that my score just plummeted point by point. After the first email I got, I took notice, thinking maybe I was rushing. And was extremely careful, replaying audio multiple times. Taking two minutes to transcribe a 20 second .08 hit. Still, my score goes down.

I submitted one last night that I KNOW was absolutely perfect, and still my score went down. WTH??? No, rejections, but I can't get my score to increase, it just keeps going down, no matter what I do. I am afraid to do anymore until I know what in the heck is going on, which kind of sucks with 4000+ hits sitting there right now. I sent Crowdsurf an email last night with no response as of yet. I often flip back to these when Crowd Virtualizer is slow, which they are at the moment, or if I have 5 or 10 minutes to spare. I am not a huge fan of doing QC tasks or Timing which is about all that is there now.

Please, if anyone has any insight on this, or has had this and heard from Crowdsurf, what is happening, please post. Are there just some bad egg Reviewers, or is this new system the problem? When I click on my "stats" it says last week my average QC score was 4.0 and this week it is 3.2. How is that possible when I am not doing anything any different? Again, no HIT rejections, am I making too big of a deal out of this? I am currently at 969, but if I get to below 900 I lose the qualification, which is unthinkable. Thanks so much!

 

Hi dirving, I have not had any such issues. My qualification went to 1004 and then stopped. I think the scoring system is off at the moment. Also, there is an alert on CrowdSurf HIT page which explains that they are resolving unjustified scores and rejections. So, I think your issue will be resolved.

 

Hi Dirving,

I had the same thing happen to me. My qual was at 1002 and it went down to 954. Not sure what's happening. I'll check website and see what's going on. I have not rec'd any emails from them. I'm not worried about it. It's possible their fixing kinks in the system or something.

 

You guys, thanks so much for posting!

I am thinking too it must be something buggy. I will keep and eye on it and hope for the best. Once again, this evening Crowd Virtualizer has NOTHING to do in the queues, so I will try a couple of Crowdsurf HITS to test the waters again. Also, I did a reply to my HIT approval from mturk about this issue two days ago and have heard nothing from Crowdsurf. Maybe I should have sent an email directly to Support, they are usually better than this about getting back to you.

 

Thank you for posting dirving. I also noticed the same issue and I'm open minded to the chance that my work could possibly not live up to the level they are looking for, but it didn't feel right. Here is the email I sent them. I haven't received any response yet...

I just wanted to send some feedback to you after receiving many -1 scores on my transcriptions on your mturk hits. I am quite careful when transcribing your hits and I'm receiving negative scores on hits that I am quite confident, were correct. In any case, I surmise that the quality of my work may not stack up the level you require, but if it becomes the case that you notice that your workforce dwindles down, there could possibly be a problem with the current system. If I had one bit of criticism before I go, it would be that workers only receive a notification of a score change without any useful feedback for future improvements on similar hits. I understand that implementing an interface that allowed reviewers to provide feedback could be cumbersome, but I think the benefit would outweigh the costs. In any case, I have appreciated working on your hits in the past and maybe I will come back to visit your hits at some point in the future, but I feel it's best I discontinue work on your hits with this current system and the negative reviews I'm receiving. All the best

 

Thanks for your feedback! Here is my update. I did five hits last night and apparently got 2 more -1 scores. I am so frustrated. I am not saying either that I am perfect. But, I will say that the hits I have been submitting in the last few days were audio replayed 2-3 times and I felt very confident with each one. I went from the week prior to an average QC score of 4.0 to now I am down to 3.17 and I am not doing anything different, and am being even more conscientious than ever. I consistently receive 100% approval rating in Crowd Virtualizer. This new system is the problem. I emailed a similar email to Crowdsurf Support directly last night and also have not heard back as of yet, which is a little frustrating as well.

I have had nothing but positive feeling for this Requester from the beginning, even when they went from .10 to .08. The volume is high for the most part and such quick approvals and consistent pay. It is such a shame to see. Hopefully we will get some response and others are notifying them of their concerns.

 

For some reason the forum didn't update me when you posted your last reply (I subscribed to the thread). I'll just check back every day I guess. Sounds like we have very similar experiences here although I'm not a member of Crowd Virtualizer. Every couple days I've randomly checked to see what their hit availability was and it looks very, very slim and at many times there are no hits available. Maybe they are working the system out, maybe they are running low on work, or who knows what else but something has definitely changed. With changes maybe it's tough for them to respond to messages so I'm giving them leeway, but it's still disappointing. Like you said, I was definitely impressed with them as a requester and actually enjoyed doing their hits at times to just zone out and relax. Maybe if we give it some time, things will work out. Especially if others are voicing their concerns like you said, so if anyone is reading this, maybe shoot them a message and if anyone receives a response, we can share it here.

 

Hi winryan-
I did finally get a response back from them, and it was completely lame. Basically they quoted the front page of the Crowdsurf website which very basically explained the new system, and that it was supposed to be very "transparent" and you never need to email them what your qc is, because your new qualification score explains everything. I shot them an email back, and said I was very disappointed in their response. I had voiced concerns in my email to them about WHY my score was repeatedly dropping and how confusing I found this when I wasn't doing anything different and felt confident that I was submitting perfect transcripts. I was looking for answers such as there are new reviewers, the computer is reviewing the transcripts and the computer keeps messing up, or even direct feedback about some of my submissions. Something. So, I was not happy with what I got from them.

The work has been pretty slim. I did submit another HIT yesterday morning that got a +10. So I am now at 974. I have submitted maybe three since then, and my score hasn't moved at all. So, not sure what to think.

The most ironic part of all of this, is they in recent months moved to a volume bonus driven system. Earn a $50.00 bonus when you knock out 1000 hits. OMG. I can't imagine what the quality of transcripts people are turning in if they are trying to knock out that many hits. If they want people submitting in that kind of volume, then I can only guess that they will expect the quality to go down. Yet, with this new system it seems to be difficult to get perfect scores. So, what gives? Obviously turkers are complaining and some must even be losing qualifications. There is a "quiz" you can take on their website to earn back your qualification of you've lost it. Holy Cow, I hope they get it together.

 

Even though it's not a resolution, I'm so glad to hear that I'm not the only one going through this. I mirror the frustration and experiences on this forum. I too was turning in transcripts that I knew were PERFECT, and they were getting rated as fair or maybe worse. I too sent an email asking for them to show me some past negative transcripts. No response, though it does sometimes take them a week or so to get back. I'll catch them some slack here. If they are having wide scale issues, then it's possible that they have a limited staff for an unusual amount of emails all coming in at the same time.

I wasn't out for the money this week. I was out to drive up my score. I would take a good five minutes to transcribe them. Checking, double and triple checking. Playing the audio three times for a final self imposed quality check. I was only going to submit them if they were perfect, and of course I needed to confirm that they were perfect more than once so I would have no room to second guess myself. Yet they were getting marked down over and over. I too concentrate on Crowdsurf mostly. It's hard for me to go back to doing other stuff since I discovered them. It has been love since first hit. But the scoring is reflecting major issues. If we are going out of our way to make them perfect and everything is getting marked down, there goes the workforce, because it must be happening across the board. There goes the engine that cranks out their product. It would just be a disaster for them and us. I really hope they fix it fast, I love doing their stuff.

 

Man, sorry to hear that BlueMoods. I sincerely doubt your quality is justified by that score. I hope they get it worked out for you.

 

My first rejection was from Crowdsurf after I've done 8,000 something hits on Mturk. The majority was with them. Maybe just under a couple thousand for other requesters. Mturk says my approval rate is at 100%. With CrowdSurf it's 99% approval. I just figured one measly rejection out of so many thousands hardly counts as 1%. Maybe they just factor it differently from Mturk, but both ways make sense. Well, I'm going to get off these forums and formulate my plan b for other stuff this week while they hopefully get it together. I can only find so many surveys everyday, and I'm not so good with sifting through forums for hits. Probably going to have to break down and write some articles on textbroker. ugh.