Do their graders want you to fail?

Discussion in 'CastingWords' started by EllieFS, May 29, 2013.

  1. deeriley

    deeriley User

    Joined:
    Mar 26, 2013
    Messages:
    53
    Likes Received:
    0
    You stand by what you said? How can you stand by what you said, when you're actually evading the very thing that you said that caused me to respond to you in the first place?

    Let's try this again: you originally (and arrogantly and presumptuously stated) that people complaining about being unfairly graded are probably not realizing that something has "flown under their radar."

    I gave you examples of where that clearly is not happening. Instead of saying, "Whoops, my bad," you respond with stuff that has nothing to do with the comments you made originally regarding grading. You pompously lecture about adhering to the Style Guide (when my examples had nothing to do with that) and then proceed to tell me about how CW has been fair to you one time when you clicked the submit button, when that is not what I was talking about, when that is not what this thread is even about. I was talking about incidents in which CW doesn't grade fairly at all, not about people accidentally clicking submit or any of that nonsense.

    So if you want to "stand by what you said," give specific examples of you getting a lower grade on a transcript than you know you deserved and CW explaining why you deserved it in a way that was 1) consistent (in other words, they didn't "flip flop" on the reasons when you challenged them) and 2) fair (downgrading for a real reason, not something petty like an em dash or a word they wanted spelled a certain way that's not in the Style Guide.)

    That's all you have to do. If you can't do that, then you will never be "standing by what you said" at all, but evading what you said by side stepping the issue.
     
    #21 deeriley, Jul 5, 2013
    Last edited by a moderator: Jul 10, 2013
  2. pwt

    pwt User

    Joined:
    Apr 28, 2012
    Messages:
    191
    Likes Received:
    0
    If CastingWords says it's wrong, then it's wrong. This is a company that wants things done a certain way. It's annoying, granted, to have to learn to do things differently for CastingWords than you'd do just about anywhere else, but that's the job. And this is a job. When your boss tells you that you did it wrong by company policy, then you did it wrong. Doesn't matter if company policy is 1+1=3. If that's what they want you to do, that's what you do.

    That's what we do.

    CW is always right. That's how it works. If CW wants it spelled XZY, then always spell it XZY for them and you'll never have another problem with that word or style.

    It can feel unfair when it's a word or style that's not addressed in their Style Guide. That's why I don't do long HITs. An 8 instead of a 9 on a two minute clip hurts a lot less than a 21 minute clip. Ask them to address it in the next revision and they might very well do that.

    That bites, and you've got my sympathy. But it's appropriate. Grade 9's are for flawless transcripts and 8's are for something just less than perfect. You spelled something wrong according to what this company wanted, so by definition it wasn't perfect.

    We've all been there. Perfect is tough.

    That really sucks, but thank you for sharing your experience. I've never had to transcribe a software version number before and because of you, I'll get right it right if/when I do. Thanks.

    There are many different experiences working for CastingWords. I've never gotten the impression that CW plays games of any sort. The more I work for them the fewer errors I make, the higher grades and PPT I get. That's been my experience. There are so many things that aren't covered in the Style Guide that you've just got to slog through lots of work until you're on the same page as they are.

    I'm on auto-approval for 8s these days, maybe 5% of my work gets graded by mTurkers and checked. About half of that 5% are 8s and half are 9s, which is up significantly from last year when they were about all 8s. I'm not that far, I think, from either auto-approval for 9s or at least a PPT that qualifies for expedited work. There are people who already have both. They are proof that nobody is playing games. I know people that get all 9s or mostly 9s. It is possible. Those expedited hits that require a PPT of 99+ don't vanish within seconds of being posted for nothing.
     
  3. _dogmeat

    _dogmeat Member

    Joined:
    Aug 4, 2013
    Messages:
    7
    Likes Received:
    0
    What I don't really get is how they deduct PPT points. I mean, In 30 odd transcriptions, I've had mostly 8s and 9s, and a few 7s. Average is about 8.4. I started submitting 8s and 9s pretty much right off the bat.

    But for every 7 I've gotten, they deducted a point from my PPT. I've had 3 7s and I've had my PPT dropped three times now, for every single 7 up till now. What the hell? I thought they only reduce PPT for 6s and below, not for 7s. I'm really curious how people on this forum got to 90+, if I get downgraded every time I get a 7, and I don't receive a point when I get a 9 (or several 9s in a row). It just isn't fair.
     
  4. pwt

    pwt User

    Joined:
    Apr 28, 2012
    Messages:
    191
    Likes Received:
    0
    I don't think it's as simple as adding a point here and taking one away there. It's not as if a PPT score can just keep getting higher and higher if all you get are 9s. It's probably closer to a weighted average.

    Mine rarely moves. I think I went a whole year between it moving from 93 to a 94.
     
  5. _dogmeat

    _dogmeat Member

    Joined:
    Aug 4, 2013
    Messages:
    7
    Likes Received:
    0
    Yeah, but I still think their grading system isn't fair. For example, you can drop very rapidly in rating, but you can't rise up rapidly. That, to me, is an unfair system. Imagine you were always submitting 7s. And 10% of the time you submit a 6 and 10% of the time you submit an 8 or a 9.

    Now, every time you submit a 6, you will lose a point, until a time comes where you reach the dreaded PPT qual of 79, and you can't work for them anymore. Yet your average would be around 7.0, even higher if you have some 9s in there. You're doomed to fail that way.

    They should devise a better system, where your PPT qual is affected more when you perform better. Because, right now, it's skewed toward screwing you. Also, automate the upgrade, so they don't manually have to increase your PPT. If you can automatically go down, you should be able to go up too.

    Good workers should be treated fairly. Now they're only giving us (the new ones to their site, at least) only the stick. Where the **** is my carrot? I love carrots. They're tasty and orange and stuff. Who doesn't like carrots?
     
  6. pwt

    pwt User

    Joined:
    Apr 28, 2012
    Messages:
    191
    Likes Received:
    0
    That's how averages work. This work set:

    ...would average out to 8.4. Add a 6 to it like this:

    ...and it drops to 8.1. But add another 9 instead:

    ...and it only increases to 8.45.

    To move an average by a lot you need to have a big outlier. Neither 8s or 9s are big outliers if your average is already 8.4. A 7 is 1.4 points off the average so it's going to move the average down by a lot all by itself. To move it up by that amount would require a 9.8 (practically speaking a grade 10), which CW doesn't hand out anymore.

    So I mean...that's how averages work. If you're already producing work that's nearly the highest grade, then more high quality work is only going to produce marginal improvements because you're mostly already there.

    It's not really a matter of fair or unfair, it's just math.

    What will help is not producing grade 7 transcripts (obviously) because once you're over the hump nobody should be producing 7s. I haven't produced a 7 since July of 2012 and I only had three all that year. That's 500 transcripts this year and 231 last year and only three were 7s. The other thing that will help is getting a lot of transcripts done. Even a grade 5 (rejection) wouldn't dent my PPT at this point.

    If you want help avoiding 7s, let me know. I'll help anyone I can.
     
  7. _dogmeat

    _dogmeat Member

    Joined:
    Aug 4, 2013
    Messages:
    7
    Likes Received:
    0
    Thanks for the explanation, pwt. Even though I know that, it kinda makes more sense when you spell it out like that. And I don't really have a problem with the 7s. I know what I have to improve upon and I'm actively working on it.

    Most of the time it's finding that right balance between removing filler and adding it in. Sometimes I go all nazi on the fillers and remove everything, and sometimes I leave a lot. Sometimes it gets a 9, sometimes a 7 for leaving out important parts of the conversation. I'm working on it and I think I'm starting to get the hang of it.

    I would really love to see this forum more alive, though. We don't seem to have as an active community as other mTurk sub-forums around here. Maybe something like a General Chat thread would be nice? Where some of the more experienced transcribers share tips and thoughts. That would be cool.
     
  8. pwt

    pwt User

    Joined:
    Apr 28, 2012
    Messages:
    191
    Likes Received:
    0
    This is a good site. Although the site itself is largely inactive, the owner put a book out on transcribing on mTurk which has a lot of good information in it.

    CastingWords thinks very highly of it and I do as well.
     
  9. electricaltill

    Joined:
    Jan 19, 2013
    Messages:
    48
    Likes Received:
    0
    I must say I find it rather a black box as to when PPT jumps are given. I've done plenty of approving, and have seen the system propose PPT jumps and reductions for reasons that are not clear.

    I'll also say again it took a lot, lot, lot of work to get the PPT to the level of being auto-graded at 9 (it was 96 or 97 for me, I think). I'm talking something like 300, 400 transcripts. Having said that I also found the system to be easier on me the higher I got. I can't even remember the last time one of my transcripts was human graded, and I certainly can't remember the last time I even got a comment from a human grader.

    At the beginning though, there was a lot of crap to wade through. Constant trying to learn from graders, keeping a log of things not in the style guide, and frequent emails to CW support (shout-out to Mike here, who kept his cool when I was, frankly, kind of an ass to him at first) about graders and editors doing stuff that didn't agree with the style guide. All I can say is persevere, because the system isn't easy on you at first, and "at first" can last quite a while.
     
  10. _dogmeat

    _dogmeat Member

    Joined:
    Aug 4, 2013
    Messages:
    7
    Likes Received:
    0
    Just got my PPT qual increased by 4 points :) I guess there is some good to be had after all. Funny thing is, I was waiting to get approval for a transcription that I had done 2 or 3 days ago. I started to think that it wasn't going to get approved or something. And just now I logged in and see some awesome news :)

    Express transcriptions, here i come (Soon! Very Soon!) :D
     
  11. pwt

    pwt User

    Joined:
    Apr 28, 2012
    Messages:
    191
    Likes Received:
    0
    Nice. Well done.
     
    #31 pwt, Aug 5, 2013
    Last edited by a moderator: Aug 5, 2013
  12. _dogmeat

    _dogmeat Member

    Joined:
    Aug 4, 2013
    Messages:
    7
    Likes Received:
    0
    Okay, now I don't get it.

    Why the decrease again? I'm starting to think someone's seriously ******* with me. Have you guys experienced something similar to this? PPT decrease for an 8? What's next, they ban me unless I start submitting straight 9s?
     
  13. pwt

    pwt User

    Joined:
    Apr 28, 2012
    Messages:
    191
    Likes Received:
    0
    So you got a bump of +4, and then shortly after a drop of -1?

    That's pretty odd. Everybody's experience is unique and that certainly seems to be the case here, because my PPT went without changing for more than a year during one stretch.

    The best thing you can do is contact CastingWords support and ask them for more details, or just send them whatever questions you have. They won't tell you how the PPT is calculated, I'm sure, but they might tell you what's causing the jitter.

    I'm curious where you saw that message about your PPT changing. I've never gotten a notice before, I've always noticed it when specifically looking to see if it was changed in the qualification section of mTurk. Did you do that work through the Workshop or something?
     
  14. electricaltill

    Joined:
    Jan 19, 2013
    Messages:
    48
    Likes Received:
    0
    pwt, if you have an account on workers.castingwords.com (if you don't, you can create one with your turker ID, which links to your turking activities), you can see these message on the feedback section. They are also shown on the dashboard of the workshop.

    _dogmeat, this happens from time to time. You're not just dealing with CW, you're dealing with the turkers who grade/approve, who can be inconsistent (source: I am one of them). This has happened to me, more than once. After I qualified for expediteds, and had submitted nothing but 9s for a week, my grade dropped 2 points without explanation, which prevented me doing expediteds.

    Speak to CW support, explain the situation, and hopefully you will get the PPT drop rescinded. In my case, someone who shouldn't have been able to do approval jobs was able to do one of mine, and dropped my PPT. The system isn't perfect.
     
  15. pwt

    pwt User

    Joined:
    Apr 28, 2012
    Messages:
    191
    Likes Received:
    0
    I do, but I don't use the Workshop for a couple of reasons not relevant to this. So all my feedback ends up on turkers.castingwords.com. I get the feedback, I've just never been notified about a PPT change like that.

    I don't check feedback anymore because I don't get any (well I get very, very little) because of the auto-approvals. I wonder if maybe my last PPT change in March was in the turkers.cw feedback somewhere. I'll have to check it out.

    Thanks.
     
  16. _dogmeat

    _dogmeat Member

    Joined:
    Aug 4, 2013
    Messages:
    7
    Likes Received:
    0
    Okay, wrote support, we'll see what happens. Also, I've seen this thing happen when one particular grader was involved. Does that that have something to do with it?

    EDIT: Okay, CW are now definitely on my Top 3 of companies with great support. They already responded, and it wasn't really what I expected. They said the PPT drop is merely an attention getter, not a punitive drop. They bumped it back up :p
     
    #36 _dogmeat, Aug 7, 2013
    Last edited by a moderator: Aug 7, 2013
  17. electricaltill

    Joined:
    Jan 19, 2013
    Messages:
    48
    Likes Received:
    0
    I hardly get any either, for the same reason. If you are looking for the PPT notification on workers.castingwords, it doesn't have its own line in the feedback table, but it appears in the feedback for a job (i.e. you have to click through to the feedback screen for the individual job).

    It would be easier for you to look on the workshop for PPT bumps, because they have their own line in the feedback table.
     
  18. pwt

    pwt User

    Joined:
    Apr 28, 2012
    Messages:
    191
    Likes Received:
    0
    I looked through the feedback (all the way through, right to the payment page) for the 24th, 25th, and 26th (update came on the 25th) and don't see it. Maybe it got adjusted in a different way, when I didn't have any work pending or something.

    Anyhoo, thanks.
     
  19. Henrik

    Henrik User

    Joined:
    Jan 18, 2013
    Messages:
    1,621
    Likes Received:
    0
    CW's instructions clearly and specifically mention the word "gray" and that it should always be spelled that way and not "grey".

    When I do an employee review and an employee has done the exact opposite of my explicit instructions--even on something minor--that's coming up in his performance review and affecting his raise.

    That's just how it goes sometimes.

    I've had nothing but good things go on with CW. Great requester.
     
  20. dancingfingers

    Joined:
    Jun 21, 2013
    Messages:
    61
    Likes Received:
    0

    Actually they are one of the worse so called " transcription" companies on MTurk. Sadly, many " noobs" think they are learning how to transcribe but noting could be further from the truth. REAL transcription companies do not allow you to use [xx] consistenly yet worry about how " gray" " grey" is spelled, both are correct, if they want it spelled a specific way that's the job of the " editor" NOT the transcriptionist.

    I have transcribed for many years and I have to tell you NO other transcription company " dings" you for the above " grey" " gray" .

    I will tell you that I am a reviewer with one company on MTurk and I can always tell when someone has come from CW with al the [xx]s in their so -called " transcriptions, but hey... they got the spaces right and the spelling of the word " grey" corect!

    And, I will NEVER forget the CW review that I took on and had to correct over 85% of the transcription. The turker not only changed the city name, but also omitted words and added their own words, but again hey they got those spaces right and probably spelled ' grey" correctly. < rolls eyes>

    I freelance for a company that does not pay you if you have [xx] without a really good explanation... and I do mean good!

    CW maybe good for typists or data entry clerks, but they are NOT a good platform for those that want to learn to make transcription a career.
     
    #40 dancingfingers, Aug 15, 2013
    Last edited by a moderator: Aug 15, 2013

Share This Page