MTurk HIT DataBase (replaces MTurk Extended HIT Search)

Discussion in 'Scripts' started by ThirdClassIntMasterTurker, Oct 3, 2012.

  1. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    This is rewrite of MTurk Extended HIT Search with some extra features. It now uses IndexedDB (http://en.wikipedia.org/wiki/Indexed_Database_API) to store the HITs.

    It also implements some ideas from 'HIT history Check'-thread that use of IndexedDB made possible. See screenshots on userscripts page: MTurk HIT DataBase (http://userscripts.org/scripts/show/149548).

    Other new features: pending/projected earnings (with progress bar), CSV export/import, requester notes and requester blocking (you can block specific HITs).

    Current version is: 1.2 (2012-12-11)

    Works on Firefox and latest Chrome (tested on 23.0.1271.64)
     
    #1 ThirdClassIntMasterTurker, Oct 3, 2012
    Last edited by a moderator: Dec 11, 2012
  2. TurkworkOrange

    Joined:
    Sep 27, 2012
    Messages:
    434
    Likes Received:
    0
    Hi ThirdClassIntMasterTurker, so I've just loaded this up this week and I like the potential of it, but had a quick question about the initial load in of information. When I first update the database do I have to go to each individual day? The output I'm looking at has a lot of my previous requesters, but also is definitely missing a bunch. Out of roughly 1,000 HITs done, it's pulling info for about 390. How do I make sure it grabs everything?
    Thanks!
     
  3. Moutheater

    Moutheater User

    Joined:
    Sep 22, 2012
    Messages:
    1,452
    Likes Received:
    0
    I'd also like to know this please.
     
  4. naturegirl

    naturegirl User

    Joined:
    Aug 30, 2012
    Messages:
    701
    Likes Received:
    0
    It's actually not working for me at all in Chrome. The monkeys say they are working, but no data is pulled. Darned simians!

    Also, it apparently played havoc with the auto-approval script. I finally had to disable it and will probably go back to the Extended HIT Search script (not sure right now whether that's the official name) that I was using before. Loved that.
     
  5. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    You don't need to go to individual days. At first update it should get all the data that is available from amazon. That is last 45 Days. After that it only refetches status pages that have changed. You still have to update it manually every now and then, it doesn't get new HITs automatically.

    Have you done more than 390 HITs within last 45 days?
    You can find that from bottom of the submitted column on your status page (https://www.mturk.com/mturk/status). Thats the number of HITs there should be in DB after first update. Requester Overview button should then list all the requesters and show how many HITs there are in DB.
     
  6. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    It works on chrome 23 beta. That I have tested. I hope they get it stable soon.

    Sometimes it helps if you change the running order of scripts. But of course it they both want to change same part of the page that doesn't help.

    Yes, Extended HIT Search is earlier version. I made this MTurk HIT DataBase as a separate script because current chrome still have problems with it. You can keep using earlier version and try HIT DataBase again later.

    But DataBase version is definitely the better one. It saves all HITs and should be more scalable. Extended HIT Search just searches last 45 days, HIT DataBase potentially all the HITs you have ever done!
     
  7. naturegirl

    naturegirl User

    Joined:
    Aug 30, 2012
    Messages:
    701
    Likes Received:
    0
    Thank you. You're always so helpful!

    I think I just finished my first 45 days (like, I'm on day 46 or 47), so that means that days 1 and 2 are forever lost to me, right, because HIT DataBase is not working and Extended HIT Search can no longer pull it?

    In other words, am I correct that if you don't start using HIT DataBase in your first 45 days, you will never be able to get the data that is any earlier than 45 days before the day that you started using it?
     
  8. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    Well, Extended HIT Search doesn't currently remove any data so technically extra days are still there in its cache. And there will (almost certainly) be import function in MTurk HIT DataBase that gets old Extended HIT Search data and puts it in the new database.

    But yes, mturk only shows last 45 days. It used to be only 30 days if I remember correctly.

    If you want to make sure that you have almost all HIT data available in some form. You could temporarily install Firefox and HIT DataBase and then export HITs in CSV-format. Or just manually copy them to spredsheet from Extended HIT Search.
     
  9. TurkworkOrange

    Joined:
    Sep 27, 2012
    Messages:
    434
    Likes Received:
    0
    Yes, definitely. My status reports 1056 approved HITs, but the database only has 425. I am only just out of 45 days doing it. When I first did the update yesterday it took a couple minutes and then I did a couple updates during the day.
    It's useful in that it's a quick visual way to see recent ones I've done and that's basically what I wanted. I'm just not sure why it's not picking up the majority of HITs, so still leaves me questioning if I've done certain HITs before or not.
     
  10. naturegirl

    naturegirl User

    Joined:
    Aug 30, 2012
    Messages:
    701
    Likes Received:
    0
    Thank you! Unfortunately, I clean my cache out all the time, so I guess maybe that's why my monkeys are always working so hard...sounds like my first few days are lost, then. I'll see what I can do from here!
     
  11. Danthony

    Danthony User

    Joined:
    Jan 15, 2012
    Messages:
    334
    Likes Received:
    0
    I have not downloaded a single app of yours yet.
    Just wanted to say thanks from the community for developing apps for us
    Look forward to trying out your hard work
     
  12. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    Update to 0.17 and see if it helps. It has some bug fixes and error checks. It should at least give some error message if something goes wrong.
     
  13. TurkworkOrange

    Joined:
    Sep 27, 2012
    Messages:
    434
    Likes Received:
    0
    YES! now we're talking, numbers are virtually the same. Great job! Really appreciate the work you've put into this. I haven't fully played around with everything (notes, blocks, etc.), but will report back when I do. Turktastic!
     
  14. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    MTurk HIT DataBase 0.19

    Main interface is now on dashboard. I also added pending earnings to dashboard and summary of all requesters with pending HITs. And previous version added a button to import HIT data from Extended HIT Search.
     
  15. TurkworkOrange

    Joined:
    Sep 27, 2012
    Messages:
    434
    Likes Received:
    0
    Hi ThirdClassIntMasterTurker, quick question - does the database only keep information for the last 45 days or will it keep all data going forward? So, say I started recording HITs into the database in August, would those HITs now fall out of the database, or remain there indefinitely?
    Also thanks again for the script - it works brilliant and I use it every day.
     
  16. ewd76

    ewd76 User

    Joined:
    Oct 8, 2012
    Messages:
    1,133
    Likes Received:
    0
    I see the link to the extended Hit script, but not the link for the Mturk Hit Database script.
     
  17. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    No 45 day limit, it keeps all HITs until somebody/something deletes the database. IndexedDB shouldn't have any size limits either. (When database size grows to 50MB browser may ask extra confirmation.)

    If you are using Firefox you can make backups by copying indexedDB/https+++www.mturk.com from Firefox configuration directory to somewhere safe.
     
  18. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
  19. schambers

    schambers New Member

    Joined:
    Oct 29, 2012
    Messages:
    1
    Likes Received:
    0
    I want to say Thank You for all your work on this! I just started Turking and I love the script!
     
  20. ThirdClassIntMasterTurker

    Joined:
    Sep 8, 2012
    Messages:
    86
    Likes Received:
    0
    MTurk HIT DataBase should now work on stable Chrome!

    I just tested it quickly on Chrome 23.0.1271.64 (on Linux) and after few minor GUI updates it looks OK. Some features work better on Firefox, like progress bar while updating database, but some features seem to be faster on Chrome.

    Get version 0.32 from here: http://userscripts.org/scripts/show/149548.
     

Share This Page