Topic
10,516 pages of Snaps... can I have them all?
So, yes, I downloaded the almost 600 Snaps that I took personally. But. I want them ALL. According to my rough calculations, there have been approximately 220,836 (plus or minus some) publicly available published snaps on the site.
Barring going through all the pages manually and saving them individually, is there an easy way for me to save them/download them? Sure, they're not all MY memories, but I want to save them anyway.
Replies
- People might be willing to send your their ZIP files.
- I'd be willing to do that.
- I have published all my snaps at www.box.com/s/wd6yz486hzl91...
-
I think the best you're going to do at this point is ask people to upload their snaps or look at places like the Facebook group or Flickr pool. (although a download of all the snaps ever would be pretty darn cool)
Here are my snaps on dropbox and here are ALLCAPS PARTY's snaps. - I'm guessing somebody's probably developed an addon for a browser for you to be able to save all the images in a page/region. Just a guess though. Other than that, I would think you would have to go through every single page and save as every single little snap, if you really wanted all the published snaps.
- Here are mine.
-
I have written a little script which does that: It crawls through all the "recent snaps" pages and saves the pictures (only the largest version of each) and all the information on the snap's page (caption, location, date, comments, ...)
I have asked staff whether it is ok with them to do that and if it is, I will start it as soon as I get an answer and publish what I got as soon as it is done. - If they compressed them Tiny Spec could make a nice chunk of coin by doing a dvd of not just our snaps but a few videos as well.... ???
- @scheijan - I will anxiously await updates, then :) thank you for the effort!
- I got an ok from Little Poundcake - the script is running now. It will take a while to download all of them, I will post here from time to time to keep you updated on the progress.
- Woohoo!
- Fantastic! I was hoping to get all the snaps.
- omg i want the snaps too!
- OMW! You are a legend Scheijan! If you need me to buy you a dropbox account to store/make them available please let me know.
- *applauds Scheijan and Little Poundcake* :D
-
I'm not really sure how to distribute them yet. I have crawled >1500 of the pages now, I have roughly 32k snaps. Looking at the sizes I will end up somewhere between 100GB and 200GB.
Also right now I'm only "dumping" all the infos (author, date, location, ...) in some kind of internal format. I will have to run over all of those again (local) to add them to a database and/or convert them to some kind of human readable format. -
When I woke up today I had >125k images, the script will still run for a while. I'm also getting a few hundred errors so far, most of them because the author's account has been deleted. I will check those error files once the script is finished, maybe I will have to correct something and download those missing snaps afterwards.
So far it looks like I won't need as much space as I estimated: I have more than 50% now and it "only" takes up ~56GB so far... - I've been looking through the pages of snaps that my friends have "liked," thus manually downloading most of the best snaps
-
Soo.... All the snaps are downloaded now. 220649 snaps, ~95GB. I'm writing a script to analyse the ~1500 errors I got on my way - most of those will be deleted accounts, I can't do anything about that. I'll see if there were any other problems that I can fix to get any missing snaps.
And now the question is how to distribute it. I was thinking about using BitTorrent, if there are a few people who would join in that would be a good way to distribute them all.
And another question is what to do with the meta data (author, location, date, comments, ...). The way it is stored now most people can't really use it. Eventually I will fill a database with it so that I can set up a small website where one can search for snaps and browse. And also to make some kind of statistics page - most snapped location, people with the most snaps, most snapped Glitchen, and so on. But for now - if any of you is interested in the meta data: What format would be useful? - As I said I don't mind hosting it on my dropbox. I can torrent it from you or something and stick it up on my account or can get you an account for a couple of months. The bottom rung of paid accounts is 102GB.
- Could it be split into several torrents? I'd be happy to help seed a torrent of it but i'm not sure I could keep the whole thing on my laptop at once.
- This is pretty cool, scheijan :3 I'd love to see it once everything's all figured out.
- @scheijan - I got a lot of errors too while downloading favorited snaps- it seems to either be deleted acct's or just deleted snaps, unfortunately.
- I can just imagine a Glitch Photosynth to explore the streets of Glitch again.
- I could help host a torrent file as well. It will be a week or so till I'm back from my Glitchmas break in the land of the Yetis, but if it is still needed then I'd love to help.
- oh my GOSH. This is...this is INCREDIBLE. I'd be happy to help seed the torrent -- is that even the right term? LOL I barely know how to use it, but yes please! You're amazing, schejian!
- did the person who is downloading the snaps block anyone? if they did, they probably wont see all the snaps
- It even happens just when scrolling through the individual pages... you can see the preview of the pics, but clicking on them just gives you an error. I don't think if you've got the person blocked you'd necessarily see the preview though.
-
Regarding the meta data, you could write a script/program that adds a white box at the bottom of the images with the meta data printed into the box? There are several open source JPEG/PNG libraries that should make this pretty straightforward. Not sure if your familiar with Java, lots of image/database libraries for Java you could use.
Just a thought,
Cheers,
~T -
One other thought, you could load the meta information into the EXIF data for other types of programmatic parsing...
~T - I can also seed the torrent. Thanks Scheijan.
- I would be happy to seed the torrent as well! I'm really excited about having all the snaps. :)
- As I mentioned before I will seed and will also host the entire collection on dropbox for those people who don't know how to torrent.
-
Also volunteering to seed! I think BitTorrent is an excellent solution due to the fact that it can be resumed in the event of a disconnect, and also paused if necessary, thus making the large file available to people with slower connections as well.
As to the meta data, I think EXIF data is readable with Windows 7 (I've never bothered to check), so that would be an excellent solution for me. I also like the idea of the white box at the bottom, however I would be concerned about possible quality loss with the automatic editing and saving of the files.
This is exciting, thanks so much, scheijan! - Could the person who created the "Glitch Goodies" torrent please put it back up for a while? Or anyone who was able to grab the whole thing? There are a bunch of us who are stuck at 89.9%. Thank you.
-
I will look into the EXIF idea - I don't really like the "white box thingy". That way the data can't easily be used by anyone else anymore (and yes, I am aware that one could use an OCR to extract it again, but that's rather silly).
Thanks for the hint Kristabel, some snaps weren't downloaded because I had someone blocked, those are being added right now... - Torrent is probably the best distribution solution. I like an EXIF idea for the data, as reusability and being able to create interesting apps or sites form the data in the future with preserved data would be awesome.
- I would love a torrent of the snaps, and am totally willing to seed them, too.
- We should probably be starting to exchange outside info for communication, or join the Facebook group as the forums are possibly closed in under 2 days. www.facebook.com/groups/307...
- Please don't limit info exchange to Facebook. Much as I love Glitch and would like to be informed about this project, I won't join Facebook for any reason.
- I will keep posting here as long as we have the forum.
- And when we don't anymore, how can we get in touch with you? :)
- I am very inexperienced in all this. Can you tell me if I will be able to look through the snaps once they are all available, and choose the ones I want? My computer couldn't handle all of them!
- Perhaps they could be hosted in a flickr account? Or something similar I guess...
- I can put them on my flickr and/or dropbox for always after I download them via scheijan's torrent. I can also break them up, offer as a whole, zip, whatever people want.
- @Cleops - Parts would be hugely appreciated, if you had to time to do that. Much as I want the whole enchilada, I don't really have the space for downloading all of it at once on my HD.
- If anyone is not on facebook you can message me through glitch (or the email in my profile) and i can keep you up to date on the progress of this when the forum closes.
-
Not sure if a method of distribution has been decided on for this, but one of my friends has given me permission to use his seedbox to download and seed this indefinitely, if torrents should be decided upon as the method.
(also, a +1 to giving the necessary information over not-Facebook... I too have quite the aversion to using it) - +1 for not-Facebook info distribution. Would love love love a Torrent of these photos. Great work, scheijan!
- @ Mercedes Kimura thank you for sharing some really great pictures!
- @Miss Bobbit - I am seeding the Glitch Goodies everyday but also have it here on my dropbox: www.dropbox.com/sh/pqp7lhvn...