Using wget to download all the files from a site

Forum Attachments

Using wget to download all the files from a site

Roguebantha 19 Jan 2013 02:46

I want to use wget (using Linux) to download every single zip file from a site. Nothing else, just the zip files…but I want to do it for every single link on the site. Basically, I want it to spider through the entire site, looking for every single zip file, and download it.

I'm certain this is possible, but I'm not quite sure how…right now I'm using
wget -r -A zip http://comeandlive.com/downloads/

To download all the zip files, but I feel like that's not getting them all, just the ones available straight from the page.

---

Boba Foxx (guest) 19 Jan 2013 04:56

I can think of a couple different file crawlers in python, but none that are a terminal command for linux. The main problem is that (IIRC) Linux's wget doesn't have the power of searching through a whole buncha subpages; it can only grab the specified filetypes that are directly linked to on apage.

Roguebantha 19 Jan 2013 18:25

Posted on cemetech too, looks like someone wrote a script for it. Should have the problem solved :)

---