What should I use to crawl/download/archive an entire website? It's all simple static pages (no JavaScript), but has lots of links to download small binary files, which I also want to preserve. Any OS -- just want the best tools.
@cancel It can only follow HTML code, naturally, it'll follow all hyperlinks regardless of data type.
@blindcoder No, it's not downloading .zip files that are linked from .html files.
@cancel @blindcoder are the binary files hosted on the same domain as the html+images+css? I think wget needs explicit options to allow getting from multiple domains in recursive mode, and it might also need options to limit recursion depth in that case to avoid downloading the whole internet...
@mathr @blindcoder They're on the same domain. It looks like either wget requires the file extensions be added to a list of accepted file extensions, or also that robots.txt be ignored. I can do the latter, but I'm not sure how to do the former, because there are many varied file extensions I want to back up. Is there a way to wildcard it?
@blindcoder It only seems to download .html, images, css, etc.