advanced code snippet search

We Recommend

bash Cookbook: Solutions and Examples for bash Users bash Cookbook: Solutions and Examples for bash Users
bash Cookbook teaches shell scripting the way Unix masters practice the craft. It presents a variety of recipes and tricks for all levels of shell programmers so that anyone can become a proficient user of the most common Unix shell -- the bash shell -- and cygwin or other popular Unix emulation packages.

Posted By

noah on 02/16/08


Tagged

auth archive recursive download cookies mirror automation agent scraping authorization


Versions (?)


Who likes this?

1 person has marked this snippet as a favorite

kyokutyo


WGet entire site with wget -pkr

Published in: Bash 

URL: http://lifehacker.com/software/top/geek-to-live--mastering-wget-161202.php

Download and archive an entire Web site, starting with the given page and recursing down 1 level. Adjust how many levels deep by changing the numeric argument given after -l

Won't follow @import links in CSS.

Plain TextExpand 
  1. #Get page.com and each page it links to as well as linked assets like images and CSS.  Change hyperlinks to point to the locally downloaded pages.
  2. wget -pkr -l 1 http://site
  3.  
  4.  
  5. #Same as above but also follow links to other domains.
  6. wget -Hpkr -l 1 http://site
  7.  
  8. #Same as the first example, but use a cookie
  9. wget -pkr -l 1 --no-cookies --header "Cookie: JSESSIONID=12345" https://securesite

Report this snippet 

You need to login to post a comment.