Snipplr

(Bash) LDAP/NTLM authentication on the command line with curl - noah

Thu, 27 Mar 2008 11:33:30 GMT

Note that curl will not follow redirects. Will prompt interactively for password if the -u option is omitted.

(Bash) WGet entire site with wget -pkr - noah

Sat, 16 Feb 2008 21:42:46 GMT

Download and archive an entire Web site, starting with the given page and recursing down 1 level. Adjust how many levels deep by changing the numeric argument given after -l Won't follow @import links in CSS.

(Bash) Scrape Google from the command line - noah

Sun, 09 Dec 2007 21:16:58 GMT

This code is POC only -- actually using it would violate Google's TOS, which forbids scraping. It is published here for educational value only. Hypothetically, the following command should return a list of the top 500 or so hits in Google for mysite.com. The results will be prepended with digits, followed by a dot and some whitespace (Lynx adds these).

(Bash) check linked pages for Tidy validation errors, on the command line - noah

Tue, 13 Nov 2007 23:47:28 GMT

Given a list of HTML links (most likely a saved Google results page) check each linked page and report if Tidy complains that its doctype declaration is missing. Besides DOCTYPE, other strings to search for include "discarding", "lacks value", "Error:"

(Bash) Validate a list of Web pages with Tidy, on the command line - noah

Tue, 13 Nov 2007 22:54:37 GMT

(Bash) Download linked JPEGs from a Web page, on the command line - noah

Fri, 02 Nov 2007 22:57:45 GMT

The following command will download all the files with a JPG extension that are linked from http://flickr.com.