Snipplr - noah

(Bash) Scrape Google from the command line

Sun, 09 Dec 2007 21:16:58 GMT

This code is POC only -- actually using it would violate Google's TOS, which forbids scraping. It is published here for educational value only. Hypothetically, the following command should return a list of the top 500 or so hits in Google for mysite.com. The results will be prepended with digits, followed by a dot and some whitespace (Lynx adds these).

(Perl) scraper

Tue, 03 Jul 2007 22:48:36 GMT

For a while I used this to scrape weather.com. Then they changed their HTML and my script broke.

(Perl) Aggregate RSS feeds

Tue, 03 Jul 2007 22:45:27 GMT

Expects a newline-delimited list of RSS feed urls. Prints an HTML 4 page containing a grid of modules. The modules contain hyperlinked feed headlines. Very fragile, any invalid feed will break the whole aggregator.