Snipplr

(Perl) check for broken links (shell one-liner) - noah

Sat, 15 Aug 2009 08:55:19 GMT

Retrieves links from a remote HTML page, then checks the response code of each link. Duplicated links are only checked once, and anchors are ignored. That is `http://foo` and `http://foo#bar` are considered to be the same URL, and thus `http://foo` will only be checked once; even if both URLs occur on the page. Note that if the command produces too much output for you, you can filter down to just the *broken* links (if any) by piping the output of the entire one-liner, to `grep -v "200 OK"` ## Dependencies These must be installed on your system: 0. [Perl](http://perl.org) 0. [lwp-request](http://search.cpan.org/~gaas/libwww-perl-5.831/bin/lwp-request) 0. [sort](http://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html 'GNU sort sorts lines in a text file') 0. [uniq](http://www.gnu.org/software/coreutils/manual/html_node/uniq-invocation.html 'GNU uniq filters duplicate items from a list') ## Troubleshooting **Check the quotes.** The command below has strings wrapped in *double* quotes (") which is appropriate if you are using a *Windows* shell. **If you are using a Mac or Linux shell** then you need to change the double quotes around the strings in the command below, to *single* quotes ('). Then all should work fine. It is good to keep in mind always that the Windows shell wants strings to be double-quoted, while Unix-ish shells want strings to be single-quoted.

(Perl) The Onion's What Do You Think? Scrape to RSS - noah

Sat, 06 Sep 2008 23:08:32 GMT

Old and busted but still gets the job done.

(Perl) Mother 3 Translation Blog RSS Tidy Up - noah

Sat, 06 Sep 2008 23:03:31 GMT

This one I like. Scrapes the blog and fixes their RSS feed to include images, video, mini updates and whatnot. Images are busted and too lazy to fix. Left up to an exercise for the reader.

(Perl) The Onion Infographic Scrape to RSS - noah

Sat, 06 Sep 2008 22:59:51 GMT

Scrapes the infographic to RSS. Old and busted.

(Perl) Girl With a One Track Mind Scraper - noah

Sat, 06 Sep 2008 22:58:36 GMT

Didn't bother fixing it but works well enough to scrape Girl With a One Track Mind

(Perl) Get Your War On Scrape to RSS - noah

Sat, 06 Sep 2008 22:57:22 GMT

Really old and busted Get Your War On scraper but it still works so there.

(Perl) Broken Gallup Election 2008 Polls Scrape to RSS - noah

Sat, 06 Sep 2008 22:56:19 GMT

Scrapes the presidential stuff into an RSS feed.

(Perl) Engrish Scrape to RSS - noah

Sat, 06 Sep 2008 22:52:28 GMT

Scrapes Engrish to RSS.

(Perl) APOD Scrape to RSS - noah

Sat, 06 Sep 2008 22:49:22 GMT

Scrapes the Astronomy Picture of the Day and packages it into a messy RSS feed.

(Perl) Achewood Scrape to RSS - noah

Sat, 06 Sep 2008 22:47:26 GMT

Generates an RSS feed from Achewood's stuff.

(Perl) New York Times Scrape to RSS - noah

Sat, 06 Sep 2008 22:44:32 GMT

Crawls all over the NYT RSS feed to glue entire articles together for your enjoyment. Requires registration. Pick a feed URL from http://topics.nytimes.com/top/reference/timestopics/index.html and have at it. perl nyt.pl --user=aksdhf --pass=aksdfhasouidf --url=http://asiudhfasdfj

(Perl) scraper - noah

Tue, 03 Jul 2007 22:48:36 GMT

For a while I used this to scrape weather.com. Then they changed their HTML and my script broke.