(Perl) check for broken links (shell one-liner) - noah

Sat, 15 Aug 2009 08:55:19 GMT

Retrieves links from a remote HTML page, then checks the response code of each link. Duplicated links are only checked once, and anchors are ignored. That is `http://foo` and `http://foo#bar` are considered to be the same URL, and thus `http://foo` will only be checked once; even if both URLs occur on the page. Note that if the command produces too much output for you, you can filter down to just the *broken* links (if any) by piping the output of the entire one-liner, to `grep -v "200 OK"` ## Dependencies These must be installed on your system: 0. [Perl](http://perl.org) 0. [lwp-request](http://search.cpan.org/~gaas/libwww-perl-5.831/bin/lwp-request) 0. [sort](http://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html 'GNU sort sorts lines in a text file') 0. [uniq](http://www.gnu.org/software/coreutils/manual/html_node/uniq-invocation.html 'GNU uniq filters duplicate items from a list') ## Troubleshooting **Check the quotes.** The command below has strings wrapped in *double* quotes (") which is appropriate if you are using a *Windows* shell. **If you are using a Mac or Linux shell** then you need to change the double quotes around the strings in the command below, to *single* quotes ('). Then all should work fine. It is good to keep in mind always that the Windows shell wants strings to be double-quoted, while Unix-ish shells want strings to be single-quoted.

Snipplr

(Perl) check for broken links (shell one-liner) - noah