http://snipplr.com/language/python/tags/crawler Recent snippets posted on Snipplr.com en-us Sun, 19 May 2013 11:01:17 GMT http://snipplr.com/view/66987/proxy-authentication-in-crawler/

]]> Sat, 01 Sep 2012 07:14:55 GMT http://snipplr.com/view/66987/proxy-authentication-in-crawler/ http://snipplr.com/view/65893/a-simple-spider-using-scrapy/ A simple spider using the [scrapy module](http://scrapy.org/) to get the text, title, url, author, and date of some poems. Although this is written with poems in mind, with some minor customization, it can be applied to a wider variety of scraping projects where the data desired is not dynamically generated.

]]> Sun, 01 Jul 2012 17:54:57 GMT http://snipplr.com/view/65893/a-simple-spider-using-scrapy/