Advanced Search

Rendered Javascript Crawler With Scrapy and Selenium RC

/ Published in: Python
Save to your folder(s)

Expand | Embed | Plain Text

Copy this code and paste it in your HTML 
  1. # Many times when crawling we run into problems where content that is rendered on the page is generated with Javascript and therefore scrapy is unable to crawl for it (eg. ajax requests, jQuery craziness).  However, if you use Scrapy along with the web testing framework Selenium then we are able to crawl anything displayed in a normal web browser.
  2. # 
  3. # Some things to note:
  4. # You must have the Python version of Selenium RC installed for this to work, and you must have set up Selenium properly.  Also this is just a template crawler.  You could get much crazier and more advanced with things but I just wanted to show the basic idea.  As the code stands now you will be doing two requests for any given url.  One request is made by Scrapy and the other is made by Selenium.  I am sure there are ways around this so that you could possibly just make Selenium do the one and only request but I did not bother to implement that and by doing two requests you get to crawl the page with Scrapy too.
  5. # 
  6. # This is quite powerful because now you have the entire rendered DOM available for you to crawl and you can still use all the nice crawling features in Scrapy.  This will make for slower crawling of course but depending on how much you need the rendered DOM it might be worth the wait.
  7.  
  8. from scrapy.contrib.spiders import CrawlSpider, Rule
  9. from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  10. from scrapy.selector import HtmlXPathSelector
  11. from scrapy.http import Request
  12.  
  13. from selenium import selenium
  14.  
  15. class SeleniumSpider(CrawlSpider):
  16. 	name = "SeleniumSpider"
  17. 	start_urls = ["http://www.domain.com"]
  18.  
  19. 	rules = (
  20. 		Rule(SgmlLinkExtractor(allow=('\.html', )), callback='parse_page',follow=True),
  21. 	)
  22.  
  23. 	def __init__(self):
  24. 		CrawlSpider.__init__(self)
  25. 		self.verificationErrors = []
  26. 		self.selenium = selenium("localhost", 4444, "*chrome", "http://www.domain.com")
  27. 		self.selenium.start()
  28.  
  29. 	def __del__(self):
  30. 		self.selenium.stop()
  31. 		print self.verificationErrors
  32. 		CrawlSpider.__del__(self)
  33.  
  34. 	def parse_page(self, response):
  35. 		item = Item()
  36.  
  37. 		hxs = HtmlXPathSelector(response)
  38. 		#Do some XPath selection with Scrapy
  39. 		hxs.select('//div').extract()
  40.  
  41. 		sel = self.selenium
  42. 		sel.open(response.url)
  43.  
  44. 		#Wait for javscript to load in Selenium
  45. 		time.sleep(2.5)
  46.  
  47. 		#Do some crawling of javascript created content with Selenium
  48. 		sel.get_text("//div")
  49. 		yield item
  50.  
  51. # Snippet imported from snippets.scrapy.org (which no longer works)
  52. # author: wynbennett
  53. # date  : Jun 21, 2011
  54.  

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.