Posted By

scrapy on 09/01/12


Tagged

javascript gtk downloader webkit middleware scrapy downloader-middleware


Versions (?)

Who likes this?

2 people have marked this snippet as a favorite

cberetta
lyroge


Rendered/interactive javascript with gtk/webkit/jswebkit


 / Published in: Python
 

  1. # This is a downloader middleware that can be used to get rendered javascript pages using webkit.
  2. #
  3. # this could be extended to handle form requests and load errors, but this is the bare bones code to get it done.
  4. #
  5. # the advantage over the selenium based approachs I've seen is that it only makes one request and you don't have to set up selenium.
  6.  
  7. from scrapy.http import Request, FormRequest, HtmlResponse
  8.  
  9. import gtk
  10. import webkit
  11. import jswebkit
  12.  
  13. class WebkitDownloader( object ):
  14. def process_request( self, request, spider ):
  15. if( type(request) is not FormRequest ):
  16. webview = webkit.WebView()
  17. webview.connect( 'load-finished', lambda v,f: gtk.main_quit() )
  18. webview.load_uri( request.url )
  19. gtk.main()
  20. js = jswebkit.JSContext( webview.get_main_frame().get_global_context() )
  21. renderedBody = str( js.EvaluateScript( 'document.documentElement.innerHTML' ) )
  22. return HtmlResponse( request.url, body=renderedBody )
  23.  
  24. # Snippet imported from snippets.scrapy.org (which no longer works)
  25. # author: jdwilson
  26. # date : Sep 06, 2011
  27.  

Report this snippet  

You need to login to post a comment.