Posted By

scrapy on 09/01/12


Tagged

downloader scrapy http11


Versions (?)

Spoof requests as HTTP/1.1


 / Published in: Python
 

  1. # This recipe was taken from the old wiki.
  2. #
  3. # You can make Scrapy send HTTP/1.1 requests by overriding the Scrapy HTTP Client Factory, with the following (undocumented) setting:
  4. #
  5. # DOWNLOADER_HTTPCLIENTFACTORY = 'myproject.downloader.HTTPClientFactory'
  6.  
  7. from scrapy.core.downloader.webclient import ScrapyHTTPClientFactory, ScrapyHTTPPageGetter
  8.  
  9. class PageGetter(ScrapyHTTPPageGetter):
  10.  
  11. def sendCommand(self, command, path):
  12. self.transport.write('%s %s HTTP/1.1\r\n' % (command, path))
  13.  
  14. class HTTPClientFactory(ScrapyHTTPClientFactory):
  15.  
  16. protocol = PageGetter
  17.  
  18. # Snippet imported from snippets.scrapy.org (which no longer works)
  19. # author: pablo
  20. # date : Sep 16, 2011
  21.  

Report this snippet  

You need to login to post a comment.