Spoof requests as HTTP/1.1


/ Published in: Python
Save to your folder(s)



Copy this code and paste it in your HTML
  1. # This recipe was taken from the old wiki.
  2. #
  3. # You can make Scrapy send HTTP/1.1 requests by overriding the Scrapy HTTP Client Factory, with the following (undocumented) setting:
  4. #
  5. # DOWNLOADER_HTTPCLIENTFACTORY = 'myproject.downloader.HTTPClientFactory'
  6.  
  7. from scrapy.core.downloader.webclient import ScrapyHTTPClientFactory, ScrapyHTTPPageGetter
  8.  
  9. class PageGetter(ScrapyHTTPPageGetter):
  10.  
  11. def sendCommand(self, command, path):
  12. self.transport.write('%s %s HTTP/1.1
  13. ' % (command, path))
  14.  
  15. class HTTPClientFactory(ScrapyHTTPClientFactory):
  16.  
  17. protocol = PageGetter
  18.  
  19. # Snippet imported from snippets.scrapy.org (which no longer works)
  20. # author: pablo
  21. # date : Sep 16, 2011
  22.  

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.