Advanced Search

Spoof requests as HTTP/1.1

/ Published in: Python
Save to your folder(s)

Expand | Embed | Plain Text

Copy this code and paste it in your HTML 
  1. # This recipe was taken from the old wiki.
  2. # 
  3. # You can make Scrapy send HTTP/1.1 requests by overriding the Scrapy HTTP Client Factory, with the following (undocumented) setting:
  4. # 
  5. #     DOWNLOADER_HTTPCLIENTFACTORY = 'myproject.downloader.HTTPClientFactory'
  6.  
  7. from scrapy.core.downloader.webclient import ScrapyHTTPClientFactory, ScrapyHTTPPageGetter
  8.  
  9. class PageGetter(ScrapyHTTPPageGetter):
  10.  
  11.     def sendCommand(self, command, path):
  12.         self.transport.write('%s %s HTTP/1.1
  13. ' % (command, path))
  14.  
  15. class HTTPClientFactory(ScrapyHTTPClientFactory):
  16.  
  17.      protocol = PageGetter
  18.  
  19. # Snippet imported from snippets.scrapy.org (which no longer works)
  20. # author: pablo
  21. # date  : Sep 16, 2011
  22.  

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.