Posted By

scrapy on 09/01/12


Tagged

crawler authentication proxy scrapy proxy-authenticate


Versions (?)

Proxy Authentication in Crawler


 / Published in: Python
 

  1. # To authenticate the proxy, you must set the Proxy-Authorization header. You *cannot* use the form http://user:pass@proxy:port in request.meta['proxy']
  2.  
  3. import base64
  4.  
  5. proxy_ip_port = "123.456.789.10:8888"
  6. proxy_user_pass = "awesome:dude"
  7.  
  8. request = Request(url, callback=self.parse)
  9.  
  10. # Set the location of the proxy
  11. request.meta['proxy'] = "http://%s" % proxy_ip_port
  12.  
  13. # setup basic authentication for the proxy
  14. encoded_user_pass=base64.encodestring(proxy_user_pass)
  15. request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass
  16.  
  17. # Snippet imported from snippets.scrapy.org (which no longer works)
  18. # author: redtricycle
  19. # date : Nov 21, 2011
  20.  

Report this snippet  

You need to login to post a comment.