Posted By

zwidnypublic on 05/23/14


Tagged

agent scrapy


Versions (?)

Using goagent agent in Scrapy


 / Published in: Python
 

Scrapy crawl with goagent agent I say you goagent list address: http://127.0.0.1:8087 and you create a scrapy project named: myscrapy. and you pwd is myscrapy

  1. # file: myscrapy/settings.py
  2. ...
  3.  
  4. USER_AGENT = 'http://127.0.0.1:8087'
  5. DOWNLOADER_MIDDLEWARES = {
  6. 'myscrapy.middlewares.MyProxyMiddleware': 100,
  7. 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
  8. 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
  9. }
  10. ...
  11.  
  12. # file: myscrapy/middlewares.py
  13. from myscrapy.settings import USER_AGENT
  14.  
  15. class MyProxyMiddleware(object):
  16. def process_request(self, request, spider):
  17. request.meta['proxy'] = USER_AGENT

Report this snippet  

You need to login to post a comment.