Advanced Search

Using goagent agent in Scrapy

/ Published in: Python
Save to your folder(s)
Scrapy crawl with goagent agent
I say you goagent list address: http://127.0.0.1:8087
and you create a scrapy project named: myscrapy.
and you pwd is myscrapy

Expand | Embed | Plain Text

Copy this code and paste it in your HTML 
  1. # file: myscrapy/settings.py
  2. ...
  3.  
  4. USER_AGENT = 'http://127.0.0.1:8087'
  5. DOWNLOADER_MIDDLEWARES = {
  6.     'myscrapy.middlewares.MyProxyMiddleware': 100,
  7.     'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
  8.     'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
  9. }
  10. ...
  11.  
  12. # file: myscrapy/middlewares.py
  13. from myscrapy.settings import USER_AGENT
  14.  
  15. class MyProxyMiddleware(object):
  16.     def process_request(self, request, spider):
  17.         request.meta['proxy'] = USER_AGENT

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.