Revision: 66561
Updated Code
at November 25, 2014 13:04 by zwidnypublic
Updated Code
# file: myscrapy/settings.py ... USER_AGENT = 'http://127.0.0.1:8087' DOWNLOADER_MIDDLEWARES = { 'myscrapy.middlewares.MyProxyMiddleware': 100, 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110, 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None, } ... # file: myscrapy/middlewares.py from myscrapy.settings import USER_AGENT class MyProxyMiddleware(object): def process_request(self, request, spider): request.meta['proxy'] = USER_AGENT
Revision: 66560
Initial Code
Initial URL
Initial Description
Initial Title
Initial Tags
Initial Language
at May 23, 2014 18:04 by zwidnypublic
Initial Code
# file: myscrapy/settings.py ... USER_AGENT = 'http://127.0.0.1:8087' DOWNLOADER_MIDDLEWARES = { 'myscrapy.middlewares.MyProxyMiddleware': 100, 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110, 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None, } ... # file: myscrapy/middlewares.py from genecards.settings import USER_AGENT class MyProxyMiddleware(object): def process_request(self, request, spider): request.meta['proxy'] = USER_AGENT
Initial URL
Initial Description
Scrapy crawl with goagent agent I say you goagent list address: http://127.0.0.1:8087 and you create a scrapy project named: myscrapy. and you pwd is myscrapy
Initial Title
Using goagent agent in Scrapy
Initial Tags
Initial Language
Python