Advanced Search

Return to Snippet

Revision: 66561
at November 25, 2014 13:04 by zwidnypublic
Updated Code 
# file: myscrapy/settings.py
...

USER_AGENT = 'http://127.0.0.1:8087'
DOWNLOADER_MIDDLEWARES = {
    'myscrapy.middlewares.MyProxyMiddleware': 100,
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
}
...

# file: myscrapy/middlewares.py
from myscrapy.settings import USER_AGENT

class MyProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = USER_AGENT

Revision: 66560
at May 23, 2014 18:04 by zwidnypublic
Initial Code 
# file: myscrapy/settings.py
...

USER_AGENT = 'http://127.0.0.1:8087'
DOWNLOADER_MIDDLEWARES = {
    'myscrapy.middlewares.MyProxyMiddleware': 100,
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
}
...

# file: myscrapy/middlewares.py
from genecards.settings import USER_AGENT

class MyProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = USER_AGENT
Initial URL 









Initial Description





Scrapy crawl with goagent agent
I say you goagent list address: http://127.0.0.1:8087
and you create a scrapy project named: myscrapy.
and you pwd is myscrapy









Initial Title





Using goagent agent in Scrapy









Initial Tags














Initial Language





Python