Posted By

scrapy on 09/01/12


Tagged

application service twisted scrapy


Versions (?)

scrapy-service.tac (Twisted Application Framework)


 / Published in: Python
 

  1. # This is a tac file that can be used for launching Scrapy as a service (in the [Twisted Application Framework](http://twistedmatrix.com/documents/current/core/howto/application.html) ) using twistd.
  2. #
  3. # You can start the service with:
  4. #
  5. # twistd -ny scrapy-service.tac
  6. #
  7. # And then schedule spiders with:
  8. #
  9. # scrapy queue add myspider
  10.  
  11. from twisted.application.service import Service, Application
  12. from twisted.python import log as txlog
  13.  
  14. from scrapy import log
  15. from scrapy.crawler import Crawler
  16. from scrapy.conf import settings
  17.  
  18. class CrawlerService(Service):
  19.  
  20. def startService(self):
  21. settings.overrides['QUEUE_CLASS'] = settings['SERVER_QUEUE_CLASS']
  22. self.crawler = Crawler(settings)
  23. self.crawler.install()
  24. self.crawler.start()
  25.  
  26. def stopService(self):
  27. return self.crawler.stop()
  28.  
  29. def get_application(logfile, loglevel=log.DEBUG):
  30. app = Application("Scrapy")
  31. app.setComponent(txlog.ILogObserver, \
  32. log.ScrapyFileLogObserver(open(logfile, 'a'), loglevel).emit)
  33. CrawlerService().setServiceParent(app)
  34. return app
  35.  
  36. application = get_application('scrapy.log')
  37.  
  38. # Snippet imported from snippets.scrapy.org (which no longer works)
  39. # author: pablo
  40. # date : Aug 26, 2010
  41.  

Report this snippet  

You need to login to post a comment.