Posted By

scrapy on 09/01/12


Tagged

mysql scrapy pipeline adbapi


Versions (?)

Who likes this?

1 person have marked this snippet as a favorite

hahahaha


MySQL Pipeline


 / Published in: Python
 

  1. # Cannot use this to create the table, must have table already created
  2.  
  3. from twisted.enterprise import adbapi
  4. import datetime
  5. import MySQLdb.cursors
  6.  
  7. class SQLStorePipeline(object):
  8.  
  9. def __init__(self):
  10. self.dbpool = adbapi.ConnectionPool('MySQLdb', db='mydb',
  11. user='myuser', passwd='mypass', cursorclass=MySQLdb.cursors.DictCursor,
  12. charset='utf8', use_unicode=True)
  13.  
  14. def process_item(self, item, spider):
  15. # run db query in thread pool
  16. query = self.dbpool.runInteraction(self._conditional_insert, item)
  17. query.addErrback(self.handle_error)
  18.  
  19. return item
  20.  
  21. def _conditional_insert(self, tx, item):
  22. # create record if doesn't exist.
  23. # all this block run on it's own thread
  24. tx.execute("select * from websites where link = %s", (item['link'][0], ))
  25. result = tx.fetchone()
  26. if result:
  27. log.msg("Item already stored in db: %s" % item, level=log.DEBUG)
  28. else:
  29. tx.execute(\
  30. "insert into websites (link, created) "
  31. "values (%s, %s)",
  32. (item['link'][0],
  33. datetime.datetime.now())
  34. )
  35. log.msg("Item stored in db: %s" % item, level=log.DEBUG)
  36.  
  37. def handle_error(self, e):
  38. log.err(e)
  39.  
  40. # Snippet imported from snippets.scrapy.org (which no longer works)
  41. # author: redtricycle
  42. # date : Nov 21, 2011
  43.  

Report this snippet  

Comments

RSS Icon Subscribe to comments
Posted By: hahahaha on August 21, 2014

Hi. How can it used to create the database and table if they do not exist.

You need to login to post a comment.