• code snippets
  • all code snippets/
  • popular code snippets/
  • your code snippets
• extras
• blog
• about snipplr

• Big /
• Small

login / Register

Share Snipplr

Tools

• Textmate Bundle
• WordPress Plugin
• Snippets

Popular Tags

Everyone's Recent Python Snippets Tagged 'scrapy'

• All / 
• JavaScript / 
• HTML / 
• PHP / 
• CSS / 
• Ruby / 
• Objective C

1. Middleware to avoid re-visiting already visited items

   Python middleware scrapy saved by 1 person
   posted on September 1, 2012 by scrapy

2. soccerway.com soccer matches data extraction

   Python spider scrapy soccer
   posted on September 1, 2012 by scrapy

3. Parsley Spider

   Python parsley scrapy parselet
   posted on September 1, 2012 by scrapy

4. Run Scrapy crawler in a thread

   Python thread scripts scrapy
   posted on September 1, 2012 by scrapy

5. Upload file (uber upload)

   Python spider scrapy file-upload uberupload-upload-file-mime
   posted on September 1, 2012 by scrapy

6. Use a django project with virtualenv from Scrapy

   Python django virtualenv scrapy saved by 1 person
   posted on September 1, 2012 by scrapy

7. Self-contained script to crawl a site [updated: scrapy 13.0dev]

   Python scrapy single-script
   posted on September 1, 2012 by scrapy

8. Monkey patch for images pipeline

   Python images monkeypatch scrapy pipeline
   posted on September 1, 2012 by scrapy

9. scrapy-service.tac (Twisted Application Framework)

   Python application service twisted scrapy
   posted on September 1, 2012 by scrapy

10. DropElderMiddleware

    Python scrapy
    posted on September 1, 2012 by scrapy

11. Using Scrapy crawler with a blocking API from a thread

    Python blocking api Threads scrapy
    posted on September 1, 2012 by scrapy

12. Downloader middleware to redirect to rel=canonical urls

    Python rel downloader canonical middleware scrapy
    posted on September 1, 2012 by scrapy

13. Text-file Spider Queue

    Python file spider Queue scrapy
    posted on September 1, 2012 by scrapy

14. URL Canonicalizer Spider middleware

    Python url scrapy c18n canonicalization spider-middleware
    posted on September 1, 2012 by scrapy

15. Using Scrapy from a script

    Python scrapy
    posted on September 1, 2012 by scrapy

16. Scrapy snippet to gather RSS feeds on a page(using feedparser)

    Python scrapy saved by 1 person
    posted on September 1, 2012 by scrapy

17. robot exclusion rules parser

    Python robotstxt robots scrapy downloadmiddleware
    posted on September 1, 2012 by scrapy

18. Eurovision.tv year archive parser

    Python parser spider scrapy crawlspider eurovision
    posted on September 1, 2012 by scrapy

19. Redirect Scrapy log messages to standard Python logger

    Python logging scrapy
    posted on September 1, 2012 by scrapy

20. Sitemap Spider

    Python spider sitemap scrapy basespider
    posted on September 1, 2012 by scrapy

21. Rendered Javascript Crawler With Scrapy and Selenium RC

    Python javascript spider selenium scrapy crawlspider saved by 1 person
    posted on September 1, 2012 by scrapy

22. rendered javascript with webdrivers

    Python javascript rendered scrapy webdrivers
    posted on September 1, 2012 by scrapy

23. Rendered/interactive javascript with gtk/webkit/jswebkit

    Python javascript gtk downloader webkit middleware scrapy downloader-middleware
    posted on September 1, 2012 by scrapy

24. Script to print crawl tree of a spider run

    Python script tree crawl scrapy
    posted on September 1, 2012 by scrapy

25. Spoof requests as HTTP/1.1

    Python downloader scrapy http11
    posted on September 1, 2012 by scrapy

26. Avoid downloading pages which exceed a certain size

    Python download size limit scrapy
    posted on September 1, 2012 by scrapy

27. Use a random user agent for each request

    Python random user agent useragent middleware scrapy
    posted on September 1, 2012 by scrapy

28. make and filter the log file into a html file

    Python html script log scrapy
    posted on September 1, 2012 by scrapy

29. My approach to stats extension

    Python stats extensions scrapy
    posted on September 1, 2012 by scrapy

30. Async Twisted Db Pipeline

    Python database Asynchronous twisted scrapy adbapi
    posted on September 1, 2012 by scrapy

31. Raising CloseSpider

    Python import scrapy closespider
    posted on September 1, 2012 by scrapy

32. Proxy Authentication in Crawler

    Python crawler authentication proxy scrapy proxy-authenticate
    posted on September 1, 2012 by scrapy

33. MySQL Pipeline

    Python mysql scrapy pipeline adbapi
    posted on September 1, 2012 by scrapy

34. Django and Scrapy without using DjangoItem

    Python scrapy
    posted on September 1, 2012 by scrapy

35. Submit scraped items to Message Queue (amqp)

    Python Carrot scrapy amqp message-queue
    posted on September 1, 2012 by scrapy

36. CrawlPathMiddleware: easily control the crawl path of a spider

    Python scrapy
    posted on September 1, 2012 by scrapy

37. Scrapy pipeline class to store scraped data in MongoDB

    Python mongodb scrapy pipeline
    posted on July 1, 2012 by hlongmore

38. A simple spider using scrapy

    Python crawler spider scraping scrapy saved by 1 person
    posted on July 1, 2012 by hlongmore