• code snippets
  • all code snippets/
  • popular code snippets/
  • your code snippets
• extras
• blog
• about snipplr

• Big /
• Small

login / Register

Share Snipplr

Tools

• Textmate Bundle
• WordPress Plugin
• Snippets

Popular Tags

Everyone's Recent Python Snippets

• All / 
• JavaScript / 
• HTML / 
• PHP / 
• CSS / 
• Ruby / 
• Objective C

1. Self-contained script to crawl a site [updated: scrapy 13.0dev]

   Python scrapy single-script
   posted on September 1, 2012 by scrapy

2. DropElderMiddleware

   Python scrapy
   posted on September 1, 2012 by scrapy

3. Monkey patch for images pipeline

   Python images monkeypatch scrapy pipeline
   posted on September 1, 2012 by scrapy

4. Using Scrapy from a script

   Python scrapy
   posted on September 1, 2012 by scrapy

5. URL Canonicalizer Spider middleware

   Python url scrapy c18n canonicalization spider-middleware
   posted on September 1, 2012 by scrapy

6. Text-file Spider Queue

   Python file spider Queue scrapy
   posted on September 1, 2012 by scrapy

7. Downloader middleware to redirect to rel=canonical urls

   Python rel downloader canonical middleware scrapy
   posted on September 1, 2012 by scrapy

8. Using Scrapy crawler with a blocking API from a thread

   Python blocking api Threads scrapy
   posted on September 1, 2012 by scrapy

9. scrapy-service.tac (Twisted Application Framework)

   Python application service twisted scrapy
   posted on September 1, 2012 by scrapy

10. robot exclusion rules parser

    Python robotstxt robots scrapy downloadmiddleware
    posted on September 1, 2012 by scrapy

11. Scrapy snippet to gather RSS feeds on a page(using feedparser)

    Python scrapy saved by 1 person
    posted on September 1, 2012 by scrapy

12. Redirect Scrapy log messages to standard Python logger

    Python logging scrapy
    posted on September 1, 2012 by scrapy

13. Eurovision.tv year archive parser

    Python parser spider scrapy crawlspider eurovision
    posted on September 1, 2012 by scrapy

14. Sitemap Spider

    Python spider sitemap scrapy basespider
    posted on September 1, 2012 by scrapy

15. Rendered Javascript Crawler With Scrapy and Selenium RC

    Python javascript spider selenium scrapy crawlspider saved by 1 person
    posted on September 1, 2012 by scrapy

16. rendered javascript with webdrivers

    Python javascript rendered scrapy webdrivers
    posted on September 1, 2012 by scrapy

17. Rendered/interactive javascript with gtk/webkit/jswebkit

    Python javascript gtk downloader webkit middleware scrapy downloader-middleware
    posted on September 1, 2012 by scrapy

18. Script to print crawl tree of a spider run

    Python script tree crawl scrapy
    posted on September 1, 2012 by scrapy

19. Spoof requests as HTTP/1.1

    Python downloader scrapy http11
    posted on September 1, 2012 by scrapy

20. Avoid downloading pages which exceed a certain size

    Python download size limit scrapy
    posted on September 1, 2012 by scrapy

21. Use a random user agent for each request

    Python random user agent useragent middleware scrapy
    posted on September 1, 2012 by scrapy

22. make and filter the log file into a html file

    Python html script log scrapy
    posted on September 1, 2012 by scrapy

23. My approach to stats extension

    Python stats extensions scrapy
    posted on September 1, 2012 by scrapy

24. Async Twisted Db Pipeline

    Python database Asynchronous twisted scrapy adbapi
    posted on September 1, 2012 by scrapy

25. Raising CloseSpider

    Python import scrapy closespider
    posted on September 1, 2012 by scrapy

26. Proxy Authentication in Crawler

    Python crawler authentication proxy scrapy proxy-authenticate
    posted on September 1, 2012 by scrapy

27. MySQL Pipeline

    Python mysql scrapy pipeline adbapi
    posted on September 1, 2012 by scrapy

28. Django and Scrapy without using DjangoItem

    Python scrapy
    posted on September 1, 2012 by scrapy

29. Submit scraped items to Message Queue (amqp)

    Python Carrot scrapy amqp message-queue
    posted on September 1, 2012 by scrapy

30. CrawlPathMiddleware: easily control the crawl path of a spider

    Python scrapy
    posted on September 1, 2012 by scrapy

31. MIT - 6.00 Problem Set 0

    Python
    posted on August 16, 2012 by jsovine

32. create buffer using gdal in python

    Python gdal
    posted on August 10, 2012 by mbsweng2

33. SMTP sink server

    Python debug server smtp django
    posted on August 3, 2012 by cphoover

34. Use email addresses for user name

    Python authentication django
    posted on August 3, 2012 by cphoover

35. Getting shovel to work in windows so python methods can be run like individual executables

    Python python windows batch shovel
    posted on July 22, 2012 by pckujawa

36. Improved ARC4 (IARC4)

    Python stream Encryption Cryptography cipher RC4 ARC4 RC4A RC4-drop8192 VMPC IARC4
    posted on July 20, 2012 by weilawei

37. [Project Euler] Exercise 02

    Python project Euler
    posted on July 18, 2012 by akus85

38. [Project Euler] Exercise 01

    Python project Euler
    posted on July 18, 2012 by akus85

39. TextBlock

    Python
    posted on July 9, 2012 by NunAttack

40. Scrapy pipeline class to store scraped data in MongoDB

    Python mongodb scrapy pipeline
    posted on July 1, 2012 by hlongmore