• code snippets
  • all code snippets/
  • popular code snippets/
  • your code snippets
• extras
• blog
• about snipplr

• Big /
• Small

login / Register

Share Snipplr

Tools

• Textmate Bundle
• WordPress Plugin
• Snippets

Popular Tags

Everyone's Recent Snippets Tagged 'robots'

• All / 
• JavaScript / 
• HTML / 
• PHP / 
• CSS / 
• Ruby / 
• Objective C

1. robot exclusion rules parser

   Python robotstxt robots scrapy downloadmiddleware
   posted on September 1, 2012 by scrapy

2. MODx TV for setting page visibility for robots

   HTML seo Sem modx robots evo rev0
   posted on November 13, 2011 by hotdiggity

3. Search engine meta description for robots noindex nofollow

   HTML html meta nofollow robotstxt noindex robots
   posted on August 30, 2011 by stur

4. Robots.txt - Disallow Certain File Type

   Other filetype robots
   posted on July 28, 2011 by batfan

5. Robots.txt - Disallow Certain Query String

   Other String query robots
   posted on July 28, 2011 by batfan

6. Magento set robots in Dev server

   PHP magento robots
   posted on July 21, 2011 by necode

7. Simple Robots.txt Examples

   Apache server apache admin robots
   posted on June 18, 2011 by ryarwood

8. Google noindex as Apache directive

   Other header textmate apache httpd seo append noindex robots directive
   posted on March 4, 2010 by michaellehmkuhl