Snipplr

(Python) robot exclusion rules parser - scrapy

Sat, 01 Sep 2012 07:15:30 GMT

(HTML) MODx TV for setting page visibility for robots - hotdiggity

Sun, 13 Nov 2011 12:25:55 GMT

Create a new TV (under elements tab) called "Robots" and past the following into the input field. Then apply it to all the templates of the pages you wish to see this TV option. Set default to: Set delimiter to: || Add the TV to the template header as [[*robots]] (v2+) or [*robots*] (v1 or less). There's probably a more eloquent way of scripting with a snippet using instead the default MODx template variables of "Cacheable" and "Searchable"!

(HTML) Search engine meta description for robots noindex nofollow - stur

Tue, 30 Aug 2011 20:16:24 GMT

(Other) Robots.txt - Disallow Certain File Type - batfan

Thu, 28 Jul 2011 08:11:12 GMT

(Other) Robots.txt - Disallow Certain Query String - batfan

Thu, 28 Jul 2011 08:10:03 GMT

(PHP) Magento set robots in Dev server - necode

Thu, 21 Jul 2011 00:06:55 GMT

Magento first steps - for dev server. Creating Robots.txt file.

(Apache) Simple Robots.txt Examples - ryarwood

Sat, 18 Jun 2011 02:50:13 GMT

(Other) Google noindex as Apache directive - michaellehmkuhl

Thu, 04 Mar 2010 09:33:05 GMT