Snipplr

(Java) Simple User Agent (Mobile and Smartphone) Detection and Redirction with PHP - Klostrag

Sat, 22 Dec 2012 01:56:10 GMT

This is similar to the one for PHP but modified for JAVA.

(Python) Use a random user agent for each request - scrapy

Sat, 01 Sep 2012 07:15:13 GMT

(Python) A Clueless Agent Generator for Python 3.2 - weilawei

Mon, 07 May 2012 00:01:14 GMT

This is an implementation of a clueless agent generator which creates self-decrypting clueless agents as described in "Environmental Key Generation towards Clueless Agents" by J. Riordan and B. Schneier. It requires Python 3.2 and PyCrypto of a recent build (tested with 2.4 and higher). To use, pass a python file (or other file) to be encrypted, followed by a series of "observations" on the command line. These observations are hashed to yield the encryption key. A signature is generated by hashing the key, and this signature will be expected to be present in the target environment. Pipe the resulting agent to a file or see the agent code directly on stdout. Additionally, there is an is_debug flag that can be specified (see the source) or tweaked in the resulting agent, to be more verbose. To attempt decryption/execution of a clueless agent, simply run the generated python script (agent) and pass a set of observations on the command line. If the hash of the hash of the observations match the signature, the hash of the observations will be used as the decryption key. If the signature does not match, the agent will exit with no output. The code previously directly exec()'d the resulting code, however, it simply outputs to stdout now. The resulting code would otherwise execute directly in-line, at that location in the program, which has many undesirable consequences. Piping it to a file and executing, piping it to a memory-backed temporary file and executing it, or placing the resulting code directly in memory afterward and then executing it, are all ways to run the code contained within. This makes it fundamentally little different from encrypting a file directly, except that the key is environmentally generated, perhaps by a daemon that feeds environmental observations on the command line to the agent. Note, you can encrypt more than Python scripts, and agents can be made to contain themselves. $ ./agent_generator.py plaincode.py 0 > cipheragent.py $ ./agent_generator.py cipheragent.py some more observations > double_agent.py $ ./double_agent.py wrong observations --nothing here-- $ ./double_agent.py some more observations > cipheragent_2.py --cipheragent_2.py now holds the same content as cipheragent.py-- $ ./cipheragent.py 0 > plaincode_2.py --plaincode_2.py now holds the same content as plaincode.py-- $ ./plaincode_2.py --should yield the same as-- $ ./plaincode.py

(MySQL) (Player - Player Agent) Images - karthi

Wed, 22 Feb 2012 18:58:34 GMT

(Player - Player Agent) Images

(PHP) Detect user agent - i0ni

Sun, 23 Jan 2011 02:19:09 GMT

(PHP) How to easily detect mobile devices - dropthenerd

Thu, 20 Jan 2011 02:14:13 GMT

(PHP) PHP Based Access Blocking Version 2.0 - MMDeveloper

Tue, 29 Dec 2009 11:12:12 GMT

This is a new version of a previous snippet I wrote. This version is smaller, more efficient, and allows the use of regular expressions. To use: $blocker = new blocker(); if ($blocker->isBlocked === true) { //blocked user, do whatever } else {} OR if you want to test IF a certain IP or hostname or user agent WOULD be blocked: $blocker = new blocker(); list($isBlocked, $reasonsBlocked) = $blocker->testIfBlocked([userIP], [userAgent], [userHostname]); if ($isBlocked === true) { print_r($reasonsBlocked); } else {} The 3 arguments are optional, but if specified, must be specified in that order. Any unspecified arguments will default to the values of the user executing the call. For example, if you do not specify a user agent and host name, and you're the one executing the test, it will use YOUR user agent and host name. An array of reasons why the test said you failed will be returned (along with a boolean true/false as to if the criteria would be blocked or not).

(Perl) lwp-request - noah

Mon, 25 Feb 2008 12:28:41 GMT

(Bash) Download an entire site with wget -pkr - noah

Sat, 16 Feb 2008 21:42:46 GMT

## Where to Get Even More WGet Hacks See also these killer `wget` [hacks by Jeff Veen.](http://www.veen.com/jeff/archives/000573.html) ## The WGet Hacks Here are a couple of recipes to **download and archive an entire Web site,** starting with the given page and recursing down 1 level. Adjust how many levels deep by changing the numeric argument given after -l ## Pitfalls As of 2008, WGet doesn't follow @import links in CSS.

(Perl) submit a POST query with parameters - noah

Tue, 03 Jul 2007 22:57:13 GMT

from The Perl Black Book, Holzner, p. 1247

(Perl) scraper - noah

Tue, 03 Jul 2007 22:48:36 GMT

For a while I used this to scrape weather.com. Then they changed their HTML and my script broke.

(Perl) Grab linked files from a list of web pages - noah

Tue, 03 Jul 2007 22:31:30 GMT

## how to use `perl grabit.pl urls_for_download.txt` Expects as argument the name of a file containing a newline-delimited list of URLs: http://example.com/coolstuff http://example.com/coolstuff/fun http://example.com/videos/explosions When invoked, launches an interactive shell that asks what type of file should be downloaded. Then downloads all the files that are linked from each of the listed Web pages. Note that the location of the download folder is hard-coded to `c:/windows/desktop/grabit/` so you may want to change that before trying. This script is also [available on Github](http://github.com/textarcana/scrapers/blob/643e6e7cb349fa94cbc3fc88e1d55c7b6a262d11/grabit.pl) ## Wait! Do you know about WGet and Curl? This script is legacy. People seem to like it (hey, I still use it) but today I would probably not write my own tool to download multiple files off remote sites. Instead I would likely just use a command-line Web browser like [WGet](http://lifehacker.com/software/top/geek-to-live--mastering-wget-161202.php 'Gina Trapani of Lifehacker, on the way of the WGet ninja') or Curl. [LWP-Request would also do the trick](http://snipplr.com/view/4063/download-linked-jpegs-from-a-web-page-on-the-command-line/) ## do not comment your code like this! For a great explanation of the rather baroque commenting style I was using circa 2001, see [Steve Yegge's excellent article on code style: *Portait of a n00b.*](http://steve-yegge.blogspot.com/2008/02/portrait-of-n00b.html) Of course, when I sit down to write a Perl script today, I [use POD](http://snipplr.com/view/18611/perl-pod-embedded-documentation-example/) to format and publish my comments.