Advanced Search

Strip punctuation from text.

/ Published in: PHP
Save to your folder(s)
When processing text for a search engine or analysis tool, code needs to strip out punctuation, formatting, spacing, and control characters to reveal indexable text. In international text there are hundreds of these characters, and some should be removed in one context, but not in another. This tip shows how.

Expand | Embed | Plain Text

Copy this code and paste it in your HTML 
  1. function strip_punctuation( $text )
  2. {
  3.     $urlbrackets    = '\[\]\(\)';
  4.     $urlspacebefore = ':;\'_\*%@&?!' . $urlbrackets;
  5.     $urlspaceafter  = '\.,:;\'\-_\*@&\/\\\\\?!#' . $urlbrackets;
  6.     $urlall         = '\.,:;\'\-_\*%@&\/\\\\\?!#' . $urlbrackets;
  7.  
  8.     $specialquotes  = '\'"\*<>';
  9.  
  10.     $fullstop       = '\x{002E}\x{FE52}\x{FF0E}';
  11.     $comma          = '\x{002C}\x{FE50}\x{FF0C}';
  12.     $arabsep        = '\x{066B}\x{066C}';
  13.     $numseparators  = $fullstop . $comma . $arabsep;
  14.  
  15.     $numbersign     = '\x{0023}\x{FE5F}\x{FF03}';
  16.     $percent        = '\x{066A}\x{0025}\x{066A}\x{FE6A}\x{FF05}\x{2030}\x{2031}';
  17.     $prime          = '\x{2032}\x{2033}\x{2034}\x{2057}';
  18.     $nummodifiers   = $numbersign . $percent . $prime;
  19.  
  20.     return preg_replace(
  21.         array(
  22.         // Remove separator, control, formatting, surrogate,
  23.         // open/close quotes.
  24.             '/[\p{Z}\p{Cc}\p{Cf}\p{Cs}\p{Pi}\p{Pf}]/u',
  25.         // Remove other punctuation except special cases
  26.             '/\p{Po}(?<![' . $specialquotes .
  27.                 $numseparators . $urlall . $nummodifiers . '])/u',
  28.         // Remove non-URL open/close brackets, except URL brackets.
  29.             '/[\p{Ps}\p{Pe}](?<![' . $urlbrackets . '])/u',
  30.         // Remove special quotes, dashes, connectors, number
  31.         // separators, and URL characters followed by a space
  32.             '/[' . $specialquotes . $numseparators . $urlspaceafter .
  33.                 '\p{Pd}\p{Pc}]+((?= )|$)/u',
  34.         // Remove special quotes, connectors, and URL characters
  35.         // preceded by a space
  36.             '/((?<= )|^)[' . $specialquotes . $urlspacebefore . '\p{Pc}]+/u',
  37.         // Remove dashes preceded by a space, but not followed by a number
  38.             '/((?<= )|^)\p{Pd}+(?![\p{N}\p{Sc}])/u',
  39.         // Remove consecutive spaces
  40.             '/ +/',
  41.         ),
  42.         ' ',
  43.         $text );
  44. }

URL: http://nadeausoftware.com/articles/2007/9/php_tip_how_strip_punctuation_characters_web_page#Removinglineparagraphandwordseparators

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.