Posted By

level09 on 05/13/11


Tagged

curl php blocking filegetcontents googlebot


Versions (?)

byPAss curl / file_get_contents blocking for scrapping


 / Published in: PHP
 

  1. $userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
  2. $curl = curl_init($url);
  3. curl_setopt($curl, CURLOPT_USERAGENT, $userAgent);
  4. curl_setopt($curl, CURLOPT_AUTOREFERER, true);
  5. curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
  6. curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1 );
  7. curl_setopt($curl, CURLOPT_TIMEOUT, 2 );
  8. $html = curl_exec( $curl );
  9. curl_close( $curl);

Report this snippet  

You need to login to post a comment.