Can servers block curl requests?

by

Many web servers want to block HTTP requests forged by something else than a browser, to prevent bots abuses.
If you want to simulate/pretend your request from a browser, you at least have to:

  1. Pass the exact same headers than your browsers (use ie Firebug to get them)

    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

  2. Change the user agent (name of the browser)

    curl_setopt($ch, CURLOPT_USERAGENT, $agent);

  3. Enable cookies (for eg redirection and session handling)

    curl_setopt ($ch, CURLOPT_COOKIEJAR, $file);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);

  4. Add referers 

    curl_setopt($curl, CURLOPT_REFERER, 'http://www.google.com');
curl_setopt($curl, CURLOPT_AUTOREFERER, true);

And pray you haven’t missed anything!

More Related Contents:

  1. We need a script to create a backup of all the files, database and email using third party Cpanel details in PHP
  2. How do I make a request using HTTP basic authentication with PHP curl?
  3. How to POST JSON Data With PHP cURL?
  4. how to upload file using curl with PHP [closed]
  5. How to catch cURL errors in PHP
  6. Downloading a large file using curl
  7. Curl and PHP – how can I pass a json through curl by PUT,POST,GET
  8. Send file via cURL from form POST in PHP
  9. How do I install cURL on Windows?
  10. Scrape web page contents
  11. PHP cURL error code 60
  12. How do I use arrays in cURL POST requests
  13. Persistent/keepalive HTTP with the PHP Curl library?
  14. PHP Curl – Cookies problem
  15. How do I install the ext-curl extension with PHP 7?
  16. cURL , get redirect url to a variable
  17. curl and ping – how to check whether a website is either up or down?
  18. PHP multipart form data PUT request?
  19. PHP cURL HTTP CODE return 0
  20. 999 Error Code on HEAD request to LinkedIn
  21. PHP cURL how to add the User Agent value OR overcome the Servers blocking cURL requests?
  22. understanding php curl_multi_exec
  23. Get file content via PHP cURL [closed]
  24. How to set a maximum size limit to PHP cURL downloads?
  25. Make cURL output STDERR to file (or string)
  26. PHP cURL: CURLOPT_CONNECTTIMEOUT vs CURLOPT_TIMEOUT
  27. How can I get the destination URL using cURL?
  28. Escaping CURL @ symbol with PHP
  29. FTP upload file to distant server with CURL and PHP uploads a blank file
  30. How can I send SOAP XML via Curl and PHP?

Leave a Comment