403 Forbidden with Java but not web browser?

by

You just need to set user agent header for it to work:

URLConnection connection = new URL("https://www.google.com/search?q=" + query).openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.connect();

BufferedReader r  = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));

StringBuilder sb = new StringBuilder();
String line;
while ((line = r.readLine()) != null) {
    sb.append(line);
}
System.out.println(sb.toString());

The SSL was transparently handled for you as could be seen from your exception stacktrace.

Getting the result amount is not really this simple though, after this you have to fake that you’re a browser by fetching the cookie and parsing the redirect token link.

String cookie = connection.getHeaderField( "Set-Cookie").split(";")[0];
Pattern pattern = Pattern.compile("content=\\\"0;url=(.*?)\\\"");
Matcher m = pattern.matcher(response);
if( m.find() ) {
    String url = m.group(1);
    connection = new URL(url).openConnection();
    connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
    connection.setRequestProperty("Cookie", cookie );
    connection.connect();
    r  = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));
    sb = new StringBuilder();
    while ((line = r.readLine()) != null) {
        sb.append(line);
    }
    response = sb.toString();
    pattern = Pattern.compile("<div id=\"resultStats\">About ([0-9,]+) results</div>");
    m = pattern.matcher(response);
    if( m.find() ) {
        long amount = Long.parseLong(m.group(1).replaceAll(",", ""));
        return amount;
    }

}

Running the full code I get 2930000000L as a result.

More Related Contents:

  1. Why do we multiply 2 inside the for loop?
  2. What is x after “x = x++”?
  3. Retain precision with double in Java
  4. How to paginate Firestore with Android?
  5. How to use Key Bindings instead of Key Listeners
  6. How to redirect multiple types of users to their respective Activities?
  7. Why does Java prohibit static fields in inner classes?
  8. How to find unused/dead code in java projects [closed]
  9. Recommended way to get hostname in Java
  10. How can I write a Java application that can update itself at runtime?
  11. Why are the RSA-SHA256 signatures I generate with OpenSSL and Java different?
  12. How do I add one month to current date in Java?
  13. Java: how to import a jar file from command line
  14. Java regex for support Unicode?
  15. Enhanced ‘for’ loop causes an ArrayIndexOutOfBoundsException
  16. could not set the column width to zero i.e. not made column invisible
  17. MessageBodyProviderNotFoundException while running jar from command line
  18. How to change background color of JTabbedPane?
  19. Copy directory from a jar file
  20. Autowired Environment is null
  21. Empty an ArrayList or just create a new one and let the old one be garbage collected? [duplicate]
  22. Disabling Log4J Output in Java
  23. Who is calling the Java Thread interrupt() method if I’m not?
  24. How to initialize MongoClient once in Spring Boot and use its methods?
  25. Is SecureRandom thread safe?
  26. How to properly close MappedByteBuffer?
  27. FunctionalInterface Comparator has 2 abstract methods
  28. Should I use composite primary keys or not?
  29. Generalized Suffix Tree Java Implementation [closed]
  30. How to use Postgres JSONB datatype with JPA?

Leave a Comment