Can Jsoup simulate a button press?

by

According to the HTML source of http://google.com the “I am feeling lucky” button has a name of btnI:

<input value="I'm Feeling Lucky" name="btnI" type="submit" onclick="..." />

So, just adding the btnI parameter to the query string should do (the value doesn’t matter):

http://www.google.com/search?hl=en&btnI=1&q=your+search+term

So, this Jsoup should do:

String url = "http://www.google.com/search?hl=en&btnI=1&q=balusc";
Document document = Jsoup.connect(url).get();
System.out.println(document.title());

However, this gave a 403 (Forbidden) error. 

Exception in thread "main" java.io.IOException: 403 error loading URL http://www.google.com/search?hl=en&btnI=1&q=balusc
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:387)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:364)
    at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:143)
    at org.jsoup.helper.HttpConnection.get(HttpConnection.java:132)
    at test.Test.main(Test.java:17)

Perhaps Google was sniffing the user agent and discovering it to be Java. So, I changed it:

String url = "http://www.google.com/search?hl=en&btnI=1&q=balusc";
Document document = Jsoup.connect(url).userAgent("Mozilla").get();
System.out.println(document.title());

This yields (as expected):

The BalusC Code

The 403 is however an indication that Google isn’t necessarily happy with bots like that. You might get (temporarily) IP-banned when you do this too often.

More Related Contents:

  1. can someone explain what this code is doing
  2. Jsoup Java HTML parser : Executing Javascript events
  3. How to “scan” a website (or page) for info, and bring it into my program?
  4. How do I preserve line breaks when using jsoup to convert html to plain text?
  5. jsoup posting and cookie
  6. How to connect via HTTPS using Jsoup?
  7. JSoup character encoding issue
  8. Problems submitting a login form with Jsoup
  9. Jsoup Cookies for HTTPS scraping
  10. Jsoup get dynamically generated HTML
  11. How to add proxy support to Jsoup?
  12. Sending POST request with username and password and save session cookie
  13. How to fill a form with Jsoup?
  14. How to parse XML with jsoup
  15. Jsoup connection with basic access authentication
  16. Android java.lang.NoClassDefFoundError: org.jsoup.Jsoup
  17. Getting Jsoup to support dynamically generated html by JavaScript
  18. Jsoup SocketTimeoutException: Read timed out
  19. How to post form login using jsoup?
  20. Why is char[] preferred over String for passwords?
  21. Mapping PostgreSQL JSON column to a Hibernate entity property
  22. Is it unnecessary to put super() in constructor?
  23. Get all of the Classes in the Classpath
  24. How to decode JSON with unknown field using Gson?
  25. How does memory reordering help processors and compilers?
  26. Signal handling using “TERM”
  27. Reader#lines() parallelizes badly due to nonconfigurable batch size policy in its spliterator
  28. Can’t rid of ‘T’ in LocalDateTime
  29. Get application context from non activity singleton class
  30. How can I get a PublicKey object from EC public key bytes?

Leave a Comment