Simplest way to correctly load html from web page into a string in Java

by

An extremely common error is the failure to correctly convert an HTTP response from bytes to characters. To do this, you have to know the character encoding of the response. Hopefully, this is specified as a parameter in the “Content-Type” parameter. But putting it in the body itself, as an “http-equiv” attribute in a meta tag is also an option.

So, it is surprisingly complicated to load a page into a String correctly, and even 3rd party libraries like HttpClient don’t offer a general solution.

Here’s a simple implementation that will handle the most common case:

URL url = new URL("http://stackoverflow.com/questions/1381617");
URLConnection con = url.openConnection();
Pattern p = Pattern.compile("text/html;\\s+charset=([^\\s]+)\\s*");
Matcher m = p.matcher(con.getContentType());
/* If Content-Type doesn't match this pre-conception, choose default and 
 * hope for the best. */
String charset = m.matches() ? m.group(1) : "ISO-8859-1";
Reader r = new InputStreamReader(con.getInputStream(), charset);
StringBuilder buf = new StringBuilder();
while (true) {
  int ch = r.read();
  if (ch < 0)
    break;
  buf.append((char) ch);
}
String str = buf.toString();

More Related Contents:

  1. Remove HTML tags from a String
  2. What are the pros and cons of the leading Java HTML parsers? [closed]
  3. How can I efficiently parse HTML with Java?
  4. Java HTML Parsing [closed]
  5. What HTML parsing libraries do you recommend in Java [closed]
  6. HTML/XML Parser for Java [closed]
  7. Reading HTML file to DOM tree using Java
  8. Parse JSON from HttpURLConnection object
  9. How can I parse a HTML string in Java?
  10. How to send HTTP request in java? [duplicate]
  11. Swing HTML drawString
  12. How to “scan” a website (or page) for info, and bring it into my program?
  13. Parsing an arithmetic expression and building a tree from it in Java
  14. How to read json file into java with simple JSON library
  15. how to download image from any web page in java
  16. By default, JSF generates unusable IDs, which are incompatible with the CSS part of web standards
  17. Javascript parser for Java [closed]
  18. Getting java gui to open a webpage in web browser
  19. Evaluate String as a condition Java
  20. Parsing time strings like “1h 30min”
  21. How to log into Facebook programmatically using Java?
  22. JEditorPane with inline image
  23. Set img src from Byte Array
  24. Why can’t this SimpleDateFormat parse this date string?
  25. Pure Java HTML viewer/renderer for use in a Scrollable pane [closed]
  26. Spring MVC Multiple ModelAttribute On the Same Form
  27. Instantiate a class object with constructor that accepts a string parameter?
  28. An HtmlUnit alternative for android?
  29. Encode URL query parameters
  30. How can I extract the text of HTML5 Constraint validation in https://www.phptravels.net/ website using Selenium and Java? [duplicate]

Leave a Comment