(See UPDATE below, first/accepted solution didn’t met the android requirement, but is left for reference.) Desktop Solution HtmlUnit doesn’t seem able to handle this site (often the case, lately). So I don’t have a plain java solution either, but you could use PhantomJS: download the binary for your os, create a script file, start the … Read more
Besides the username, password and the cookies, the site requeires two additional values for the login – VIEWSTATE and EVENTVALIDATION. You can get them from the response of the first Get request, like this – Document doc = loginForm.parse(); Element e = doc.select(“input[id=__VIEWSTATE]”).first(); String viewState = e.attr(“value”); e = doc.select(“input[id=__EVENTVALIDATION]”).first(); String eventValidation = e.attr(“value”); And … Read more
The charset attribute is missing in HTTP response Content-Type header. Jsoup will resort to platform default charset when parsing the HTML. The Document.OutputSettings#charset() won’t work as it’s used for presentation only (on html() and text()), not for parsing the data (in other words, it’s too late already). You need to read the URL as InputStream … Read more
If you want to do it the right way, and/or you need to deal with only one site, then you basically need to grab the SSL certificate of the website in question and import it in your Java key store. This will result in a JKS file which you in turn set as SSL trust … Read more
When you login to the site, it is probably setting an authorised session cookie that needs to be sent on subsequent requests to maintain the session. You can get the cookie like this: Connection.Response res = Jsoup.connect(“http://www.example.com/login.php”) .data(“username”, “myUsername”, “password”, “myPassword”) .method(Method.POST) .execute(); Document doc = res.parse(); String sessionId = res.cookie(“SESSIONID”); // you will need … Read more
The real solution that preserves linebreaks should be like this: public static String br2nl(String html) { if(html==null) return html; Document document = Jsoup.parse(html); document.outputSettings(new Document.OutputSettings().prettyPrint(false));//makes html() preserve linebreaks and spacing document.select(“br”).append(“\\n”); document.select(“p”).prepend(“\\n\\n”); String s = document.html().replaceAll(“\\\\n”, “\n”); return Jsoup.clean(s, “”, Whitelist.none(), new Document.OutputSettings().prettyPrint(false)); } It satisfies the following requirements: if the original html contains newline(\n), … Read more
Use a HTML parser like Jsoup. This has my preference above the other HTML parsers available in Java since it supports jQuery like CSS selectors. Also, its class representing a list of nodes, Elements, implements Iterable so that you can iterate over it in an enhanced for loop (so there’s no need to hassle with … Read more
JSoup is just an HTML parser/”tidyfier” – not a browser emulator. To interact with HTML pages (execute javascript, fill out forms, etc.) you should use a tool like HtmlUnit or Selenium.
JSoup is an HTML parser, not some kind of embedded browser engine. This means that it’s completely unaware of any content that is added to the DOM by Javascript after the initial page load. To get access to that type of content you will need an embedded browser component, there are a number of discussions … Read more
I don’t think anyone will take the time to explain the code for you. A good opportunity for you to do some debugging. ArrayIndexOutOfBounds comes from response.split(“\”extract\”:\””)[1]. There is no guarantee that the String response can be split into at least 2 parts. Add a check to avoid the error. Instead of… String result = … Read more