Nginx 403 error: directory index of [folder] is forbidden

If you have directory indexing off, and is having this problem, it’s probably because the try_files you are using has a directory option: location / { try_files $uri $uri/ /index.html index.php; } ^ that is the issue Remove it and it should work: location / { try_files $uri /index.html index.php; } Why this happens TL;DR: … Read more

Problem HTTP error 403 in Python 3 Web Scraping

This is probably because of mod_security or some similar server security feature which blocks known spider/bot user agents (urllib uses something like python urllib/3.3.0, it’s easily detected). Try setting a known browser user agent with: from urllib.request import Request, urlopen req = Request(‘http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1’, headers={‘User-Agent’: ‘Mozilla/5.0’}) webpage = urlopen(req).read() This works for me. By the way, … Read more

403 Forbidden with Java but not web browser?

You just need to set user agent header for it to work: URLConnection connection = new URL(“https://www.google.com/search?q=” + query).openConnection(); connection.setRequestProperty(“User-Agent”, “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11”); connection.connect(); BufferedReader r = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName(“UTF-8”))); StringBuilder sb = new StringBuilder(); String line; while ((line = r.readLine()) != null) { sb.append(line); } … Read more

403 Forbidden vs 401 Unauthorized HTTP responses

A clear explanation from Daniel Irvine: There’s a problem with 401 Unauthorized, the HTTP status code for authentication errors. And that’s just it: it’s for authentication, not authorization. Receiving a 401 response is the server telling you, “you aren’t authenticated–either not authenticated at all or authenticated incorrectly–but please reauthenticate and try again.” To help you … Read more