Hopefully, this is an exhaustive list of things to check when your site won’t scrape:
1) Is your site on a spam blacklist?
This is rare, but Facebook and most other tools won’t parse your site at all if it shows up on a spammer blacklist.
I use https://admin.uribl.com/ as a checker. If your site is listed, you need to find and clean the malware on your site, then follow the instructions from the blacklist owner(s) to remove your site. If the problem is that you’ve got a host who is a known spammer, you’ll need to change hosts. It’s going to take a few days for this to work its way through the system before any site will scrape your site again.
2) Is your (X)HTML valid?
Facebook’s parser is very strict. If the headers sent by your web server or your HTML isn’t valid, Facebook will not parse your site. To test this in detail, use the Markup Validator from the W3C. You have to resolve all of the errors before Facebook will parse your page.
Some of the most common errors I have seen are:
- Invalid string sent in the headers
- Mismatch between the
character-encodingsent in the header and the<meta charset>tag in the document. - Invalid or incorrect
<!DOCTYPE> - Whitespace before the ` tag
- Malformed HTML tags, especially in the
<head> - Tags closed with
>instead of/>in XHTML documents
3) Are you redirecting your visitors with JavaScript?
The Facebook parser does not execute JavaScript. If you want to redirect a visitor to custom content, you need to do this with a server-side script.
4) Is your server refusing connections to non-browsers?
This is harder to diagnose, but some servers are set to return a 500:Server Error or 403:Forbidden for any non-browser visitor.
5) Does the Facebook site tell you your link is blocked?
Log into Facebook and attempt to share a link on your timeline. If your site appears in the Facebook internal blacklist, you will get a message telling you the site is blocked. On this dialog, there is a form where you can mark this as a false positive and request a review of your site.
If you end up on this list, Facebook users are blocking your postings or marking them as spam. That probably originates in your content. What you think is SEO is probably spamdexing or the content you are sharing is offensive or polarizing to some users, or you’re just sharing the same stuff over and over again.
Once you have fixed the error, visit the Facebook Debugger again. A manual visit to the debugger clears Facebook’s cache for that URL. Give things a few minutes for Facebook to push the updates to all servers, and then try again.