This is a reCAPTCHA (version 2, check out demos here), which is shown to you as the owner of the page does not want you to automatically crawl the page.
Your options are the following:
Option 1: Stop crawling or try to use an official API
As the owner of the page does not want you to crawl that page, you could simply respect that decision and stop crawling. Maybe there is a documented API that you can use.
Option 2: Automate/Outsource the captcha solving
There is an entire industry which has people (often in developing countries) filling out captchas for other people’s bots. I will not link to any particular site, but you can check out the other answer from Md. Abu Taher for more information on the topic or search for captcha solver.
Option 3: Solve the captcha yourself
For this, let me explain how reCAPTCHA works and what happens when you visit a page using it.
How reCAPTCHA (v2) works
Each page has an ID, which you can check by looking at the source code, example:
<div class="g-recaptcha form-field" data-sitekey="ID_OF_THE_WEBSITE_LONG_RANDOM_STRING"></div>
When the reCAPTCHA code is loaded it will add a response textarea to the form with no value. It will look like this:
<textarea id="g-recaptcha-response" name="g-recaptcha-response" class="g-recaptcha-response" style="... display: none;"></textarea>
After you solved the challenge, reCAPTCHA will add a very long string to this text field (which can then later be checked by the server/reCAPTCHA service in the backend) when the form is submitted.
How to solve the captcha yourself
By copying the value of the textarea field you can transfer the “solved challenge” from one browser to another (this is also what the solving services to for you). The full process looks like this:
- Detect if the page uses reCAPTCHA (e.g. check for
.g-recaptcha) in the “crawling” browser - Open a second browser in non-headless mode with the same URL
- Solve the captcha yourself
- Read the value from:
document.querySelector('#g-recaptcha-response').value - Put that value into the first browser:
document.querySelector('#g-recaptcha-response').value="..." - Submit the form
Further information/reading
There is not much public information from Google how exactly reCAPTCHA works as this is a cat-and-mouse game between bot creators and Google detection algorithms, but there are some resources online with more information:
- Official docs from Google: Obviously, they just explain the basics and not how it works “in the back”
- InsideReCaptcha: This is a project from 2014 which tries to “reverse-engineer” reCAPTCHA. Although this is quite old, there is still a lot of useful information on the page.
- Another question on stackoverflow: This question contains some useful information about reCAPTCHA, but also many speculative (and very likely) outdated approaches on how to fool a reCAPTCHA.