The API key itself is most probably a one way hash of the domain the key is associated with and a secret only the Google API server knows about. It may contain some other pieces of well-known (to Google of course) information. When you make a request from that domain, the API server takes the domain the request comes from and makes that same one way hash calculation and compares the two values.
For Ajax calls, they most probably use the referrer to get the domain of the document host. While the referrer can be spoofed, ultimately in order to use the API, you need to get Google javascript to execute in the document. At this point, this javascript can verify that indeed the document that invoked the Ajax API call originated from the target server. This is also spoofable of course, provided you have your own DOM implementation or on the fly modification of the script. However, this spoofing needs to happen on the client side and the chances that the website that wants to use Google API will be able to spoof the client software are quite small.
Note that since the API is essentially free, they could’ve offered anonymous access to their API as well. Apparently Google’s intent is not to protect unauthorized access to it, but to ensure that they can gather as much data as possible about that data usage and be able to associate that usage with other data they’ve collected about the target domain. As such, I wouldn’t expect the API key verification to be much more complex than what I described above – the ROI on more advanced approach is too low.
And of course there’s also the concern of possible XSS attacks through their API. But I don’t believe their API key is tied too much into any anti-XSS code they have.