How to get an AJAX get-request to wait for the page to be rendered before returning a response?

In order for the AJAX get to “wait for the page to be rendered”, it would actually have to fully process the page, fetching and running all the included CSS and javascript files. That’s not easy and not recommended. Fortunately, you don’t need to do that anyway.

Here are three better ways to approach this kind of problem:

1. The resource page (mpdining.rewardsnetwork.com, for this question) might have an API. If it does, find it and use it. This is your best bet, if it’s available.

2. Analyze the resource page’s javascript and/or AJAX requests. Use GM_xmlhttpRequest() to directly fetch just the payload data, instead of trying to parse the resource page.

   Sometimes this process is fairly easy, but some sites require complex interaction and/or authentication.

3. Load the resource page in a hidden iframe; set your Greasemonkey script to run on both the resource page and the master page and to relay the desired data using postMessage().

   This approach will almost always work, although you may have to prevent some pages from attempting to “bust out” of the iframe.

---

Using a hidden iframe to get data from a cross-domain, resource page:

Greasemonkey scripts will run on both a normal page and on pages within iframes. In fact, you can set the same script to run on both, and on multiple domains.

If a master page and an iframed resource page are both running GM script(s), the script instances can communicate with each other, cross-domain, using postMessage().

For example, suppose we have a site, fiddle.jshell.net/9ttvF/show, that contains travel data, and we want to mash-up that site with matching data from a resource site, jsbin.com/ahacab, that uses AJAX to get its payload data.

The target (master) site looks like this:

The resource site looks like this at first:

Then finishes like this:

The following script:

1. Loads the resource page in a hidden iframe.
2. Starts a second instance of itself running on the iframed page.
3. Waits for the iframed page to finish, processing the results as desired.
4. Sends the desired payload data to the GM script running on the target (master) page.
5. The target-page’s script then inserts the payload data to complete the mash-up.

// ==UserScript==
// @name     _Cross-site, AJAX scrape demo
// @include  http://fiddle.jshell.net/9ttvF/show/
// @include  http://jsbin.com/ahacab*
// @require  http://ajax.googleapis.com/ajax/libs/jquery/1.8.3/jquery.min.js
// @require  https://gist.github.com/raw/2625891/waitForKeyElements.js
// @grant    GM_addStyle
// ==/UserScript==

if (/fiddle\.jshell\.net/i.test (location.host) ) {
    console.log ("***Master-page start...");

    /*--- Inform the user.
    */
    $("#plainResults").before (
        '<div id="gmFetchRez">Greasemonkey is fetching results from jsbin.com...</div>'
    );

    /*--- Setup to process messages from the GM instance running on the iFrame:
    */
    window.addEventListener ("message", receiveMessage, false);

    /*--- Load the resource site in a hidden iframe.
    */
    $("body").append ('<iframe src="http://jsbin.com/ahacab" id="gmIframe"></iframe>');
}
else {
    console.log ("***Framed start...");
    /*--- Wait for the AJAXed-in content...
    */
    waitForKeyElements ("#results table.rezTable", sendResourcePageData);
}

function sendResourcePageData (jNode) {
    console.log ("Results found!  Sending them to the main window...");

    window.top.postMessage (jNode.html(), "*");
}

function receiveMessage (event) {
    if (event.origin != "http://jsbin.com")     return;

    $("#gmFetchRez").html (event.data);
}

//--- Use CSS to control appearances.
GM_addStyle ( "                                 \
    #gmIframe {                                 \
        display:            none;               \
    }                                           \
    #gmFetchRez {                               \
        background:         lightYellow;        \
        border:             3px double red;     \
        padding:            1em;                \
    }                                           \
" );

---

The final result looks like this, with the script installed and running: