fancy box work with iframes in beforeShow function?

You could also use the beforeShow callback option, but you may need to cancel all the transitions (set nextSpeed and prevSpeed to 0).

The transition speed seems to be creating the jumping effect using the afterShow callback or avoiding to get the correct value using the beforeShow callback.

You may also need to update to fancybox version v2.0.6.

Additionally, you could also simplify your script without using external variables like:

$(document).ready(function() {
 $("a.fancybox").fancybox({
  openEffect : 'elastic',
  closeEffect : 'elastic',
  fitToView: false,
  nextSpeed: 0, //important
  prevSpeed: 0, //important
  beforeShow: function(){
  // added 50px to avoid scrollbars inside fancybox
   this.width = ($('.fancybox-iframe').contents().find('html').width())+50;
   this.height = ($('.fancybox-iframe').contents().find('html').height())+50;
  }
 }); // fancybox
}); // ready

See DEMO here