Mocking $modal in AngularJS unit tests

When you spy on the $modal.open function in the beforeEach,

spyOn($modal, 'open').andReturn(fakeModal);

or 

spyOn($modal, 'open').and.returnValue(fakeModal); //For Jasmine 2.0+

you need to return a mock of what $modal.open normally returns, not a mock of $modal, which doesn’t include an open function as you laid out in your fakeModal mock. The fake modal must have a result object that contains a then function to store the callbacks (to be called when the OK or Cancel buttons are clicked on). It also needs a close function (simulating an OK button click on the modal) and a dismiss function (simulating a Cancel button click on the modal). The close and dismiss functions call the necessary call back functions when called.

Change the fakeModal to the following and the unit test will pass:

var fakeModal = {
    result: {
        then: function(confirmCallback, cancelCallback) {
            //Store the callbacks for later when the user clicks on the OK or Cancel button of the dialog
            this.confirmCallBack = confirmCallback;
            this.cancelCallback = cancelCallback;
        }
    },
    close: function( item ) {
        //The user clicked OK on the modal dialog, call the stored confirm callback with the selected item
        this.result.confirmCallBack( item );
    },
    dismiss: function( type ) {
        //The user clicked cancel on the modal dialog, call the stored cancel callback
        this.result.cancelCallback( type );
    }
};

Additionally, you can test the cancel dialog case by adding a property to test in the cancel handler, in this case $scope.canceled:

$scope.modalInstance.result.then(function (selectedItem) {
    $scope.selected = selectedItem;
}, function () {
    $scope.canceled = true; //Mark the modal as canceled
    $log.info('Modal dismissed at: ' + new Date());
});

Once the cancel flag is set, the unit test will look something like this:

it("should cancel the dialog when dismiss is called, and $scope.canceled should be true", function () {
    expect( scope.canceled ).toBeUndefined();

    scope.open(); // Open the modal
    scope.modalInstance.dismiss( "cancel" ); //Call dismiss (simulating clicking the cancel button on the modal)
    expect( scope.canceled ).toBe( true );
});