Generating non-repeating random numbers in JS

If I understand right then you’re just looking for a permutation (i.e. the numbers randomised with no repeats) of the numbers 1-10?
Maybe try generating a randomised list of those numbers, once, at the start, and then just working your way through those?

This will calculate a random permutation of the numbers in nums:

var nums = [1,2,3,4,5,6,7,8,9,10],
    ranNums = [],
    i = nums.length,
    j = 0;

while (i--) {
    j = Math.floor(Math.random() * (i+1));
    ranNums.push(nums[j]);
    nums.splice(j,1);
}

So, for example, if you were looking for random numbers between 1 – 20 that were also even, then you could use:

nums = [2,4,6,8,10,12,14,16,18,20];

Then just read through ranNums in order to recall the random numbers.

This runs no risk of it taking increasingly longer to find unused numbers, as you were finding in your approach.

EDIT: After reading this and running a test on jsperf, it seems like a much better way of doing this is a Fisher–Yates Shuffle:

function shuffle(array) {
    var i = array.length,
        j = 0,
        temp;

    while (i--) {

        j = Math.floor(Math.random() * (i+1));

        // swap randomly chosen element with current element
        temp = array[i];
        array[i] = array[j];
        array[j] = temp;

    }

    return array;
}

var ranNums = shuffle([1,2,3,4,5,6,7,8,9,10]);

Basically, it’s more efficient by avoiding the use of ‘expensive’ array operations.

BONUS EDIT: Another possibility is using generators (assuming you have support):

function* shuffle(array) {

    var i = array.length;

    while (i--) {
        yield array.splice(Math.floor(Math.random() * (i+1)), 1)[0];
    }

}

Then to use:

var ranNums = shuffle([1,2,3,4,5,6,7,8,9,10]);

ranNums.next().value;    // first random number from array
ranNums.next().value;    // second random number from array
ranNums.next().value;    // etc.

where ranNums.next().value will eventually evaluate to undefined once you’ve run through all the elements in the shuffled array.

Overall this won’t be as efficient as the Fisher–Yates Shuffle because you’re still splice-ing an array. But the difference is that you’re now doing that work only when you need it rather than doing it all upfront, so depending upon your use case, this might be better.