Exponentiation by squaring still “works” for modulo exponentiation. Your problem isn’t that 2 ^ 168277
is an exceptionally large number, it’s that one of your intermediate results is a fairly large number (bigger than 2^32), because 673109 is bigger than 2^16.
So I think the following will do. It’s possible I’ve missed a detail, but the basic idea works, and this is how “real” crypto code might do large mod-exponentiation (although not with 32 and 64 bit numbers, rather with bignums that never have to get bigger than 2 * log (modulus)):
- Start with exponentiation by squaring, as you have.
- Perform the actual squaring in a 64-bit unsigned integer.
- Reduce modulo 673109 at each step to get back within the 32-bit range, as you do.
Obviously that’s a bit awkward if your C++ implementation doesn’t have a 64 bit integer, although you can always fake one.
There’s an example on slide 22 here: http://www.cs.princeton.edu/courses/archive/spr05/cos126/lectures/22.pdf, although it uses very small numbers (less than 2^16), so it may not illustrate anything you don’t already know.
Your other example, 4000111222 ^ 3 % 1608
would work in your current code if you just reduce 4000111222
modulo 1608
before you start. 1608
is small enough that you can safely multiply any two mod-1608 numbers in a 32 bit int.