RGB distance in the euclidean space is not very similar to “average human perception”.
You can use YUV color space, it takes into account this factor :
| Y' | | 0.299 0.587 0.114 | | R |
| U | = | -0.14713 -0.28886 0.436 | | G |
| V | | 0.615 -0.51499 -0.10001 | | B |
You can also use the CIE color space for this purpose.
EDIT:
I shall mention that YUV color space is an inexpensive approximation that can be computed via simple formulas. But it is not perceptually uniform. Perceptually uniform means that a change of the same amount in a color value should produce a change of about the same visual importance.
If you need a more precise and rigourous metric you must definitely consider CIELAB color space or an another perceptually uniform space (even if there are no simple formulas for conversion).