The first step would be to define the colors you want to compare to.
The second step is to find the smallest distance from your color to one of the colors you chose in the previous step. In order to be able to measure that distance you need an Euclidian space in which to model colors.
Of course the simple choice would be the RGB space
And the distance between two colors C1(r1, g1, b1) and C2(r2, g2, b2) would be
sqrt( (r1 – r2)2 + (g1 – g2)2 + (b1 – b2)2 ).
But if you need more precision it would be better to use the Hue-Chroma-Lightness bicone space, a derivative of the HSL cylinder.
In the RGB space things were straight forward as R, G and B where each on a separate axis. In HCL we need to compute the coordinates on each of the axis.
First of all we compute the chroma (which is a bit different from saturation) as:
Chroma = max(Red, Green, Blue) – min(Red, Green, Blue)
Then we normalize our H, C and L value so that H goes from 0 to 2 (to cover a circle if we multiply by PI and take radians as the unit), C goes from 0 to 1 (the radius of the trigonometric circle) and L goes from -1 (Black) to 1 (White).
Next we take z = L without any transformations as it is clear from the image that it goes along the vertical axis.
We can easily observe that for a color, Chroma is the distance from the z axis and Hue is the angle. So we get
x = C * cos(H*PI) and
y = C * sin(H*PI)
At this point x, y and z will all be in [-1, 1] and the distance between two colors will be, using the same formula as above,
sqrt( (x1 – x2)2 + (y1 – y2)2 + (z1 – z2)2 ).
To get even more precision and find the closest color according to human color perception you could use the CIE-L*ab modeling space and compute the distance with one of these algorithms. The principles are the same as for the two cases presented above, only the algorithms are more complex.
Update (7 years later)
Finally xkcd featured a comic that I can use in this post!
