Let’s take a look at HSV color space:
You need white, which is close to the center and rather high. Start with
sensitivity = 15
lower_white = np.array([0,0,255-sensitivity])
upper_white = np.array([255,sensitivity,255])
and then adjust the threshold to your needs.
You might also consider using HSL color space, which stands for Hue, Saturation, Lightness. Then you would only have to look at lightness for detecting white and recognizing other colors would stay easy. Both HSV and HSL keep similar colors close. Also HSL would probably prove more accurate for detecting white – here is why: