CV – Extract differences between two images

One problem in your code is cv::threshold which only uses 1 channel images. Finding the pixelwise “difference” between two images in only grayscale often leads to unintuitive results.

Since your provided images are a bit translated or the camera wasnt stationary, I’ve manipulated your background image to add some foreground:

background image:

foreground image:

code:

    cv::Mat diffImage;
    cv::absdiff(backgroundImage, currentImage, diffImage);

    cv::Mat foregroundMask = cv::Mat::zeros(diffImage.rows, diffImage.cols, CV_8UC1);

    float threshold = 30.0f;
    float dist;

    for(int j=0; j<diffImage.rows; ++j)
        for(int i=0; i<diffImage.cols; ++i)
        {
            cv::Vec3b pix = diffImage.at<cv::Vec3b>(j,i);

            dist = (pix[0]*pix[0] + pix[1]*pix[1] + pix[2]*pix[2]);
            dist = sqrt(dist);

            if(dist>threshold)
            {
                foregroundMask.at<unsigned char>(j,i) = 255;
            }
        }

giving this result:

with this difference image:

in general it is hard to compute a complete foreground/background segmentation from pixel-wise difference interpretations.

You will probably have to add postprocessing stuff to get a real segmentation, where you start from your foreground mask. Not sure whether there are any stable universal solutions yet.

As berak mentioned, in practice it won’t be enough to use a single background image, so you will have to compute/manage your background image over time. There are plenty of papers covering this topic and afaik no stable universal solution yet.

here are some more tests. I converted to HSV color space: cv::cvtColor(backgroundImage, HSVbackgroundImagebg, CV_BGR2HSV); cv::cvtColor(currentImage, HSV_currentImage, CV_BGR2HSV); and performed the same operations in this space, leading to this result:

after adding some noise to the input:

I get this result:

so maybe the threshold is a bit too high. I still encourage you to have a look at HSV color space too, but you might have to reinterpret the “difference image” and rescale each channel to combine their difference values.