If you read the documentation it is mentioned this function implements the algorithm of:
Suzuki, S. and Abe, K., Topological Structural Analysis of Digitized
Binary Images by Border Following. CVGIP 30 1, pp 32-46 (1985)
OpenCV is open source if you want to see how this is implemented just need to read the code:
https://github.com/opencv/opencv/blob/master/modules/imgproc/src/contours.cpp#L1655
The paper is available here.