Zhang Zhengyou calibration Opencv realization , Calibration process and image coordinates are converted to world coordinates
Copyright notice ： This article is the original article of blogger , No reprint without permission of Blogger .
Before using the camera , Camera calibration is the first step , The reason is that we know the internal and external parameters of the camera through calibration , After getting the internal and external parameter matrix, the photos taken by the camera can be corrected , The image with less distortion can be obtained . and
The input of camera calibration is the corner coordinates of the multi frame pictures taken by the camera , And the spatial coordinates of all corners on the calibration board image ( commonly Z The axis is assumed to be Z=0). The output of the camera after calibration is the internal and external parameters of the camera .
Calibration process for Zhang Zhengyou camera calibration ：1, Get the corner coordinates of multiple images 2, Obtain the spatial coordinates of all corners on the calibration board image 3, Camera calibration 4, The calibration results were evaluated
5, Thirdly, the calibration board image is used to correct 6, Image coordinate system to world coordinate system
1, Get the corner coordinates of multiple images
Take photos of the calibration image continuously , choice 10-20 Frame picture , Then the corner information is extracted .( Here we need to pay attention to the calibration image needs to be clear )
The selected corner function is the image ：findChessboardCorners Function extraction corner
findChessboardCorners Function prototype ：
* //! finds checkerboard pattern of the specified size in the image
* CV_EXPORTS_W bool findChessboardCorners( InputArray image, Size patternSize,
* OutputArray corners,
* int flags=CALIB_CB_ADAPTIVE_THRESH+CALIB_CB_NORMALIZE_IMAGE ); Parameter analysis ：
①Image, The chessboard picture taken in Mat image , It has to be 8 Bit gray or color image ;
②patternSize, The number of rows and rows of interior corners on each chessboard , Normally , The number of rows should not be the same , It is convenient to identify the direction of the calibration board for the subsequent calibration program ;
③corners, It is used to store the coordinates of the detected inner corner image , The general elements are Point2f Is represented by a vector of ：vector<Point2f>
④flags： It is used to define different processing methods for finding inner corner on chessboard , There are default values .
2, The sub-pixel corner information is further extracted for each frame
In order to improve the calibration accuracy , It is necessary to extract sub-pixel information from the initial corner information , Reducing camera calibration bias , The common methods are cornerSubPix
cornerSubPix Function prototype ：
* CV_EXPORTS_W void cornerSubPix( InputArray image, InputOutputArray corners,
* Size winSize, Size zeroZone,
* TermCriteria criteria ); Parameter analysis ：①Image, The chessboard picture taken in Mat image , Better 8 Bit gray image , Higher detection efficiency ;
corners, Initial corner coordinate vector , At the same time as the output of sub-pixel coordinate position , So it needs to be floating-point data , The general elements are Pointf2f/Point2d Is represented by a vector of ：vector<Point2f/Point2d>
③winSize, Half the size of the search window ;
zeroZone, Half the size of the dead zone , The dead zone is the area where the central position of the search area is not summed . It is used to avoid some possible singularities of autocorrelation matrix . When the duty is （-1,-1） There is no dead zone ;
⑤criteria, Define the termination conditions of the iterative process for finding corners , It can be a combination of iteration times and corner accuracy ;
3, Obtain the spatial coordinates of all corners on the calibration board image ( That is, the actual coordinates of the object )
Obtain the 3D coordinates of the world coordinate system in the same way as step 1 objectPoints, It's represented by vectors vector<vector<Point3f>>
4, Camera calibration
After obtaining the image coordinates of the inner corner of the chessboard calibration map and the spatial coordinates of the calibration board image , You can use it calibrateCamera Function to calibrate , Calculation of camera internal and external parameters ,
calibrateCamera Function prototype ：
* CV_EXPORTS_W double calibrateCamera( InputArrayOfArrays objectPoints,
* InputArrayOfArrays imagePoints,
* Size imageSize,
* CV_OUT InputOutputArray cameraMatrix,
* CV_OUT InputOutputArray distCoeffs,
* OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs,
* int flags=0, TermCriteria criteria = TermCriteria(
* TermCriteria::COUNT+TermCriteria::EPS, 30, DBL_EPSILON) ); Parameter analysis ：
①objectPoints, Is a 3D point in the world coordinate system . when in use , The vector of the vector of a 3D coordinate point should be input , Namely vector<vector<Point3f>>
object_points. According to the size of a single black and white matrix on the chessboard , Calculated （ initialization ） The world coordinates of each interior corner ;
②imagePoints, It is the image coordinate point corresponding to each interior corner point . and objectPoints equally , Should be entered vector<vector<Point2f>>
image_points_seq Formal variable ;
③imageSize, Is the pixel size of the image , This parameter should be used in the calculation of camera intrinsic parameter and distortion matrix ;
④cameraMatrix Is the internal parameter matrix of the camera . Enter one Mat cameraMatrix that will do , as Mat
⑤distCoeffs Is the distortion matrix . Enter one Mat distCoeffs=Mat(1,5,CV_32FC1,Scalar::all(0));
⑥rvecs Is the rotation vector ; One should be entered Mat Type vector, Namely vector<Mat>rvecs;
⑦tvecs Is the displacement vector , and rvecs equally , Should be vector<Mat> tvecs;
⑧flags Is the algorithm used in calibration . There are several parameters as follows ：
CV_CALIB_USE_INTRINSIC_GUESS： When this parameter is used , stay cameraMatrix There should be in the matrix fx,fy,u0,v0 Estimated value of . Otherwise , Will initialize (u0,v0） The center of the image , It is estimated by using the least square method fx,fy.
CV_CALIB_FIX_PRINCIPAL_POINT： The optical axis points are fixed during optimization . When CV_CALIB_USE_INTRINSIC_GUESS Parameters are set , The optical axis point will remain at the center or at an input value .
CV_CALIB_FIX_ASPECT_RATIO： fixed fx/fy The ratio of , Only will fy As variable , Carry out optimization calculation . When CV_CALIB_USE_INTRINSIC_GUESS Not set ,fx and fy Will be ignored . only fx/fy The ratio is used in the calculation .
CV_CALIB_ZERO_TANGENT_DIST： Setting tangential distortion parameters （p1,p2） Is zero .
CV_CALIB_FIX_K1,…,CV_CALIB_FIX_K6： The corresponding radial distortion remains unchanged in the optimization .
CV_CALIB_RATIONAL_MODEL： calculation k4,k5,k6 Three distortion parameters . If not set , Only the others are calculated 5 Distortion parameters .
Before using this function for calibration operations , The coordinates of each interior point on the chessboard are initialized , The result of calibration is to generate the camera's internal parameter matrix cameraMatrix, Camera's 5 Distortion coefficients distCoeffs, In addition, each image will generate its own translation vector and rotation vector .
5, The calibration results were evaluated
The method to evaluate the calibration results is to get the camera internal and external parameters , The three-dimensional points of space are re projected , The coordinates of the new projection points on the image are obtained , Calculate the deviation between the projection coordinates and the sub-pixel corner coordinates , The smaller the deviation , The better the calibration result is .
The function of back projection of 3D coordinate points in space is projectPoints, The function prototype is ：
* CV_EXPORTS_W void projectPoints( InputArray objectPoints,
* InputArray rvec, InputArray tvec,
* InputArray cameraMatrix, InputArray distCoeffs,
* OutputArray imagePoints,
* OutputArray jacobian=noArray(),
* double aspectRatio=0 );
Parameter analysis :①objectPoints, Is the 3D point coordinates in the camera coordinate system ;
②rvec Is the rotation vector , Each image has its own selection vector ;
③tvec Is the displacement vector , Each image has its own translation vector ;
④cameraMatrix The internal parameter matrix of the camera is obtained ;
⑤distCoeffs Is the distortion matrix of the camera ;
⑥imagePoints It is the coordinate point on the image corresponding to each interior corner point ;
⑦jacobian It's the Jacobian determinant ;
⑧aspectRatio Is an optional parameter related to the sensitive unit of the camera sensor , If set to not 0, Then the function defaults to the dx/dy It's fixed , The Jacobian matrix will be adjusted accordingly ;
6, Thirdly, the calibration board image is used to correct
Using the obtained internal and external parameters of the camera data , The image distortion can be corrected , use undistort Function implementation .
undistort Function prototype :
* CV_EXPORTS_W void undistort( InputArray src, OutputArray dst,
* InputArray cameraMatrix,
* InputArray distCoeffs,
* InputArray newCameraMatrix=noArray() );
Parameter analysis :①src, input parameter , Original image representing distortion ;
②dst, Corrected output image , Same type and size as input image ;
③cameraMatrix Is the internal parameter matrix of the camera obtained previously ;
④distCoeffs Is the camera distortion matrix obtained previously ;
⑤newCameraMatrix, Default with cameraMatrix bring into correspondence with ;
7, Image coordinate system to world coordinate system
The perspective transformation is used to transform the image coordinate system into the world coordinate system , Calibration of homography matrix by Zhang Zhengyou , The world coordinate system can be obtained by the perspective transformation principle .
Refer to big man blog : Zhang Zhengyou calibration ：
Perspective transformation principle can refer to the following blog :
Reprinted from ：https://blog.csdn.net/xiaomifanhxx/article/details/79560693