Determine all camera parameters by a simultanous minimization process.
camera_calibration performs the calibration of a camera. For this, known 3D model points (with coordinates NX, NY, NZ) are projected in the image and the sum of the squared distance between these projections and the corresponding image points (with coordinates NRow, NCol) is minimized.
If the minimization converges, the exact interior (CamParam) and exterior (NFinalPose) camera parameters are determined by this minimization algorithm. The parameters StartCamParam and NStartPose are used as initial values for the minimization process. Since this algorithm simultaneously handles correspondences between image and model points from different images, it is also called multi-image calibration.
In general, camera calibration means the exact determination of the parameters that model the (optical) projection of any 3D world point P(w) into a (sub-)pixel [r,c] in the image. This is important, if the original 3D pose of an object has to be computed using an image (e.g., measuring of industrial parts).
The projection consists of multiple steps: First, the point p(w) is transformed from world into camera coordinates (points as homogeneous vectors, compare affine_trans_point_3d):
/ \ / x \ / \ / \ | p(c) | = | y | = | R t | * | p(w) | | | | z | | | | | \ 1 / \ 1 / \ 0 0 1 / \ 1 /
Then, the point is projected into the image plane, i.e., onto the sensor chip.
For the modeling of this projection process that is determined by the used combination of camera, lens, and frame grabber, HALCON provides the following three 3D camera models:
Area scan pinhole camera: The combination of an area scan camera with a lens that effects a perspective projection and that may show radial distortions.
Area scan telecentric camera: The combination of an area scan camera with a telecentric lens that effects a parallel projection and that may show radial distortions.
Line scan pinhole camera: The combination of a line scan camera with a lens that effects a perspective projection and that may show radial distortions.
For area scan cameras, the projection of the point p(c) that is given in camera coordinates into a (sub-)pixel [r,c] in the image consists of the following steps: First, the point is projected into the image plane, i.e., onto the sensor chip. If the underlying camera model is an area scan pinhole camera, i.e., if the focal length passed in CamParam is greater than 0, the projection is described by the following equations:
/ x \
p(c) = | y |
\ z /
u = Focus * x / z
v = Focus * y / z
In contrast, if the focal length is passed as 0 in CamParam, the camera model of an area scan telecentric camera is used, i.e., it is assumed that the optics of the lens of the camera performs a parallel projection. In this case, the corresponding equations are:
/ x \
p(c) = | y |
\ z /
u = x
v = y
The following equations perform the radial distortion of the points:
u' = (2*u) / (1+sqrt(1-4*Kappa*(u^2+v^2))) v' = (2*v) / (1+sqrt(1-4*Kappa*(u^2+v^2)))
Finally, the point is transformed from the image plane coordinate system into the image coordinate system, i.e., the pixel coordinate system:
c = u' / Sx + Cx
r = v' / Sy + Cy
For line scan cameras, also the relative motion between the camera and the object must be modeled. In HALCON, the following assumptions for this motion are made:
1) the camera moves with constant velocity along a straight line 2) the orientation of the camera is constant 3) the motion is equal for all images
The motion is described by the motion vector V = (Vx,Vy,Vz)' that must be given in [meter/scanline] in the camera coordinate system. The motion vector describes the motion of the camera, assuming a fixed object. In fact, this is equivalent to the assumption of a fixed camera with the object travelling along -V.
The camera coordinate system of line scan cameras is defined as follows: The origin of the coordinate system is the center of projection. The z-axis is identical to the optical axis and directed so that the visible points have positive z coordinates. The y-axis is perpendicular to the sensor line and to the z-axis. It is directed so that the motion vector has a positive y-component. The x-axis is perpendicular to the y- and z-axis, so that the x-, y-, and z-axis form a right-handed coordinate system.
As the camera moves over the object during the image acquisition, also the camera coordinate system moves relatively to the object, i.e., each image line has been imaged from a different position. This means, there would be an individual pose for each image line. To make things easier, in HALCON, all transformations from world coordinates into camera coordinates and vice versa are based on the pose of the first image line only. The motion V is taken into account during the projection of the point p(c) into the image. Consequently, only the pose of the first image line is returned by the operators find_marks_and_pose and camera_calibration.
For line scan pinhole cameras, the projection of the point p(c) that is given in the camera coordinate system into a (sub-)pixel [r,c] in the image is defined as follows:
Assuming
/ x \
p(c) = | y |,
\ z /
the following set of equations must be solved for m,
u', and t:
m * D * u' = x - t * Vx
-m * D * pv = y - t * Vy
m * Focus = z - t * Vz
with
1
D = -----------------------
1 + Kappa*(u'*u' + pv*pv)
pv = Sy*Cy
This already includes the compensation for radial distortions.
Finally, the point is transformed into the image coordinate system, i.e., the pixel coordinate system:
c = u' / Sx + Cx
r = t
The total of 14 camera parameters for area scan cameras and 17 camera parameters for line scan cameras, respectively, can be divided into the interior and exterior camera parameters:
Interior camera parameters:
These parameters describe the characteristics of the used
camera, especially the dimension of the sensor itself and the
projection properties of the used combination of lens, camera, and
frame grabber.
For area scan cameras, the above described camera model contains
the following 8 parameters:
Focus: Focal length of the lens. 0 for telecentric lenses.
Kappa: Distortion coefficient to model the pin-cushion- or
barrel-shaped distortions caused by the lens.
Sx: Scale factor.
For pinhole cameras, it corresponds to the horizontal
distance between two neighboring cells on the sensor.
For telecentric cameras, it represents the horizontal
size of a pixel in world coordinates.
Attention: This value increases if the image is
subsampled!
Sy: Scale factor.
For pinhole cameras, it corresponds to the vertical
distance between two neighboring cells on the sensor.
For telecentric cameras, it respresents the vertical
size of a pixel in world coordinates.
Since in most cases the image signal is sampled
line-synchronously, this value is determined by the
dimension of the sensor and needn't be estimated
for pinhole cameras by the calibration process.
Attention: This value increases if the image is
subsampled!
Cx: Column coordinate of the image center point (center
of the radial distortion).
Cy: Row coordinate of the image center point (center of
the radial distortion).
ImageWidth: Width of the sampled image. Attention: This
value decreases if the image is subsampled!
ImageHeight: Height of the sampled image. Attention: This
value decreases if the image is subsampled!
For line scan cameras, the above described camera model contains
the following 11 parameters:
Focus: Focal length of the lens.
Kappa: Distortion coefficient to model the pin-cushion- or
barrel-shaped distortions caused by the lens.
Sx: Scale factor, corresponds to the horizontal distance
between two neighboring cells on the sensor.
Attention: This value increases if the image is
subsampled!
Sy: Scale factor. During the calibration, it appears only
in the form pv = Sy*Cy. pv describes the distance of
the image center point from the sensor line in [meters].
Attention: This value increases if the image is subsampled!
Cx: Column coordinate of the image center point (center
of the radial distortion).
Cy: Distance of the image center point (center of the
radial distortion) from the sensor line in [scanlines].
ImageWidth: Width of the sampled image. Attention: This
value decreases if the image is subsampled!
ImageHeight: Height of the sampled image. Attention: This
value decreases if the image is subsampled!
Vx: X-component of the motion vector.
Vy: Y-component of the motion vector.
Vz: Z-component of the motion vector.
Note that the term focal length is not quite correct and would be
appropriate only for an infinite object distance. To simplify matters,
always the term focal length is used even if the image distance is meant.
Exterior camera parameters:
These 6 parameters describe the 3D pose, i.e., the position and
orientation, of the world coordinate system relative to the camera
coordinate system. For line scan cameras, the pose of the world
coordinate system refers to the camera coordinate system of the first
image line. Three parameters describe the translation, three the
rotation. See create_pose for more information about 3D poses. Note
that camera_calibration operates with all types of 3D poses for
NStartPose. When using the standard calibration plate, the world
coordinate system is defined by the coordinate system of the calibration
plate which is located in the middle of the surface of the calibration
plate, its z-axis pointing into the calibration plate, its x-axis to the
right, and it y-axis downwards.
The use of camera_calibration leads to some questions, which are dealed with in the following sections:
How to generate a appropriate calibration plate?
The simplest method to determine the interior parameters of a camera is the use of the standard calibration plate as generated by the operator gen_caltab. You can obtain high-precision calibration plates in various sizes and materials from your local distributor. In case of small distances between object and lens it may be sufficient to print the calibration pattern by a laser printer and to mount it on a cardboard. Otherwise -- especially by using a wide-angle lens -- it is possible to print the PostScript file on a large ink-jet printer and to mount it on a aluminum plate. It is very important that the mark coordinates in the calibration plate description file correspond to the real ones on the calibration plate with high accuracy. Thus, the calibration plate description file has to be modified in accordance with the measurement of the calibration plate!
How to take a set of suitable images?
If you use the standard calibration plate, you can proceed in the following way: With the combination of lens (fixed distance!), camera, and frame grabber to be calibrated a set of images of the calibration plate has to be taken, see open_framegrabber and grab_image. The following items have to be considered:
- At least a total of 10 to 20 images should be taken into account.
- The calibration plate has to be completely visible (incl. border!).
- Reflections etc.\ on the calibration plate should be avoided.
- Within the set of images the calibration plate should appear in
different positions and orientations: Once left in the image,
once right, once (left and right) at the bottom, once (left or
right) at the top, from different distances etc. At this, the
calibration plate should be rotated around its x-
and/or y-axis, so the perspective distortions of the
calibration pattern are clearly visible. Thus, the exterior
camera parameters (camera pose with regard of the calibration
plate) should be set to a large variety of different values!
- The calibration plate should fill at least a quarter of the
whole image to ensure the robust detection of the marks.
How to extract the calibration marks in the images?
If a standard calibration plate is used, you can use the operators find_caltab and find_marks_and_pose to determine the coordinates of the calibration marks in each image and to compute a rough estimate for the exterior camera parameters. The concatenation of these values can directly be used as initial values for the exterior camera parameters (NStartPose) in camera_calibration.
Obviously, images in which the segmentation of the calibration plate (find_caltab) has failed or the calibration marks haven't been determined successfully by find_marks_and_pose should not be used.
How to find suitable initial values for the interior camera parameters?
The operators find_marks_and_pose (determination of initial values for the exterior camera parameters) and camera_calibration require initial values for the interior camera parameters. These parameters can be provided by a appropriate text file (see read_cam_par) which can be generated by write_cam_par or can be edited manually.
For area scan cameras, the following should be considered for the
initial values of the single parameters:
Focus: The initial value is the nominal focal length of the
the used lens, e.g., 0.008 m.
Kappa: Use 0.0 as initial value.
Sx: The initial value for the horizontal distance between
two neighboring cells depends on the dimension of
the used chip of the camera (see technical
specifications of the camera). Generally, common chips
are either 1/3''-Chips (e.g., SONY XC-73, SONY XC-777),
1/2''-Chips (e.g., SONY XC-999, Panasonic WV-CD50), or
2/3''-Chips (e.g., SONY DXC-151, SONY XC-77). Notice:
The value of Sx increase, if the image is subsampled!
Appropriate initial values are:
Full image (768*576) Subsampling (384*288)
1/3"-Chip 0.0000055 m 0.0000110 m
1/2"-Chip 0.0000086 m 0.0000172 m
2/3"-Chip 0.0000110 m 0.0000220 m
The value for Sx is calibrated, since the video signal
of a camera normally isn't sampled pixel-synchronously.
Sy: Since most off-the-shelf cameras have quadratic pixels,
the same values for Sy are valid as for Sx. In contrast
to Sx the value for Sy will NOT be calibrated for pinhole
cameras, because the video signal of a camera normally
is sampled line-synchronously. Thus, the initial value
is equal to the final value. Appropriate initial values are:
Full image (768*576) Subsampling (384*288)
1/3"-Chip 0.0000055 m 0.0000110 m
1/2"-Chip 0.0000086 m 0.0000172 m
2/3"-Chip 0.0000110 m 0.0000220 m
Cx and Cy: Initial values for the coordinates of the image center
is the half image width and half image height.
Notice: The values of Cx and Cy decrease if the image
is subsampled! Appropriate initial values are:
Full image (768*576) Subsampling (384*288)
Cx 384.0 192.0
Cy 288.0 144.0
ImageWidth and ImageHeight: These two parameters are determined by the
the used frame grabber and therefore are not calibrated.
Appropriate initial values are, for example:
Full image (768*576) Subsampling (384*288)
ImageWidth 768 384
ImageHeight 576 288
For line scan cameras, the following should be considered for the
initial values of the single parameters:
Focus: The initial value is the nominal focal length of the
the used lens, e.g., 0.008 m.
Kappa: Use 0.0 as initial value.
Sx: The initial value for the horizontal distance between
two neighboring cells can be taken from the technical
specifications of the camera.
Typical initial values are 7e-6 m, 10e-6 m, and 14e-6 m.
Notice: The value of Sx increase, if the image is subsampled!
Sy: The initial value for the size of a cell in the direction
perpendicular to the sensor line can also be taken from
the technical specifications of the camera.
Typical initial values are 7e-6 m, 10e-6 m, and 14e-6 m.
Notice: The value of Sx increase, if the image is subsampled!
In contrast to Sx, the value for Sy will NOT be calibrated
for line scan cameras, because it appears only in the
form pc = Sy*Cy. Therefore, it cannot be determined
separately.
Cx: The initial value for the x-coordinate of the image center
is the half image width.
Notice: The values of Cx decreases if the image is subsampled!
Appropriate initial values are:
Image width: 1024 2048 4096 8192
Cx: 512 1024 2048 4096
Cy: The initial value for the y-coordinate of the image center
can normally set to 0.
ImageWidth and ImageHeight: These two parameters are determined by the
used frame grabber and therefore are not calibrated.
Vx, Vy, Vz: The initial values for the x-, y-, and z-component of the
motion vector depend on the image acquisition setup.
Assuming a camera that looks perpendicularly onto a
conveyor belt, and that is rotated around its optical axis
such that the sensor line is perpendicular to the conveyor
belt, i.e., the y-axis of the camera coordinate system is
parallel to the conveyor belt, the initial values Vx = Vz = 0.
The initial value for Vy can then be determined, e.g., from
a line scan image of an object with known size (e.g.,
calibration plate, ruler):
Vy = LM / LR
with:
LM = Length of the object in object coordinates [meter]
LR = Length of the object in image coordinates [rows]
If, compared to the above setup, the camera is rotated at
30 degrees around its optical axis, i.e., around the z-axis
of the camera coordinate system, the above determined
initial values must be changed as follows:
Vx_rot_z & = & sin(30) * Vy
Vy_rot_z & = & cos(30) * Vy
Vz_rot_z & = & Vz = 0
If, compared to the first setup, the camera is rotated at
20 degrees around the x-axis of the camera coordinate
system, the following initial values result:
Vx_rot_x & = & Vx = 0
Vy_rot_x & = & cos(20) * Vy
Vz_rot_x & = & sin(20) * Vy
The quality of the initial values for Vx, Vy, and Vz are
crucial for the success of the whole calibration. If they
are not precise enough, the calibration may fail.
Which camera parameters have to be estimated?
The input parameter EstimateParams is used to select which camera parameters to estimate. Usually this parameter is set to 'all', i.e., all 6 exterior camera parameters (translation and rotation) and all interior camera parameters are determined. If the interior camera parameters already have been determined (e.g., by a previous call to camera_calibration) it is often desired to only determine the pose of the world coordinate system in camera coordinates (i.e., the exterior camera parameters). In this case, EstimateParams can be set to 'pose'. This has the same effect as EstimateParams = ['alpha','beta','gamma','transx','transy','transz']. Otherwise, EstimateParams contains a tuple of strings indicating the combination of parameters to estimate. In addition, parameters can be excluded from estimation by using the prefix ~. For example, the values ['pose','~transx'] have the same effect as ['alpha','beta','gamma','transy','transz']. Whereas ['all','~focus'] determines all internal and external parameters except the focus, for instance. The prefix ~ can be used with all parameter values except 'all'.
What is the order within the individual parameters?
The length of the tuple NStartPose corresponds to the number of calibration images, e.g., using 15 images leads to a length of the tuple NStartPose equal to 15*7=105 (15 times the 7 exterior camera parameters). The first 7 values correspond to the pose of the calibration plate in the first image, the next 7 values to the pose in the second image, etc.
This fixed number of calibration images has to be considered within the tuples with the coordinates of the 3D model marks and the extracted 2D marks. If 15 images are used, the length of the tuples NRow and NCol is 15 times the length of the tuples with the coordinates of the 3D model marks (NX, NY, and NZ). If every image consists 49 marks, the length of the tuples NRow and NCol is 15*49=735, while the length of the tuples NX, NY, and NZ is 49. The order of the values in NRow and NCol is ``image after image'', i.e., using 49 marks the first 3D model point corresponds to the 1st, 50th, 99th, 148th, 197th, 246th, etc. extracted 2D mark.
The 3D model points can be read from a calibration plate description file using the operator caltab_points. Initial values for the poses of the calibration plate can be determined by applying find_marks_and_pose for each image. The tuple NStartPose is set by the concatenation of all these poses.
What is the meaning of the output parameters?
If the camera calibration process is finished successfully, i.e., the minimization process has converged, the output parameters CamParam and NFinalPose contain the computed exact values for the interior and exterior camera parameters. The length of the tuple NFinalPose corresponds to the length of the tuple NStartPose.
The representation types of NFinalPose correspond to the representation type of the first tuple of NStartPose (see create_pose). You can convert the representation type by convert_pose_type.
The computed average errors (Errors) give an impression of the accuracy of the calibration. The error values (deviations in x and y coordinates) are measured in pixels.
Must I use a planar calibration object?
No. The operator camera_calibration is designed in a way that the input tuples NX, NY, NZ, NRow, and NCol can contain any 3D/2D correspondences, see the above paragraph explaining the order of the single parameters.
Thus, it makes no difference how the required 3D model marks and the corresponding extracted 2D marks are determined. On one hand, it is possible to use a 3D calibration pattern, on the other hand, you also can use any characteristic points (natural landmarks) with known position in the world. By setting EstimateParams to 'pose', it is thus possible to compute the pose of an object in camera coordinates! For this, at least three 3D/2D-correspondences are necessary as input. NStartPose can, e.g., be generated directly as shown in the program example for create_pose.
The minimization process of the calibration depends on the initial values of the interior (StartCamParam) and exterior (NStartPose) camera parameters. The computed average errors Errors give an impression of the accuracy of the calibration. The errors (deviations in x and y coordinates) are measured in pixels.
|
NX (input_control) |
real-array -> real |
| Ordered tuple with all x coordinates of the calibration marks (in meters). | |
|
NY (input_control) |
real-array -> real |
| Ordered tuple with all y coordinates of the calibration marks (in meters). | |
|
NZ (input_control) |
real-array -> real |
| Ordered tuple with all z coordinates of the calibration marks (in meters). | |
|
NRow (input_control) |
real-array -> real |
| Ordered tuple with all row coordinates of the extracted calibration marks (in pixels). | |
|
NCol (input_control) |
real-array -> real |
| Ordered tuple with all column coordinates of the extracted calibration marks (in pixels). | |
|
StartCamParam (input_control) |
real-array -> real / integer |
| Initial values for the interior camera parameters. | |
|
NStartPose (input_control) |
pose-array -> real / integer |
| Ordered tuple with all initial values for the exterior camera parameters. | |
|
EstimateParams (input_control) |
string -> string / integer |
| Camera parameters to be estimated. | |
| Default value: 'all' | |
| List of values: 'all', 'pose', 'alpha', 'beta', 'gamma', 'transx', 'transy', 'transz', 'focus', 'kappa', 'cx', 'cy', 'sx', 'sy', 'vx', 'vy', 'vz' | |
|
CamParam (output_control) |
number-array -> real / integer |
| Interior camera parameters. | |
|
NFinalPose (output_control) |
pose-array -> real / integer |
| Ordered tuple with all exterior camera parameters. | |
|
Errors (output_control) |
real-array -> real |
| Average error distances in pixels. | |
* read calibration images
read_image(Image1, 'calib-01')
read_image(Image2, 'calib-02')
read_image(Image3, 'calib-03')
* find calibration pattern
find_caltab(Image1, Caltab1, 'caltab.descr', 3, 112, 5)
find_caltab(Image2, Caltab2, 'caltab.descr', 3, 112, 5)
find_caltab(Image3, Caltab3, 'caltab.descr', 3, 112, 5)
* find calibration marks and start poses
find_marks_and_pose(Image1, Caltab1, 'caltab.descr',
[0.008, 0.0, 0.000011, 0.000011, 384, 288, 768, 576],
128, 10, 18, 0.9, 15.0, 100.0, RCoord1, CCoord1,
StartPose1)
find_marks_and_pose(Image2, Caltab2, 'caltab.descr',
[0.008, 0.0, 0.000011, 0.000011, 384, 288, 768, 576],
128, 10, 18, 0.9, 15.0, 100.0, RCoord2, CCoord2,
StartPose2)
find_marks_and_pose(Image3, Caltab3, 'caltab.descr',
[0.008, 0.0, 0.000011, 0.000011, 384, 288, 768, 576],
128, 10, 18, 0.9, 15.0, 100.0, RCoord3, CCoord3,
StartPose3)
* read 3D positions of calibration marks
caltab_points('caltab.descr', NX, NY, NZ)
* camera calibration
camera_calibration(NX, NY, NZ, [RCoord1, RCoord2, RCoord3],
[CCoord1, CCoord2, CCoord3],
[0.008, 0.0, 0.000011, 0.000011, 384, 288, 768, 576],
[StartPose1, StartPose2, StartPose3], 'all',
CamParam, NFinalPose, Errors)
* write interior camera parameters to file
write_cam_par(CamParam, 'campar.dat')camera_calibration returns 2 (H_MSG_TRUE) if all parameter values are correct and the desired camera parameters have been determined by the minimization algorithm. If necessary, an exception handling is raised.
camera_calibration is reentrant and processed without parallelization.
find_marks_and_pose, caltab_points, read_cam_par
write_pose, pose_to_hom_mat3d, disp_caltab, sim_caltab
find_caltab, find_marks_and_pose, disp_caltab, sim_caltab, write_cam_par, read_cam_par, create_pose, convert_pose_type, write_pose, read_pose, pose_to_hom_mat3d, hom_mat3d_to_pose, caltab_points, gen_caltab
Calibration
