Construction of intelligent car vision system calibration room / 360 degree surround view

The visual system calibration of intelligent vehicles is the cornerstone for accurate perception in autonomous driving and intelligent cockpits (such as 360 degree surround view). Due to the fact that the camera captures two-dimensional pixels, while vehicle decision-making requires the real physical position in three-dimensional space (such as lane lines, obstacle distances), it is necessary to establish a mathematical mapping relationship of "pixel coordinate camera coordinate body coordinate/world coordinate" through calibration.

The complete calibration process of intelligent automotive vision systems is usually divided into three dimensions: production line/static calibration, dynamic/online calibration, and multi-sensor joint calibration. The following is a detailed explanation of the core calibration process:

1. Classification of core calibration parameters

Before entering the process, we need to know what exactly calibration is:

Intrinsic Parameters: Determine the geometric distortion and focal length properties inside the camera lens. For example, eliminating the "fisheye" distortion of wide-angle lenses and straightening curved images.

Exterior Parameters: Determine the specific position and orientation of the camera in the vehicle coordinate system (installed X, Y, Z coordinates and heading angles Pitch, Yaw, Roll).

2. Production line/static calibration process (mainstream foundation)

Static calibration is usually performed in the calibration room before the car leaves the factory or after maintenance at the 4S store. The environment is controlled and the accuracy is extremely high.

(1) Preparation and environmental setup

Calibration tool: Prepare a calibration board with a specific geometric pattern (such as a checkerboard, dot array diagram, or specialized panoramic calibration cloth).

Environmental requirements: Ensure uniform lighting and avoid overexposure caused by strong light or inability to extract image feature points due to low light.

Vehicle status: The vehicle is parked on a level surface, with tire pressure adjusted to the standard value, and the vehicle needs to be completely stationary (to prevent suspension shaking).

(2) Internal parameter calibration (usually completed at the supplier's location or before loading)

Data collection: The camera points at the calibration board and takes multiple photos (usually 10-20) from different angles and distances.

Algorithm solving: The industry generally adopts the Zhang Zhengyou calibration method. The algorithm identifies corner points (feature points) on the calibration board and uses mathematical matrices to solve for the camera's focal length (f_x, f_y), principal point position (u-0, v-0), and distortion coefficient (radial and tangential distortion).

(3) External parameter calibration (must be performed after loading)

Location layout: Place the calibration plate or calibration cloth accurately at known relative positions around the vehicle (using a total station or laser rangefinder to determine the position of the calibration plate relative to the vehicle axle and centerline).

Feature extraction: The various cameras of the vehicle capture images containing calibration plates, and the algorithm automatically detects the corresponding feature points of the calibration plates in the images (such as the corner points of the four calibration cloths in the surround view system).

External parameter matrix calculation: Based on the "pixel coordinates" of feature points in the image and their "real physical coordinates" relative to the vehicle in reality, calculate the rotation matrix R and displacement vector T of each camera relative to the vehicle coordinate system.

Splicing and inspection (using 360 degree surround view as an example): Use the calculated extrinsic to project the images of the four cameras, front, back, left, and right, onto a bird's-eye view (BEV) perspective, and check if the stitching gaps (such as ground lines) are aligned and if there are any ghosting.

3. Dynamic calibration/online calibration process (optimized during driving)

During the driving process of a car, due to long-term bumps, temperature changes causing thermal expansion and contraction, or changes in suspension load, the camera's external camera will experience slight displacement. Online calibration can achieve "calibration while driving".

[Vehicle Driving] ->[Detecting Natural Features (Lane Lines/Road Signs)] ->[Extracting Features and Integrating with Kinematics] ->[Online Iterative Updating of External Parameter Matrix]

(1) Feature triggering and extraction

When the vehicle is driving on regular roads such as highways and urban overpasses, visual algorithms will extract natural feature points from the environment in real time, with the most commonly used being lane markings (assuming that the lane markings are parallel and extend to infinity in front of the vehicle) and vanishing points.

(2) Multi sensor fusion and state estimation

By combining the vehicle's IMU (Inertial Measurement Unit) and speed sensor (Wheel Speed Gauge), estimate the vehicle's own attitude changes.

(3) Iterative calculation and correction

Algorithms such as Kalman filtering or beam based adjustment (BA) continuously fine tune and correct the pitch and yaw angles of the camera by comparing the image changes between the current frame and historical frames, as well as the geometric constraints of lane lines, to achieve seamless dynamic refresh.

4. Joint calibration process (cross sensor alignment)

Modern smart cars not only have cameras, but also LiDAR and millimeter wave radar. Align their data in the same spatial coordinate system in order to perform pre fusion perception.

Taking Camera to LiDAR (Forward Camera and Lidar Calibration) as an example:

(1) Common view area acquisition: Place a specially designed calibration plate (usually with high reflectivity patterns or specific geometric shapes) that can be captured clearly by a camera and scanned by a laser radar in front of the car.

(2) Data extraction: Extract the pixel coordinates of the corners or edges of the calibration plate from the camera's 2D image.

Extract the three-dimensional spatial feature points corresponding to the calibration plate from the three-dimensional point cloud of the LiDAR (using point cloud reflection intensity or plane fitting).

(3) Constraint solving: Find the best match between two sets of feature points, use optimization algorithms such as least squares, and calculate the 3D-to-2D transformation matrix from the LiDAR coordinate system to the camera coordinate system.

(4) Alignment effect verification: Project the colored point cloud of the LiDAR onto the image of the camera, and observe whether the contour of the point cloud perfectly fits the objects such as vehicles, buildings, and vehicles in the image.

5. Calibration result verification and quality assessment

After calibration is completed, it must undergo rigorous indicator evaluation before it can be put into business use:

Reprojection Error: The calculated 3D spatial points are re projected back into a 2D image using calibration parameters, and the deviation from the actual pixel points in the image is calculated. In autonomous driving, the average reprojection error is usually required to be less than 0.5 to 1 pixel.

Parameter stability check: Check whether the relative geometric relationship between the left and right binocular cameras or the surround view camera conforms to the rigid common sense of mechanical structure, to prevent the algorithm from getting stuck in local optimal solutions and calculating "deformed" parameters.