Multi-sensor fusion calibration for intelligent vehicles

The multi-sensor fusion calibration of intelligent vehicles is a crucial step for the auto drive system to achieve "spatial synchronization" and "temporal synchronization". Laser radar (LiDAR), camera, millimeter-wave radar (Radar), and inertial measurement unit/global navigation satellite system (IMU/GNSS) each have their own independent coordinate systems. The purpose of fusion calibration is to calculate the relative position and orientation relationship (extrinsic matrix) between them, allowing the vehicle to perfectly overlay the two-dimensional images it "sees", the three-dimensional point clouds it "scans", and the speed and distance it "measures".

The following are the core processes and mainstream methods for multi-sensor fusion calibration in intelligent vehicles:

1. Core preparation before calibration

Before starting mathematical calculations, it is necessary to ensure that the hardware and environment are in a standard state:

(1)Time stamp alignment (time synchronization): This is the prerequisite for spatial calibration. Typically, PTP (Gigabit Ethernet Time Synchronization Protocol) or GPS/PPS second pulse is used to align the sampling time of all sensors to the microsecond level, ensuring that "they see the world at the same moment".

(2)Vehicle status inspection: The vehicle is parked in a standard horizontal calibration room, with tire pressure and suspension at standard loads, and the vehicle is completely stationary.

(3)Deployment of calibration objects: According to the calibration requirements, arrange specially made calibration objects around the vehicle.

(4)Common calibration objects: calibration plates with high reflectivity patterns, 3D boxes, cylinders, or specially made triangular reflectors (commonly used for millimeter-wave radar).

2. Classification of core fusion calibration processes

Fusion calibration is typically performed based on the "pairwise alignment" of sensor combinations, ultimately unifying them into the vehicle/base link coordinate system.

(1) Joint calibration of LiDAR and camera (LiDAR-to-Camera)

This is the core calibration for autonomous driving, used to achieve the pre-fusion of point cloud and image (color point cloud).

[Extract 2D feature points from image] ──┐

 ├─> [Search for correspondence] ├─> [Minimize reprojection error] ├─> [Output extrinsic matrix [R|T]]

[Extract 3D feature points from point cloud] ──┘

① Feature extraction:

Camera side: The algorithm identifies the coordinates ((u, v)) of the corners, edges, or central pixels of the calibration board in the two-dimensional image.

LiDAR side: The algorithm extracts the coordinates ((X, Y, Z)) of the edges or center points of the calibration board in three-dimensional space based on the reflection intensity of the point cloud (or geometric plane fitting).

② Mathematical solution: This is a typical 3D-to-2D Perspective-n-Point (PnP) problem. Using the least squares method or nonlinear optimization, the rotation matrix R and translation vector T are calculated.

③ Online/dynamic alignment: During daily driving, by utilizing natural features such as utility poles on both sides of the road and building edges, the alignment between image edges and point cloud edges is optimized (e.g., using mutual information method) to fine-tune the extrinsic parameters in real-time.

(2)Joint calibration of LiDAR and millimeter-wave radar (LiDAR-to-Radar)

Millimeter-wave radar can measure speed and has strong penetration, but its point cloud is extremely sparse and noisy; LiDAR point cloud is dense. Their goal is to achieve "spatial trajectory alignment".

①Static method (calibration chamber): Place a corner reflector with high reflectivity at a specific location. Millimeter-wave radar appears as a strong signal point on the energy map, and lidar can also accurately identify the three-dimensional contour of the corner reflector. The external parameters are calculated by matching the center points of the two.

② Dynamic method (road driving): The vehicle drives on an open road, tracking a moving target ahead (such as another vehicle). Using algorithms like Kalman Filter, the trajectory of the object tracked by the LiDAR and the trajectory of the dynamic point measured by the millimeter-wave radar are matched, and the external parameters are solved through trajectory correlation.

3. Joint calibration of LiDAR/camera and inertial measurement unit (Sensor-to-IMU)

Align the perception sensor with vehicle kinematics to address the issue of "how to compensate the sensor during vehicle shaking and bumps".

① Hand-eye calibration method: It belongs to the typical AX = XB matrix solving problem.

② Calibration process: The vehicle is instructed to follow a specific trajectory (such as driving in an "8" shape, experiencing frequent acceleration and deceleration, and making turns). The IMU records the angular velocity and acceleration of the vehicle in real time to estimate its own motion trajectory. Simultaneously, the LiDAR estimates a motion trajectory through the point cloud odometry (LOAM) or the camera estimates a motion trajectory through the visual odometry (VO).

③ Solution: The algorithm calculates the relative position of the sensor with respect to the IMU by aligning the rotation and displacement of these two motion trajectories.

3. Integrated joint debugging and closed-loop control of multiple sensors

After all pairwise calibrations are completed, global closed-loop optimization is required:

(1)Unify coordinate system: Convert all relative external parameters to the body center coordinate system (such as the point where the rear axle center is projected onto the ground).

(2)Error consistency check: Ensure a closed loop in the transmission chain. For example, the IMU position calculated from LiDAR ightarrow Camera ightarrow IMU should be consistent with the result directly calculated from LiDAR ightarrow IMU. If there is a discrepancy, perform global adjustment on the entire topological network using graph optimization.

IV. Calibration quality assessment and effect verification

After calibration, strict quantitative indicators must be used to verify its accuracy:

①True value comparison method: With the aid of a high-precision, wide-range total station or laser tracker, manually measure the physical absolute positions of the marked points on each sensor housing in the vehicle body coordinate system, and compare them with the extrinsic parameter matrix calculated by the algorithm. The typical requirements for extrinsic parameter accuracy are: distance error < 1~2 cm, angle error < 0.1°~0.2°.

② Point cloud back projection (color fusion projection): The 3D point cloud collected by the LiDAR is projected onto the 2D image of the camera through the calculated extrinsic matrix. If the point cloud colors at the contours of utility poles, street signs, and vehicles are perfectly aligned with the image edges, without any displacement or misalignment, it indicates excellent calibration quality.

③ Spatio-temporal consistency test: During dynamic driving of a real vehicle, check whether the 3D box detected by the camera, the point cloud clustering box of the LiDAR, and the target point of the millimeter-wave radar for the same obstacle (such as the preceding vehicle) completely overlap in the Bird's Eye View (BEV) and do not undergo severe jitter due to vehicle bumps.