Dense Depth Labels from LiDAR–Camera Fusion

Project overview

Area	Details
Goal	Generate denser depth ground truth for synchronized camera images
Sensors	Calibrated camera and LiDAR streams
Key method	Multi-frame point-cloud registration and reprojection

I developed a ground-truth generation pipeline for dense depth estimation. A single LiDAR scan leaves most image pixels without a depth value, so the workflow registers nearby scans, aligns them to the target camera frame, and projects the fused cloud back into the image.

Pipeline

1. Transform the target LiDAR scan into its synchronized camera frame using the calibrated intrinsics and extrinsics.
2. Register 60 neighboring LiDAR scans with iterative closest point (ICP).
3. Align the target calibrated scan with the registered multi-frame point cloud.
4. Apply the recovered transformation and project the fused point cloud into the image plane.
5. Store the projected values as dense depth ground truth for model training and evaluation.

I also investigated RGB-guided depth completion and a multi-view stereo path using camera poses, but prioritized LiDAR-frame fusion because the ground truth should remain grounded in measured geometry rather than depend heavily on image appearance.

Sensor path

Step	Detail
Calibration	Transform LiDAR points into the corresponding camera frame using intrinsics and extrinsics
Fusion	Register 60 LiDAR frames with ICP
Alignment	Recover the transform between the target frame and registered multi-frame cloud
Projection	Project the fused cloud into the image to create dense depth ground truth

Key constraint

Accurate timestamps are essential. Camera–LiDAR synchronization errors propagate through registration and projection, reducing the reliability of the generated labels even when the geometric calibration is correct.

Result

LiDAR depth density increased from 6% with a single frame to 58% after fusing 60 frames.