Dual-Camera 3D Tennis Scene Understanding
A multi-model perception system that maps court geometry, players, and fast-moving ball trajectories into real-world 3D coordinates.
Project overview
I developed the visual-perception layer for a dual-camera tennis system that unifies court, player, and ball understanding in a common 3D court coordinate system. The main challenge was not any single detector: it was turning several specialized model outputs into a synchronized and geometrically consistent view of a live match.
My responsibilities
- Built the multi-model inference and fusion pipeline for synchronized camera streams.
- Developed temporal ball detection with ball-diameter prediction for a small, fast-moving target.
- Integrated grid-guided court keypoint detection and person detection with task-specific attributes.
- Connected camera-pose estimation, homography, and stereo geometry to real-world court coordinates.
- Prepared the model path for ONNX, TorchScript, and edge deployment.
System design
| Stage | Responsibility |
|---|---|
| Court perception | Detect court landmarks and recover the known tennis-court geometry |
| Camera localization | Estimate each camera’s 3D position and orientation in court coordinates |
| Player understanding | Detect people and distinguish active players, bystanders, gestures, and distance attributes |
| Ball perception | Combine temporal, full-frame, center-crop, and cross-camera inference branches |
| Geometric fusion | Map detections into a shared coordinate system and reconstruct player positions and ball trajectory |
Why multiple ball branches
A tennis ball occupies very few pixels and changes position quickly. The pipeline therefore combines evidence from full-frame context, a higher-resolution center crop, temporal inference, and projections from the second camera. Merging these branches makes the perception layer less dependent on a single view or scale.
Outcome
The resulting scene representation exposes camera pose, 3D player positions, and a reconstructed 3D ball trajectory. It provides the common visual foundation needed by downstream match analytics such as player movement analysis, active-player tracking, automated line calling, and live scene understanding.
Technology
Python, PyTorch, YOLO, temporal object detection, keypoint detection, person-attribute prediction, homography, camera-pose estimation, stereo geometry, ONNX, and TorchScript.