Two-Frame Temporal Player Detection System
A six-channel person detector that combines a current frame with a configurable earlier background frame, using matched crops and optional camera-motion compensation.
Problem and approach
I built a person detector for court-view video that uses an earlier image of the scene as context. The extra view helps the model learn foreground changes around players while retaining information about the net and background.
Foreground and background input
The input is a current RGB frame plus an earlier RGB frame, cropped at the same location and concatenated into six channels. The background is a past frame, not a guaranteed empty-court image or a computed subtraction mask.
The temporal gap is configurable. The reference video and image-sequence runners default to a 30-frame offset, so this is not restricted to adjacent frames. The model returns detections for the current foreground frame, rather than separate detections for both inputs.
Preprocessing and motion compensation
Training images can store foreground and background vertically in a single file; the loader splits that image and converts the pair into channel-stacked RGB. Inference also accepts raw videos or preprocessed pairs.
Both frames receive matching resize and crop operations. Optional KLT alignment compensates for camera motion before pairing. Predicted boxes are transformed from the crop back into the foreground image’s coordinates, with confidence scores available to downstream consumers.
Deployment and evolution
The project includes training, validation, PyTorch inference, export adapters, and edge inference paths. Its output is person location and confidence. The later person-attribute detector adds depth, hand-raise probability, and player-role prediction as a separate model design.
