Auto-Labeled Weed Detection and Species Classification

Project overview

Area	Details
Product context	Perception for an autonomous lawn-care robot
Teacher pipeline	HQ-SAM mask generation and EVA-CLIP classification
Deployment model	Compact YOLOv11n detector

I built an automated labeling pipeline to turn unlabeled lawn imagery into training data for a smaller deployment model. Large foundation models acted as offline teachers; the resulting annotations were then used to train a compact detector suited to the robot’s runtime constraints.

Model Roles

Component	Role
YOLOv11n	Small deployable weed detector
HQ-SAM	High-quality mask generation and refinement
EVA-CLIP	Region and weed-species classification

Label-generation pipeline

1. Generate candidate masks with HQ-SAM.
2. Filter noisy masks with contour detection and minimum area thresholds.
3. Merge small related masks, apply 5x5 dilation, and recover cleaner bounding boxes.
4. Classify each candidate region with EVA-CLIP text prompts such as weed, grass, and soil.
5. Refine selected masks using bounding-box prompts to HQ-SAM.
6. Export image-wise weed bounding boxes into a dataset-level JSON annotation file.

Species-classification path

• Crop detected weed regions from the image.
• Classify crops with QuanSun/EVA-CLIP-E_psz14_plus_s9B.
• Evaluated a larger EVA-CLIP model in place of the initial CLIP backbone for stronger region classification.

Design rationale

Using the heavier models offline separates annotation quality from deployment cost. The edge model receives task-specific training labels without carrying the teachers’ compute footprint onto the robot. A future iteration would fine-tune the mask-generation stage once enough reviewed examples are available.

Demonstration

The embedded demonstration shows this perception work in the context of the wider autonomous lawn-care robot.