Abstract: Existing small object detection methods for safety helmets, self-rescuers, and mining lamps show insufficient adaptability to small targets and unstable sample matching during training under underground coal mine conditions where low light and backlight coexist. Based on the YOLOv11n framework, a small object detection model for complex lighting conditions in underground coal mines, named LSD-YOLO, was proposed. In the neck network, a Lighting-Aware Spatial and Channel Adaptive Modulation (LASCAM) module was introduced to perform channel-wise affine compensation and spatial saliency modulation for feature responses under low-light and backlight scenarios. A Frequency-Aware Small-Object Pyramid Module (FSPM) was designed to enhance the detail representation of small objects through multi-scale frequency decomposition and high-frequency modulation. A Low-Light and Small-Object Detection Friendly Loss (LSD-Loss) was designed to enhance the learning signal of valid samples, and a Scale-Adaptive Task-Aligned Distribution (SATAD) strategy was introduced so that the positive sample matching process was adaptively adjusted according to object scale, thereby improving training stability and the utilization efficiency of small object samples. The results showed that LSD-YOLO achieved excellent detection performance, with mAP@0.5 reaching 91.6%, outperforming all comparison models. Compared with the baseline model YOLOv11n, the precision, recall, and mAP@0.5 of LSD-YOLO increased by 0.9%, 1.2%, and 3.7%, respectively, effectively improving detection performance in complex underground scenarios. In terms of model complexity, LSD-YOLO had 4.1×10⁶ parameters and 8.7 GFLOPs, which were much lower than those of RT-DETR-R18 and YOLOv11s, and the inference speed reached 104.1 frames/s, which met real-time detection requirements. The mAP@0.5 of LSD-YOLO was improved by 0.1% and 0.2% compared with RT-DETR-R18 and YOLOv11s, respectively, indicating a good balance between detection accuracy and model complexity.