Aiming at the problems of low accuracy of crack monitoring and insufficient efficiency of damage assessment in health monitoring of civil structures, a crack detection and damage analysis system based on YOLO algorithm for civil structures is designed in the study. The crack feature extraction capability is enhanced by optimizing the YOLOX network structure, introducing lightweight convolution and coordinate attention mechanism. At the same time, a new MODSLayer module is designed, which enables the model to extract features in high dimensions of MODL-Head. The system integrates image acquisition and processing, multi-scale crack detection, geometric parameter quantification, and damage assessment modules, and realizes the automation of the whole process from detection to analysis. The MOD-YOLO algorithm F1 score is 0.521, which is 3.6% to 18.6% higher than the comparison algorithm, and the mAP reaches 58.491%, which is also much higher than the comparison algorithm. The results of this paper’s model for crack length as well as width are basically consistent with the real results, with an average relative error of 0.96% and 1.21%, respectively. The system constructed in the study detected that the length of crack No. 7 has achieved the maximum value of 1938.7 mm, and the angle with the base surface reaches 75.7°, which may continue to grow longitudinally. In this paper, the system found that the nonlinear coefficient of the civil engineering construction and the length of the crack has a pattern of “surge-slowly increasingdecreasing”, which indicates that the model has a high sensitivity to recognize the damage degree of the cracks in the civil engineering structure.