Paying attention to the changing situation of college students’ emotional state is the basis for orderly psychological intervention. This paper constructs a three-stage abnormal behavior detection model for college students, which includes target detection, multi-target tracking and abnormal behavior detection. The YOLOv5s detection module, which is small in size and fast in operation, is selected to detect students’ behavioral and emotional information on the premise of ensuring the completeness of feature information extraction. Based on the deflationary dot product self-attention method, continuous emotion inference of students under multimodal fusion is realized. Combined with the emotion recognition reasoning results, psychological intervention for abnormal students is carried out. The results found that the area of ROC curve reaches 0.9, and the effect of behavioral-emotional recognition is good. The average accuracy of the model’s emotional reasoning for five subjects was 99.54%, and it had a fast running speed and fine emotional classification effect. The scores of the 4 scales before and after the psychological intervention of abnormal students were P<0.01, and the mental health level was effectively improved after the intervention.