Physical fitness quality, as the core quality of college students in public security colleges and universities, and its correlation with college students’ vocational ability has gradually become a research hotspot in related fields in recent years. This paper identifies and judges college students’ physical training actions by designing classifiers. At the same time, the gray correlation analysis method is used to analyze and construct the relationship model between physical fitness training and vocational ability of college students in public security colleges. By reflecting the relationship between physical training and occupational ability, it provides effective data reference and target guidance for the optimization and improvement of physical training programs. The design of the classifier is based on the identification process of physical training movements of college students in public security colleges, and adopts the Support Vector Machine algorithm (SVM) as the classification method of physical training movements and behaviors. Finally, the ant colony algorithm is introduced to optimize the kernel function of SVM algorithm to improve the classification accuracy and establish the physical training action classifier based on SVM. In the analysis experiment with a total of 154 college students from a public security university, the occupational ability performance scores of the students in the low level group improved up to 8.44 points compared with the pre-training scores after targeted physical fitness training.