College libraries are important information centers in colleges and universities, which are important positions for knowledge storage, communication and discovery. With the development of social digitization and informatization, the quantity and variety of information are changing day by day, and the information needs of university user groups are also changing. Research from data collection and analysis, label extraction to portrait mining applicable to the library user portrait model, combined with hybrid recommendation algorithms from the data collection layer, processing layer, fusion layer, service application layer four levels to build a multimodal dataenabled smart library structure. The K-means algorithm was used to cluster and analyze library users, and four types of users were formed: “pragmatic”, “youthful”, “recreational” and “curious”. The accuracy, recall and comprehensive value F, which are commonly used to evaluate the recommendation effect, are chosen to validate the recommendation results, and the hybrid recommendation integrating user profiles is higher than the traditional collaborative filtering recommendation in recall and F value, and the recommendation effect is good.