Listening ability is the core part of English language application in practice, so the evaluation of its teaching effect and the optimization of teaching design are the focus of improving the quality of English teaching in colleges and universities. In this paper, the sophomore English majors in college I are selected as the research object, and the English listening teaching evaluation system is established through the form of questionnaire survey. Then a logistic regression analysis model is established, which is used as an analysis method of the effect of English listening teaching in colleges and universities. With the support of the logistic regression model, the evaluation model of English listening teaching effect in colleges and universities is proposed. Through the logistic regression analysis of multiple influencing factors of teaching effect, the column-line diagram model is constructed for the evaluation of teaching quality. In the correlation analysis between different influencing factors and teaching effect, the correlation coefficient between teaching effect and teaching content is the highest among the four influencing factors at 0.936, which is more significant. The study points out that English listening teaching in colleges and universities should pay attention to the design and arrangement of classroom teaching content, and promote the improvement of English listening teaching effect by setting reasonable teaching content.