The rapid development of the construction industry and the deepening reform of spiritual civilization construction, the public’s aesthetic level is also improving, which puts forward more diversified demands for housing building appearance design and decoration. The article proposes an aesthetic evaluation method of modern housing building appearance design with VR eye movement data tracking. Semantic difference analysis is used to design the questionnaire of modern housing building appearance design, and factor analysis is used to obtain the main influencing factors of housing building appearance design. VR eye-tracking technology is introduced to integrate Chinese painting art and modern housing building exterior design in a virtual scene, and the eye movement data of the subjects are obtained, so as to analyze the influence of the integration of Chinese painting art and modern housing building exterior design on the subjects’ gaze time and visual comfort. The mean value of the subjects’ evaluation scores on the housing building exterior design fused with Chinese painting art was 3.283 points, and the longest total time spent on the modern housing building exterior fused with Chinese painting art was about 2750.94 s, and the shortest total time spent on the simple building exterior was only 175.98 s. When the color purity and luminance of the Chinese painting art and the modern housing building exterior design exceeded 82.1% or 65%, respectively, it would significantly reduce the subjects’ visual comfort. Incorporating the art of Chinese painting into the architectural appearance design of modern housing requires choosing relatively warm colors and diverse elemental symbols as a way of highlighting its aesthetic value in architectural appearance.