With the continuous iteration of algorithms in the field of deep learning, generative AI design is ushering in a revolutionary change. In this project, we study the controllability path of generative AI, take Stable Diffusion as the base model of image generation, use the improved LoRA method and Controlnet to fine-tune its control, and realize the image generation method based on Stable Diffusion ControlNet. The experimental results show that the image quality of the image generation model designed in this paper obtains better performance in IS, FID and other evaluation indexes with reasonable scale parameter and sampling step size. 8.83% and 76.07% of IS and FID values are improved compared with the Stable Diffusion1 model when scale parameter is set to 8, and 1000 sampling steps are used to achieve better image quality than the Stable Diffusion1 model. The minimum FID value of 1.18 is obtained when the sampling step is 1000, which verifies the effectiveness of the Stable Diffusion ControlNet network designed in this paper. The silk scarf pattern generated by the model scores 5.36 and 5.47 in artistic aesthetics and normality, and the generated pattern is clear and of excellent quality. In addition, the model meets the requirements of practical applications in terms of computational efficiency and hardware cost. The results show that the proposed Stable Diffusion ControlNet model can be used as a generative AI method with good fine-tuning.
