With the continuous development of weather prediction technology, the traditional single data source prediction model has been difficult to meet the demand of increasingly complex weather changes. In this paper, an intelligent weather prediction model based on the fusion of cloud radar and weather sensor data is proposed, which utilizes a combination of Kalman filter algorithm and deep learning model for weather forecasting. First, the Kalman filter algorithm is used to invert the cloud radar echo data, and the inversion accuracy is improved by optimizing the parameters, with the lowest temperature deviation reaching 3.2 K. Then, based on the multimodal fusion of weather prediction model, the temporal and spatial dependencies in the meteorological data are modeled using the Transformer encoder-decoder architecture, which further improves the prediction accuracy. The experimental results show that the model in this paper performs better in the evaluation indexes of RMSE, MSE and MAPE compared with the LSTM and RNN models, with an RMSE of 2.6483, a MAPE of 0.0229, and an R² value close to 1, which makes the prediction results the closest to the real values. The model shows significant advantages in multimodal data fusion and provides an effective solution in the field of weather prediction.