On this page

Research on optimization scheme for large-scale digital media data processing based on distributed computing architecture

By: Tao Liu 1, Meiling Yang 2
1Department of Journalism and Communication, Anhui Vocational College of Press and Publishing, Hefei, Anhui, 230601, China
2Hefei Transportation Comprehensive Administrative Law Enforcement Detachment, Hefei Transportation Bureau, Hefei, Anhui, 230601, China

Abstract

The explosive growth of digital media data makes the traditional centralized processing architecture face serious challenges in computational efficiency and storage cost. This paper responds to the demand for efficient processing of large-scale digital media data and launches a research and analysis based on distributed computing architecture. Combing the feature clustering process of distribution sets and features of association rule data, as well as the distributed data frequent item clustering collection process. At the same time, the CDMDP protocol with advanced encryption technology, distributed storage mechanism and smart contract features is designed to effectively realize the distribution and protection of digital data content. Combining logical search tree and parallel algorithm SFUPM-SP, Spark-based parallel mining algorithm for distributed computing of big data is proposed as a processing and optimization method for large-scale digital media data. In the system platform built by this paper’s method, the average execution time of K-Means algorithm on data is only 17.9 seconds, which demonstrates the effectiveness and feasibility of this paper’s method.