Load Balance Optimization of Distributed Massive Database Information Acceptance and Processing in the Internet of Things Scenario

Authors

  • Jialiang Wang Department of Information and Engineering, Sichuan Tourism University, Cheng Du 610072, China

Abstract

In order to make full use of the resources of each service node in a cluster to improve the overall performance of the cluster, it is necessary to select the appropriate load balancing technology and efficient load balancing algorithm to allocate client access requests. Focusing on the load imbalance problem of Spark, this paper proposes an adaptive task scheduling strategy based on Spark cluster to achieve the load balancing optimization of Spark clusters. This strategy uses the heuristic algorithm of the ant colony simulated annealing fusion algorithm to optimize the task scheduling strategy of the Spark cluster according to the node’s current load and computing resources. This achieves the appropriate allocation of tasks for the purpose of load balancing optimization, thereby improving the cluster’s task-processing efficiency. In order to achieve dynamic load balancing on the Reduce side, a dynamic and lightweight division strategy is adopted. This strategy combines the load information for the dynamic design of the sampling scale and the lightweight design of the sampling method, and combines the sampling data and node performance to determine the number of Reducers. In addition, a division strategy is formulated based on the partition analysis of the sampling results and load information. After experimentation and analysis, it is concluded that the optimization technology has significantly improved the parallel computing performance of the Map Reduce cluster.

Keywords: load balancing; ant colony-simulated annealing algorithm; distributed massive database; Internet of Things (IoT) scenario

Cite As

J. Wang, "Load Balance Optimization of Distributed Massive Database Information Acceptance and
Processing in the Internet of Things Scenario", Engineering Intelligent Systems, vol. 29 no. 5, pp. 279-
289, 2021.


Published

2021-09-01