Load Balance Optimization of Distributed Massive Database Information Acceptance and Processing in the Internet of Things Scenario

Authors

  • Jialiang Wang Department of Information and Engineering, Sichuan Tourism University, Cheng Du 610072, China

Abstract

In order to make full use of the resources of each service node in a cluster to improve the overall performance of the cluster, it is necessary to select the appropriate load balancing technology and efficient load balancing algorithm to allocate client access requests. Focusing on the load imbalance problem of Spark, this paper proposes an adaptive task scheduling strategy based on Spark cluster to achieve the load balancing optimization of Spark clusters. This strategy uses the heuristic algorithm of the ant colony simulated annealing fusion algorithm to optimize the task scheduling strategy of the Spark cluster according to the node’s current load and computing resources. This achieves the appropriate allocation of tasks for the purpose of load balancing optimization, thereby improving the cluster’s task-processing efficiency. In order to achieve dynamic load balancing on the Reduce side, a dynamic and lightweight division strategy is adopted. This strategy combines the load information for the dynamic design of the sampling scale and the lightweight design of the sampling method, and combines the sampling data and node performance to determine the number of Reducers. In addition, a division strategy is formulated based on the partition analysis of the sampling results and load information. After experimentation and analysis, it is concluded that the optimization technology has significantly improved the parallel computing performance of the Map Reduce cluster.

Keywords: load balancing; ant colony-simulated annealing algorithm; distributed massive database; Internet of Things (IoT) scenario

Cite As

J. Wang, "Load Balance Optimization of Distributed Massive Database Information Acceptance and
Processing in the Internet of Things Scenario", Engineering Intelligent Systems, vol. 29 no. 5, pp. 279-
289, 2021.


Downloads

Published

2021-09-01

Issue

Vol. 29 No. 5 (2021): International Journal of Engineering Intelligent Systems

Section

General Submission

License

The submission of a paper implies that, if accepted for publication, it will not be published elsewhere in the same form, in any language, without the prior consent of the publisher. Before publication, authors are requested to assign copyright to CRL Publishing Ltd. This allows CRL to sanction photocopying, and to authorize the reprinting of issues or volumes according to demand. Authors' traditional rights will not be jeopardized by assigning Copyright in this way, as they retain the right to reuse the material following publication, and to veto third-party publication.