Research on Resource Prediction and Load Balancing Strategies Based on Big Data in Cloud Computing Platform

Abstract

Reasonable and accurate resource estimation and good load balancing play a decisive role in the operational performance of large, high-load cloud platforms. This article proposes an intelligent scheduling framework that considers resource estimation, scheduling optimization, and data isolation. In terms of resource estimation, a hybrid prediction model based on LightGBM and LSTM was developed to model key indicators, including CPU, memory, and disk I/O, in a time series context. Experimental results have shown that the average absolute percentage error (MAPE) of the model on the Alibaba Cloud Tianchi dataset is 7.8%. In terms of load balancing optimization, a reinforcement learning method based on Deep Q-Network (DQN) was introduced to achieve dynamic scheduling and resource reallocation of multitasking. In terms of monitoring, closed-loop data collection and decision support are accomplished through Prometheus and Grafana. In order to improve security and model stability in multi-tenant environments, an isolation mechanism combining virtual network segmentation and access control lists (ACLs) is proposed. Tests on enterprise-level private cloud platforms have shown that the framework has increased resource utilization by 22.4% and reduced average response time by 17.3% under peak loads. The specific test results have demonstrated good practicality and utility.