As the Red Hat OpenStack Platform has progressed in recent years to cater to a diverse array of customer requirements, the need for scalability has grown increasingly crucial. Clients rely on the Red Hat OpenStack Platform to provide a robust and adaptable cloud infrastructure, and as its utilization expands, so does the necessity for deploying more extensive clusters.
Hardware and Configuration
To enhance the platform to over 1000 virtual instances, a supermicro server was employed as a baremetal undercloud host with 32 total cores and 64 threads. Utilizing a 25G control plane/provisioning NIC, the setup comprised of 3 baremetal controllers, 7 baremetal CephStorage nodes, and over 250 baremetal dell servers as hypervisors for 1150 virtual instances. Additionally, 30 baremetal dell servers were allocated for NFV instances with SRIOV and OVNDPDK capabilities.
Test Approach
The main objective was to scale beyond 1000 virtual instances to evaluate the impact on performance and scalability, identify bottlenecks, and address any issues encountered.
- Undercloud deployment and configuration of adequate control plane subnet ranges for large-scale overcloud deployment
- Incremental scaling in batches of 100 nodes until surpassing 1000+ instances
- Running stack operations and dynamic workload tests to simulate real-time tasks
Findings
Scaling in increments of 100 virtual instances revealed challenges such as open file limit errors, timeouts during stack operations, and heat stack creation failures. In response to these issues, adjustments to various configurations such as open file limits and template sizes were made to enhance performance and mitigate congestion.
Grafana snapshot: illustrates heat-engine CPU consumption during stack creation
Workload Assessment
Dynamic workload testing using Browbeat facilitated the simulation of real-world cloud usage patterns. By creating and evaluating various scenarios, performance and responsiveness of critical components were assessed. Browbeat’s ability to generate dynamic workloads enabled the examination of RHOSP’s handling of real-time activities on a large scale.
Final Thoughts
Conducting performance and scalability testing with over 1160 virtual instances demonstrates a dedication to pushing the boundaries of the Red Hat OpenStack Platform 17.1. The rigorous testing and optimizations undertaken exhibit a commitment to excellence and a continuous effort to enhance scalability and performance in large-scale cloud deployments.
