Benchmarking experts are looking at the difficult problem of measuring the performance of the part of cloud computing known as Infrastructure-as-a-Service. We all use the cloud, but it's hard to measure the performance of cloud computing. I looked into this a long time ago using Primate Labs' Geekbench 3 and other programs, which frankly weren't ideal for testing cloud computing performance. But now Principled Technologies and the BenchmarkXPRT Development Community have just published the CloudXPRT Preview, a free benchmark, which seeks to accurately measure the performance of the cloud in the Infrastructure-as-a-Service (IaaS) model.
The CloudXPRT beta program includes web micro-services and data analytics workloads. Testers can use the metrics from the two workloads to compare the performance of the IaaS stack - both hardware and software. With this information, you can decide which cloud provider and a stack of bricks are right for you.
Workload reports performance in terms of transactions per second, which testers can use to directly compare IaaS stacks and to assess whether a particular configuration can meet their needs and service level agreements (SLAs). Agreement).
The workload calculates the training time of the XGBoost machine learning model. This is a gradation framework for machine learning-based regression and classification problems. The goal of the workload in CloudXPRT is to assess how well an IaaS stack enables XGBoost to speed up and optimize model training. The workload indicates the latency and throughput rates.
CloudXPRT asks you to run your tests with the following configuration:
• Test systems must be running Ubuntu 18.04.
• You must use Docker containers managed by Kubernetes .
• The benchmark requires relatively high-end servers. For functional testing, the physical nodes or VMs under test must meet the following minimum specifications: 16 logical or virtual CPUs; 8 GB of RAM; 10 GB of available hard-disk space (20 GB for data analysis workload).
CloudXPRT can be configured to run on your on-premises data center, Amazon Web Services (AWS) , Google Cloud Platform, or Microsoft Azure .
It's not easy to measure the performance of the public cloud
It's not easy to measure the performance of the public cloud. So much depends on the exact configuration of the cloud you are using and your workloads; it is difficult to make objective judgments. And hardly any of the benchmarks of legacy servers and systems are up to the challenge of the cloud.
As Bill infectious, co-founder of Principled Technologies and benchmarking expert explains, “I don't currently see a direct competitor to CloudXPRT. It appears that people mostly use old benchmarks in new contexts with varying levels of relevance. You can certainly run a bunch of SPEC CPU workloads on a cloud, but ... "
Bill Catching adds, “Existing datacenter benchmarks make it difficult to understand how applications will perform on a given IaaS infrastructure. CloudXPRT uses cloud domestic components on a hardware and software stack to provide end-to-end performance metrics that allow users to choose the best configuration for their business. ”
Bridge the gap between performance testing and using benchmarks created for another purpose
That said, ThousandEyes , a cloud computing infrastructure analysis company, provides a useful annual assessment of the performance of the global network of public cloud providers .
CloudXPRT's goal is, according to Bill Catching, "to help fill that gap between performance testing using your real-world applications and using some existing benchmarks created for another purpose." One of the areas where we are still trying to improve is to make things run more smoothly across different cloud service providers ”.
It is still a first attempt. But he hopes it will be useful and that with the help of others it will be even more so soon. “I don't think a benchmark has to be as big a business as it once was. The ability to use existing open-source code and higher-level interfaces make things a. We hope that once CloudXPRT becomes available, we will receive code contributions or suggestions from OEMs and other industry players. "
Users are encouraged to use CloudXPRT freely and to help shape effortlessly the future of this benchmark with their feedback. Analyst can also freely access the source code of CloudXPRT .
According to analyst firm Canalys , companies want to know if they are making the most of their money, after spending a record $ 107 billion on infrastructure services in 2019, an increase of 37 % compared to the previous year.