Distributed SQL database biz Cockroach Labs has analysed public cloud performance and concluded that Google Cloud Platform wins on throughput, although AWS is ahead on CPU performance and network latency.
Cockroach Labs first reported on cloud performance in 2018, saying: “We are committed to building a cloud neutral product, and we run test clusters on all three leading US cloud providers.”
The researchers measured CPU, network, storage, and online transaction processing (OLTP) performance. At the time, the researchers reported that “AWS outperforms GCP on nearly every criteria we tested – including cost.” Azure was not tested in 2018. In 2020 a new test showed relatively close performance from AWS, Azure and GCP, the conclusion being that “GCP shows dramatic improvement in the 2020 Cloud Report edging out AWS and Azure on price per performance of TPC-C but slightly underperforming AWS and Azure on max tpmC available on a three node cluster.”
How about 2021? The researchers benchmarked 54 different virtual machine types and performed nearly 1,000 benchmark runs. The benchmark scripts are open source here. None of the three providers have been shamed, with the detailed results showing wins for all three in some categories. This was also a test limited to VMs, and did not cover the myriad other services on offer.
The high-level overview reported that AWS provided the most cost-efficient machine and the best network latencies, and that the ARM-based Graviton processor “performed best in the multi-core CPU benchmark”. AWS was behind though on storage I/O and single-core CPU.
A key metric in the report is TPM (throughput per minute). TPM “in practical terms, measures the number of orders processed per minute,” the researchers said.
Azure “had comparable raw throughput (TPM) performance with GCP and AWS,” surpassed AWS in storage I/O performance, and led on storage IOPS and write latency with its ultra disks – but this came at a price, with Azure being the “least cost-efficient cloud provider in terms of $/TPM.”
GCP had the best single-core CPU performance and “the most throughput at every level,” the report concluded, outperforming AWS and Azure “in network and storage I/O (read and write) throughput.”
Two key charts show the most cost-efficient machine, based on cost per TPM, and the maximum obtainable TPM from a virtual machine.
AWS has a small but significant advantage in cost efficiency, while Azure competes well for best performance but at substantially higher cost
On cost efficiency over three years, AWS was hailed as the winner at 0.81 $/TPM, followed by GCP at 0.92 $/TPM and Azure at 0.95 $/TPM. For top performance, though, GCP won with TPM of 37,048, followed by Azure at 36,952 and AWS at 36,627 – though, once again, Azure was the most expensive at 1.49 $/TPM compared to GCP’s 1.01 $/TPM and AWS at 0.97 $/TPM.
Azure’s best performance is achieved using its Ultra Disk product.
The AWS Graviton2 processors scored better than AMD alternatives for multi-CPU scaling but not by a huge margin. “GCP and Azure machines produced scores that were 5 per cent and 7 per cent lower respectively,” said the researchers.
GCP was highly praised for its network throughput. “GCP’s top-performing machine had 165 per cent and 237 per cent more throughput than AWS and Azure respectively,” said the report. This is not so much a technical advantage, but that “GCP simply made more bandwidth available.” The researchers were disappointed with AWS in this respect, and said that “some AWS network-optimized machines… were expected to have up to 25 Gbps in network throughput but averaged 5 Gbps in our tests.” These AWS results, they said, were “admittedly surprising” and the team ran 10 independent experiments to confirm them.
The detailed report is good reading not only for those comparing cloud providers, but also for people trying to puzzle out the pros and cons of different VM types, from the huge number of variants available.
Overall, Cockroach Labs said that “declaring a winner was much harder” than in years past. GCP came top on the basis of “the best overall performance across the set of benchmarks”, but AWS won on cost-efficiency, which could be a more important metric for some cases, while Azure scored for its Ultra Disk storage in cases where cost is not the key consideration.
A caveat is that the rapid pace of change in the details of machine types on offer mean that a report like this is soon out of date. “Each of the cloud providers showed overall growth in machine performance since the 2020 report,” the researchers said, confirming that the trend is for performance to improve. ®