The Age of Big Compute™
Houston, we have a problem — in computing.
For sure, we have come a long way. The iPhone in your hand is more powerful than all the computing power combined that sent Neil Armstrong to the moon more than half a century ago. But today, we are hitting a hard ceiling in computational power — computational power we need for creating revolutionary breakthroughs in science, business and society. At a company born to solve problems using quantum computing and other advanced technology, we are feeling it intensely.
There are three essential elements in computing: data — the information source; model — the algorithms to process the data; and compute — the device that executes the instructions of the models to process the data. These three elements need to advance in tandem to create breakthroughs. If any one of them becomes a bottleneck, the whole innovation cycle stalls.
Case in point: The field of AI started in the 1950s, but AI did not become widely practical until after 2012. Why? Big Data emerged in the mid-2000s, providing fertile training materials for models; AI deep learning models such as convolutional neural networks became more sophisticated and mature after decades of research; and then in 2012, Geoffrey Hinton at the University of Toronto found GPU, a massively-parallel-small-compute architecture originally designed for gaming, to be super effective for neural networks calculation. When all three elements — model, data, compute — were in place in 2012, the world of AI exploded with a big bang.
Since the term “Big Data” was coined in 2008, data has grown exponentially. Big Data is now a $70 billion industry on its own and affects virtually every other industry.
Meanwhile, models have been growing exponentially into “Big Models”.
In computer vision or natural language processing, for example, increases in accuracy often demand exponential increases in model size. As we push for new AI breakthroughs such as autonomous driving or conversational AI, we have created bigger and bigger models. For example, the latest language model, GPT-3, has 175 billion parameters, more than 100 times bigger than its predecessor GPT-2.
Computational models for optimization and simulation are the bedrock for scientific discovery and economic efficiency. As we push the knowledge boundaries in pandemic prevention, material discovery, supply chain optimization, financial engineering, and other high-impact applications, more scenarios need to be explored and more data points need to be incorporated. But linear growth of problem size often drives the search space of those models to grow exponentially larger. Many are so large that they are beyond solvable, even by today’s most powerful supercomputers.
As data and models have grown exponentially, compute can no longer keep up.
The golden era of Moore’s Law is over: we can no longer double the computing power in silicon chips every 18 months, simply because we have shrunken the circuit so much that it has approached the physical limitation: the latest 2nm silicon transistor is only 10 atoms wide. Specialized silicon chips aimed at speeding up AI calculations from GPU, TPU, to Neuromorphic chips are all bound by this hard reality.
An alternative approach to gain more computing power is to go parallel. For example, the GPT-3 model is distributed in 7500 computer nodes. High Performance Computing (HPC) clusters aggregate more computing nodes with high-speed fabric, and break down computationally complex workloads into smaller tasks so that they can be distributed across these nodes. But it is not feasible to scale computing power exponentially simply by parallel processing. If GPT-4 is going to be 500 times bigger than GPT-3, as expected, it will need 4 million computer nodes. In addition, some super-complex computational problems can hardly be broken down into parallel tasks. They need to be executed sequentially.
Quantum computing is a game changer, and the development of the technology – both hardware and software — is advancing at an accelerating pace. Leveraging mind-boggling phenomena in quantum mechanics that are not intuitive to the classical world, quantum computers can perform computation in a way beyond what is possible with classical computers. Quantum computers and algorithms bring the potential to exponentially boost computing power and solve our hardest problems in machine learning, optimization, and simulations.
But quantum computers have limitations. They aren’t fault-tolerant yet and will always need to work with classical computers to be effective. For example, in the flagship quantum chemistry algorithm for use on near-term quantum computers, the Variational Quantum Eigensolver (VQE), less than 5% of its sub-tasks run on quantum computers; the rest run on classical machines. Also, we have many distinct types of quantum computers, from trapped ions, superconducting qubits, photonic qubits to neutral atoms, each with its own strengths. We need classical computers to select and send tasks to the quantum computer type best suited to execute them.
Big Data fuels the growth of Big Models, from AI/ML to optimization and simulation; quantum algorithms can help optimize AI/ML models, and AI/ML models can also help create better quantum algorithms, for instance, by helping optimize quantum circuit parameters. The interplay of these classical and quantum fields will inevitably create a fusion of heterogeneous, computationally complex applications, such as quantum machine learning or quantum natural language processing.
This new breed of applications is composed of both classical and quantum algorithms, tasks that can be executed in parallel and tasks that must be executed sequentially. These tasks ingest large volumes of data from distributed sources including public and private clouds, and need to be executed by distinct and distributed – often highly fragmented — compute resources. For example, a quantum machine learning application we pioneered at Zapata – Enhanced Combinatorial Optimization with Quantum Generative Models– may need to run on GPUs, CPUs, as well as quantum computers.
To meet the needs of these heterogeneous and computationally complex applications, and to keep up with the exponential growth of data and models, enterprises need Big Compute.
Specifically, we need a new computing paradigm that intelligently orchestrates computing workflows across a hybrid of quantum and classical resources — NISQ and Fault-tolerant quantum computers, supercomputers, high performance computers, as well as GPU, TPU, and other emerging architectures. This paradigm needs to optimize computing tasks across these heterogeneous computing resources, to streamline and accelerate the workflow of data and models across public clouds, edge, and on-prem environments.
At Zapata, we built Orquestra® to enable enterprises to harness the power of quantum and Big Compute. As the operating system for Big Compute, it is a unified software platform for developing quantum-ready applications® and deploying them on a hybrid spectrum of quantum and classical resources. It delivers on the following Nine Principles of Big Compute that enterprises need to solve their most computationally complex problems:
For the reasons laid out in this blog post, we believe the next decade will be The Age of Big Compute. With Moore’s Law running up against physical limitations and models becoming exponentially bigger, Data, Models, and Compute need to advance in tandem. Quantum computing will be the largest step-change for Big Compute and power the revolutionary breakthroughs that accompany it.
To learn more about Big Compute and what your organization can do now to take advantage of it today – and in the future as quantum devices and software become more powerful and useful — contact us.
[In a follow-up to this post, we will break out the Nine Principles of Big Compute in more detail and provide more context about how they manifest today and what to look for in the future as all the elements progress. Beyond post #2, we are also developing additional content in the form of videos, webinars and more. Stay tuned via our blog, Twitter and LinkedIn to stay up to date.]
