AI is growing quickly.

Possibly too quickly…

More and more parameters are being used to train the Large Language Models (LLMs) that AI applications like ChatGPT are being built on top of. Their growth has been exponential, and possibly unsustainable.

The original GPT LLM had 1.5 billion parameters.

GPT-3 had 175 billion parameters.

GPT-4, OpenAI’s current LLM, was trained with a staggering 1.7 trillion parameters.

GPT-5 is still in planning stages, but it could have ten times as many parameters as its predecessor, clocking in at 17.5 trillion.

Other generative AIs are scaling at similar rates. They are getting bigger and more complex in order to become more capable, so the number of parameters used to train their neural networks is growing quickly from one generation to another.

At a rate of 10x growth between generations, optimizing parameters to wring out the maximum performance from a large language model like GPT is critical, because as the model complexity increases, so do the hardware and energy requirements.

Which is really, really expensive!

OpenAI is reported to be raising cash by selling shares at a valuation of $86 billion.

And they’re going to need it, as the company will only make about $1.3 billion this year, while merely running ChatGPT is estimated to cost the company $700,000 each and every day.

That figure is just to run ChatGPT for end users, not train it, which is also incredibly expensive.

OpenAI’s models were trained using Nvidia hardware, which is in short supply, and costs an arm and a leg even if you are lucky enough to find it.

GPT-4 was trained on 25,000 Nvidia A100 GPUs for about 3 months. GPT-5 will require far more if the training is to take a similar reasonable amount of time. Some estimates are for OpenAI’s next version to require half a million more powerful GPUs to do the job over a 3 month period.

That would be half of all the latest model accelerators, H100s, that Nvidia produces—for a single project. And the cost would be astronomical. The list price for an H100 accelerator is in the $30,000 range, meaning that acquiring close to 500,000 GPUs would come in at the princely sum of $150 billion.

That’s not even accounting for any other hardware like CPUs, server chassis, networking and interconnect equipment, racks, cooling, memory, storage, and all the other hardware and physical plant required for such a massive project.

An Unstoppable Force

Nvidia doesn’t have much of an incentive to cut prices since it is selling every unit it can make right now. This explains OpenAI’s move toward potentially designing and manufacturing its own hardware under former Apple product designer Jony Ive.

Doing this could help greatly reduce the cost of training future LLM’s at OpenAi, and not only through cutting out Nvidia’s hefty profits. OpenAI could also carefully tune the characteristics of its hardware to closely match its computational requirements, so that it could get more training done for every compute cycle.

My back of the envelope calculation earlier might not line up with what it will eventually cost OpenAI to build GPT-5, but it underscores a big problem we’re going to start seeing soon, and that’s the physical and economic limits to building more advanced AIs using current technology.

We’re already seeing it.

On the one hand though, there are positive factors that help reduce costs at play. For example, Moore’s Law still lives, meaning every couple of years or so, the cost of computational performance halves, thanks to improving semiconductor manufacturing techniques that can scale down components and pack more elements on a silicon wafer.

As that law has played out over the past 60 years, it has enabled ever more powerful chips, bringing down prices and putting powerful and affordable computing on everyone’s desk and in everyone’s pocket.

There’s also potential for improved AI training optimization, which currently is in the works.This would mean that we could figure out ways to train AIs that require less of a computational burden.

But these two factors aren’t improving at nearly the same rate as the growth in the size of the models. So if AI is to keep improving at the rate it has been I think we are going to need some sort of breakthrough, or series of breakthroughs.

The faster a chip can cycle, the more work it can do in a given amount of time.

The problem with using current silicon-based electronics is that, even though we have denser and denser chips (which improves performance), they don’t cycle faster.

In fact, we reached a practical limit decades ago.

Even now, clock rates for even the most advanced chips, including Nvidia H100s, run at under 2 gigahertz—that’s two billion cycles per second. Other chips, like Intel CPUs, can run somewhat faster.

But there is no big gain to be had here...the highest clock rate ever recorded as a Guiness World Record was 8.42 gigahertz, and they had to bathe the chip in liquid helium to do it. At higher clock rates, the chips just get too hot for silicon to handle.

There are other materials, like silicon carbide and gallium nitride, that you can run hotter and theoretically run faster. They are currently used in specialized applications like power electronics that generate more heat.

But even though they can take the heat, it doesn’t mean you have a great situation. That heat is all wasted energy, and data centers already consume over 2% of our electricity supply (and rising).

That might mean we’ll have to ditch the electronics that have been our computing mainstay for many decades and do something completely different, like turning to light.

Decades ago using light to transmit data revolutionized our communications infrastructure and made the internet possible. Fiber optic lines carried this tech update to you on beams of light for at least part of the distance.

Light could also be used to perform calculations. MIT researchers recently created new processors that work using photons of light rather than electrons to perform computational operations.

They claim these electro-optical chips can compute up to 100 times faster than existing processors in AI learning systems, plus, they can run 100 times more efficiently, meaning less wasted energy in the form of heat.

Another solution could be to ditch the classical binary digital computer entirely, whether using electrons or photons to manipulate bits, and do something truly radical.

That’s what quantum computing represents, where each computational component accesses additional degrees of capacity by ditching the binary bit and using the quantum computational qubit instead.

If that breakthrough occurs, as I expect it will, it won’t just be off to the races for more powerful AI models, it will be a revolution that changes everything.