Without him, Google would not do as well. Amin Vahdat oversees the global infrastructure that powers Search, YouTube, Google Cloud but especially Gemini. On the occasion of the Raise Summit, he tells JDN the other side of the AI race.
JDN. DeepMind designs Google’s AI models. Your team builds the infrastructure that allows them to be trained and run. How do you work together?
Amin Vahdat. We work hand in hand with DeepMind. This is one of the most interesting aspects of my job: doing vertical co-design between the teams that design the models and those that build the hardware and software infrastructure necessary to train them, serve them and, now, run agents. When a subject is critical, we can bring together five, ten or twenty people in the same room, regardless of their original hierarchy. They work side by side, screen to screen, around a whiteboard, define an objective, designate someone responsible and move forward until they find a solution.
When you design infrastructure for AI, are you first looking to make the models faster or cheaper to run?
The answer, unfortunately, is: both. I wish I could choose just one lens, because that would make our job a lot simpler. But we can’t afford it. The quality of the model is absolutely critical: it must produce the best possible answer. But if this model consumes all our resources, it becomes unusable on a large scale. Even if you have the best model in the world, if you can only provide one answer per day, it doesn’t work. So the challenge is to find the right balance between quality, performance and efficiency. It’s a constant compromise. We try to identify the best point on this curve, the one that provides the best possible experience, while using resources efficiently.
When you train a model like Gemini, what is the real technical wall today: the computing power, the memory, the network or the way in which the model is designed?
“At DeepMind, teams have hundreds, sometimes thousands, of ideas for improving models.”
It’s all of this at once. The architecture of the model is obviously decisive. At DeepMind, teams have hundreds, sometimes thousands, of ideas for improving models. But they do not directly test them on a very large scale. They first make them compete on a small scale, a bit like in a tournament. The best ideas then move on to the next level, and then the next one again. In the end, only the strongest can be integrated into a future version of Gemini. It is a very rigorous scientific process, based on successive hypothesis tests. When we then launch training on tens or hundreds of thousands of chips, we already have strong confidence in the choices made.
But you also have to think about how this model will deploy on hardware. A large model does not fit on a single chip. It is therefore necessary to cut it up, distribute it between the chips, decide how much memory to place on each circuit, how much calculation to allocate to each chip, what network bandwidth to plan for and how to manage the exchanges between them. The big question is also latency. Can we continue to do useful calculations locally while a chip waits for a result from another part of the system, sometimes at the other end of a cluster of 100,000 chips? This is called latency hiding. All these decisions, model architecture, memory, calculation, network, latency, are linked. And it is their balance that determines real performance on a very large scale.
With AI agents, a single query can trigger hundreds of calls to a model. Is inference becoming as complex as training?
Yes, we are clearly there. With agents, it’s no longer just a matter of calling a model and getting back a response. A single task can require hundreds or even thousands of steps: calling a model, parsing its response, retrieving data from storage, putting it back into context, and then restarting a new series of operations. At this point, we are no longer just coordinating TPUs or GPUs. We also need to coordinate CPUs, the data center network and storage. And if just one element becomes a bottleneck, the entire agent response slows down. These slowdowns accumulate very quickly and can propagate throughout the system.
This is also what makes optimization very difficult. There can be 2x, 3x, or 4x performance gains simply by identifying the right bottleneck and removing it. But there is no perfect solution, because there is no one type of workload. We must therefore design algorithms capable of handling a very wide variety of situations, while maintaining the best possible level of efficiency.
Added to this is the question of energy. In some countries, electricity demand peaks only a few weeks a year, for example during the hottest or coldest periods. We work with electricity suppliers on demand response mechanisms: they warn us in advance, and we temporarily reduce our consumption in certain data centers. This further complicates the scheduling problem. It is no longer enough to know what computing resources are available. You also need to know how much energy is available at a particular location, and distribute workloads accordingly. For agents, this means that the infrastructure must take into account compute, network, storage and available energy.
Does this mean that energy is now becoming one of the main obstacles to the development of AI?
Yes, it is both one of our greatest challenges and one of our greatest opportunities. The electricity network still has significant capacity, but we need to think about its use differently. Historically, data centers have been designed to achieve extremely high levels of reliability. For this, we built a lot of redundancy: two power supplies each capable of providing full power, with very fast switching systems. It was expensive, but acceptable, because the energy was less critical and cheaper.
“We are now making performance per watt a central metric”
Today the situation is changing. If we accept a slightly lower level of reliability for some workloads, we can sometimes access much more power, faster and more efficiently. This is a very important decision. The electricity network is sized for the few weeks of highest demand in the year. If, during these periods, we can reduce our consumption, or even reinject locally produced energy, we are helping the network instead of simply constraining it. This is why we now make performance per watt a central metric. Not just performance per dollar or raw performance, but performance delivered for each watt consumed. When you optimize with this constraint, you don’t design hardware and software in the same way.
As generative models get bigger and bigger, could inference end up costing more than training?
Looking at history, I think the answer is yes. In the early days of the Internet and web search, the majority of resources, computing, storage, network, were used to build the index. Responding to user queries accounted for a smaller share of capacity. Over time, this was reversed. Today, we no longer really talk about building a new web index on an ad hoc basis: users expect almost real-time access to the entire Internet. User service has therefore become the dominant part of the capacity.
I think we are at a similar inflection point with AI. The training remains enormous and will continue to be so. But inference takes off very quickly, to the point where it approaches a 50/50 balance with training. Ultimately, if the objective is to deliver value to users based on models, then inference must logically become the dominant part of the investment.
The adoption of AI is progressing very heterogeneously. How do you anticipate the needs for chips and data centers, particularly for markets like France or Europe?
This is an extremely difficult problem. Capacity planning is a very important part of my team’s work. The difficulty is that we are facing exponential growth. And predicting an exponential accurately is almost impossible. In more mature markets, like traditional computing before the arrival of agents, demand was easier to anticipate. If it grew by 30% per year, you could plan for 35% or 40% to keep a margin.
“A customer can go from a few TPUs or GPUs to a demand for 10,000 chips very quickly”
With AI, it’s very different. A customer can go from a few TPUs or GPUs to a demand for 10,000 chips very quickly. So we have to constantly plan and replan. When a customer, internal or external, requests more capacity from us, we first seek to extract more efficiency from the existing infrastructure. Then, we work with our suppliers and partners to accelerate the arrival of new capacities: can what was supposed to happen in three months happen in two months, or even in one month? It is a continuous process, 24 hours a day, 7 days a week: forecasting, replanning, optimizing existing assets, finding new capacities and working with the entire supply chain.
Jeff Bezos wants to send data centers into space, particularly for AI training. Is this a credible lead or still science fiction?
“At Google, we are exploring space with the Suncatcher project”
This is a very interesting direction. At Google, we are also exploring this avenue with a project called Suncatcher. I think this is one of the most exciting, but also one of the most challenging, paths for the future of compute infrastructure. Space would make it possible to respond to some of the energy challenges. Solar energy is much more efficient: there is no cloud cover, less attenuation linked to the atmosphere, and certain orbits allow almost continuous exposure to the sun. The panels can therefore produce much more efficiently than on Earth.
But the challenges are considerable. Cooling, for example, may seem simpler in space, but in reality it is extremely complex. Reliability is also a major subject: today, these infrastructures require maintenance. In space, we must therefore completely rethink redundancy and repair. Finally, there is the question of networking. We can’t just send a huge data center into orbit. It would probably be necessary to divide the infrastructure into several modules and make them communicate with each other, for example with optical links in free space. All these problems are technically affordable, but they are still very difficult. It’s a major gamble. And it’s clearly a bet that interests us.
