Why AI Data Centers Need a New Infrastructure Strategy

The AI industry has embraced a simple idea about data centers: If AI needs more computing power, build bigger facilities and pack them with more chips. But as the infrastructure race accelerates, that assumption is becoming harder to defend. Power availability, grid capacity, cooling, water and the physical limits of getting new facilities online are emerging as constraints that raw compute cannot solve.

The question matters because data centers are long-term investments being built around a technology that is changing at remarkable speed. The infrastructure designed for training enormous models may not be what businesses need as inference workloads grow, models become more efficient and AI moves closer to where data and users actually reside.

There are also business questions hiding underneath the engineering one: How much compute does a company actually need? Where should it run? What happens when power costs change or a new generation of hardware makes today’s architecture less attractive? And how should CEOs think about infrastructure that needs to remain useful even as the AI workloads it supports evolve?

Members of the Senior Executive AI Think Tank, a curated group of experts specializing in machine learning, generative AI and enterprise AI applications, offer a range of perspectives on what should come next. In the discussion that follows, they examine whether bigger data centers are really the answer, where power and efficiency fit into the equation, why inference deserves a different infrastructure strategy and which overlooked factors—from memory and networking to reliability and data governance—could determine whether an AI investment delivers lasting business value.

Reuse Can Beat New Construction

Lynn Comp, Head of AI Center of Excellence at Intel, approaches the issue from the intersection of technology strategy and business value. She challenges the assumption that every AI workload requires a brand-new, larger facility.

“If the data center is not meant to be used mostly for AI training, I find the assumption that ‘bigger is better’ and ‘new construction is the only way forward’ to be impractical,” Comp says.

For inference workloads, she says executives should instead balance location and latency against available power and opportunities to reuse existing facilities.

“The better approach for inferencing balances location and latency, power availability and practical reuse of already constructed facilities that can deploy resources in a more timely manner,” she says.

That argument matters because energy infrastructure often moves more slowly than technology.

“Building new buildings does not result in the fastest availability of compute capacity,” she adds.

Trade Scale for Efficiency and Sustainability

Will Conaway, President of Tuxedo Cat Consulting, sees raw scale as an incomplete measure of AI competitiveness.

“A key misconception in the AI industry is the belief that simply scaling up data centers with more powerful hardware and higher energy consumption will guarantee superior AI performance and competitiveness,” Conaway says.

Instead, he argues that energy efficiency, sustainability and cooling should become core infrastructure decisions.

“If I were a CEO, I would invest in decentralized, modular data centers powered by renewable energy sources, and prioritize advanced cooling technologies to reduce environmental impact,” he says.

Microsoft reports that newer direct-to-chip cooling designs can avoid more than 125 million liters of water per facility annually, illustrating how facility design can materially change the economics and environmental footprint of AI infrastructure.

“Emphasizing efficiency and sustainability, rather than raw scale,” Conaway says, “would not only lower operational costs but also future-proof infrastructure against regulatory and environmental challenges.”

“AI architectures will evolve faster than data centers. Flexibility, energy efficiency and utilization will ultimately matter more than sheer scale.”

– Dileep Rai, Manager of Oracle Cloud Technology at Hachette Book Group (HBG)

SHARE IT

Optimize the System, Not the Chip

Dileep Rai, Manager of Oracle Cloud Technology at Hachette Book Group (HBG), argues that compute density is becoming an increasingly poor proxy for infrastructure quality.

“The assumption I would challenge is that AI infrastructure should be optimized primarily for maximum compute density,” Rai says.

He adds that the real constraint is “the system around the chips: power availability, cooling, grid capacity, water, network latency and utilization.”

That shifts the executive metric from hardware acquisition to useful output. Rai would optimize for “cost and intelligence delivered per watt,” while combining cloud, edge infrastructure, specialized accelerators and geographically distributed capacity.

“I would also avoid overbuilding for today’s models,” he says. “AI architectures will evolve faster than data centers. Flexibility, energy efficiency and utilization will ultimately matter more than sheer scale.”

If the hardware and model landscape can change materially during the useful life of a building, flexibility becomes an economic asset.

Build for Inference, Not Just Training

Vivek Kumkar, Sr. GenAI Leader at Amazon Web Services (AWS), believes the industry’s fixation on giant training facilities risks overlooking where enterprise AI value will increasingly be created.

“While industry attention is fixated on gigawatt data centers for monolithic training, the true economic battleground is production inference at scale,” Kumkar says. “Training demands localized power density, but enterprise value lives in high-availability, low-latency inference serving millions of daily workloads.”

His recommendation is to separate the two architectures rather than treat training and inference as interchangeable workloads. That means investing in distributed inference fleets, domain accelerators, edge caching and dynamic model serving.

“The winning strategy is not about who has the largest power plug,” Kumkar says. “It is about delivering the lowest latency and cost per request while honoring regional data sovereignty and regulatory compliance.”

For executives, this is a reminder that infrastructure should follow workload economics—not industry headlines about the largest AI cluster.

Avoid the Five-Year Obsolescence Trap

Rodney Mason, Chief Marketing Officer at Minty, takes a skeptical view of the rush toward enormous centralized facilities.

“The industry operates as if centralized hyper-scale data centers are the answer,” Mason says. “But power availability, grid constraints and latency will increasingly determine where AI lives.”

His alternative is a more distributed model.

“A distributed, energy-flexible infrastructure that combines hype-scale capacity with regional facilities, diverse power sources, storage and workload-aware scheduling built at the pace that computing power is actually advancing, is a better path forward,” he says.

Mason’s concern is ultimately capital allocation: “Spending trillions of dollars on massive centralized facilities that will be outdated in five years is a strategic miscalculation.”

For CEOs, Mason’s lesson is simple: Don’t confuse the availability of capital with the availability of durable demand.

“I believe the real bottleneck is not compute; it’s reliability, observability and trust in the data ecosystem powering AI.”

– Gaurav Rastogi, Senior Director of Enterprise Data Analytics, Data Science and Strategic Insights at Hertz

SHARE IT

Put Reliability and Trust Ahead of More GPUs

Gaurav Rastogi, Senior Director of Enterprise Data Analytics, Data Science and Strategic Insights at Hertz, challenges another assumption: that compute is automatically the primary bottleneck.

“The AI industry assumes bigger data centers automatically create better outcomes,” Rastogi says. “I believe the real bottleneck is not compute; it’s reliability, observability and trust in the data ecosystem powering AI.”

That distinction is critical for enterprises because an expensive AI system can still produce poor business outcomes if its data is unreliable, its performance is difficult to monitor or its governance is inconsistent.

“If I were a CEO today, I would invest first in intelligent data infrastructure: end-to-end observability, automated governance, workload optimization and energy-efficient architectures,” he says.

The implication is that executives should measure infrastructure not only by throughput but by whether it produces reliable, explainable and repeatable business results.

Start With the Energy Model

Geetha Kumari Kommepalli, Founder, CTO and Chief AI Officer at Thriven Advisory, argues that executives should stop treating data center construction as primarily a technology or capital-allocation decision.

“The AI industry is making a fundamental mistake by treating data centers primarily as a compute and capital-allocation problem,” Kommepalli says. “They are equally an energy, grid-capacity, cooling, water, supply-chain and operating-model challenge.”

Her proposed decision sequence begins with the physical environment.

“If I were a CEO making infrastructure decisions today, I would not start by asking, ‘How many GPUs should we buy?’ I would first assess power availability, grid access, cooling requirements, permitting, resilience and the business value of each workload,” she says.

Her next step would be to build a flexible infrastructure portfolio “combining cloud, colocation, owned capacity, workload-specific architectures and smaller optimized models,” noting that successful leaders will be the ones aligning their AI ambitions with “reliable energy, infrastructure resilience, financial discipline and measurable business outcomes.”

Make Power a Strategic Capability

David Obasiolu, AI Security, Governance and Systems Consultant at Vliso AI, believes site selection itself needs to be rethought.

“The industry often treats compute as the primary constraint, when power, grid access, cooling and permitting are becoming equally strategic,” Obasiolu says.

He notes that if he were a CEO, he wouldn’t choose sites based mainly on land and tax incentives.

“I would prioritize power availability, grid resilience, water efficiency, modular expansion and proximity to low-cost energy,” he says. “I would also design for heterogeneous compute so every workload does not require the most expensive GPU infrastructure.”

That is a significant shift in how infrastructure investments are traditionally evaluated. A cheap site is not necessarily a cheap AI site if the facility cannot obtain reliable power or expand when demand changes.

“The winning data center strategy will optimize the entire system, not just the chips,” he says.

Solve the Energy Constraint Before Buying Chips

Kiran Palla, Chief Information Officer at CogniwareAI, is particularly direct about what he sees as the industry’s central mistake.

“The industry is wrong to assume data centers can keep scaling by simply adding more compute,” Palla says. “I’m firmly against that view.”

He believes the real problem lies elsewhere.

“The real bottleneck is power, not GPUs, and ignoring this is the biggest strategic mistake AI leaders are making,” he says.

Palla’s recommendation to CEOs begins with energy procurement rather than accelerator procurement.

“I’d rebuild my infrastructure strategy around energy first: Secure long-term generation, co-develop renewable assets, build in surplus-power regions and treat power procurement as a core competency,” he says. “AI growth will be determined by who solves the energy constraint, not who buys the most chips.”

“The industry is heavily prioritizing compute scaling; however, enough memory at all hierarchies is equally, if not more, important for effective throughput.”

– Piyush Lakhawat, Senior Member of Technical Staff at Salesforce

SHARE IT

Don’t Forget Memory and the Network

Piyush Lakhawat, Senior Member of Technical Staff at Salesforce, argues that compute-centric planning can miss other bottlenecks that emerge as workloads change.

“The industry is heavily prioritizing compute scaling,” Lakhawat says. “However, enough memory at all hierarchies is equally, if not more, important for effective throughput.”

He also emphasizes the infrastructure between compute resources.

“Architecture flexibility, network latency and network bandwidth at all connections is very important as well,” he says.

The key executive insight is that infrastructure performance is dynamic. A configuration that works for today’s traffic distribution can become inefficient as workloads evolve, models grow or application architectures change.

“Any of these can be a bottleneck if the input traffic flow changes its distribution, or any of the compute infrastructure evolves,” Lakhawat says. “And both of these are a question of when, not if.”

That makes flexibility less of a technical preference and more of a capital-preservation strategy.

“If I were making an AI infrastructure decision, I would keep these in mind and build my systems so they are flexible and robust for different future scenarios.”

Actionable Strategies for AI Infrastructure Leaders

• Consider existing facilities before building new ones. For inference workloads, location, latency and available power can make reuse a faster and more practical path to capacity than new construction.
• Prioritize efficiency and sustainability alongside performance. Modular facilities, renewable energy and advanced cooling can reduce operating costs and help protect infrastructure investments against future environmental and regulatory pressures.
• Optimize for useful intelligence, not GPU count. Evaluate infrastructure based on cost and intelligence delivered per watt, while keeping architectures flexible enough to adapt as AI models evolve.
• Build inference infrastructure for the workloads it actually serves. Separate training and inference strategies, with distributed capacity, specialized accelerators, edge caching and dynamic model serving where they make economic sense.
• Be skeptical of infrastructure built around today’s AI boom. Balance hype-scale capacity with regional facilities, diverse power sources, storage and workload-aware scheduling to reduce the risk of expensive infrastructure becoming obsolete.
• Invest in reliability and observability before simply adding compute. Strong data infrastructure, automated governance and workload optimization can deliver more business value than additional GPUs when reliability and trust are the real constraints.
• Treat infrastructure as an energy and operating-model decision. Assess power availability, grid access, cooling, water, permitting and resilience before deciding how much compute to buy.
• Make power availability a core site-selection criterion. Prioritize grid resilience, water efficiency, modular expansion and access to low-cost energy rather than relying primarily on land costs or tax incentives.
• Build around energy constraints, not chip availability. Secure long-term generation, explore renewable power partnerships and consider regions with surplus power before committing to major AI capacity.
• Plan for memory and networking as carefully as compute. Account for memory hierarchy, network latency and bandwidth so changing workloads or infrastructure do not turn an otherwise powerful system into a bottleneck.

Focus on Flexibility, Not Just Scale

The common thread across the Think Tank’s perspectives is not an argument against large data centers or powerful hardware. It is an argument against treating scale as the strategy.

AI infrastructure is becoming an interconnected system of chips, memory, networks, cooling, power, data, software, facilities and governance. The companies that manage those dependencies as one economic system will be better positioned than those that simply accumulate compute.

For CEOs, start with the workload and the desired business outcome, then work backward to the infrastructure required to deliver it. In an environment where energy, capital and technology are all constrained, flexibility may prove more valuable than size.