From AI Innovation to Adoption: The Hardware Technologies to Watch

The AI hardware race is about to get a lot more interesting. For years, the industry has focused on building faster, more powerful chips to keep up with increasingly demanding AI models. But what if the next breakthrough isn’t a faster processor at all? What if it comes from rethinking how computers handle memory, move data or even where AI processing happens? From chips that use light to move information to processors that bring computing closer to memory, a new wave of architectures is challenging the way we’ve traditionally thought about AI infrastructure.

Members of the Senior Executive AI Think Tank, a curated group of experts specializing in machine learning, generative AI and enterprise AI applications, weigh in on the emerging hardware architectures they believe could have the greatest impact on AI computing and what will ultimately separate transformative technologies from those that struggle to gain commercial traction.

Rethink Memory: The Case for Composable Architectures

As AI inference becomes increasingly memory-intensive, simply adding more processing power may not be the most effective way to improve performance. Pawan Anand, Portfolio Head of Communications, Media and Entertainment at LTM, points to the growing demands of context windows, multimodal workloads and agent memory as reasons conventional server architectures may become less efficient.

“I would watch composable memory fabrics as closely as the next accelerator,” he says. “AI inference is becoming increasingly memory-bound.”

Technologies such as Compute Express Link (CXL) could allow processors and accelerators to share large memory pools rather than trapping available capacity inside individual servers. This could help organizations scale memory and computing resources independently, reducing wasted infrastructure spending.

However, Anand cautions that performance improvements alone will not establish commercial viability.

“Commercial adoption will depend less on peak benchmark gains and more on latency, software support, reliability and whether the architecture lowers cost per useful token at rack scale,” he says.

“A chip that wins a benchmark but requires customers to rebuild their software stack faces a steep commercial hurdle.”

– Dr. Aditya Vikram Kashyap, Vice President of Firmwide Innovation at Morgan Stanley

SHARE IT

Bring Computing Closer to Data With Advanced Chip Architectures

Dr. Aditya Vikram Kashyap, Vice President of Firmwide Innovation at Morgan Stanley, sees three technologies with particularly strong potential: 3D chiplet architectures, silicon photonics and in-memory computing.

“They target a critical constraint,” Kashyap says, “the energy and time spent moving data between memory and processors.”

Three-dimensional integration can bring computing components closer together, while silicon photonics uses light to transmit information between chips. In-memory computing takes the concept further by performing certain operations within or near memory itself.

These approaches are not equally mature. Kashyap sees optics and advanced packaging as closer to commercial scale, while analog computing still faces challenges involving noise and precision.

The commercial evaluation, he emphasizes, must consider the entire system.

“Commercial adoption will depend on cost per useful token at matched accuracy and latency, measured across the entire system, including cooling, networking and software migration.”

His advice for enterprise buyers is to look beyond isolated processor specifications. Manufacturing yields, reliability, supply capacity and compatibility with existing software environments can determine whether a promising architecture becomes a practical investment.

As he notes, “A chip that wins a benchmark but requires customers to rebuild their software stack faces a steep commercial hurdle.”

Treat Talent as a Technology Adoption Strategy

Divya Parekh, Founder of executive coaching brand DivyaParekh.com, offers a perspective that technology procurement teams sometimes overlook: Hardware adoption is also a workforce challenge.

“Every new chip architecture eventually runs into a staffing question,” Parekh says. “Who in the company knows how to program it, and how long will it take to hire the next three people who can?”

The commercial potential of emerging technologies such as photonic and in-memory computing depends not only on engineering breakthroughs but also on the development of accessible programming environments and a qualified talent pool.

“Today’s dominant platforms owe much of their lead to years of graduate students and engineers who learned on them,” she says. “That familiarity is hard to displace, even with better silicon.”

Monitor early workforce signals, including university courses, open-source development and job postings that specifically reference emerging architectures.

For technology leaders, these indicators can help determine whether an innovation is developing the ecosystem needed for sustainable adoption.

Invest in Hardware That Can Adapt as AI Evolves

Andre Shojaie, Founder of HumanLearn, says the challenge is a growing mismatch between hardware development cycles and the pace of AI innovation. Organizations often invest in infrastructure designed for workloads they understand today, even though tomorrow’s models may have very different requirements.

“I believe one of the most interesting shifts in AI hardware will be the move toward architectures that can evolve with the intelligence they support,” Shojaie says.

He sees reconfigurable computing as a potential solution, allowing systems to adapt as AI models and workloads change rather than relying on fixed hardware configurations.

“Today, we design chips for workloads we can anticipate,” he says. “But AI capabilities, behaviours and demands are changing faster than hardware development cycles.”

For companies making substantial infrastructure investments, adaptability has direct financial implications. Hardware that becomes obsolete as model architectures evolve can create expensive replacement cycles and limit the return on capital.

Shojaie stresses that flexibility must still deliver measurable business value.

“Commercial adoption will depend on whether this flexibility can deliver measurable value without sacrificing reliability, energy efficiency or cost.”

Enterprise leaders should therefore evaluate new infrastructure not only on its current performance but also on its ability to accommodate changing models, workloads and deployment environments.

“The biggest shift won’t be one exotic chip; it’s the model moving onto the device. ”

– Matan Mishan, Senior Vice President, Agentic at Dot Compliance, Inc.

SHARE IT

Move AI Onto Devices and Make Intelligence Ubiquitous

Matan Mishan, Senior Vice President, Agentic at Dot Compliance, Inc., believes the most consequential hardware shift may be the movement of AI models from centralized infrastructure onto individual devices.

“The biggest shift won’t be one exotic chip; it’s the model moving onto the device,” Mishan says. “IoT put connectivity into everything; once open-weight models compete locally, we get AIoT.”

Neural processing units (NPUs) and specialized edge silicon will help make this transition possible by enabling AI inference directly on devices, reducing dependence on remote computing resources.

The business case could be significant. Gartner predicts that small, task-specific AI models will be used at least three times more frequently than general-purpose LLMs by 2027.

“Benchmarks won’t decide adoption,” he says. “Energy per inference, tooling that makes shipping a model to a million devices as routine as shipping firmware, and trust will.”

He also highlights a security consideration: An OTA channel that updates AI models introduces another potential attack surface.

For enterprise leaders, the lesson is to assess edge AI as a complete operational strategy, incorporating device management, model updates, cybersecurity and lifecycle support.

Make Local AI Economical for Everyday Business

Nikhil Jathar, CTO of AvanSaber Inc., points to Apple’s M5 architecture as an example of the direction worth watching, combining unified memory with neural accelerators in each GPU core.

“I would watch architectures that bring memory and AI processing closer together,” Jathar says.

The broader opportunity is making local AI more practical, allowing organizations to run useful workloads without depending entirely on expensive centralized infrastructure.

However, Jathar emphasizes that performance must be evaluated against actual business requirements rather than theoretical capabilities.

“Commercial adoption will depend on whether a customer’s actual model fits, responds quickly and runs affordably through a full working day.”

He recommends testing workloads that reflect real operating conditions, including long documents and multiple concurrent users. Organizations should also account for software migration and ongoing support when calculating total costs.

For midsize businesses, this is particularly important. A technically impressive system may still be impractical if it requires a specialized infrastructure team to maintain.

As Jathar explains, “For a midsize business, the winning chip is the one that makes a useful workflow affordable without requiring a specialist infrastructure team.”

Democratize AI Through Hybrid Computing

Lynn Comp, Head of AI Center of Excellence at Intel, sees the future of AI computing through the lens of an earlier technology transformation: the democratization of computing.

She compares today’s large-scale AI infrastructure deployments with the mainframe era, when computing initially required substantial, centralized systems.

“A great deal of extrapolating and exploring has occurred with big, large-scale hardware systems,” Comp says. “I look at this like the mainframe rollout.”

In her view, computing became transformative for broader populations when miniaturization and wider accessibility made the technology practical beyond specialized environments.

“When did computing truly explode for the masses? After miniaturization and tech democratization occurred,” she says.

She believes AI infrastructure could follow a similar trajectory, moving from enormous centralized systems toward a combination of smaller, lower-power devices and large-scale frontier computing.

“All technology inflections start with very bulky solutions, but achieving the vision requires hybrid solutions—from low power and small size through the largest frontier systems,” Comp says.

For enterprise leaders, this suggests that the future will not necessarily belong to one dominant hardware architecture. Instead, successful organizations will combine centralized accelerators, local processing and intermediate computing resources according to workload requirements.

“Adoption will depend on whether it is easy to use, cheap per result and works with existing systems.”

– Yogesh Malik, CEO of Way2Direct B.V.

SHARE IT

Optimize for Results, Not Computing Power

Yogesh Malik, CEO of Way2Direct B.V., believes the next significant shift will involve reducing the amount of computing power required to produce useful results.

Rather than continuing to prioritize increasingly powerful hardware, he expects greater attention to move toward efficient inference, edge computing and near-memory processing.

“The biggest shift is needing far less compute to get a result,” Malik says.

Malik believes that edge and near-memory computing will become increasingly relevant as businesses seek to reduce unnecessary data movement and dependence on centralized infrastructure.

“Adoption will depend on whether it is easy to use, cheap per result and works with existing systems.”

For enterprise buyers, the takeaway is to evaluate AI hardware against measurable business outcomes, including cost per completed task, deployment complexity and integration requirements.

Keep AI Accelerators Busy Across Changing Workloads

Vivek Kumkar, Sr. GenAI Leader at Amazon Web Services (AWS), highlights a less obvious hardware bottleneck: the work required to prepare multimedia data before AI inference begins.

For multimodal applications, video and images must be decoded, resized and transferred into memory. These processes can delay inference and leave expensive accelerators underutilized.

“Everyone is racing to speed up the math inside the model,” Kumkar says. “For multimodal AI, the bottleneck often comes earlier.”

He sees potential in architectures that integrate media decoding, preprocessing and inference on a single package, reducing unnecessary movement between separate chips.

“Benchmarks will not decide adoption. Utilization will,” he says.

Kumkar notes that a processor that performs exceptionally well under ideal conditions may still be commercially inefficient if it cannot switch between workloads quickly or maintain appropriate customer isolation.

“The chip that wins is the one that stays busy,” he says.

For enterprise technology leaders, this means evaluating accelerator utilization, workload scheduling, preprocessing efficiency and multitenant capabilities alongside conventional performance metrics.

Prioritize Hardware Longevity Over Short-Term Performance

Goran Paun, Principal and Creative Director at ArtVersion, believes adaptability will become one of the most important measures of AI hardware value.

“AI workloads are evolving faster than traditional hardware cycles,” Paun says. “That makes architectures that can remain useful across changing models, inference patterns and deployment environments especially interesting.”

He sees potential in reconfigurable systems, chiplets and tighter memory integration because they can give organizations greater flexibility as technical requirements change.

But Paun believes the most important commercial consideration is not necessarily raw processing speed. It is the ability to protect infrastructure investments over time.

“The commercial test will be longevity as much as raw performance,” he says.

Companies investing heavily in AI infrastructure need confidence that today’s hardware advantages will remain relevant as models become smaller, more specialized or increasingly deployed at the edge.

Paun cautions against architectures that require organizations to repeatedly rebuild surrounding systems as AI technology changes.

“The hardware that wins may be the hardware that adapts without requiring the surrounding software, infrastructure and operating model to be rebuilt every time AI changes direction.”

10 Tips for Navigating Emerging AI Hardware

• Evaluate memory efficiency alongside processing power. Explore composable memory, CXL and near-memory architectures to determine whether they can reduce infrastructure waste and improve cost per useful token.
• Assess the entire computing system, not individual chips. Include cooling, networking, software migration, manufacturing reliability and supply capacity when comparing emerging architectures.
• Build workforce readiness before committing to unfamiliar hardware. Monitor university curricula, open-source communities and job-market demand to determine whether the talent ecosystem is developing around a new architecture.
• Prioritize adaptability in long-term infrastructure investments. Evaluate reconfigurable computing and flexible architectures based on their ability to accommodate changing models and workloads without expensive system replacements.
• Develop an edge AI strategy. Consider local inference, NPUs and OTA model updates while accounting for device security, energy requirements and lifecycle management.
• Test hardware against actual business workflows. Use representative models, long documents, concurrent users and full-day operating requirements to assess performance and affordability.
• Plan for a hybrid computing environment. Combine low-power devices, edge infrastructure and centralized frontier systems rather than assuming one architecture will meet every requirement.
• Measure cost per completed task. Compare the economics of AI inference based on useful business outcomes rather than peak theoretical processing performance.
• Optimize utilization across multimodal workloads. Evaluate how effectively infrastructure handles data preparation, model switching, scheduling and customer isolation.
• Make infrastructure longevity a procurement priority. Favor technologies that can evolve with AI models and deployment requirements while minimizing software and operational disruption.

Preparing for the Next Era of AI Computing

AI hardware is moving into an exciting period of experimentation, and the decisions companies make over the next few years could have a lasting impact on their ability to innovate and compete. For technology leaders, that means paying attention to developments that may not yet be on their procurement radar, asking tougher questions about infrastructure costs and performance, and giving emerging technologies room to prove their value. Some architectures will take years to mature. Others may find their way into enterprise environments sooner than expected.

The challenge for business leaders is keeping pace with a field that continues to evolve while making infrastructure investments that stand the test of time. That requires a willingness to experiment, a clear understanding of business needs and an honest assessment of what new hardware can deliver. The organizations that approach this next chapter with curiosity and a long-term perspective will be in a stronger position to turn the next wave of AI computing advances into real business opportunities.