{"id":13680,"date":"2026-10-21T08:59:57","date_gmt":"2026-10-20T23:59:57","guid":{"rendered":"https:\/\/news.skhynix.com\/en\/?p=13680"},"modified":"2026-10-21T08:59:57","modified_gmt":"2026-10-20T23:59:57","slug":"ai-ecosystem-series-ep4","status":"publish","type":"post","link":"https:\/\/news.skhynix.com\/en\/ai-ecosystem-series-ep4\/","title":{"rendered":"[AI Ecosystem] The semiconductor paradigm shifts toward memory"},"content":{"rendered":"
\n

As AI workloads evolve toward inference and agentic AI, the criteria for system performance are also changing. Computing power alone is no longer enough. Where data is stored, how quickly it moves, and how efficiently it is processed now have a direct impact on the performance and efficiency of AI systems.<\/span><\/p>\n

<\/div>\n

This article series explores the transformation of the AI ecosystem, where software, data center infrastructure, semiconductors, and memory technologies work together. It also examines why memory semiconductors have become a foundational technology for the
\nAI era and outlines SK hynix\u2019s vision for building the next-generation AI memory ecosystem.<\/span><\/p>\n

<\/div>\n

Part 4 features Raja Koduri, CEO of OXMIQ Labs, who has led GPU and AI accelerator architecture development at Apple, AMD, and Intel to explore why the focus of AI semiconductor design is shifting toward memory.<\/span><\/p>\n

<\/div>\n

\u2460 The paradigm shift in AI computing<\/span>
\n\u2461 The real bottleneck: Data, not compute<\/span>
\n\u2462 Redesigning infrastructure: Why architecture determines performance<\/span>
\n\u2463 The semiconductor paradigm shifts toward memory \u2014 Raja Koduri, founder and CEO of OXMIQ Labs<\/span><\/strong>
\n\u2464 Completing AI: Physical intelligence and the role of memory<\/span><\/p>\n<\/div>\n

<\/div>\n
\n
\n
<\/div>\n

The era of the data supply bottleneck<\/h3>\n
<\/div>\n<\/div>\n<\/div>\n

The key determinant of AI infrastructure performance is shifting from the capabilities of individual compute devices to data flow across the entire system. Performance and efficiency increasingly depend on how GPUs, memory, networks, and storage work together. This shift is also changing the fundamental questions that semiconductor designers need to address. To supply data faster and reduce unnecessary back-and-forth movement, the architecture of chips, packages, and memory itself must evolve.<\/p>\n

The center of gravity in chip design is shifting toward memory. This is not a temporary technology trend, but rather a structural transition driven by physical limitations. Following Moore\u2019s Law*<\/span>, the semiconductor industry has continued to integrate more compute circuitry onto a single die, and today\u2019s AI accelerators have reached petaflops*<\/span>-level performance. Yet the bandwidth required to supply data to these compute devices has not kept pace.<\/p>\n

* <\/span>Moore\u2019s Law: An observation proposed by Gordon Moore in 1965 that the number of components integrated into an integrated circuit has historically increased at regular intervals. It is one of the semiconductor industry\u2019s best-known empirical principles
\n* <\/span>Petaflops: A unit of computing performance equivalent to 10\u00b9\u2075, or one quadrillion, floating-point operations per second. It is commonly used to describe the performance of AI accelerators and supercomputers<\/div>\n

Computation itself has become relatively inexpensive. Multiplication and addition operations require only a few femtojoules*<\/span> of energy. Reading data from on-chip SRAM*<\/span>, however, consumes considerably more energy, while retrieving data from off-chip DRAM*<\/span> can cost hundreds or even thousands of times more. The farther data must travel from the compute device, the greater the time and energy costs.<\/p>\n

* <\/span>Femtojoule(fJ): A unit of energy equal to one quadrillionth(10\u207b\u00b9\u2075) of a joule. In semiconductors, it is used to describe the very small amount of energy consumed by individual computations or data accesses
\n* <\/span>On-chip SRAM, off-chip DRAM: On-chip SRAM is high-speed memory located on the same chip as the compute device, while off-chip DRAM is located outside the chip. In general, data access latency and energy costs increase as memory is located farther from the compute device<\/div>\n

As the integration density of compute units on a chip has grown significantly, the performance bottleneck has gradually shifted toward data supply. To keep compute devices fully utilized, the required data must arrive on time. No matter how much compute circuitry is integrated, its full performance cannot be utilized if the data arrives too late.<\/p>\n

The same challenge applies to memory capacity. The on-chip memory hierarchy, from registers to shared caches, is fast but limited in data capacity. Today\u2019s AI models, by contrast, handle far larger volumes of parameters and intermediate data. As a result, data must be brought in from HBM or DDR, displaced after use, and retrieved again when needed. This repeated streaming adds to both the time and energy burden on the system.<\/p>\n

AI semiconductor competition is therefore shifting from simply increasing compute speed toward reliably supplying enough data to keep available compute resources continuously utilized. In large-scale training and large-batch inference, compute resources may appear to be the bottleneck, while in low-latency, small-batch inference, memory bandwidth can limit performance. Even with the same silicon, the location of the bottleneck changes depending on the workload. This is why memory is increasingly becoming a starting point for AI architecture design.<\/p>\n

<\/div>\n
\n
\n
<\/div>\n

Transformers and the changing memory hierarchy<\/h3>\n
<\/div>\n<\/div>\n<\/div>\n

Computer architectures have long relied on caches to reduce this burden. Caching is based on the assumption that data will be reused. Once data is brought closer to the compute device, it can be accessed multiple times, spreading out the cost of retrieving it in the first place across multiple operations. This assumption worked well for conventional computing workloads. Even relatively small amounts of on-chip SRAM could substantially reduce DRAM accesses, making caching one of the most effective ways to improve computer performance for decades.<\/p>\n

However, transformers*<\/span>, which now dominate today\u2019s mainstream AI models, have challenged this assumption because they process vast numbers of parameters and long contexts. As model parameters have grown significantly, it has become difficult to keep all weights on-chip. During inference, each weight may be used only a limited number of times before new data is required. This reduces data reuse, making it more difficult for caches to sufficiently offset the cost of initially retrieving the data. As a result, systems repeatedly read data from DRAM or HBM.<\/p>\n

* <\/span>Transformer: An AI model architecture that processes relationships among elements of input data using attention mechanisms. It serves as the foundation of modern large language models(LLM)<\/div>\n

The burden grows as context length increases. The KV cache*<\/span> expands in proportion to context length, while attention*<\/span> operations increase both computation and memory accesses. Agents handling long contexts must also retrieve state information at each stage of a task. When large numbers of these agents operate simultaneously, conventional memory architectures centered on DDR or GDDR can struggle to handle the resulting data flow.<\/p>\n

* <\/span>Key-value cache(KV cache): A memory area where an LLM stores the results of attention calculations for previous tokens and reuses them when generating subsequent tokens. As context length increases, the KV cache also grows, increasing memory capacity and bandwidth requirements
\n* <\/span>Attention: A mechanism that calculates which information within the input context an AI model should focus on. It is a core component of the transformer architecture, with computation and memory access requirements increasing as context length grows<\/div>\n

More recent optimization techniques such as FlashAttention*<\/span>, PagedAttention*<\/span>, and speculative decoding*<\/span> have also emerged to alleviate this bottleneck*<\/span>. While their approaches differ, they share the same objective: reducing memory accesses and data movement. These techniques reduce the amount of data transferred between the compute die and DRAM, easing the memory burden during inference. Even with more efficient memory access, however, the physical cost of moving data cannot be eliminated entirely. The direction of semiconductor design is therefore clear: Data needs to be placed closer to the transistors where computation takes place. As transformers become more widely adopted, architectures that reduce the physical distance between data and computation will become increasingly important.<\/p>\n

* <\/span>FlashAttention: An optimization technique that improves processing efficiency by reducing memory accesses during attention operations. It is used to alleviate memory bottlenecks in LLMs handling long contexts
\n* <\/span>PagedAttention: An attention optimization technique that improves memory utilization by managing the KV cache in non-contiguous pages
\n* <\/span>Speculative decoding: A technique for accelerating LLM inference in which a smaller model first generates candidate tokens that are then verified by a larger model
\n* <\/span>Tri Dao et al., \u201cFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,\u201d NeurIPS 2022; Woosuk Kwon et al., \u201cEfficient Memory Management for Large Language Model Serving with PagedAttention,\u201d SOSP 2023; Yaniv Leviathan, Matan Kalman, and Yossi Matias, \u201cFast Inference from Transformers via Speculative Decoding,\u201d ICML 2023<\/div>\n
<\/div>\n
\n
\n
<\/div>\n

HBM and data supply architecture<\/h3>\n
<\/div>\n<\/div>\n<\/div>\n

High Bandwidth Memory(HBM) emerged as the first major architectural solution to this challenge and has become a key memory layer supporting the performance of today\u2019s AI accelerators.<\/p>\n

HBM uses an architecture in which multiple DRAM dies are vertically stacked and connected to the compute device through through-silicon vias*<\/span>\u00a0and an interposer*<\/span>. This significantly reduces the distance between compute and memory, from centimeters to millimeters. As a result, bandwidth increases substantially while the energy required for data transfer decreases. This architecture has also played an important role in the dramatic advances in AI accelerators over the past decade.<\/p>\n

* <\/span>Through-silicon via(TSV): A microscopic vertical electrical connection that links stacked semiconductor dies. In HBM, TSVs are a key interconnect technology that enables data movement between multiple DRAM layers
\n* <\/span>Interposer: A substrate-like structure that connects multiple semiconductor chips, such as GPUs and HBM, within a single package. It is used to create shorter, wider data paths between chips<\/div>\n

\"\"
\n<\/span><\/p>\n

These design challenges have repeatedly emerged in real-world product development. At AMD in 2015, the author <\/span>was involved in<\/span> the development of an HBM-based GPU. Later, at Intel<\/span>*<\/span>, he led the development of \u201cPonte Vecchio,\u201d which integrated 47 chiplets<\/span>*<\/span> and four HBM2e stacks into a single package<\/span>*<\/span>.<\/span>\u00a0These projects demonstrated that factors such as signal integrity, power delivery, thermal management, and mechanical stability are critical determinants of actual performance and overall product quality.<\/p>\n

* <\/span>Chiplet: A design approach that divides a large chip into multiple smaller functional dies and connects them within a package. It can improve yield, scalability, and design flexibility
\n* <\/span>Intel official materials on Ponte Vecchio architecture<\/div>\n

HBM has addressed much of the data supply challenge facing AI accelerators. The challenges ahead, however, are more complex. HBM bandwidth is constrained by a combination of factors, including the interface area available around the compute die, signal transmission speed, power delivery, and thermal design. Constraints also arise within the stack itself. As multiple DRAM dies are stacked and signals travel through TSVs and interconnects, longer and more complex signal paths can increase signal loss and interference. This also affects power consumption and heat generation. As HBM advances from one generation to the next, simply adding more bandwidth has become an insufficient means to sustain performance gains.<\/p>\n

For this reason, the industry\u2019s focus is expanding from memory itself to the system as a whole. The scope of HBM design is also broadening to encompass the base die*<\/span>, package, interconnects, thermal design, and the characteristics of actual workloads. Performance increasingly depends on how closely and reliably memory and compute devices can be connected within a single system.<\/p>\n

* <\/span>Base die: A die located at the bottom of an HBM stack that provides connectivity and control between the memory dies and external logic or the package<\/div>\n
<\/div>\n
\n
\n
<\/div>\n

Advanced packaging and memory integration<\/h3>\n
<\/div>\n<\/div>\n<\/div>\n

As memory becomes increasingly central to system performance, the role of packaging is also changing. Packaging is no longer simply a back-end assembly process. It is now an integral part of the system architecture, involving the co-design of compute dies, memory stacks, interconnects, and power delivery structures, as well as thermal paths that dissipate generated heat. How these elements are arranged and connected directly affects bandwidth, latency, energy efficiency, and reliability.<\/p>\n

2.5D and 3D integration, chiplets, interposers, and hybrid bonding*<\/span> are all approaches designed to reduce the physical distance between data and computation. Simply placing multiple functional blocks within a single package is not enough. Performance and efficiency depend on how closely and reliably those blocks are connected and how well the interconnect architecture aligns with the data flow of actual workloads.<\/p>\n

* <\/span>Hybrid bonding: A packaging technology that directly bonds copper-to-copper connections instead of using conventional microbumps, enabling denser die-to-die connectivity. It is used to shorten data movement paths while providing wider data paths and greater power efficiency within the same area<\/div>\n

In large-scale chip projects, seemingly minor details can often escalate into performance degradation or reliability issues. PCB resonance*<\/span> can destabilize signal and power delivery at specific frequencies, while microbump fatigue*<\/span> can reduce interconnect reliability under repeated thermal and mechanical stress. Thermal interactions between different dies can also create localized hot spots, potentially degrading performance or shortening product lifespan. The effectiveness of memory-centric design therefore depends on how precisely these granular issues are identified and addressed during the design phase.<\/p>\n

* <\/span>Printed circuit board(PCB) resonance: A phenomenon in which the interconnect and power delivery structures of a printed circuit board can destabilize signals at specific frequencies. In semiconductor systems with high-speed signaling, it can affect both performance and stability
\n* <\/span>Microbump fatigue: The accumulation of damage in microbumps, the fine interconnects between semiconductor dies, caused by repeated temperature changes and mechanical stress. It can affect interconnect reliability and lifespan<\/div>\n

\"\"<\/span><\/p>\n

Hybrid bonding is a technology that enables shorter, denser connections between dies. Replacing conventional microbumps with fine-pitch direct copper-to-copper bonding allows more connections to be formed within the same area. This reduces the electrical overhead associated with contacts and interconnects along the signal path while providing wider data paths within the same area. While microbumps can become a bottleneck in high-speed signal transmission, hybrid bonding connects dies more closely and densely, reducing energy consumption per bit. As a result, highly integrated 3D architectures that more tightly combine logic and memory are emerging as a viable alternative.<\/p>\n

At the same time, the boundaries between chips, packages, and systems are becoming increasingly blurred. In the past, GPU compute performance and HBM bandwidth were key metrics for assessing system competitiveness. Today, system designers must also consider how to allocate compute, memory, and interconnect resources within a limited footprint. Some workloads benefit from greater memory capacity, while others require shorter, wider interconnect paths that can move more data within the same area. AI semiconductor design is therefore moving away from assembling general-purpose components toward integrated architectures tailored to the characteristics of actual workloads.<\/p>\n

<\/div>\n
\n
\n
<\/div>\n

PIM and near-data processing<\/h3>\n
<\/div>\n<\/div>\n<\/div>\n

The technologies discussed so far are converging on a single direction: bringing data closer to compute devices. Larger caches, HBM, and advanced packaging technologies such as hybrid bonding all aim to reduce the physical distance between data and computation. Taking this a step further leads to another strategy for reducing data movement: Instead of moving data to the compute device, some computation is performed directly where the data resides.<\/p>\n

Processing-in-memory*<\/span> is an extension of this approach. In conventional computing architectures, data is brought to the compute device for processing. PIM, by contrast, moves some computation to where the data resides. By performing certain operations within DRAM, on an HBM base die, or near other memory technologies, PIM can reduce unnecessary back-and-forth data transfers.<\/p>\n

* <\/span>Processing-in-memory(PIM): A technology that performs some computation within or close to memory instead of moving data to a compute device. It can improve performance and power efficiency in AI workloads where data movement costs are high<\/div>\n

Rather than replacing CPUs or GPUs, PIM is well suited to complementing existing accelerators for workloads where the cost of data movement exceeds the cost of computation. For example, when performing repetitive, relatively simple operations on large tensors*<\/span> containing vast amounts of data, the cost of moving that data over long distances within the memory hierarchy can have a greater impact on overall system efficiency than the computation itself. For such workloads, performing computation within or near memory can reduce energy consumption by 10 to 100 times, depending on the workload*<\/span>. The primary advantage of PIM lies less in increasing compute performance itself but in reducing data movement.<\/p>\n

* <\/span>Tensor: A multidimensional array used to represent data processed by AI models, including images, model weights, and intermediate computation results. AI computation involves repeatedly processing these large data arrays
\n* <\/span>Raja Koduri, \u201cThe Shift Toward Memory,\u201d shared for SK hynix AI Ecosystem series, 2026<\/div>\n

This shift is elevating the memory hierarchy to the starting point of system design. System efficiency can vary significantly depending on data placement, which operations are performed where the data resides, and when to transfer data to a central accelerator. Future memory-centric architectures*<\/span> are likely to move closer to a model in which computation is brought to the data, rather than data being moved to the compute device.<\/p>\n

* <\/span>Memory-centric architecture: An approach that moves beyond architectures centered on transferring data to compute devices, instead co-designing compute, packaging, and interconnects around where data resides and how the memory hierarchy is organized<\/div>\n