{"id":12572,"date":"2026-08-31T08:59:33","date_gmt":"2026-08-30T23:59:33","guid":{"rendered":"https:\/\/news.skhynix.com\/en\/?p=12572"},"modified":"2026-08-31T08:59:33","modified_gmt":"2026-08-30T23:59:33","slug":"ai-infrastructure-insight-ep2","status":"publish","type":"post","link":"https:\/\/news.skhynix.com\/en\/ai-infrastructure-insight-ep2\/","title":{"rendered":"[AI Infrastructure Insight] Why faster GPUs alone can\u2019t deliver AI performance"},"content":{"rendered":"
\n

AI is no longer defined by a single model or chip. For AI to operate effectively in real-world services and industrial applications, it takes faster compute, greater memory bandwidth, higher-performance networking, more efficient storage, and stable power and cooling working together. This is why competition in AI is shifting beyond individual technologies to the design and operation of the entire infrastructure.<\/p>\n

<\/div>\n

In the AI Infrastructure Insight<\/strong> series, we explore how the architecture behind AI infrastructure is evolving \u2014 from compute, memory, storage, networking, and power and cooling to the way they come together as a unified system \u2014 and what this means for the industry.<\/p>\n

<\/div>\n

[Series overview]<\/strong>
\n\u2460 What is changing inside AI data centers?
\n\u2461 Why faster GPUs alone can\u2019t deliver AI performance<\/strong>
\n\u2462 Why power and cooling have become the next challenge for AI infrastructure
\n\u2463 How AI infrastructure will be designed for the future<\/p>\n<\/div>\n

Does using a faster GPU*<\/span> result in faster AI operations? GPUs and AI accelerators*<\/span> are core technologies that have enabled the training of large-scale models and the processing of complex inference requests. In real-world service environments, however, AI performance is not determined by compute speed alone.<\/p>\n

* <\/span>Graphics processing unit(GPU): A parallel processing device originally developed for graphics computation. Its ability to process large volumes of data simultaneously has led to its widespread use in AI training and inference
\n* <\/span>AI accelerator: A semiconductor or computing device designed to rapidly perform the large-scale computations required for AI training and inference. Examples include GPUs, NPUs, and TPUs<\/div>\n

Even when a processor is ready to perform calculations, an expensive accelerator can be left waiting if data is not supplied in time, reducing overall system efficiency.
\nAn AI system can therefore deliver its full performance only when accelerator compute capability, data delivery speed, and data movement paths work together effectively.<\/p>\n

The research supports this point.
\nThe paper \u201cAI and Memory Wall,\u201d*<\/span> by researchers from UC Berkeley, ICSI, and LBNL, found that over the past 20 years, peak FLOPS*<\/span> in server hardware increased by approximately threefold every two years, while DRAM bandwidth*<\/span> increased by only 1.6 times and interconnect*<\/span> bandwidth by 1.4 times.
\nWhile compute performance has advanced rapidly, the ability to supply and move data has progressed at a comparatively slower pace. As this gap widens, bottlenecks*<\/span> in memory and data movement paths \u2014 not just compute performance \u2014 may become more pronounced in AI systems.<\/p>\n

* <\/span>AI and Memory Wall: A study by researchers from UC Berkeley, ICSI, and LBNL analyzing the gap between the rate of growth in compute performance and the growth of memory and interconnect bandwidth in AI systems
\n* <\/span>Peak floating point operations per second(FLOPS): The theoretical maximum floating-point compute performance of a computing system. FLOPS refers to the number of floating-point operations that can be performed per second
\n* <\/span>Bandwidth: A measure of the amount of data that can be transferred over a given period of time. It is commonly used to describe data transfer performance in memory and networks.
\n* <\/span>Interconnect: An architecture or technology that connects components such as servers, chips, memory, and accelerators, enabling data to move between them
\n* <\/span>Bottleneck: A point that limits overall system performance. In AI systems, bottlenecks can occur not only in processors but also in data delivery, memory, storage, and networks<\/div>\n
\"\"<\/div>\n
<\/div>\n
\n
\n
<\/div>\n

Data never stays in one place<\/h3>\n
<\/div>\n<\/div>\n<\/div>\n

In an AI system, data does not remain stagnant in one place. Training data, model parameters*<\/span>, user requests, contextual information, search results, and compute results move between different locations depending on the stage of processing. Some data is retrieved from high-capacity storage, while other data passes through system memory before reaching memory closer to the accelerator. Once processing is complete, the results are stored again or transmitted over the network to other servers and services.<\/p>\n

* <\/span>Model parameter: An internal value adjusted as an AI model learns, determining how the model interprets inputs and generates outputs<\/div>\n
\n

\u2611\ufe0e Where does data travel before AI answers?<\/span><\/strong><\/p>\n

<\/div>\n

When a user enters a question into a chatbot or AI search engine, the data center processes more than just the user\u2019s request. Previous conversational context, along with search results, relevant documents, and other information the model may need, is gathered as required. This data moves through storage and memory before reaching layers closer to the accelerator. Once computation is complete, the result is stored again or returned to the user over the network.<\/p>\n

<\/div>\n

As the use of AI search and chatbots grows, these inference requests are occurring more frequently and at greater scale. Ultimately, the response speed experienced by users depends not only on model performance but also on how quickly the necessary data can be gathered and moved.<\/p>\n

<\/div>\n

As of July 2025, 18 billion messages were exchanged on ChatGPT each week, according to OpenAI, while Gemini surpassed 1 billion monthly users after August 2026.<\/p>\n

<\/div>\n

\"\"<\/p>\n

\u25b2 Stargate Data Center Source : OpenAI, \u300cStargate Advances with Partnership with Oracle\u300d(2025)<\/p>\n<\/blockquote>\n

<\/div>\n

Within this flow, each layer plays a different role. Storage retains large volumes of information, while system memory provides a workspace for the entire server. Memory located close to the accelerator rapidly supplies the information needed by the processor, while networks connect multiple servers and accelerators into a single system.<\/p>\n

AI performance, therefore, is not guaranteed simply by having the necessary data available. Even when using the same GPU, systems can deliver different real-world performance depending on how efficiently data is supplied and moved.<\/p>\n

<\/div>\n
\n
\n
<\/div>\n

Bottlenecks beyond compute<\/h3>\n
<\/div>\n<\/div>\n<\/div>\n

Bottlenecks in AI systems do not occur in a single location. No matter how fast a processor is, insufficient memory bandwidth can prevent the system from supplying enough data when it is needed. In this case, an accelerator may be ready to compute but still spend time waiting for data, reducing actual utilization rates.<\/p>\n

Storage can also become a bottleneck* <\/span>point. When an AI service needs to retrieve large volumes of documents, images, logs, user histories, or search results, response speeds slow down if the required data cannot be read quickly enough.
\nAs the number of requests for inference services grows, such delays can directly degrade service quality.<\/p>\n

Similar challenges arise during large-scale AI training. Training an AI model requires checkpointing*<\/span>, which periodically saves the model\u2019s intermediate training state. If storage fails to save or retrieve these large volumes of data quickly enough, latency can occur across the entire training cluster. This is why MLCommons*<\/span> separately measures storage performance for AI and machine learning training workloads through its MLPerf Storage*<\/span> benchmark. Data management is a core element of AI training and inference, making storage performance an increasingly important part of AI infrastructure.<\/p>\n

* <\/span>Checkpoint: A process or set of saved data that periodically records the current state of AI model training, allowing training to resume from the saved point following an interruption or failure
\n* <\/span>MLCommons: A global AI engineering consortium that develops industry-standard benchmarks for measuring the performance and efficiency of AI systems
\n* <\/span>MLPerf Storage: A suite of MLCommons benchmarks that evaluates the performance of storage systems supporting AI and machine learning workloads. It measures how data reads and delivery, as well as checkpoint saving and recovery during AI training, affect system efficiency<\/div>\n

Networks are another potential source of bottlenecks. In large-scale AI training or high-performance inference, multiple servers and accelerators exchange data simultaneously to divide and process a single task. If data transmission between servers are delayed or network bandwidth is insufficient, slowdowns in one part of the system can reduce overall processing speed. It is not enough for an individual device to be fast. Multiple devices need to exchange data efficiently and operate as a unified system.<\/p>\n

Power and cooling can also constrain performance. High-performance equipment requires a stable power supply and effective thermal management to sustain peak performance. Part 3 of this series will explore these issues in greater detail, but the key point is clear: Bottlenecks and performance limitations can arise throughout the system \u2014 from data movement paths involving memory, storage, and networks to power and cooling infrastructure.<\/p>\n

\"\"<\/p>\n

<\/div>\n
\n
\n
<\/div>\n

Data architecture improves accelerator utilization<\/h3>\n
<\/div>\n<\/div>\n<\/div>\n

The key to AI infrastructure lies not only in securing more accelerators but also in designing systems that keep those accelerators continuously engaged in computation. When data is supplied too slowly, accelerator utilization*<\/span> falls, reducing performance relative to the investment made. This is why even systems equipped with high-performance hardware may fail to perform at their full potential.<\/p>\n

* <\/span>Accelerator utilization: The proportion of time a GPU or AI accelerator is actively used for computation. Delays in data delivery can leave accelerators waiting and reduce utilization<\/div>\n

To prevent this, data paths need to be designed so that data can be supplied where and when it is needed. Frequently used data should be placed close to processors, while large volumes of data should be retrieved efficiently through storage and networks. This is due to the fact that it is impossible to place all information in the fastest location. Ultimately, the location and movement paths of data must be meticulously designed with speed, capacity, cost, and power efficiency in mind.<\/p>\n

Software coordinates this architecture for real-world operations. Accelerator utilization and overall performance vary depending on which tasks are processed first, when data is retrieved, and how they are both distributed across multiple devices. Even with fast hardware, inefficient scheduling and placement can create new bottlenecks.<\/p>\n

Designing to minimize data movement can also improve power efficiency. Intel explains that while the process of moving data within a system consumes energy, it does not directly contribute to computation itself. This is why data should travel shorter distances and be placed closer to where it is needed.<\/p>\n

Within this architecture, memory serves as a critical layer that supplies data where and when it is needed, helping accelerators remain continuously engaged in computation. How efficiently compute and memory are connected can therefore determine real-world system performance.<\/p>\n

\"\"<\/div>\n
\n
\n
<\/div>\n

AI infrastructure competitiveness depends on data flow<\/h3>\n
<\/div>\n<\/div>\n<\/div>\n

This approach of designing systems around data paths is bringing about changes in the competitive landscape of the AI infrastructure industry. While fast GPUs, high memory bandwidth, high-capacity storage, and high-speed networks are all important individually, they are insufficient if they operate in isolation. AI systems can deliver real-world performance only when data moves smoothly from storage to the processor, and results are delivered where they are needed.<\/p>\n

Consequently, collaboration methods across the entire industry ecosystem are also shifting. There are limitations to maximizing overall system performance if semiconductor, server, networking, cloud, and storage companies each optimize only their own components independently. As AI workloads grow larger and more complex, collaboration based on the entire data flow becomes crucial. This is due to the fact that required infrastructure architecture varies depending on which models customers use, what data they process, and what level of response speed is required.<\/p>\n

The role of memory companies is expanding as well. Beyond merely supplying products, they increasingly need the capabilities to jointly design data flows within customer systems and determine what memory architectures are required from a system-level perspective encompassing processors, storage, and networks.<\/p>\n

Compute remains essential. Faster GPUs and AI accelerators will continue to be core components of AI infrastructure. However, translating their capabilities into real-world performance requires optimizing the flow of data across the system as well.<\/p>\n

Looking ahead, understanding AI infrastructure requires examining not only the speed of processors but also the ways in which data moves through the system. Memory, storage, networks, and power and cooling should not be considered as independent components but as organically interconnected parts of a single system that keep data flowing without interruption.<\/p>\n

The need for faster GPUs remains as essential as ever. But the question has now evolved one step further: How quickly and efficiently does the data reach the GPU for processing?<\/p>\n

The next bottleneck determining AI performance begins with that question.<\/p>\n

\n

<References<\/strong>><\/strong><\/p>\n