When someone tells me they are planning an AI workload, the conversation often turns to GPUs pretty quickly. I understand why. But before we talk about hardware, I want to know what the application needs to do.

Is it classifying documents or summarizing text? How many people will use it, and how quickly does it need to respond? Where does the data live? Once we understand the task and choose a model that can handle it, we can ask: Does this workload actually need a GPU?

I brought that question up during US Signal’s annual Cloud + AI Readiness Virtual Summit using a small language model as an example. For a narrow task, CPU-based infrastructure, including ARM-based compute, may meet the application’s needs. Whether it does depends on the model and how it performs under the workload you expect to run.

Start with the job, then test the model

“AI workload” can mean many things. Training a large model, serving an interactive assistant, and using a small model to classify incoming documents place different demands on infrastructure.

For the small-model example, I would first test whether the model does the job well enough. If it does, I would run it on the hardware options under consideration and measure how it performs. A model that responds quickly to one request may slow down when many people use the application at once.

That is why I would look at response time, the number of simultaneous requests it can handle, and how much memory it needs. If a CPU-based deployment meets those requirements, it belongs in the comparison. If it cannot, a GPU may be necessary. The decision comes from testing the application, not from assuming every model needs the same hardware.

Understand how much you will use it

The next question is how the workload behaves over time. Will people use it occasionally, or will it process a steady stream of requests throughout the day? How much capacity will it need when usage peaks?

Those answers affect the cost comparison. A hosted API may be practical when usage is low or difficult to predict. For a workload with steady demand, running a model on cloud or dedicated compute may offer a different cost profile. But infrastructure costs are only part of the picture. If your team operates the model, it also has to deploy it, monitor performance, maintain it, and plan for capacity.

I would compare those options using the usage the organization actually expects. An estimate based on a small pilot can look very different once the application is used across the business.

Follow the data

I also want to know what information the application will use and where that information is stored. A model summarizing internal documents or working with customer data may need access to systems that are already in a particular environment.

If you plan to send that information to an external AI service, review what data will be transferred and the provider’s applicable terms and controls. If you plan to run the model closer to your data, check whether that environment can deliver the performance and capacity the application needs.

Neither choice should be made on location alone. Data handling, performance, and cost all have to work together.

Compare CPU and GPU against the same requirements

Once we know the task, model, expected demand, and data requirements, we can make a useful hardware comparison. Run the model under realistic conditions. Measure its response time and capacity, then compare the full cost of each option.

A GPU may be the right choice for a larger model, higher concurrency, or a response time the CPU cannot meet. For a smaller, task-specific model, CPU infrastructure may do the job. ARM-based compute is one option worth testing when the model and its supporting software are compatible.

Start with what the application needs to accomplish and choose a model that can handle it. Then test the model against the performance, usage, and data requirements of the application. That will tell you whether a GPU earns its place in the architecture, or whether a simpler compute option will meet the need.

Explore OpenCloud ARM Compute to learn about one CPU-based option for small language models and AI inference.