Why AI Agents Run on the Machine, Not the Cloud

Most AI agents today live in the cloud. A sensor reports a value, the application sends it over the internet to a data center, a large model responds, the answer travels back. For a chat in the browser, that is fine. For a machine that has to react in milliseconds, at a site without a stable connection, or in a plant whose data must not leave the building, it is not.

Here the question flips. Why should the agent compute in the cloud at all, rather than directly on the device? An Edge Agent inverts the topology. The intelligence runs where the data is created, on the controller, the industrial PC, the small board next to the machine. Four sober reasons make the case.

Latency means reactions in milliseconds, not seconds

Every trip to the cloud costs time. There is network latency out, the response back, plus the processing in the data center. For a plant that should react to a threshold, that is a problem. On-device AI saves the entire round-trip. The evaluation happens on the same device the sensor is attached to.

The architecture matters here. The agent never sits inside the hard real-time control loop. The deterministic controller keeps that. The agent works as an “agent as a tool”. It reads values, makes a decision and writes parameters, such as a new setpoint, back in a buffered way. The control loop runs without interruption. The reaction stays fast without putting the stability of the control at risk.

Cost without a continuous stream into the cloud

A cloud agent costs on every call, in compute time, token stream and bandwidth. For a device that processes sensor data around the clock, this adds up to a running line item that never stops. Local AI in industry reverses that. The evaluation runs on existing hardware, without paying per inference.

This is made possible by an intelligence cascade. It runs simple rules, then classical machine learning, then a small language model (SLM, 1 to 3 billion parameters) directly on the device, then a mid-size open-weight model, and only as the exception a large cloud model. A large share of industrial tasks fit an on-device SLM. The large model becomes the exception, not the rule.

Connectivity for plants that are rarely reliably online

Many industrial environments are offline, air-gapped, or only intermittently connected, for example a ship at sea, a factory floor without plant Wi-Fi at the machine, or a remote field site. A cloud agent stops the moment the connection drops, precisely when no one is on site watching.

An Edge Agent is offline-capable by default. The engine runs standalone on the device, with durable local memory for state and history. The honest qualification belongs here. Operation is fully self-contained only with a local model. A cloud model, web search and a remote knowledge retriever make outbound calls. They stay optional and the exception. Run the agent with a local SLM and it keeps going even through days without a link, syncing as soon as a line is back.

Data sovereignty keeps data in-house

The moment sensor data, process values or images move into a third-party cloud, you give up control, legally and technically. For many operators that is a deal-breaker, not only because of GDPR, but because process data are trade secrets.

On-device AI keeps the data where it is created. Processing happens locally, execution is encapsulated by container isolation, and image distribution runs signed. The auditable, bounded graph makes it traceable at any time what the agent does. These properties are designed, as an architecture characteristic, for the EU Cyber Resilience Act. Security here is not a bolt-on afterthought but part of how it is built.

An example, the remote pumping station

A processing and pumping station sits far out, connected only over a shaky cellular link. For several hours a day it is effectively offline. A cloud solution would be blind exactly when something happens.

On the local industrial PC runs an Edge Agent. A threshold trigger with deadband watches pressure and flow. When a value crosses the limit, a local SLM classifies the situation from the current context and the local history, writes an adjusted setpoint to the controller in a buffered way, and records the event in durable memory. The actual control loop runs deterministically the whole time. When the connection returns, the collected events are transmitted to the central system. No data point is lost, no decision waits on the cloud.

How it works at the edge

Technically, a graph-first runtime carries this. The workflow graph is the program. The AI is just one node within it. That makes the flow deterministic, inspectable and bounded, not a black-box agent acting freely, but a clearly staked-out path.

The same single binary runs on any hardware, from a Raspberry Pi through a Jetson board to an industrial PC or x86 server. New hardware means redeploy, not rewrite. Hardware inputs and outputs such as GPIO, ADC/DAC/PWM, UART/serial and MQTT are first-class nodes against real Linux drivers. Triggers range from interval and time alarm through threshold-with-deadband and pin-edge to an incoming MQTT message.

The small language model runs via a local, OpenAI-compatible inference server (e.g. llama.cpp) right next to the engine. A knowledge base and semantic search supply context from your own documents. Durable local memory holds state across restarts. A visual builder and CLI build the graphs. The open-source core is usable without an account, and the backend is the management and fleet layer on top, including an agent registry, fleet orchestration and federated learning, which improves models across many identical devices without centralizing raw data. From Hardware to Intelligence.

Key Takeaways

• Latency. Evaluation happens on the device, with no cloud round-trip. The agent only writes buffered parameters and never sits in the hard control loop.
• Cost. An intelligence cascade keeps most tasks on an on-device SLM, with no continuous token stream into the cloud.
• Connectivity. The engine is offline-capable. It is fully self-contained with a local model, while cloud calls stay optional and the exception.
• Data sovereignty. Data stays local. Local execution, container isolation and signed images are designed, as architecture, for the EU Cyber Resilience Act.
• One binary, any hardware. From Raspberry Pi to industrial PC, redeploy instead of rewrite.