How Does Reinforcement Learning Make Predictions?

Reinforcement Learning (RL) is formulated as a loop between an agent and its environment. An agent is a computer program that makes decisions and learns from their observed effects on its environment. The environment is what the agent interacts with - it comprises everything external to the agent that is relevant to the control task. At each step, the agent observes the state of its environment and selects an action. The environment responds with two things: a new state and a reward scoring what just happened.

Mapping RL to process control

In process control, the RL loop maps onto the plant directly:

• State — the plant's sensor readings at that moment. Inlet and outlet concentrations, flows, pH, temperature, equipment status, etc.
• Action — the setpoints the agent is permitted to move. A chemical dose rate, a pump speed, a blower output, etc.
• Reward — a number expressing how well the control task is being carried out at that moment in time.

The environment is the plant itself. The agent holds no schematic and no physics model of it. It reads the instruments, moves setpoints, and reads the instruments again.

Reward is the part operators specify, though not as a number. RLTune accepts three kinds of specification, written in the engineering units of the plant:

• Constraints — process variables that must stay within bounds, such as an outlet concentration below a regulatory limit or a pH held inside a safe band. Bounds are zoned, with a red zone marking the region the agent is required to keep out of.
• Goals — target conditions to reach and hold, such as driving an outlet reading down to a threshold.
• Optimization — the quantity to push on once constraints and goals are satisfied, typically chemical or energy consumption.

RLTune converts that specification into the single scalar reward the agent learns from, preserving the priority ordering: constraints before goals, goals before optimization.

What the agent is trying to maximize, and over how long

An agent that maximized each reward as it arrived would be solving a different problem than the plant poses, because actions taken now change the states observed later (i.e. changes the future). A dose set this minute determines what the outlet looks like forty minutes from now, and therefore what decisions remain available at that time.

So the quantity reinforcement learning maximizes is not a single reward but the return — the rewards accumulated after the action:

Read left to right: the reward one step after the action counts fully, the reward two steps after counts γ times as much, the reward three steps after counts γ² times as much, and so on. The discount factor γ sits between 0 and 1, so each additional step of lookahead contributes less than the one before it, and the sum stays finite.

γ controls how far ahead the agent looks. At γ = 0 every term after the first disappears and the return collapses to R_{t+1}, the reward from the very next observation. As γ approaches 1, consequences well into the future carry nearly as much weight as immediate ones.

What the agent learns is a policy — a decision-making rule assigning an action to every state it might encounter — that maximizes the average return from each of those states. The word average carries weight here. A plant is noisy, and the same action taken twice in the same state will not produce the same return, so the agent optimizes the average outcome rather than any individual trajectory.

A discount factor is an awkward thing to configure. It is dimensionless, its useful range is compressed against 1, and the difference between 0.99 and 0.999 is an order of magnitude in reach with no intuitive reading. To better enable end-users to intuitively configure an agent, RLTune exposes the performance window instead: a duration, in the same units as the rest of the timing configuration, describing how far past an action its consequences are still collected. The system converts that duration into the discount factor the learning algorithm requires. A plant engineer who knows that a dose change takes roughly forty minutes to fully express at the outlet can configure that number directly.

Value functions are where the prediction happens

The agent cannot compute a return before acting. The return is a fact about a future that has not yet occurred. What it can do is predict it, and that prediction is called a value function.

A value function predicts the return the agent expects to accumulate from a given plant state onward. An action-value function predicts the return that follows from taking a specific action in a specific state, which is the more operationally useful form — it is defined over exactly the choice the agent has to make.

Action selection then reduces to comparison. Given the current sensor readings, the agent evaluates candidate setpoints against its action-value predictions and selects the one expected to produce the highest return. These estimates are learned from the interaction stream itself, revised by every step of experience, so they sharpen as the agent accumulates time on the plant.

A policy is only as good as the predicted returns it selects against, and those estimates are only defined over the horizon the performance window sets.

A different kind of prediction than a language model makes

Large Language Models (LLMs) are also prediction systems, which makes the word ambiguous in an industrial setting. An LLM predicts the next token in a sequence, conditioned on the tokens before it. An RL agent predicts the accumulated numerical consequence of an action on a physical process, conditioned on the observed state of that process.

The difference shows up most clearly in how each prediction system is trained. An LLM's prediction is graded against text data and human feedback. An RL agent's prediction is graded against how the process actually evolved after the agent acted (the sequence of observed sensor readings).

Both systems produce a number they were trained to be accurate about. Only one of them is trained to be accurate about the plant.

An agent that sees one step ahead

Setting the performance window to its shortest possible value collapses the return to a single reward. This is called a bandit problem. From the agent's perspective, it selects an action, receives one reward, and the situation resets. There is no carryover between decisions, and the agent's only job is to identify the action with the highest immediate payoff.

Bandit algorithms are well understood and they learn quickly, because credit assignment is trivial. The reward that arrives is unambiguously attributable to the action just taken. Nothing else could have caused it.

Applied to an industrial control setting such as chemical dosing, this formulation produces two specific and repeatable failures.

Consider an agent scored only on setpoint tracking error at the next observation. The outlet is above target. The action that minimizes error at the next step is the largest available chemical dose increase — drive the actuator hard, kill the error fast. The overshoot arrives two or three steps later, outside the single step that enters the agent's return, so the action records as a success. In the meantime, the outlet still reads above target, and the same logic prescribes another maximum increase, then another, each one compounding the overshoot already in transit. When it finally lands, the error changes sign and the agent reverses to the largest available decrease with identical conviction. The controller oscillates between its action limits, and it oscillates deliberately, because every individual action is optimal under the objective it was given.

The second failure is quieter. Suppose the process is currently in good shape: every constraint is satisfied and every goal is met. The only term left in the objective is the optimization — reduce chemical consumption, or reduce energy. Scored one step ahead, the best available action is to cut the feed rate to its lower bound. Nothing in the next observation degrades, because the effect of underdosing has not reached the outlet yet. Three steps later the untreated load arrives and the process enters a state that no single-step score ever penalized.

The agent is not malfunctioning in either case. It is maximizing exactly what it was told to maximize. The objective simply ends before the consequences do.

This is what the performance window governs. It determines how far a consequence can reach back toward the action that caused it before its weight decays to nothing.

Why a longer window is not simply better

If the window only controlled how much of the future the agent accounts for, the correct setting would be as long as possible. But it also controls how much unrelated variation enters the learning signal.

Over a forty-minute window, the rewards the agent collects reflect its own dose change. They also reflect an influent load shift, an upstream unit coming back online, a temperature change, and a crew washing down a channel. None of those were caused by the action, but all of them land inside the window and affect the return.

Extending the window adds terms to the sum. The later terms carry less weight through the discount, but they carry proportionally more disturbance and proportionally less of the action's own effect, since the action's influence decays as the process settles while the disturbance rate does not. The agent's learning signal is the gap between predicted and observed return, so the further out the window reaches, the more of that gap is noise.

Concretely, a good action and a mediocre one start producing returns that overlap heavily. Telling them apart requires many more observations of each. This is the credit assignment problem, and it is the real cost of a long performance window.

In simulation, that cost is compute. On live process equipment, it is wall-clock time. An agent that needs four times as much data to resolve the same distinction needs four times as many action periods to acquire it, and those elapse at the speed of the plant. A performance window set far longer than the process response yields a controller that learns slowly and may fail to converge within the deployment, while adding no foresight the process dynamics can use.

Setting the performance window

RLTune derives the discount factor from the performance window in the timing configuration, alongside the observation and action periods. Configuring a successful agent requires setting the performance window long enough that an action's full response falls inside it, and no longer than that. An agent scored on a window matched to its process learns a policy that accounts for the effects its actions actually cause, and stops there. That is the entire purpose of the performance window.

Start from the process rather than from a step count. If a dose change requires transport time through the contact basin plus settling time at the sensor, the window needs to cover both. A window shorter than the transport delay alone scores the agent on observations that contain no information about its actions’ effects at all. At commissioning, measure the process response time and set the performance window from it. Revisit that measurement periodically, and whenever the process dynamics change.

Avoid anchoring the window to operational schedules. A diurnal load cycle is not a consequence of the agent's actions, and setting the window to span it adds a full day of disturbance variance to every return without adding anything the agent can act on.

The interaction between the performance window and the action period also affects agent learning. The action period is how often the agent is allowed to move its setpoints, and a shorter one means more decisions share credit for the same physical outcome. Say a dose change takes forty minutes to fully express at the outlet, so the performance window is set to forty minutes. At a ten-minute action period, that window spans four agent actions: the effect of the first action appears in its own return as well as the returns of the next three actions, and credit for the outcome is shared, to varying degrees, among all four. At a one-minute action period, credit is shared among forty actions. The more actions that fall within the performance window, the more difficult it is for the agent to learn the effect of any particular one.

Two diagnostics distinguish the failure modes in production:

• Persistent overshoot or oscillation, with individually reasonable-looking actions. The window is too short. The agent is scoring well on a horizon that ends before the process finishes responding.
• Short-term behavior that looks sensible but performance that improves slowly or inconsistently. The window is likely too long. The agent is attempting to attribute disturbance variation to its own actions.

The performance window is the setting that decides what kind of controller you get. If you set it too short, the agent optimizes against a response it never sees. The agent learns to drive actuators to their upper and lower limits to achieve short term goals without accounting for their long term consequences. If you set it too long, the agent spends the deployment trying to separate its own effect from the plant's ordinary variation. Matched to the process, it learns a policy that accounts for what its actions actually cause, and stops there.