What Water Treatment Operators Told Us About Trusting AI

AI-based controllers are achieving exciting results in industrial process control. Most of the work on such systems, RLCore’s included, goes into making them perform well and safely. Whether an operator will actually let the controller run is a separate question, and it gets less attention.

Change management and the associated adoption curve for novel industrial control strategies deserve more attention. To dive deeper into how operators think about this, we surveyed 43 of them, in collaboration with Amii and Community Circle. Most of the operators surveyed work in rural and Indigenous communities in British Columbia. 

Our survey methodology described three ways of running a plant: a human operator working remotely, a conventional automated control system, and an AI system that learns and adjusts on its own. We told operators that all three approaches had the same safety record.

We then asked how useful each would be and how likely they would be to trust it. Operators found all three systems equally useful. However, they did not trust them equally. Eighty-five percent said they would trust the remote human, 70 percent the conventional automation, and 45 percent the AI learning controller. Holding the safety record constant did not close that gap, so something other than safety was doing the work.  When we asked, operators were fairly direct about what it was.

Operator reservations were about trust in AI-based control broadly. When we asked them to name the barriers to adopting a system like this, 82 percent named the trustworthiness of the system itself. Cost came a distant second at 40 percent, and lack of training third at 30 percent.

The shape of the trust numbers is also informative. Conventional automation lost fifteen points relative to the human; the AI lost another twenty-five. In other words, the larger penalty was not for handing control to a machine, which plants have been doing for decades, but for handing it to a machine whose behaviour can change. Unlike a conventional control loop, a learning system's behaviour depends partly on what experience it has seen, and there is no fixed specification to check it against.

An example makes the difficulty concrete. A learning controller will sometimes move a setpoint away from what the operator would have chosen. This is because these controllers run mini "science experiments" in order to figure out what optimal control is — in the vocabulary of the algorithm this is called exploration, and learning is impossible without it. 

This behaviour has a good explanation in the system's terms, and a machine learning engineer who trusts a learning algorithm will not be worried if the algorithm chooses to explore. However, this behaviour could be surprising to plant operators since it is the controller choosing a worse setting, on purpose.

What would help build trust in learning systems? Fortunately the operators surveyed identified the potential levers. Asked what would help them build trust in an AI system, some said it should be tested thoroughly before deployment, to be easy to interact with and report on itself, and to show them how to override it. When asked about training, the most common request was for instruction on how the system actually works. 

If we think broadly about these requests, we can put them into three categories:

1. What will the controller do before anyone has seen it act?
2. Can operators still interact with and guide the system?
3. Is the system explainable?

The first question is answered before the controller reaches the plant. At RLCore our agents are trained offline, on historical data from the plant, before they are deployed. The exploration described above is not eliminated, but it is less necessary since the agent already has a working understanding of the plant. What arrives at the plant is a policy that already behaves sensibly and refines itself from there.

The second question is about whether the operator remains in charge. RLCore tackles this in three ways. First, the controller's goal is set by the plant operators: our agents are configured by specifying targets for process variables and a final cost objective, in the terms an operator already uses, such as holding a chlorine residual or keeping turbidity below a limit with as little chemical as possible. Second, the agent’s freedom is bounded: guardrails, which are defined by the plant operations team, place hard limits on the actions the agent may take and on the process variables it may move. Third, the operators can step in at any time to guide the agent towards better control strategies.

The third question, explainability, is harder for a learning system than for a conventional control loop, and is not fully solved yet (but is an area of active research at RLCore). But two things go a long way. Our agent’s goals are written in the operator's vocabulary, and we communicate which goals the agent pursues and when. These two facts mean it is clear to operators what the goals driving the agent’s behaviour are. 

Explainability is also where the training request lands. An operator who knows that the controller explores, within limits they set, toward a target they chose, does not necessarily read an odd setpoint as a fault. 

That being said, trust in a learning controller will be built the way trust in a new colleague is built: incrementally, with the stakes kept low until there is a track record. That is a change-management problem as much as an engineering one, and it is one the developer and the plant work through together rather than one handing it to the other. The case studies we publish are part of that effort: deployments operators can read about before they are asked to trust anything. Visit our rlcore.ai to explore them  here: https://rlcore.ai/customers