Autonomous Chlorine Dosing for Manganese and Iron removal
Even in developed nations like Canada, numerous rural and Indigenous communities face persistent water advisories. A major obstacle to building and operating water treatment plants in these remote communities is the lack of trained operators to maintain and oversee the plants.
At RL Core Technologies, we believe that intelligent automation, powered by Artificial Intelligence (AI), offers a sustainable solution to this crisis. In partnership with Amii, Google.org, and Community Circle, we set out to demonstrate the viability of Reinforcement Learning (RL) for autonomously controlling key water treatment processes.
This case study details RL Core’s successful deployment of an RL agent at a mobile water treatment plant in BC, Canada, to autonomously control chlorine dosing. Our approach maximized the effectiveness of the water treatment while minimizing chemical usage, demonstrating a clear path towards more efficient and reliable water treatment operations globally.
The municipal water source—six groundwater wells—is rich in contaminants like manganese and iron. To remove these metals, the water is treated with sodium hypochlorite (chlorine) and then directed through a GreensandPlus filter.
The challenge lies in precision dosing. Operators must constantly balance two competing demands:
- Dose enough chlorine to fully oxidize manganese and iron for removal by the filter.
- Minimize chlorine dosing to lower costs and limit the formation of harmful byproducts.
This balancing act is complicated by several factors:
- Variable Water Quality: The water composition constantly shifts due to seasonality and environmental disturbances.
- Filter Conditioning: The GreensandPlus filter's effectiveness degrades over time, requiring operators to adapt the chlorine dose as the filter becomes "unconditioned."
- Time Delays: A significant retention time (approximately 30 minutes) exists between the chlorine injection point and the downstream sensor that measures its effect.
Our goal was to create an autonomous system that could adjust chlorine dosing in real-time, adapting to these complex and constantly changing dynamics to maintain optimal performance.
Since manganese sensors were unavailable, an Oxidation Reduction Potential (ORP) sensor downstream of the filter was used as a proxy. An ORP reading of 350 mV was determined to indicate the minimum chlorine needed for complete manganese oxidation, maximizing filter effectiveness while minimizing chemical waste.
The agent's concrete task was to maintain the ORP setpoint of 350 mV by adjusting the Flow Pace Multiplier (FPM)—the setting that controls the chlorine pump speed.
Markov Decision Process (States, Actions, and Rewards)
RL Core modeled the chlorine dosing problem as a Markov Decision Process (MDP), the standard framework for Reinforcement Learning:
Action: The manipulated variable in the control problem. The RL agent selected the chlorine pump’s FPM every 30 minutes, allowing time for the chlorine retention period to elapse and effects to be observed.
State: Sensory information describing the current state of the process, which the agent uses to select its action. In this case, the agent considered the current ORP, FPM, and flow rate, as well as their trends. Trends were vital for inferring water composition, filter conditioning, and the effect of delayed actions.
Reward: A signal indicating how well the agent is solving the control problem. The agent learns to solve the control problem by discovering which actions maximize reward in each state. The reward function was the error between the observed ORP and 350 mV.
Training Strategy: Offline and Online Learning
To ensure safety and performance from day one, we employed a two-phase training approach:
- Offline Pre-Training: We first generated a dataset of operational data using an exploration agent that systematically varied the FPM over a week. This data was used to pre-train our RL agent offline. This step gave the agent a strong initial understanding of the plant’s dynamics before deployment.
Online Deployment and Fine-Tuning: Once deployed, the agent continuously learned from its new experience, using the collected data to fine-tune its action selections. Continual learning allowed the agent to adapt to the slow degradation of physical components and seasonal water changes.
Sustained Setpoint Tracking
The RL agent was deployed with full autonomy, operating without human intervention.
The agent quickly increased the ORP to the 350 mV setpoint and maintained stable, precise control for the two-week deployment, keeping values within +/- 5 mV of the target:
Intelligent Adaptability
The agent’s ability to adapt to changes in dynamics was one of the most exciting outcomes:
- Flow Rate Fluctuation: When an operator suddenly increased the flow rate (from 7 L/s to 16 L/s), the agent immediately sensed the resulting increase in ORP and rapidly decreased the FPM to counteract the surge, bringing the ORP back to the setpoint. This showed proactive control, preventing the system from over-dosing.
Depleted Filter Reaction: When the GreensandPlus filter required reconditioning, the agent correctly identified that its normal actions were no longer effective. It intelligently reacted by ramping the FPM up to its maximum value in an attempt to counteract the filter's decreased efficiency, acting as a clever, temporary stop-gap until a human operator could intervene.
Bottom Line
RL Core Technologies successfully developed and deployed an intelligent agent that maintained water quality autonomously for two weeks, adapted to sudden changes, and reacted intelligently to mechanical failures. The autonomous chlorine dosing project is a testament to the power of Reinforcement Learning in solving complex, real-world control problems in resource-constrained environments.
As we look to the future, integrating autonomous control agents and smart alert systems into water treatment plants promises to alleviate the strain of labor shortages, reduce resource consumption, and ensure consistent delivery of clean drinking water for all communities.
The agent quickly increased the ORP to the 350 mV setpoint and maintained stable, precise control for the two-week deployment, keeping values within +/- 5 mV of the target. Additionally, the agent adjusted the FPM over time to track the ORP setpoint. (1 / 2)