Reinforcement Learning Applied to an Electric Water Heater: From Theory to Practice

Reference

F. Ruelens, B.J. Claessens, S. Quaiyum, B. De Schutter, R. Babuška, and R. Belmans, "Reinforcement Learning Applied to an Electric Water Heater: From Theory to Practice," IEEE Transactions on Smart Grid, vol. 9, no. 4, pp. 3792-3800, 2018.

Abstract

Electric water heaters have the ability to store energy in their water buffer without impacting the comfort of the end user. This feature makes them a prime candidate for residential demand response. However, the stochastic and non-linear dynamics of electric water heaters, makes it challenging to harness their flexibility. Driven by this challenge, this paper formulates the underlying sequential decision-making problem as a Markov decision process and uses techniques from reinforcement learning. Specifically, we apply an auto-encoder network to find a compact feature representation of the sensor measurements, which helps to mitigate the curse of dimensionality. A well-known batch reinforcement learning technique, fitted Q-iteration, is used to find a control policy, given this feature representation. In a simulation-based experiment using an electric water heater with 50 temperature sensors, the proposed method was able to achieve good policies much faster than when using the full state information. In a lab experiment, we apply fitted Q-iteration to an electric water heater with eight temperature sensors. Further reducing the state vector did not improve the results of fitted Q-iteration. The results of the lab experiment, spanning 40 days, indicate that compared to a thermostat controller, the presented approach was able to reduce the total cost of energy consumption of the electric water heater by 15%.

Downloads

Bibtex entry

@article{RueCla:16-028,
author={F. Ruelens and B.J. Claessens and S. Quaiyum and B. {D}e Schutter and R. Babu{\v{s}}ka and R. Belmans},
title={Reinforcement Learning Applied to an Electric Water Heater: {From} Theory to Practice},
journal={IEEE Transactions on Smart Grid},
volume={9},
number={4},
pages={3792--3800},
year={2018},
doi={10.1109/TSG.2016.2640184}
}

Go to the publications overview page.
This page is maintained by Bart De Schutter. Last update: February 21, 2026.