Data center temperature control is a critical process for maintaining high quality of service. While maintaining temperatures in appropriate temperature bands is paramount, it is also important to consider that every heating or cooling strategy has associated environmental and economic costs. For example, a cooling strategy that makes excessive or unnecessary use of air conditioning will consume more energy than a strategy that better leverages free cooling. As long as operational constraints are satisfied, opportunities to discover temperature control solutions that minimize energy utilization are highly valuable. Co-developed by the Vector Institute and TELUS, this repository provides a solution for training a reinforcement learning (RL) agent to perform temperature control in a room. Click here for detailed documentation.

• Innovative discretization method: The provided solution leverages an innovation in state space design proposed by TELUS called Hyperspace Neighbour Penetration (HNP), which allows slowly-changing continuous variables to be approximated as discrete variables.
• Support for building control environments: The HNP package has built-in support for Sinergym - a building control simulation environment, as well as Beobench - a toolkit providing unified access to building control environments.
• Ready-to-use RL agents: A HNP-enabled Q-Learning agent and two baseline agents are provided in the package.

A key assumption in HNP is that continuous state spaces with very incremental, locally linear transitions can be effectively discretized into relatively coarse ranges or bands of values as tiles. When this type of coarse discretization is applied to a slowly-changing variable (e.g. temperature), it can lead to situations where an action (e.g. changing the setpoints) results in such a small change to the subsequent state observation that no discrete state transition has actually occurred. A naïve solution could be to increase the granularity of the state space, i.e. to consider much smaller changes in temperature as distinct elements of the state space, but it is computationally expensive or impossible to establish an extremely granular grid system. Alternatively, HNP computes multiple values from tile boundaries and then aggregates them using a weighted norm. This enables state-action pairs to result in steps towards other states and their corresponding values.

HNP is fully described in its foundational paper.

Sinergym is a building control environment that follows OpenAI Gym interface and uses EnergyPlus simulator. To use Sinergym, see detailed instruction on how to install here.

Beobench is a toolkit providing unified access to building control environments for RL (Sinergym also supported). It uses docker to manage all environment dependencies in the background. See detailed instruction on how to use Beobench here.

The HNP package provides the following agents:

• Random Action Agent: An agent that takes a random action
• Fixed Action Agent: An agent that always take a pre-defined action
• HNP-enabled Q-Learning Agent: A Q-learning agent with built-in HNP that allows different types of observation variables

• Python ≥ 3.9
• Sinergym == 2.2.0

To install hnp from PyPI:

pip install hnp

This is a minimalist example of using the HNP Q-Learning agent in Sinergym

import numpy as np

from hnp.agents import QLearningAgent
from hnp.environment import ObservationWrapper, create_env

config = {
    "agent": {
        "num_episodes": 100,
        "horizon": 24,
        "gamma": 0.99,
        "num_tiles": 20,
        "initial_epsilon": 1,
        "epsilon_annealing": 0.999,
        "learning_rate": 0.1,
        "learning_rate_annealing": 0.999
    },
    "env": {
        "name": "Eplus-5Zone-hot-discrete-v1",
        "normalize": True,
        "obs_to_keep": [0, 1, 8, 10],
        "mask": [0, 0, 0, 0]
    }
}

obs_to_keep = np.array(config["env"]["obs_to_keep"])
mask = np.array(config["env"]["mask"])

env = create_env(config["env"])
env = ObservationWrapper(env, obs_to_keep)

agent = QLearningAgent(
    env,
    config["agent"]["params"],
    mask,
)
agent.train()
agent.save_results()
env.close()

Detailed package documentation: HNP Docs

The code in this repository is published under 3-Clause BSD license (see LICENSE file).