Multi-objective Energy Management for We-Energy
in Energy Internet using Reinforcement Learning
Qiuye Sun
School of Information Science
and Engineering
Northeastern University
Shenyang, China
sunqiuye@ise.neu.edu.cn
Danlu Wang
School of Information Science
and Engineering
Northeastern University
Shenyang, China
1653224713@qq.com
Dazhong Ma
School of Information Science
and Engineering
Northeastern University
Shenyang, China
madazhong@ise.neu.edu.cn
Bonan Huang
School of Information Science
and Engineering
Northeastern University
Shenyang, China
huangbonan@ise.neu.edu.cn
Abstract—We-Energy is a novel energy production-storage-
consumption mode proposed for Energy Internet, where more
renewable energy can be utilized. This paper mainly focuses on
the energy management of We-Energy in Energy Internet
consisting of combined heat and power unit (CHP), photovoltaic
unit, heating only unit and storage device. To construct an
environmental-friendly and low-operating cost energy
consumption structure, a multi-objective optimization model is
proposed in this paper. Furthermore, in order to satisfy the
power and heat demands of the We-Energy simultaneously as
well as realizing minimum operating cost and pollutant emission,
an intelligent energy management system (IEMS) is presented. In
particular, reinforcement learning method has been implemented
to formulate the optimal operating strategy. Eligibility trace
theory is also been introduced to accelerate the computational
process. Finally, simulation results are given to prove the
effectiveness of the proposed optimization strategy.
Keywords—Energy management, reinforcement learning, We-
Energy, multi-objective optimization
I. INTRODUCTION
Due to the upcoming shortage of fossil fuels and increasing
concerns for environmental pollution, the current power grid is
facing the challenges of increasing power demand and
environmental protection which calls for a sustainable power
system to realize low-carbon energy consumption. As a
consequence, the concept of “Energy Internet” is proposed as a
potential solution to these issues, the conventional fuels are
replaced by renewable energy and the centralized generation is
transformed into distributed generations.
The combined of multiple types of energy is one of the
specific characteristics of Energy Internet. The Energy Internet
can be assumed as a cluster of distributed energy resources and
loads, which contains various types of energy resources such as
electricity, gas, heat and so on [1]. The use of different kind of
energy brings great benefit to Energy Internet, which allows
multiple end users to make options according to their own
power demands, hence increasing the flexibility of the power
system and weakening the impact of traditional energy supplier.
However, using distributed generations indiscriminately may
also impose undesirable effects on power system. Therefore,
issues on optimal energy management come into play. A lot of
researches concerning control and operation of power system
have been done in recent years. Several common optimization
objectives including lower cost of carbon and minimum
operating cost have been discussed in [2]. In [3], authors
proposed a smart energy management system in order to
minimize the operating cost of the micro-grid. Only electricity
is discussed during optimization process while other types of
energy resources are not considered in the paper. The authors
in [4] proposed a micro-grid scenario consists of combined
heat and power generation, as well as power and thermal
energy storage devices. And an online algorithm has been put
forward to optimize the cost of whole system.
However, the optimized economic dispatch does not always
satisfy the demands when taking pollutants emission into
account. So, multi-objective energy management has drawn
attention from researchers so as to realize optimization both
economically and environmentally. The authors in [5]
proposed an intelligent energy management system (IEMS) for
a CHP-based micro-grid, and minimized the operation cost and
the net emission simultaneously. An efficient modified
bacterial foraging optimization algorithm was used to find the
optimal set points of the system. Reference [6] proposed a
Stackelberg game-based optimization model, and a differential
evolution-based heuristic algorithm was designed to reach the
Stackelberg equilibrium. But in the previous studies, there is a
lack of consideration of specific characteristics of Energy
Internet, such as openness, sharing and peer-to-peer integration.
As there is a series of complex problems in Energy Internet
like energy management, dynamic pricing and trading and
information interaction need to be solved, a novel energy
integration mode called “We-Energy” (WE) is proposed.
According to [7], We-Energy is a fusion of energy producers,
energy storage and consumers, which breaks the traditional
energy supply pattern led by major energy supplier and realizes
open operation and energy supply complementarity.
In this paper, an optimization model based on We-Energy
in Energy Internet is proposed considering both operating cost
and environmental pollutant. We stress that recent work has
shown reinforcement learning to be very effective and suitable
for making decisions and finding optimal strategies. Thus, we
provided an intelligent energy management system (IEMS)
based on Q-learning method to find the optimal operating
strategy of a WE.
978-1-5386-2726-6/17/$31.00 ©2017 IEEE