Rising energy demand and the disproportionate utilization of fossil fuels not only result in power imbalance and economic drain but also raise environmental concerns. Under these challenging circumstances, microgrids provide a tactical solution by adopting distributed energy resources at user end. However, this solution is not effective without enough participation by these end users (prosumers) for sustainable energy growth in microgrids. This paper presents a behavioral control theory and various psychological motivational models to improve prosumers' participation up to the desired level. A framework for peers' management within a community is also presented. The coalition-based game theory is employed for fair and trustworthy inter-trading which lead to the formation of grand coalition by satisfying all the defined motivational models. Various trading systems i.e. feed in tariff system, peer-topeer trading with and without storage systems, and demand-side management-based peer-to-peer trading systems are used for energy inter-trading with minimum involvement with the grid. Finally, the proposed system is validated through simulations of various game theoretic-based peer-to-peer trading systems. Simulation results show a considerable reduction in average expenses for energy demand and carbon emissions with improved earnings for peers. INDEX TERMS peer-to-peer energy trading system, prosumers, demand-side management, game theory, motivational models I. INTRODUCTION Carbon emissions caused by disproportionate utilization of cost-inefficient fossil fuels have become a global concern. The world population is expected to rise by 50% in the next decade [1], resulting in a 25% upsurge in energy demand and a consequent huge gap between demand and supply. It seems difficult for the energy sector to meet such a high demand without the exploration of new generation techniques, the use of efficient plants, and adoption of bidirectional communication based power exchanges [1]. One of the possible solutions is to increase fossil fuel based-centralized generation; however, high capital cost, relocation problems, carbon emissions, socio-political pressure, energy security, and several other constraints make this choice less feasible. Currently, in the European Union (EU), buildings are responsible for 40% of carbon emissions, which can be reduced by using efficient home energy management system (HEMS) [2]. In HEMS, smart and energy-efficient appliances which have an impact on customer's preferences are used at the user end [3]. These energy efficiency improvements (EEI) will reduce energy demand and emissions. However, sometimes EEI has a motivational rebound effect, and raises energy demand [4]. Microgrids (MG) can also be used to provide a tactical solution by adopting distributed energy resources (DER) at user end [5]. The MG is a distributed grid having various distributed generators i.e. renewable energy resources (RERs), along with energy storage systems and interconnected loads to meet user demand. Traditional grids ar...