In the optional prisoner's dilemma (OPD), players can choose to cooperate and defect as usual, but can also abstain as a third possible strategy. This strategy models the players' participation in the game and is a relevant aspect in many settings, for example, social networks or opinion dynamics, for example, during an election abstention is an option. In this article, we provide a formulation of the OPD where we consider irrational behaviors in the population, inspired by prospect theory. Prospect theory has gained increasing popularity in recent times because it can capture aspects such as reference dependence or loss aversion, which are common in human behavior. This element is original in our formulation of the game and is incorporated in our framework through pairwise comparison dynamics. Recently, the impact of the environment has been studied in the form of feedback on the population dynamics. Another element of novelty in our work is the extension of the game-environment feedback to the OPD in two forms of dynamics, the replicator and the pairwise comparison. The contribution of this article is threefold. First, we propose a modeling framework where prospect theory is used to capture irrational behaviors in an evolutionary game with game-environment feedback. Second, we carry out the stability analysis of the system equilibria and discuss the oscillating behaviors arising from the game-environment feedback. Finally, we extend our previous results to the OPD and we discuss the main differences between the model resulting from the replicator dynamics and the one resulting from the pairwise comparison dynamics.