Robust Algorithmic Collusion

Abstract

This paper develops a formal framework to assess policies of learning algorithms in economic games. We investigate whether reinforcementlearning agents with collusive pricing policies can successfully extrapolate collusive behavior from training to the market. We find that in testing environments collusion consistently breaks down. Instead, we observe static Nash play. We then show that restricting algorithms' strategy space can make algorithmic collusion robust, because it limits overfitting to rival strategies. Our findings suggest that policy-makers should focus on firm behavior aimed at coordinating algorithm design in order to make collusive policies robust.

Behavior-based algorithmic pricing

“…The algorithms first choose their pricing strategies, and then firms implement them and charge prices. This timing is common in the literature on algorithmic pricing where firms first calibrate the properties of their algorithm, which then determines a pricing strategy based on the characteristics of the market and of competing firms among other (Hansen et al, 2021;Eschenbaum et al, 2022).…”

Value Iteration is Optic Composition

“…A proof-of-concept implementation of value iteration with open games was done in 2019 by the first author and Wolfram Barfuss 1 , implementing a model from [4] -a model of the social dilemma of emissions cuts and climate collapse as a stochastic game, or jointly controlled MDP -and verifying it against Barfuss' Matlab implementation. A far more advanced implementation of reinforcement learning using open games was developed recently by Philipp Zahn, currently closed-source, and was used for the paper [16].…”

Artificial Collusion: Examining Supracompetitive Pricing by Q-Learning Algorithms

“…Abada and Lambin (2020) are critical about the one-period price cuts used by Calvano et al (2020a) to infer reward-and-punishment schemes and suggest insufficient exploration as one of the drivers of what the authors call 'seemingly collusive outcomes'. Eschenbaum, Mellgren and Zahn (2022) criticize the claim that algorithms can be trained offline to successfully collude online in different market environments. The authors find that 'collusion consistently breaks down' (p. 1) when collusive reinforcement learning policies are extrapolated from a training environment to the market.…”