Which theories lead to a contradiction between simple reasoning principles and modelling observers' memories as physical systems? Frauchiger and Renner have shown that this is the case for quantum theory (Frauchiger and Renner 2018 Nat. Commun. 9 3711). Here we generalize the conditions of the Frauchiger-Renner result so that they can be applied to arbitrary physical theories, and in particular to those expressed as generalized probabilistic theories (GPTs) (Hardy 2001 arXiv:quant-ph/0101012; Barrett 2007 Phys. Rev. A 75 032304). We then apply them to a particular GPT, box world, and find a deterministic contradiction in the case where agents may share a PR box (Popescu and Rohrlich 1994 Found. Phys. 24 379-85), which is stronger than the quantum paradox, in that it does not rely on postselection. Obtaining an inconsistency for the framework of GPTs broadens the landscape of theories which are affected by the application of classical rules of reasoning to physical agents. In addition, we model how observers' memories may evolve in box world, in a way consistent with Barrett's criteria for allowed operations (Barrett 2007 Phys. Rev. A 75 032304; Gross et al 2010 Phys. Rev. Lett. 104 080402).Ordinary readers, forgive my paradoxes: one must make them when one reflects; and whatever you may say, I prefer being a man with paradoxes than a man with prejudices.