Brownian circuits are based on a novel computing approach that exploits quantum fluctuations to increase the efficiency of information processing in nanoelectronic paradigms. This emerging architecture is based on Brownian cellular automata, where signals propagate randomly, driven by local transition rules, and can be made to be computationally universal. The design aims to efficiently and reliably perform primitive logic operations in the presence of noise and fluctuations; therefore, a Single Electron Transistor (SET) device is proposed to be the most appropriate technology-base to realize these circuits, as it supports the representation of signals that are token-based and subject to fluctuations due to the underlying tunneling mechanism of electric charge. In this paper, we study the physical limitations on the energy efficiency of the Single-Electron Transistor (SET)-based Brownian circuit elements proposed by Peper et al. using SIMON 2.0 simulations. We also present a novel two-bit sort circuit designed using Brownian circuit primitives, and illustrate how circuit parameters and temperature affect the fundamental energy-efficiency limitations of SET-based realizations. The fundamental lower bounds are obtained using a physical-information-theoretic approach under idealized conditions and are compared against SIMON 2.0 simulations. Our results illustrate the advantages of Brownian circuits and the physical limitations imposed on their SET-realizations.