We demonstrate 230 MHz photodetection and a switching energy of merely 27 fJ by using WSe2 multilayers and a very simple device architecture. This improvement over previous, slower WSe2 devices is enabled by systematically reducing the RC constant of the devices through decreasing the photoresistance as well as the capacitance. In contrast to MoS2, reducing the WSe2 thickness toward monolayers only weakly decreases the response time, highlighting that ultrafast photodetection is also possible with atomically thin WSe2. Our work provides new insights into the temporal limits of pure transition metal dichalcogenide photodetectors and suggests that Gigahertz photodetection with these materials should be feasible.