Sustainable management of complex human-environment systems, and the essential services they provide, remains a major challenge, felt from local to global scales. These systems are typically highly dynamic and hard to predict, particularly in the context of rapid environmental change, where novel sets of conditions drive coupled socio-economic-environmental responses. Faced with these challenges, our tools for policy development, while informed by the past experience, must not be unduly constrained; they must allow equally for both the fine-tuning of successful existing approaches and the generation of novel ones in unbiased ways. We study ocean fisheries as an example class of complex human-environmental systems, and present a new model (POSEIDON) and computational approach to policy design. The model includes an adaptive agentbased representation of a fishing fleet, coupled to a simplified ocean ecology model. The agents (fishing boats) do not have programmed responses based on empirical data, but respond adaptively, as a group, to their environment (including policy constraints). This conceptual model captures qualitatively a wide range of empirically observed fleet behaviour, in response to a broad set of policies. Within this framework, we define policy objectives (of arbitrary complexity) and use Bayesian optimization over multiple model runs to find policy parameters that best meet the goals. The trade-offs inherent in this approach are explored explicitly. Taking this further, optimization is used to generate novel hybrid policies. We illustrate this approach using simulated examples, in which policy prescriptions generated by our computational methods are counterintuitive and thus unlikely to be identified by conventional frameworks.