Applications are increasingly operating on large data sets. This trend creates problems for access control, which in principle restricts the actions that subjects can perform on any item in that data set. Performance issues therefore emerge, typically for operations on entire data sets. Emerging access control models such as attribute-based access control do meet their limitations in this context. Worse, few solutions exist that addresses performance problems while supporting separation of concerns. In this paper, we present a first approach towards addressing this challenge. We propose a middleware architecture that performs policy transformations and query rewriting for externalized policies to optimize the access control process on the data set. We argue that this offers a promising approach for reducing the policy evaluation overhead for access control on large data sets.