Traffic at larger or busier urban intersections is currently coordinated using traffic signals to prevent dangerous traffic situations and to regulate the flow of traffic. In future scenarios with 100% connected automated vehicles, conventional traffic signals could be replaced, and vehicles at intersections could be seamlessly coordinated via vehicle-to-vehicle and vehicle-to-infrastructure communication. In the past two decades, many such control strategies have been presented, commonly referred to as autonomous intersection management (AIM). In recent years, an evolution from simpler first come, first served to more sophisticated optimization-based AIM strategies can be observed. Optimization-based AIM can significantly improve capacity and reduce delays as compared to slot-based strategies and conventional traffic signal control (TSC). In addition, it allows for prioritizing road users. This paper is among the first to consider pedestrians in optimization-based AIM. The proposed approach consists of a signal-free vehicle control in combination with pedestrian signal phases that are fully integrated into the optimization problem. Since the communication range of the controller is limited in real-world applications, a rolling horizon scheme is presented and explained in detail. The presented strategy is implemented and evaluated using a microscopic traffic simulation framework. Results show that vehicle delays can be significantly reduced and vehicle capacity can be increased compared to fully actuated TSC, while pedestrian waiting times are comparable. In addition, focus is put on how vehicle and pedestrian delays can be balanced in the presented setup. Three different control parameters can be adjusted, which need to be tuned based on the considered demand scenario.