Recent breakthroughs in predictive business process monitoring equip process analysts with predictions on running process instances, supporting the elicitation of proactive measures to mitigate risks that can be caused by the process instance. However, contrary to active research on providing various predictions and improving the accuracy of prediction models, the practical application of such predictions has been left to the subjective judgment of domain experts. In this work, we explore the exploitation of the insights from predictive information for the actual process improvement in practice. Concretely, we focus on improving resource allocation in business processes where the goal is to allocate appropriate resources to tasks at the proper time. Based on design science methodology, we develop a two-phase method to improve resource allocation by leveraging predictions. Based on the method, we instantiate an algorithm to optimize total-weighted completion time and evaluate its effectiveness and efficiency. From an academic standpoint, our work demonstrates the combination of predictions using machine learning and optimizations based on scheduling. From a practical standpoint, our work provides a general approach to optimize resource allocations for different objectives using predictions.