To significantly decrease fossil carbon emissions from oil refineries, a combination of climate mitigation options will be necessary, with potential options including energy efficiency, carbon capture and storage/utilization, biomass integration and electrification. Since existing refinery processes as well as many of the potential new processes are characterized by large heating demands, but also offer large opportunities for process excess heat recovery, heat integration plays a major role for energy efficient refinery operation after the implementation of such measures. Consequently, the process heat recovery systems should not only be able to handle current operating conditions, but also allow for flexibility towards possible future developments. Evaluation of the flexibility of process heat recovery measures with both these perspectives enables a more accurate screening and selection of alternative process design options. This paper proposes a new approach for assessing the trade-off between total annual cost and potential operating flexibility for the heat exchanger network in short-as well as in long-term perspectives. The flexibility assessment is based on the evaluation of a flexibility ratio (similar to the conventional flexibility index) to determine the range in which operating conditions may vary while at the same time achieving feasible operation. The method is further based on identification of critical operating points to achieve pre-defined flexibility targets. This is followed by optimization of design properties (i.e., heat exchanger areas) such that feasible operation is ensured in the critical operating points and costs are minimized for representative operating conditions. The procedure is repeated for a range of different flexibility targets, resulting in a curve that shows the costs as a function of desired flexibility ratio. The approach is illustrated by an example representing a heat exchanger network retrofit at a large oil refinery. Finally, the paper illustrates a way to evaluate the cost penalty if the retrofit is optimized for one operating point but then operated under changed conditions. Consequently, the presented approach provides knowledge about cost and flexibility towards short-term variations considering also changes in operating conditions due to long-term development.