Temporal downscaling of gridded geophysical data is essential for improving climate models, weather forecasting, and environmental assessments. However, existing methods often cannot accurately capture multi-scale temporal features, affecting their accuracy and reliability. To address this issue, we introduce an Enhanced Residual U-Net architecture for temporal downscaling. The architecture, which incorporates residual blocks, allows for deeper network structures without the risk of overfitting or vanishing gradients, thus capturing more complex temporal dependencies. The U-Net design inherently can capture multi-scale features, making it ideal for simulating various temporal dynamics. Moreover, we implement a flow regularization technique with advection loss to ensure that the model adheres to physical laws governing geophysical fields. Our experimental results across various variables within the ERA5 dataset demonstrate an improvement in downscaling accuracy, outperforming other methods.