Deep Reinforcement Learning (DRL) has demonstrated its strength in developing intelligent systems. These systems shall be formally guaranteed to be trustworthy when applied to safety-critical domains, which is typically achieved by formal verification performed after training. This train-then-verify process has two limits: (i) trained systems are difficult to formally verify due to their continuous and infinite state space and inexplicable AI components (i.e., deep neural networks), and (ii) the ex post facto detection of bugs increases both the time- and money-wise cost of training and deployment. In this paper, we propose a novel verification-in-the-loop training framework called Trainify for developing safe DRL systems driven by counterexample-guided abstraction and refinement. Specifically, Trainify trains a DRL system on a finite set of coarsely abstracted but efficiently verifiable state spaces. When verification fails, we refine the abstraction based on returned counterexamples and train again on the finer abstract states. The process is iterated until all predefined properties are verified against the trained system. We demonstrate the effectiveness of our framework on six classic control systems. The experimental results show that our framework yields more reliable DRL systems with provable guarantees without sacrificing system performance such as cumulative reward and robustness than conventional DRL approaches.

An effective way to improve the combined performance of mechanical seals is to optimize their surface textures using multi-objective optimization method. For compatibility with the multi-objective optimization algorithm, the theoretical performance of a mechanical seal is often determined using the finite-difference method (FDM). However, compared with the finite-volume method (FVM) and finite-element method (FEM), FDM is weaker for dealing with the issue of discontinuous film thickness for a textured surface. In the present study, the thermo-hydrodynamic lubrication model of a mechanical seal is modified by means of an equivalent-thickness treatment, and the accuracy of the modified lubrication model is assessed by comparing its predictions for film pressure and temperature with published FVM and FEM results, showing that the equivalent-thickness lubrication model is effective for addressing the issue of discontinuous film thickness. The present work is important in that it improves the simulation accuracy of multi-objective optimization for textured mechanical seals.