ORAD: a new framework of offline Reinforcement Learning with Q-value regularization

Abstract

This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distribution into a standard Gaussian and then samples actions conditioned on the environment state with a corresponding reverse-time SDE, like a typical diffusion policy. We show that such an SDE has a solution that we can use to calculate the log probability of the policy, yielding an entropy regularizer that improves the exploration of offline datasets. To mitigate the impact of inaccurate value functions from out-of-distribution data points, we further propose to learn the lower confidence bound of Q-ensembles for more robust policy improvement. By combining the entropy-regularized diffusion policy with Q-ensembles in offline RL, our method achieves state-of-the-art performance on most tasks in D4RL benchmarks. Code is available at https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble.

Bimetallic Sulfides with Vacancy Modulation Exhibit Enhanced Electrochemical Performance

“…[ 29 ] As for S 2p, we observed in Figure S16d (Supporting Information) that the four characteristic peaks of Ni–S and Co–S could be classified from low to high energies as 2p3/2, 2p1/2, SO 4 2− and satellite peaks. [ 30 ] In particular, contrary to monometallic sulfides, bimetallic sulfides (Figure 3i) were categorized from low to high energies into 2p3/2, 2p1/2, M‐S bonds (M = Ni /Co), and satellite peaks. [ 31 ] The higher‐energy S 2p1/2 characteristic peaks were derived from low‐coordinated sulfur vacancies.…”

Controllable Synthesis of Carbon Yolk‐Shell Microsphere and Application of Metal Compound–Carbon Yolk‐Shell as Effective Anode Material for Alkali‐Ion Batteries

“…The porous and thin hollow shell consisting of CoSe 2 ‐H‐YS facilitated the shortened ion diffusion and fast electrolyte penetration compared to the dense structured CoSe 2 ‐D‐CS. [ 48 ] CoSe 2 ‐H‐YS for SIBs exhibited reversible discharge capacities of 461, 434, 404, 376, 346, 301, 255, and 209 mA h g −1 at current densities of 0.1, 0.2, 0.5, 1.0, 2.0, 4.0, 6.0, and 8.0 A g −1 , respectively, CoSe 2 ‐H‐YS for KIBs exhibited reversible discharge capacities of 373, 348, 312, 271, 208, 154, 98, and 60 mA h g −1 at current densities of 0.1, 0.2, 0.5, 1.0, 2.0, 3.0, 4.0, and 5.0 A g −1 , respectively. In contrast, CoSe 2 ‐D‐CS exhibited low reversible capacities of 95 and 5 mA h g −1 at high current densities of 8.0 and 5.0 A g −1 for SIBs and KIBs, respectively.…”

Ni3S2/Co3S4 with controlled surface electron arrangement for high-performance aqueous energy storage

“…Generally, as the vacancy diffusion is another important ion transport means except for interstitial diffusion, 39 vacancy engineering is an efficient tactic to construct vacancies and strengthen the ion transport rate through the rational optimization of the electronic distribution and adjustment of the intrinsic electrochemical activity of the active materials. [40][41][42] Particularly, constructing sulfur vacancies in metal suldes (so as to accelerate ion diffusion kinetics by inducing localized electron centers) has been put forward by researchers. [43][44][45][46] For example, Guo's group revealed that generating sulfur vacancies in Ni 3 S 2 nanosheets can decrease the band gap, increase the occurrence of richer active sites, and thus enhance electrical conductivity.…”