JPEG is a widely used image compression standard that shows a reasonable image quality for a wide range of compression rates. However, when compressed with a low compression quality factor to increase the compression rate, it brings a large loss in the frequency domain, which turns into visible artifacts in the image domain. Accordingly, removing artifacts in JPEG-compressed images has been an essential image restoration task. While most previous methods use the information on compression quality factors available in the header of the JPEG file, we note this approach is not practical in the real-world scenario because many compressed images' metadata are not exist. To deal with this issue, we propose a new method based on a Deformable Offset Gating Network (DOGNet) and a Variational Autoencoder (VAE). We train the overall network in an end-to-end manner, where the role of the VAE is to guide the offset of the deformable convolution to flexibly deal with images compressed with diverse and unknown quality factors. Extensive experiments validate that our method achieves better or comparable results to the state-of-the-art methods in JPEG Artifact Removal.