Spherical Transformer

Abstract

Using convolutional neural networks for 360 • images can induce sub-optimal performance due to distortions entailed by a planar projection. The distortion gets deteriorated when a rotation is applied to the 360 • image. Thus, many researches based on convolutions attempt to reduce the distortions to learn accurate representation. In contrast, we leverage the transformer architecture to solve image classification problems for 360 • images. Using the proposed transformer for 360 • images has two advantages. First, our method does not require the erroneous planar projection process by sampling pixels from the sphere surface. Second, our sampling method based on regular polyhedrons makes low rotation equivariance errors, because specific rotations can be reduced to permutations of faces. In experiments, we validate our network on two aspects, as follows. First, we show that using a transformer with highly uniform sampling methods can help reduce the distortion. Second, we demonstrate that the transformer architecture can achieve rotation equivariance on specific rotations. We compare our method to other state-of-the-art algorithms using the SPH-MNIST, SPH-CIFAR, and SUN360 datasets and show that our method is competitive with other methods.

A Study on AI Literacy of University Students in Language and Language Education

“…To exclude the style information, we conduct such measurement in grayscale image format, shown in Tab. 6. We indicate such FID measurement as FID c .…”

“…To deal with such discontinuity and distortion, recent works introduce modeling in spherical domain [13,9], projecting an image to local tangent patches with minimal geometric error. It is proved that leveraging transformer architecture in 360°image modeling reduces distortions caused by projection and rotation [6]. For this reason, recent approaches [44,45] including PAVER [60], PanoFormer [48], and Text2Light [4] used the transformer achieving global structural consistency.…”

$\mathcal {S}^{2}$Net: Accurate Panorama Depth Estimation on Spherical Surface

“…We use Base to denote fusion module with direct addition, Fusion to denote fusion module with our CAF. The Base fusion module is only consisted of two simple residual convolution units from work [5] using basic spherical convolution method in previous works [14], [25].…”

“…F F N denotes feed-forward operation, and LN denotes layer norm operation [25], the two operations formulate a residuallike structure. Q i , K i and V i used in Equation 2 are generated from input X 0 and X 1 using corresponding learnable parameters W :…”

“…c) Spherical deep learning: most of these methods use dilated convolution or convolution kernel with special forms [21], [22], or compensate distortion using latitude and longitude [23], [24] in panorama image understanding tasks. There are two works that are similar to the proposed method: one work [25] utilizes the Icosahedron sampling method [26] and forms a novel transformer for representation learning of ERP images. Another work [14] utilizes HEALPix [16] as the sampling method and proposes a CNN method for this unique pixel representation.…”