Topological spin textures have attracted much attention both for fundamental physics and spintronics applications. Among them, antiskyrmions possess a unique spin configuration with Bloch-type and Néel-type domain walls due to anisotropic Dzyaloshinskii-Moriya interaction (DMI) in the noncentrosymmetric crystal structure. However, antiskyrmions have thus far only been observed in a few Heusler compounds with D2d symmetry. Here, we report a new material Fe1.9Ni0.9Pd0.2P in a different symmetry class (S4 symmetry), where antiskyrmions exist over a wide temperature region including room temperature, and transform to skyrmions upon changing magnetic field and lamella thickness. The periodicity of magnetic textures greatly depends on crystal thickness, and domains with anisotropic sawtooth fractals are observed at the surface of thick crystals, which are attributed to the interplay between dipolar interaction and DMI as governed by