Portable low-cost magnetic resonance imaging (MRI) systems have the potential to enable "point-of-care" and timely MRI diagnosis, and to make this imaging modality available to routine scans and to people in underdeveloped countries and areas. With simplicity, no maintenance, no power consumption, and low cost, permanent magnets/magnet arrays/magnet assemblies are attractive to be used as a source of static magnetic field to realize the portability and to lower the cost for an MRI scanner. However, when taking the canonical Fourier imaging approach and using linear gradient fields, homogeneous fields are required in a scanner, resulting in the facts that either a bulky magnet/magnet array is needed, or the imaging volume is too small to image an organ if the magnet/magnet array is scaled down to a portable size. Recently, with the progress on image reconstruction based on nonlinear gradient field, static field patterns without spatial linearity can be used as spatial encoding magnetic fields (SEMs) to encode MRI signals for imaging. As a result, the requirements for the homogeneity of the static field can be relaxed, which allows permanent magnets/magnet arrays with reduced sizes, reduced weight to image a bigger volume covering organs such as a head. It offers opportunities of constructing a truly portable lowcost MRI scanner. For this exciting potential application, permanent magnets/magnet arrays have attracted increased attention recently. A magnet/magnet array is strongly associated with the imaging volume of an MRI scanner, image reconstruction methods, and RF excitation and RF coils, etc. through field patterns and field homogeneity. This paper offers a review of permanent magnets and magnet arrays of different kinds, especially those that can be used for spatial encoding towards the development of a portable and low-cost MRI system. It is aimed to familiarize the readers with relevant knowledge, literature, and the latest updates of the development on permanent magnets and magnet arrays for MRI. Perspectives on and challenges of using a permanent magnet/magnet array to supply a patterned static magnetic field, which does not have spatial linearity nor high field homogeneity, for image reconstruction in a portable setup are discussed.