Motivated by the recent experiment [H. Kadau et al., arXiv:1508.05007 (2015], we study roton instability and droplet formation in a Bose-Einstein condensate of 164 Dy atoms with strong magnetic dipole-dipole interaction. We numerically solve the cubic-quintic Gross-Pitaevskii equation with dipole-dipole interaction, and show that the three-body interaction plays a significant role in the formation of droplet patterns. We numerically demonstrate the formation of droplet patterns and crystalline structures, decay of droplets, and hysteresis behavior, which are in good agreement with the experiment. Our numerical simulations provide the first prediction on the values of the three-body interaction in a 164 Dy Bose-Einstein condensate. We also predict that the droplets remain stable during the time-of-flight expansion. From our results, further experiments investigating the three-body interaction in dipolar quantum gases are required. Dipolar Bose-Einstein condensates (BECs) of atoms with large magnetic dipole moments, such as chromium [1], dysprosium [2], and erbium [3], are systems in which the longrange and anisotropic dipole-dipole interaction strongly affects their static and dynamic properties. The researches on such a dipolar system both in theories and in experiments are driven by the search for new novel phases in condensed matter physics. Structured ground states and roton excitation spectrum in a pancake-shaped trap have been studied [4][5][6][7][8][9][10]. Anisotropic expansion [11] and collapsing instability [12][13][14] have been observed in a chromium BEC. Increasing attention has also been focusing on bright solitons [15,16], anisotropic superfluidity [17,18], Faraday patterns [19], and multicomponent BECs [20][21][22][23]. A binary BEC with a strong dipole-dipole interaction exhibits instability and forms patterns similar to those in magnetic liquids, such as hexagonal, soliton-like, and labyrinthine patterns [20]. Droplet formation has also been investigated in dipolar atomic systems [24,25].In the recent experiment reported in Ref.[26], interactioninduced periodic patterns spontaneously formed in a BEC of dysprosium atoms. By using Feshbach resonance to control the ratio between the s-wave and dipole-dipole interactions, they observed discrete droplet patterns arranged in a long-lived triangular lattice. This result indicates possibility that the system possesses a stable periodic state with matterwave coherence, which is therefore a candidate of supersolidity [27][28][29]. Before exploring this possiblity, a theoretical understanding of the experimental results in Ref.[26] is required.In this Rapid Communication, we propose a theoretical model to explain the experimental results in Ref. [26]. One finds that the standard mean-field Gross-Pitaevskii model with dipole-dipole interaction cannot reproduce the experimental results; the strong Roton instability is always followed by the d-wave dipolar collapse, which hinders the droplet formation. To circumvent this problem, we propose to include the th...