All current recommendation algorithms, when modeling user–item interactions, basically use dot product. This dot product calculation is derived from matrix factorization. We argue that an inherent drawback of matrix factorization is that latent semantic vectors of users or items sometimes do not satisfy triangular inequalities, which may affect the performance of the recommendation. Recently, metric factorization was proposed to replace matrix factorization and has achieved some improvements in terms of recommendation accuracy. However, similar to matrix factorization, metric factorization still uses a simple, linear fashion. In this paper, we explore leveraging nonlinear deep neural networks to realize Euclidean distance interaction between users and items. We propose a generic Neural Metric Factorization Framework (NMetricF), which learns representations for users and items by incorporating Euclidean metric factorization into deep neural networks. Extensive experiments on six real-world datasets show that, compared to the previous recommendation algorithms based purely on rating data, NMetricF achieves the best performance.