Histopathology diagnosis is an important standard for breast tumors identifying. However, histopathology image analysis is complex, tedious and error-prone, due to the superresolution image. In recent years, deep learning technology has been successfully applied to histopathology image analysis and made great progress. The well-known deep neural networks usually have tens of million parameters, which consume much memory to deploy the state-of-the-art model. In addition, deep neural networks rely on high-performance hardware resources, which impede the deployment of state-of-the-art model on portable equipment. In this work, a novel framework which consists of a weight accumulation method and a lightweight fast neural network (FastNet) was proposed for tumor fast identification (TFI) in mobile computer-assisted devices. The weight accumulation method was designed to obtain the tissue mask regions of interest and remove the useless background area in histopathology images, which greatly reduces the redundant computation cost. Furthermore, we proposed the lightweight FastNet to improve the computational efficiency on mobile devices. A novel attention loss function was designed and applied in FastNet. The attention loss function pays more attention on the positive samples and the indistinguishable samples, which greatly improves performance. The proposed FastNet was compared with three state-of-the-art methods commonly used for image classification and object detection. Experimental results indicated that FastNet achieves highest recall of 96.94%, highest F1 score of 97.33% and highest accuracy of 97.34%, besides least trainable parameters of 0.22M and smallest floating point operations of 210M FLOPs.