In this paper, a two-stage method is proposed for predicting the catch of skipjack tuna (Katsuwonus pelamis) from a 2D sea environmental pattern. Following the assumption that sea water temperature and sea surface height (SSH) which fishermen often use for finding fishing spots has a correlation with the skipjack tuna catch, a new approach of using Faster R-CNN in object detection is proposed. The proposed method consists of two part. In the first part, taking a sea temperature map as input, Faster R-CNN extracts the candidates of where skipjack tuna would be on the map in order to imitate the behaviors of fishers. In the second part, Support Vector Regression (SVR) estimates the catch amount in each candidate. Faster R-CNN is applied to several sea environmental patterns with three different loss functions and compares each performance. The proposed model is evaluated by comparing the result with average fishers' ability on the skipjack tuna catches and several criteria for evaluating the proposed model. The results show that the proposed method is able to outperform the average fishers' ability by an average of 3%.