Modeling the mass distributions of strong gravitational lenses is often necessary in order to use them as astrophysical and cosmological probes. With the large number of lens systems (≳105) expected from upcoming surveys, it is timely to explore efficient modeling approaches beyond traditional Markov chain Monte Carlo techniques that are time consuming. We train a convolutional neural network (CNN) on images of galaxy-scale lens systems to predict the five parameters of the singular isothermal ellipsoid (SIE) mass model (lens center x and y, complex ellipticity ex and ey, and Einstein radius θE). To train the network we simulate images based on real observations from the Hyper Suprime-Cam Survey for the lens galaxies and from the Hubble Ultra Deep Field as lensed galaxies. We tested different network architectures and the effect of different data sets, such as using only double or quad systems defined based on the source center and using different input distributions of θE. We find that the CNN performs well, and with the network trained on both doubles and quads with a uniform distribution of θE > 0.5″ we obtain the following median values with 1σ scatter: Δx = (0.00−0.30+0.30)″, Δy = (0.00−0.29+0.30)″, ΔθE = (0.07−0.12+0.29)″, Δex = −0.01−0.09+0.08, and Δey = 0.00−0.09+0.08. The bias in θE is driven by systems with small θE. Therefore, when we further predict the multiple lensed image positions and time-delays based on the network output, we apply the network to the sample limited to θE > 0.8″. In this case the offset between the predicted and input lensed image positions is (0.00−0.29+0.29)″ and (0.00−0.31+0.32)″ for the x and y coordinates, respectively. For the fractional difference between the predicted and true time-delay, we obtain 0.04−0.05+0.27. Our CNN model is able to predict the SIE parameter values in fractions of a second on a single CPU, and with the output we can predict the image positions and time-delays in an automated way, such that we are able to process efficiently the huge amount of expected galaxy-scale lens detections in the near future.