Homomorphic encryption (HE) method can be used to realize arithmetic operations on encrypted data. This method, however, is limited owing to its low efficiency in performing certain functions, especially those involving several multiplications. As a solution, this paper proposes a new HE-based secure computation scheme, termed as the HE for stochastic computing (HESC); this scheme can homomorphically evaluate both the stochastic addition and multiplication operations, without any bootstrapping. This HESC scheme is constructed based on additive/multiplicative HE, which only supports homomorphic addition/multiplication, and realizes the homomorphic evaluation of stochastic multiplication. The HESC employs the features of stochastic computing (SC) for homomorphic stochastic operations, where stochastic additions and multiplications are performed using random multiplexing and bit-parallel logic operations, respectively. This paper first presents a basic HESC scheme based on additive/multiplicative HE. It then presents an efficient HESC scheme that utilizes the parallelism of lattice-based cryptography (i.e., plaintext packing and vectorized homomorphic evaluation). A new stochastic addition operation is also introduced in this study, which can be used for the HESC instantiated by lattice-based cryptography. This new stochastic addition operation significantly improves the accuracy of the HESC, albeit with the trade-off of increased ciphertext size. Accordingly, this paper also proposes a technique that can reduce the size of ciphertexts, while maintaining the accuracy of the scheme. The basic performance of the HESC implemented with various HEs is demonstrated, along with its applications in polynomial functions and an oblivious inference with a neural network. Lastly, the results thus obtained indicate that the proposed scheme is more advantageous than the conventional schemes. This paper is concluded with some implications/research directions for HESC from perspectives of cryptography and HE implementations.