Blockchain has many benefits including decentralization, availability, persistency, consistency, anonymity, auditability and accountability, and it also covers a wide spectrum of applications ranging from cryptocurrency, financial services, reputation system, Internet of Things, sharing economy to public and social services. Not only may blockchain be regarded as a by-product of Bitcoin cryptocurrency systems, but also it is a type of distributed ledger technologies through using a trustworthy, decentralized log of totally ordered transactions. By summarizing the literature of blockchain, it is found that more and more important research is to develop basic theory, for example, mathematical models (Markov processes, queueing theory and game models) for mining management and consensus mechanism, performance analysis and optimization of blockchain systems. In this paper, we develop queueing theory of blockchain systems and provide system performance evaluation. To do this, we design a Markovian batch-service queueing system with two different service stages, which are suitable to well express the mining process in the miners pool and the building of a new blockchain. By using the matrix-geometric solution, we obtain a system stable condition and express three key performance measures: (a) The average number of transactions in the queue, (b) the average number of transactions in a block, and (c) the average transaction-confirmation time. Finally, we use numerical examples to verify computability of our theoretical results. Although our queueing model here is simple only under exponential or Poisson assumptions, our analytic method will open a series of potentially promising research in queueing theory of blockchain systems.