The trend of cloud databases is leaning towards Not Only SQL (NoSQL) databases as they provide better support for scalable storage and quick retrieval of exponentially voluminous data. One of the more prominent types of NoSQL databases is document-based storage, which is being increasingly used in the dynamic cloud paradigm. However, there are inherent security issues in cloud, including remote data residency along with the non-existent control of owners over their own data. In addition to that, the inherent security features of most document-based NoSQL databases lack granular access control and robust confidentiality mechanisms. There is also a distinct lack of a comprehensive solution that effectively caters to all the security requirements of a document-oriented database in cloud. In order to overcome these issues, we propose a database security-as-a-service (DB-SECaaS) system over document-oriented database hosted in cloud, which provides authentication, fine-grained authorization, and encryption of the database objects, while ensuring that access to the data is granted only to authorized users on a need-to-know basis. The paper shows that the DB-SECaaS system strongly enhances the security of documentoriented databases on cloud, and it is thus expected to facilitate the industry to reap the benefits of NoSQL without worrying over security issues. In order to certify the abovementioned security enhancements, provided by DB-SECaaS, the paper also provides a formal analysis of DB-SECaaS using the Scyther model checker. As a proof of concept, the core functionalities of the protocol, i.e., authorization, authentication, and encryption, are formally modeled in Scyther to formally verify that the proposed framework mitigates privacy and security concerns.