Deployment of smart card-based applications is becoming prevalent these days. Security of these applications depends upon the nature of underlying hardware and software platforms, cryptographic algorithms, along with a number of other security mechanisms. To develop a secure smart card-based application, preliminary evaluation of the security of embedded software as well as how that software will work in real time is required. Over the past few years, significant improvements have been made in smart card technology for its large-scale acceptance. However, attackers are in continuous search for exploiting the unknown issues related to this technology. This survey discusses smart card technology in detail and provides a comprehensive taxonomy of security attacks on smart card-based applications and their countermeasures. In addition, it covers some of the open source tools available for its development. Finally, it provides a brief outline of some of the major open challenges and future research directions that are yet to be explored.