The Internet of Things (IoT) has provided substantial enhancements to the communication of sensors, actuators, and their controllers, particularly in the field of home automation. Home automation is experiencing a huge rise in the proliferation of IoT devices such as smart bulbs, smart switches, and control gateways. However, the main challenge for such control systems is how to maximize security under limited resources such as low-processing power, low memory, low data rate, and low-bandwidth IoT networks. In order to address this challenge the adoption of IoT devices in automation has mandated the adoption of secure communication protocols to ensure that compromised key security objectives, such as confidentiality, integrity, and availability are addressed. In light of this, this work evaluates the feasibility of MQTT-based Denial of Service (DoS) attacks, Man-in-the-Middle (MitM), and masquerade attacks on a ZigBee network, an IoT standard used in wireless mesh networks. Performed through MQTT, the attacks extend to compromise neighboring Constrained Application Protocol (CoAP) nodes, a specialized service layer protocol for resource-constrained Internet devices. By demonstrating the attacks on an IKEA TRÅDFRI lighting system, the impact of exploiting ZigBee keys, the basis of ZigBee security, is shown. The reduction of vulnerabilities to prevent attacks is imperative for application developers in this domain. Two Intrusion Detection Systems (IDSs) are proposed to mitigate against the proposed attacks, followed by recommendations for solution providers to improve IoT firmware security. The main motivation and purpose of this work is to demonstrate that conventional attacks are feasible and practical in commercial home automation IoT devices, regardless of the manufacturer. Thus, the contribution to the state-of-the-art is the design of attacks that demonstrate how known vulnerabilities can be exploited in commercial IoT devices for the purpose of motivating manufacturers to produce IoT systems with improved security.