Within an Internet of Multimedia Things, the risk of disclosing streamed video content, such as arising from video surveillance, is of heightened concern. This leads to the encryption of that content. To reduce the overhead and lack of flexibility arising from full encryption of the content, a good number of selective-encryption algorithms have been proposed in the last decade. Some of them have limitations, in terms of significant delay due to computational cost, or excess memory utilization, or, despite being energy efficient, do not provide a satisfactory level of confidentiality, due to their simplicity. To address such issues, this paper presents a lightweight selective encryption scheme, in which encoder syntax elements are encrypted with the innovative EXPer (EXtended Permutation with exclusive OR). The selected syntax elements are taken from the final stage of video encoding that is during entropy coding. As a diagnostic tool, the Encryption Space Ratio measures encoding complexity of the video relative to the level of encryption so as to judge the success of the encryption process, according to entropy coder. A detailed comparative analysis of EXPer with state-of-the-art algorithms confirms that the EXPer provides significant confidentiality with a small computational cost and negligible encryption bitrate overhead. Thus, the results demonstrate that the proposed security scheme is a suitable choice for constrained devices in an Internet of Multimedia Things environment.