This paper describes a software architecture designed to enable the evaluation of information processing and retrieval systems. The overall objective of our project is to provide an open technical framework for the integration of tools for collection, processing, analysis and communication of open source information 1 . However, enabling the integration of heterogeneous components does not make sense without a proper way to compare the capabilities of multiple tools.Thus, as part of the architecture the VIRTUOSO platform offers an evaluation framework which allows one to deploy and run evaluation kits for different use-cases.