No abstract

In this paper, we specify a generic ontology for the process domain considering the related state-of-the-art research literature. As a result, the recently built process ontology contributes to enrich semantically the terms for the (previously developed) measurement and evaluation domain ontology by means of stereotypes. One of the underlying hypothesis in this research is that the generic ontology for process can be seen as a reusable artifact which can be used to enrich semantically not only the measurement and evaluation domain ontology but also to other domains involved in different organizational endeavors. For instance, for the measurement domain, now is explicit that the measurement term has the semantic of task, the measure term has the meaning of outcome, and the metric term has the semantic of method, from the process terminological base standpoint. The augmented conceptual framework, i.e. measurement and evaluation concepts plus process concepts, has also a positive impact on the GOCAME (Goal-Oriented Context-Aware Measurement and Evaluation) strategy capabilities since ensures terminological uniformity, consistency and verifiability to its process and method specifications. In order to illustrate how the augmented conceptual framework impacts on the verifiability of GOCAME process and method specifications in addition to the consistency and comparability of results in measurement and evaluation projects, an ICT (Information and Communications Technology) security and risk evaluation case study is used.

For evaluating web applications (WebApps) and other entities in a systematic way and further fostering robust data analysis among projects, measurement and evaluation (M&E) strategies are a valuable asset in any organization. However, the evaluation of the quality of the capabilities for an integrated M&E strategy-seen as a resource-has often been neglected. We regard a M&E strategy is integrated if three coexisting capabilities are supported, namely: i) a conceptual framework, ii) a well-established process specification, and iii) an explicit methodological support. Under this premise, we conducted a case study where GQM + Strategies (Goal-Question-Metric), and GOCAME (Goal-Oriented Context-Aware Measurement and Evaluation) strategies were evaluated. The results allowed us to understand the strengths and weaknesses of both strategies. From this understanding we have planned improvement actions and implemented some changes in GOCAME as well. This paper ultimately analyzes the achieved gains after recommended changes were performed.

Organizations should establish business goals and check for their achievement in a systematic and disciplined way. In order to know if a business goal is achieved, it should be necessary to consider information need goals that also can require satisfying measurement and evaluation goals at operational level. Furthermore, if measurement and evaluation goals are not aligned with top-level business goals such as tactical or strategic level goals, the organization could waste its effort and resources. Usually, the different goals established in an organization are operationalized through projects. For a given project, strategies should be used in order to help in the goal achievement. A strategy defines a set of activities and methods to be followed for a specific goal purpose. Ultimately, to engineering all these issues in a systematic way, organizations should adopt a holistic evaluation approach supported by a set of integrated strategies. By means of a systematic literature review as research method, we have observed that very few approaches support integrated strategies and multilevel goals. To bridge this gap, we have developed a holistic quality multilevel and multipurpose evaluation approach that ties together multilevel goals, projects and integrated strategies. As contributions, this paper discusses an enhanced conceptual base (specified by ontologies) for linking business and information need goal concepts with project, strategy and nonfunctional requirements concepts. Then, it defines the step by step of our holistic quality evaluation approach, by listing the necessary activities to establish goals and projects at different organizational levels. Lastly, it specifies and illustrates evaluation scenarios for business/information need goal purposes such as understanding, improving, monitoring and controlling, comparing and selecting entities, which are supported by strategies and strategy patterns.

No abstract