Evaluation Methods and Replicability of Software Architecture Research Objects

Konersmann, Marco; Kaplan, Angelika; Kühn, Thomas; Heinrich, Robert; Koziolek, Anne; Reussner, Ralf; Jürjens, Jan; al-Doori, Mahmood; Boltz, Nicolas; Marco, Ehl; Fuchs, Dominik; Groser, Katharina; Hahner, Sebastian; Keim, Jan; Lohr, Matthias; Saglam, Timur; Schulz, Sophie; Toberg, Jan-Philipp

doi:10.1109/icsa53651.2022.00023

Cited by 12 publications

(3 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To answer our last research question and investigate which methods and which datasets are used by the 52 collected approaches during their evaluation, we adapt the evaluation method taxonomy presented by Konersmann et al (2022) for the software architecture domain. In general, not all methods are applicable to the cognitive robotics domain.…”

Section: Evaluation Methods and Benchmarkingmentioning

confidence: 99%

Commonsense knowledge in cognitive robotics: a systematic literature review

Töberg,

Ngonga Ngomo,

Beetz

et al. 2024

Front. Robot. AI

View full text Add to dashboard Cite

One of the big challenges in robotics is the generalization necessary for performing unknown tasks in unknown environments on unknown objects. For us humans, this challenge is simplified by the commonsense knowledge we can access. For cognitive robotics, representing and acquiring commonsense knowledge is a relevant problem, so we perform a systematic literature review to investigate the current state of commonsense knowledge exploitation in cognitive robotics. For this review, we combine a keyword search on six search engines with a snowballing search on six related reviews, resulting in 2,048 distinct publications. After applying pre-defined inclusion and exclusion criteria, we analyse the remaining 52 publications. Our focus lies on the use cases and domains for which commonsense knowledge is employed, the commonsense aspects that are considered, the datasets/resources used as sources for commonsense knowledge and the methods for evaluating these approaches. Additionally, we discovered a divide in terminology between research from the knowledge representation and reasoning and the cognitive robotics community. This divide is investigated by looking at the extensive review performed by Zech et al. (The International Journal of Robotics Research, 2019, 38, 518–562), with whom we have no overlapping publications despite the similar goals.

show abstract

Section: Evaluation Methods and Benchmarkingmentioning

confidence: 99%

Commonsense knowledge in cognitive robotics: a systematic literature review

Töberg,

Ngonga Ngomo,

Beetz

et al. 2024

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

“…Goal G1 shall evaluate the accuracy of the analysis results of our approach, also compared with the different analysis types discussed in Section III. This represents a typical approach to design time analysis evaluation [26]. This includes evaluating the behavior of the analysis in different variants with and without uncertainty and confidentiality violations.…”

Section: A Evaluation Goals Questions and Metricsmentioning

confidence: 99%

“…To maximize construct validity, we used a Goal Question Metric plan [2] and oriented our evaluation plan to similar approaches [7], [45]. Konersmann et al [26] state the lack of replication packages and the availability of tools used for the evaluation. To overcome this limitation and to increase reliability, we published a data set containing all evaluation data [3].…”

Section: Threats To Validitymentioning

confidence: 99%

Model-based Confidentiality Analysis under Uncertainty

Hahner

Bitschi

Walter

et al. 2023

2023 IEEE 20th International Conference on Software Architecture Companion (ICSA-C)

Self Cite

View full text Add to dashboard Cite

In our connected world, ensuring the confidentiality of the software systems we build becomes increasingly difficult. Model-based design time confidentiality analyses have been proposed to cope with this complexity early. However, the usefulness of such analyses is limited due to uncertainty about the software architecture itself and the software's execution environment. This leads to conclusions about confidentiality violations that lack both precision and comprehensiveness. Although there exist approaches to deal with design time uncertainty, existing research lacks precise statements about the impact of uncertainty on confidentiality. To address this, we include uncertainty as part of our software architectural model. We extend a data flow-based analysis to include the impact of uncertainty on confidentiality violations. The results of the case study-based evaluation show high accuracy with typical design time uncertainty. Also, our analysis yields more precise statements about the impact of uncertainty on confidentiality than the state of the art.

show abstract