Supporting Many-Objective Software Requirements Decision: An Exploratory Study on the Next Release Problem

Geng, Jiangyi; Song, Ying; Jia, Xiangyang; Zhang, Ting; Luo, Xin; Xuan, Jifeng

doi:10.1109/access.2018.2875122

Cited by 18 publications

(10 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some attention has been given to using many-objective techniques in other areas. For example, there is a recent exploratory study on using several evolutionary manyobjective optimisation approaches, including NSGA-III for the Next Release Problem [64]. Another example approach [65] looks at optimal feature selection by first optimising on one objective and then trying to improve others.…”

Section: Resultsmentioning

confidence: 99%

Using a many-objective approach to investigate automated refactoring

Mohan

Greer

2019

Information and Software Technology

View full text Add to dashboard Cite

Context: Software maintenance is expensive and so anything that can be done to reduce its cost is potentially of huge benefit. However, it is recognised that some maintenance, especially refactoring, can be automated. Given the number of possible refactorings and combinations of refactorings, a search-based approach may provide the means to optimise refactorings. Objective: This paper describes the investigation of a many-objective genetic algorithm used to automate software refactoring, implemented as a Java tool, MultiRefactor. Method: The approach and tool is evaluated using a set of open source Java programs. The tool contains four separate measures of software looking at the software quality as well as measures of code priority, refactoring coverage and element recentness. The many-objective algorithm combines the four objectives to improve the software in a holistic manner. An experiment has been constructed to compare the many-objective approach against a mono-objective approach that only uses a single objective to measure software quality. Different permutations of the objectives are also tested and compared to see how well the different objectives can work together in a multi-objective refactoring approach. The eight approaches are tested on six different open source Java programs. Results: The many-objective approach is found to give better objective scores on average than the mono-objective approach and in less time. However, the priority and element recentness objectives are both found to be less successful in multi/many-objective setups when they are used together. Conclusion: A many-objective approach is suitable and effective for optimising automated refactoring to improve quality. Including other objectives does not unduly degrade the quality improvements, but is less effective for those objectives than if they were used in a mono-objective approach.

show abstract

Section: Resultsmentioning

confidence: 99%

Using a many-objective approach to investigate automated refactoring

Mohan

Greer

2019

Information and Software Technology

View full text Add to dashboard Cite

show abstract

“…Another issue is metrics preference among researchers. We observed that some of the studies justify the reason they employ these metrics as either based on a metric's popularity (i.e., usage or related work in which it was used) or if it best fits their choice of algorithm (i.e., References [6,86,87]), while some other studies adopted some of these metrics because they are hybrids. For example, HV can cover both convergence and diversity [87], while some others avoided using more metrics because that might have led to different conclusions or threatened the validity of their results [122].…”

Section: Discussionmentioning

confidence: 99%

“…However, in the available literature, there appears to be no agreement on the number of objectives we call 'multi' or 'many' [6], and the focuses of these terminologies may create confusion.…”

Section: Introductionmentioning

confidence: 99%

Performance Evaluation Metrics for Multi-Objective Evolutionary Algorithms in Search-Based Software Engineering: Systematic Literature Review

et al. 2021

View full text Add to dashboard Cite

Many recent studies have shown that various multi-objective evolutionary algorithms have been widely applied in the field of search-based software engineering (SBSE) for optimal solutions. Most of them either focused on solving newly re-formulated problems or on proposing new approaches, while a number of studies performed reviews and comparative studies on the performance of proposed algorithms. To evaluate such performance, it is necessary to consider a number of performance metrics that play important roles during the evaluation and comparison of investigated algorithms based on their best-simulated results. While there are hundreds of performance metrics in the literature that can quantify in performing such tasks, there is a lack of systematic review conducted to provide evidence of using these performance metrics, particularly in the software engineering problem domain. In this paper, we aimed to review and quantify the type of performance metrics, number of objectives, and applied areas in software engineering that reported in primary studies—this will eventually lead to inspiring the SBSE community to further explore such approaches in depth. To perform this task, a formal systematic review protocol was applied for planning, searching, and extracting the desired elements from the studies. After considering all the relevant inclusion and exclusion criteria for the searching process, 105 relevant articles were identified from the targeted online databases as scientific evidence to answer the eight research questions. The preliminary results show that remarkable studies were reported without considering performance metrics for the purpose of algorithm evaluation. Based on the 27 performance metrics that were identified, hypervolume, inverted generational distance, generational distance, and hypercube-based diversity metrics appear to be widely adopted in most of the studies in software requirements engineering, software design, software project management, software testing, and software verification. Additionally, there are increasing interest in the community in re-formulating many objective problems with more than three objectives, yet, currently are dominated in re-formulating two to three objectives.

show abstract

“…Here, the search problem is what requirements should be fulfilled for the next release such that some goals, e.g., importance and cost, are optimal. The NRP problem has been widely studied in SBSE [35,60,105,106], from which we choose 15 releases dataset that was mined from real-world software projects and their versions. As shown in Table 4, each software project/version involves a set of randomly sampled requirements to fulfill.…”

Section: Next Release Planning (Nrp)mentioning

confidence: 99%

“…As shown in Table 4, we use two or three common objectives for all software projects/versions, namely penalty score, cost, and coverage, which have the objective functions as below [3,35,60,105,106]: where there are 𝑛 requirements and 𝑚 stakeholders. 𝐼 𝑗 , 𝐶 𝑖 , and 𝑅 𝑖 are respectively the satisfaction level of the 𝑖th requirement from the 𝑗th stakeholder, the related cost for fulfilling the 𝑖th requirement 𝑟 𝑖 , and the ratio of fulfilled requirement for 𝑟 𝑖 .…”

Section: Objectivementioning

confidence: 99%

The Weights can be Harmful: Pareto Search versus Weighted Search in Multi-Objective Search-Based Software Engineering

Chen,

2022

Preprint

View full text Add to dashboard Cite

In presence of multiple objectives to be optimized in Search-Based Software Engineering (SBSE), Pareto search has been commonly adopted. It searches for a good approximation of the problem's Pareto optimal solutions, from which the stakeholders choose the most preferred solution according to their preferences. However, when clear preferences of the stakeholders (e.g., a set of weights which reflect relative importance between objectives) are available prior to the search, weighted search is believed to be the first choice since it simplifies the search via converting the original multi-objective problem into a single-objective one and enable the search to focus on what only the stakeholders are interested in.This paper questions such a "weighted search first" belief. We show that the weights can, in fact, be harmful to the search process even in the presence of clear preferences. Specifically, we conduct a large scale empirical study which consists of 38 systems/projects from three representative SBSE problems, together with two types of search budget and nine sets of weights, leading to 604 cases of comparisons. Our key finding is that weighted search reaches a certain level of solution quality by consuming relatively less resources at the early stage of the search; however, Pareto search is at the majority of the time (up to 77% of the cases) significantly better than its weighted counterpart, as long as we allow a sufficient, but not unrealistic search budget. This is a beneficial result, as it discovers a potentially new "rule-of-thumb" for the SBSE community: even when clear preferences are available, it is recommended to always consider Pareto search by default for multi-objective SBSE problems provided that solution quality is more important. Weighted search, in contrast, should only be preferred when the resource/search budget is limited, especially for expensive SBSE problems. This, together with other findings and actionable suggestions in the paper, allows us to codify pragmatic and comprehensive guidance on choosing weighted and Pareto search for SBSE under the circumstance that clear preferences are available. All code and data can be accessed at: https://github.com/ideas-labo/pareto-vs-weight-for-sbse.

show abstract

Supporting Many-Objective Software Requirements Decision: An Exploratory Study on the Next Release Problem

Cited by 18 publications

References 37 publications

Using a many-objective approach to investigate automated refactoring

Using a many-objective approach to investigate automated refactoring

Performance Evaluation Metrics for Multi-Objective Evolutionary Algorithms in Search-Based Software Engineering: Systematic Literature Review

The Weights can be Harmful: Pareto Search versus Weighted Search in Multi-Objective Search-Based Software Engineering

Contact Info

Product

Resources

About