VEER: Enhancing the Interpretability of Model-based Optimizations

Abstract

Software comes with many configuration options, satisfying varying needs from users. Exploring those options for non-functional requirements can be tedious, time consuming, and even error-prone (if done manually). Worse, many software systems can be tuned to multiple objectives (e.g., faster response time, fewer memory requirements, decreased network traffic, decreased energy consumption, etc.). Learning how to adjust the system among these multiple objectives is complicated due to the trade-off among objectives; i.e., things that seem useful to achieve one objective could be detrimental to another objective. Consequentially, the optimizer built for one objective may have different (or even opposite) insights on how to locate good solutions from the optimizer built from another objective. In this paper, we define this scenario as the model disagreement problem.One possible solution to this problem is to find a one-dimensional approximation to the N-objective space. In this way, the case is converted to a single-objective optimization, which is naturally confusion-free. This paper demonstrates VEER, a tool that builds such an approximation by combining our dimensionality-reduction heuristic on top of one of the state-of-the-art optimizers, FLASH. VEER can explore very large configuration spaces by evaluating just a small fraction of the total number of configurations (e.g., a space of 81,000 configurations can be explored by 70 samples).The experimental result in this paper demonstrates the feasibility of our approach in terms of the on-par quality of the solution set generated by the optimizer, and the resolved model disagreement within the optimizer. Moreover, we demonstrate that VEER has an improved computational complexity compared to the original optimizer (by up to 1,000 times faster while maintaining on-par performance).

Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes

“…Our study covers seven widely used machine learning models for learning software performance, i.e., Decision Tree (DT) [46] (used by [4,8,25,41]), 𝑘-Nearest Neighbours (𝑘NN) [21] (used by [35]), Kernel Ridge Regression (KRR) [52] (used by [35]), Linear Regression (LR) [23] (used by [4,8,49]), Neural Network (NN) [53] (used by [20,26]), Random Forest (RF) [30] (used by [45,50]), and Support Vector Regression (SVR) [17] (used by [4,50]), together with five popular real-world software systems from prior work [15,16,41,44], covering a wide spectrum of characteristics and domains. Naturally, the first research question (RQ) we ask is: RQ1: Is it practical to examine all encoding methods for finding the best one under every system?…”

“…We shortlisted systems and their data from recent studies on software configuration tuning and modeling [41,44], from which we identified five systems and their environment according to the above criteria, as shown in Table 2. The five systems contain different percentages of categorical/binary and numeric configuration options while covering five distinct domains.…”