Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software

Beller, Moritz; Bholanath, Radjino; McIntosh, Shane; Zaidman, Andy

doi:10.1109/saner.2016.105

Cited by 155 publications

(155 citation statements)

References 32 publications

Supporting

Mentioning

144

Contrasting

Unclassified

Order By: Relevance

“…Compared to about 60% of state-of-the-art GITHUB projects using ASATs [7] and some 50% of projects in general doing testing [16], a number of reasons might hinder an even more-widespread use of TRAVIS CI: Famous GITHUB projects such as SCALA/S-CALA 9 often run their own CI server (see Section 2.1). 10 This exemplifies that from the 30% adoption rate, it does not follow that 70% of projects do not use CI.…”

Section: Resultsmentioning

confidence: 99%

“…While both TRAVISPOKER and TRAVISHARVESTER utilize TRAVIS CI's Ruby client for querying the API, 6 we cannot use its job log retrieval function (Job:log) due to a memory leak 7 and because it does not retrieve all build logs. We circumvented these problems by also querying an undocumented Amazon Cloud server we discovered that archives build logs.…”

Section: Toolsmentioning

confidence: 99%

“…A full CI build comprises 1) a traditional build and compile phase, 2) a phase in which automated static analysis tools (ASATs) such as FINDBUGS and JSHINT are run [6,7], and 3) a testing phase, in which unit, integration, and system tests are run. If any of these three phases fails, the whole CI build is typically aborted and regarded as broken [8].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Oops, my tests broke the build: An analysis of Travis CI builds with GitHub

Beller

Gousios

Zaidman

2016

Preprint

Self Cite

View full text Add to dashboard Cite

Continuous Integration (CI) has become a best practice of modern software development.At present, we have a shortfall of insight into the testing practices that are common in CIbased software development. In particular, we seek quantifiable evidence on how central testing really is in CI, how strongly the project language influences testing, whether different integration environments are valuable and if testing on the CI can serve as a surrogate to local testing in the IDE. In an analysis of 2,640,825 Java and Ruby builds on Travis CI, we find that testing is the single most important reason why builds fail.Moreover, the programming language has a strong influence on both the number of executed tests, their test run time and proneness to fail. The use of multiple integration environments leads to 10% more failures being caught at build time. However, testing in the CI does not seem to be a good surrogate for running tests in the IDE.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Toolsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Oops, my tests broke the build: An analysis of Travis CI builds with GitHub

Beller

Gousios

Zaidman

2016

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Analysis Tools (ASATs) analyze source or binary code without observing its run time behavior [1]. ASATs have become an integral part of today's software quality assurance practices, reflected by the increased uptake of new ASATs such as Google's Error Prone 1 or Facebook's Infer.…”

Section: Introduction Automated Staticmentioning

confidence: 99%

“…ASATs have become an integral part of today's software quality assurance practices, reflected by the increased uptake of new ASATs such as Google's Error Prone 1 or Facebook's Infer. 2 In addition to the well-known standalone ASATs like FindBugs, Checkstyle, or PMD [1], recently, new cloud services like CodeClimate 3 have emerged. They try to provide a better integration of ASATs into the development process.…”

Section: Introduction Automated Staticmentioning

confidence: 99%

UAV: Warnings from multiple Automated Static Analysis Tools at a glance

Buckers

Cao

Doesburg

et al. 2017

2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER)

Self Cite

View full text Add to dashboard Cite

Abstract-Automated Static Analysis Tools (ASATs) are an integral part of today's software quality assurance practices. At present, a plethora of ASATs exist, each with different strengths. However, there is little guidance for developers on which of these ASATs to choose and combine for a project. As a result, many projects still only employ one ASAT with practically no customization. With UAV, the Unified ASAT Visualizer, we created an intuitive visualization that enables developers, researchers, and tool creators to compare the complementary strengths and overlaps of different Java ASATs. UAV's enriched treemap and source code views provide its users with a seamless exploration of the warning distribution from a high-level overview down to the source code. We have evaluated our UAV prototype in a user study with ten second-year Computer Science (CS) students, a visualization expert and tested it on large Java repositories with several thousands of PMD, FindBugs, and Checkstyle warnings. Project Website: https://clintoncao.github.io/uav/

show abstract

Assessing code readability in Python programming courses using eye‐tracking

Segedinac,

Savić,

Zeljković

et al. 2023

Comp Applic In Engineering

View full text Add to dashboard Cite

Code readability models are typically based on the code's structural and textual features, considering code readability as an objective category. However, readability is inherently subjective and dependent on the knowledge and experience of the reader analyzing the code. This paper assesses the readability of Python code statements commonly used in undergraduate programming courses. Our readability model is based on tracking the reader's eye movement during the while‐read phase. It uses machine learning (ML) techniques and relies on a novel set of features—observational features—that capture how the readers read the code. We experimented by tracking the eye movement of 90 undergraduate students while assessing the readability of 48 Python code snippets. We trained an ML model that predicts readability based on the collected observational data and the code snippet's structural and textual features. In our experiments, the XGBoost classifier trained using observational features exclusively achieved the best results (0.85 F‐measure). Using correlation analysis, we identified Python statements most affecting readability for undergraduate students and proposed implications for teaching Python programming. In line with findings for Java language, we found that constructs related to the code's size and complexity hurt the code's readability. Numerous comments also hindered readability, potentially due to their association with less readable code. Some Python‐specific statements (list comprehension, lambda function, and dictionary comprehension) harmed code readability, even though they were part of the curriculum. Tracking students' gaze indicated some additional factors, most notably nonlinearity introduced by if, for, while, try, and function call statements.

show abstract

Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software

Cited by 155 publications

References 32 publications

Oops, my tests broke the build: An analysis of Travis CI builds with GitHub

Oops, my tests broke the build: An analysis of Travis CI builds with GitHub

UAV: Warnings from multiple Automated Static Analysis Tools at a glance

Assessing code readability in Python programming courses using eye‐tracking

Contact Info

Product

Resources

About