"Get in Researchers; We're Measuring Reproducibility": A Reproducibility Study of Machine Learning Papers in Tier 1 Security Conferences

Olszewski, Daniel; Lu, Allison; Stillman, Carson; Warren, Kevin; Kitroser, Cole; Pascual, Alejandro; Ukirde, Divyajyoti; Butler, Kevin; Traynor, Patrick

doi:10.1145/3576915.3623130

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Other3

Article2

Relationship

Self Cite0

Independent5

Authors

Journals

Cited by 5 publications

References 255 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

NeuroIDBench: An open-source benchmark framework for the standardization of methodology in brainwave-based authentication research

Chaurasia,

Fallahi,

Strufe

et al. 2024

Journal of Information Security and Applications

View full text Add to dashboard Cite

NeuroIDBench: An open-source benchmark framework for the standardization of methodology in brainwave-based authentication research

Chaurasia,

Fallahi,

Strufe

et al. 2024

Journal of Information Security and Applications

View full text Add to dashboard Cite

The Missing Link in Network Intrusion Detection: Taking AI/ML Research Efforts to Users

Dietz,

Mühlhauser,

Kögel

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Intrusion Detection Systems (IDS) tackle the challenging task of detecting network attacks as fast as possible. As this is getting more complex in modern enterprise networks, Artificial Intelligence (AI) and Machine Learning (ML) have gained substantial popularity in research. However, their adoption into real-world IDS solutions remains poor. Academic research often overlooks the interconnection of users and technical aspects. This leads to less explainable AI/ML models that hinder trust among AI/ML non-experts. Additionally, research often neglects secondary concerns such as usability and privacy. If IDS approaches conflict with current regulations or if administrators cannot deal with attacks more effectively, enterprises will not adopt the IDS in practice. To identify those problems systematically, our literature survey takes a user-centric approach; we examine IDS research from the perspective of stakeholders by applying the concept of personas. Further, we investigate multiple factors limiting the adoption of AI/ML in security and suggest technical, non-technical, and user-related considerations to enhance the adoption in practice. Our key contributions are threefold. (i) We derive personas from realistic enterprise scenarios, (ii) we provide a set of relevant hypotheses in the form of a review template, and (iii), based on our reviews, we derive design guidelines for practical implementations. To the best of our knowledge, this is the first paper that analyzes practical adoption barriers of AI/ML-based intrusion detection solutions concerning appropriateness of data, reproducibility, explainability, practicability, usability, and privacy. Our guidelines may help researchers to holistically evaluate their AI/ML-based IDS approaches to increase practical adoption.

show abstract

Revisiting Temporal Inconsistency and Feature Extraction for Android Malware Detection

Tanha,

Singh,

Knoke

2024

2024 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE)

View full text Add to dashboard Cite

"Get in Researchers; We're Measuring Reproducibility": A Reproducibility Study of Machine Learning Papers in Tier 1 Security Conferences

Cited by 5 publications

References 255 publications

NeuroIDBench: An open-source benchmark framework for the standardization of methodology in brainwave-based authentication research

NeuroIDBench: An open-source benchmark framework for the standardization of methodology in brainwave-based authentication research

The Missing Link in Network Intrusion Detection: Taking AI/ML Research Efforts to Users

Revisiting Temporal Inconsistency and Feature Extraction for Android Malware Detection

Contact Info

Product

Resources

About