Combinatorial coverage framework for machine learning in multi-domain operations

Cody, Tyler; Kauffman, Justin; Krometis, Justin; Sobien, Daniel; Freeman, Laura J.

doi:10.1117/12.2617117

Cited by 6 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are many challenges when acquiring ML systems, not the least of which concerns test and evaluation (T&E). [29][30][31][32][33][34][35][36][37][38][39][40][41][42][43] T&E is required to inform acquisition decisions, and T&E planning and function development must occur prior to conducting T&E to inform a decision.…”

Section: Applications To Support Acquisition Processesmentioning

confidence: 99%

A systems theoretic perspective on open architectures for learning systems

Cody,

Beling

2024

Disruptive Technologies in Information Sciences VIII

View full text Add to dashboard Cite

The advancement of open architecture ecosystems is fundamentally dependent on the interoperability, scalability, and adaptability of their constituent elements. As machine learning (ML) systems become increasingly integral to these ecosystems, the need for a systematic approach to engineer, deploy, and re-engineer them grows. This paper presents a novel modeling approach based on recently published, formal, systems-theoretic models of learning systems. These models serve dual purposes: first, they give a theoretical grounding to standards that govern the architecture, functionality, and performance criteria for ML systems; second, they allow for requirements to be specified at various levels of abstraction to ensure the systems are intrinsically aligned with the overall objectives of the open architecture ecosystem they belong to. Through the proposed modeling approach, we demonstrate how the adoption of standardized models can significantly enhance interoperability between disparate machine learning systems and other architectural components. Further, we relate our framework to on-going efforts such as Open Neural Network Exchange (ONNX). We identify how our approach can be used to address limitations in government acquisition processes for ML systems. The proposed systems-theoretic framework provides a structured methodology that contributes to the foundational building blocks for open architecture ecosystems for ML systems, thereby advancing the state-of-the-art in complex system integration.

show abstract

Section: Applications To Support Acquisition Processesmentioning

confidence: 99%

A systems theoretic perspective on open architectures for learning systems

Cody,

Beling

2024

Disruptive Technologies in Information Sciences VIII

View full text Add to dashboard Cite

show abstract

“…45,46 Generating campaigns in terms of sets of MDPs requires significant process engineering, 2 and the generate MDPs may be sensitive to real-world changes, 47 as machine learning is known to be, [48][49][50][51][52][53][54] motivating research into the resilience of RL-based cyber. [55][56][57][58][59][60] 3. COMPETITION FRAMEWORK…”

Section: Related Workmentioning

confidence: 99%

Lessons learned in designing a cyber reinforcement learning competition

Cody,

Meno,

Jones

et al. 2024

Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications VI

View full text Add to dashboard Cite

This paper details the lessons learned from designing a specialized competition focused on network attack simulation, specifically emphasizing lateral movement in networks. Our experience in planning the competition reveals several critical design principles that could guide the development of future competitions. Among these principles are the need for a carefully curated simulation environment that mimics real-world network behaviors and vulnerabilities without neglecting the limitations of current capabilities, the necessity for defining a clear scoring system that accurately reflects the effectiveness of different strategies while allowing for interpretability and style, and the importance of ensuring that the competition has clear roles for participants with varied levels of expertise in both cybersecurity and reinforcement learning. This paper offers lessons learned for those interested in organizing similar competitions or developing benchmarks in the area of cyber reinforcement learning. We strongly advocate for the use of network attack simulation competitions as an effective means to accelerate research, foster community collaboration, and identify emergent strategies and techniques. By outlining our design process, the challenges faced, and the solutions implemented, we provide actionable insights to assist practitioners, researchers, and government sponsors.

show abstract

“…Drawing from abstract systems theory 17 and statistical learning theory, 18 abstract learning systems theory (ALST) was recently established as a formal study of general learning systems that is expressly independent of specific solution methods. [19][20][21][22][23][24][25][26][27][28][29] ALST has been applied to systems and test engineering research areas for AIES including test harnesses, [30][31][32][33][34][35] condition-based monitoring, [36][37][38][39] cyber defense, [40][41][42][43][44][45] and cognitive communications. 46,47 In ALST, a learning system S is defined as a relation on sets…”

Section: Abstract Learning Systems Theory (Alst)mentioning

confidence: 99%

On the role of loss-driven systems analysis in assessing AI ethics

Cody,

Freeman,

Beling

2024

Assurance and Security for AI-enabled Systems

View full text Add to dashboard Cite

As machine learning (ML) models are integrated more and more into critical systems, questions regarding their ethical use intensify. This paper advocates for a loss-driven engineering approach, incorporating concepts from systems-theoretic process analysis (STPA), to identify external systems, technologies, and processes essential for ethical ML deployment and therefore crticial to assessing AI ethics. STPA facilitates a deep analysis of potential hazards and system-level vulnerabilities, generating actionable insights for designing support systems and safeguards. Resilience engineering principles can be utilized to convert these insights into testable requirements for assessing AI ethics. This innovative, multi-disciplinary approach addresses a critical gap in current ML practices by extending ethical evaluation beyond the model, offering a robust framework for the responsible development and deployment of AI technologies.

show abstract

Combinatorial coverage framework for machine learning in multi-domain operations

Cited by 6 publications

References 0 publications

A systems theoretic perspective on open architectures for learning systems

A systems theoretic perspective on open architectures for learning systems

Lessons learned in designing a cyber reinforcement learning competition

On the role of loss-driven systems analysis in assessing AI ethics

Contact Info

Product

Resources

About