Machine learning (ML) models commonly experience issues when integrated into production systems. In this podcast, researchers from the Carnegie Mellon University Software Engineering Institute and the U.S. Army AI Integration Center (AI2C) discuss Machine Learning Test and Evaluation (MLTE), a new tool that provides a process and infrastructure for ML test and evaluation. MLTE can aid organizations across the DoD in more effectively negotiating, documenting, and evaluating model and system qualities.

Improving Machine Learning Test and Evaluation with MLTE

In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Gregory Touhill, director of the SEI CERT Division, sits down with Matthew Butkovic, technical director of Cyber Risk and Resilience at CERT, to discuss ways in which CERT researchers and technologists are working to deliver rapid capability to warfighters in the Department of Defense.&nbsp;

Delivering Capability to the DoD Warfighter

A strong cyber defense is vital to &nbsp;public- and private-sector activities in the United States. In 2019, in response to an executive order to strengthen America’s cybersecurity workforce, the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) partnered with the SEI to develop and run the President’s Cup Cybersecurity Competition, a national cyber competition that identifies and rewards the best cybersecurity talent in the federal workforce. In six years, more than 8,000 people have taken part in the President’s Cup. In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Jarrett Booz, technical lead for the President’s Cup, and John DiRicco, a training specialist in the SEI’s CERT Division, sit down with Matthew Butkovic, the CERT technical director of cyber risk and resilience, to reflect on six years of hosting the cup, including challenges, lessons learned, the path forward, and publicly available resources. &nbsp;

The Best and Brightest: 6 Years of Supporting the President’s Cup Cybersecurity Competition

Warfighters in the Department of Defense (DoD) operate in high-stakes environments where security, efficiency, and speed are critical. In such environments DevSecOps has become crucial in the drive toward modernization and overall mission success. A recent study led by researchers at the Carnegie Mellon University Software Engineering Institute (SEI) examined the state of DevSecOps within the Department of Defense. In this podcast, Eileen Wrubel,&nbsp;the SEI’s Transforming Software Acquisition Policy and Practice technical director, sits down with George Lamb, director for DoD Cloud and Software Modernization in the Information Enterprise Office of the DoD CIO, which is responsible for the DoD Software Modernization Strategy and its associated implementation plan, and Bill Nichols, lead of the SEI’s Software Engineering Measurement and Analysis work. They discuss DevSecOps successes in the DoD and opportunities for scaling its impact.

Making Process Respectable Again: Advancing DevSecOps in the DoD Mission Space

A January 2025 Defense Innovation Board study on scaling nontraditional defense innovation stated, “We must act swiftly to ensure the DoD leads in global innovation and competition over AI and autonomous systems – and is a trendsetter for their responsible use in modern warfare." In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), chief technical officer Tom Longstaff discusses the SEI’s long-standing work to help the DoD rapidly scale technology including artificial intelligence (AI) and autonomous systems. &nbsp;

The Magic in the Middle: Evolving Scaled Software Solutions for National Defense

Container images are increasingly being used as the main method for software deployment, so ensuring the reproducibility of container images is becoming a critical step in protecting the software supply chain. However, in practice builds are often not reproducible due to elements of the build environment that rely on nondeterministic factors such as timestamps and external dependencies. Lack of reproducibility can lead to lack of trust, broken builds, and possibly mask hidden malware insertion. Vessel, a recent tool from the Carnegie Mellon University Software Institute (SEI), helps developers identify the difference between two container images to help sort benign from problematic issues. In this SEI Podcast, Kevin Pitstick, a senior software engineer at the SEI and Vessel’s lead developer, and Lihan Zhan, a software engineer at the SEI working on tactical and AI-enabled systems, sit down with Grace Lewis, lead of the Tactical and AI-Enabled Systems (TAS) applied research and development team at the SEI, to discuss the Vessel tool, its development, and application in mission-critical settings.  &nbsp;     &nbsp;

Understanding Container Reproducibility Challenges: Stopping the Next Solar Winds

A recent Google survey found that many developers felt comfortable using the Rust programming language in two months or less. Yet barriers to Rust adoption remain, particularly in safety-critical systems, where features such as memory and processing power are in short supply and compliance with regulations is mandatory. In our latest podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Vaughn Coates, an engineer in the SEI’s Software Solutions Division, sits down with Joe Yankel, initiative Lead of the DevSecOps Innovations team at the SEI, to discuss the barriers and benefits of Rust adoption.&nbsp;&nbsp;

The Benefits of Rust Adoption for Mission-and-Safety-Critical Systems

Kurt Vonnegut Jr. reads a poem at the start of his talk to the International Poetry Forum in Pittsburgh, Pennsylvania's Heinz Hall on Nov. 17, 1971.

Citation:International Poetry Forum. IPF Recording November 17, 1971: Kurt Vonnegut, Jr. Carlow University. 1971. https://carlow.hykucommons.org/concern/generic_works/a2f4d445-732f-421c-8a1e-b3f27d1c2bf1?locale=en

Kurt Vonnegut reads "Dear Old Tech"

Software bills of materials or SBOMs are critical to software security and supply chain risk management. Ideally, regardless of the SBOM tool, the output should be consistent for a given piece of software. But that is not always the case. The divergence of results can undermine confidence in software quality and security. In our latest podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Jessie Jamieson, a senior cyber risk engineer in the SEI’s CERT Division, sits down with Matt technical director of Risk and Resilience in CERT, to talk about how to achieve more accuracy in SBOMs and present and future SEI research on this front.   

Getting Your Software Supply Chain In Tune with SBOM Harmonization

Matt Gaston and Matt Butkovic discuss ongoing and future work in AI, including test and evaluation, the importance of hands-on experience with AI systems, and why government needs to continue partnering with industry to spur innovation in national defense.

Delivering Next-Generation AI Capabilities

How can you ever know whether an LLM is safe to use? Even self-hosted LLM systems are vulnerable to adversarial prompts left on the internet and waiting to be found by system search engines. These attacks and others exploit the complexity of even seemingly secure AI systems.  
In our latest podcast from the Carnegie Mellon University Software Engineering Institute (SEI), David Schulker and Matthew Walsh, both senior data scientists in the SEI’s CERT Division, sit down with Thomas Scanlon, lead of the CERT Data Science Technical Program, to discuss their work on System Theoretic Process Analysis, or STPA, a hazard-analysis technique uniquely suitable for dealing with AI complexity when assuring AI systems.

What Could Possibly Go Wrong? Safety Analysis for AI Systems

The SEI is a nonprofit, public–private partnership conducting R&D in software engineering, systems engineering, cybersecurity, and many other areas of computing, working to introduce private-sector innovations into government.

Improving Machine Learning Test and Evaluation with MLTE

Carnegie Mellon - Software Engineering Institute

Related tracks