Security Vulnerability Response for AI: Building Secure and Autonomous Agents

Abstract

This talk is in two parts. In the first part, I look at what security vulnerabilities in AI products look like in the current day and age of AI. I will deep dive into some past vulnerabilities in agentic systems we have encountered at Microsoft's Security Response Center and how we approach vulnerability response for AI. In the second part, I show how these real-world vulnerabilities feed into our research on building deterministically secure and autonomous AI agents. Indirect prompt injection attacks threaten AI agents that execute consequential actions, motivating deterministic system-level defenses that can provably block unsafe actions. Such defenses have historically appeared costly, reducing task completion rates and increasing reliance on human oversight. I present our recent work which introduces autonomy metrics to quantify how many consequential actions an agent can execute without human-in-the-loop approval while preserving security, and a security-aware agent design that plans jointly for task progress and policy compliance, achieving higher autonomy without sacrificing utility.