NeurIPS 2026 Evaluations & Datasets Track
CyBiasBench
Attack Bias of Agents in AI Security
* Equal contribution · † Corresponding authors
LLM-powered AI agents have evolved from answering questions to using tools, planning multi-step tasks, and carrying out complex workflows. In cybersecurity, they can now explore web applications, identify vulnerabilities, select attack techniques, and verify whether their attempts succeed.
But what happens when different AI agents are asked to assess the same system? Do they approach the task in similar ways?
One agent might begin with SQL injection, while another focuses on authentication bypass or information disclosure. These differences could be random, but they might also reflect consistent patterns in how agents make decisions.
Most existing benchmarks focus on outcomes, such as attack success rates or the number of vulnerabilities discovered. Yet agents with similar overall performance may take very different approaches. More importantly, the attacks they choose most frequently may not be the ones they perform best.
To investigate these differences, we developed CyBiasBench, a benchmark for understanding how LLM agents select cyberattack techniques and how those choices relate to their actual performance.
CyBiasBench — Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios
NeurIPS 2026 Evaluations & Datasets Track
Authors: Taein Lim*, Seongyong Ju*, Munhyeok Kim*, Hyunjun Kim†, Hoki Kim†
* Equal Contribution · † Corresponding Authors