Summary
- The NCSC added seven AI-related research questions to its problem book on 20 October, grouped around defence, AI system assurance and resilience.
- Research priorities include bounded agent authority, realistic testing of autonomous remediation, secure AI dependencies and national-scale analysis.
- The chapter invites contributions but does not establish mandatory technical standards or claim autonomous cyber defence is already proven safe.
The UK’s National Cyber Security Centre has added seven research questions on artificial intelligence to its cyber security problem book, focusing on how autonomous systems can support defensive operations without creating new sources of disruption, compromise or unaccountable decision-making. Published on 20 October, the chapter covers AI-enabled security operations, the assurance of AI systems themselves and the use of data to understand resilience across the country.
The new chapter is directed at research organisations, suppliers and government rather than at organisations meeting a new compliance duty. It asks how defensive agents can use privileged tools safely, when a human must review an action and how autonomous behaviour can be measured after deployment. The NCSC has invited feedback and examples of work contributing to those questions; it has not introduced certification, a mandatory technical control or a funding award.
Its first question concerns redesigning cyber defence workflows for agents capable of planning, invoking tools and completing multi-step tasks. Conventional operations centres were built around analysts reviewing alerts, investigating evidence and authorising changes through established escalation routes. An agent that can both analyse a problem and alter a live configuration challenges that separation of duties, particularly when the action could affect availability or the integrity of evidence.
The agency also asks how agents should exchange evidence, coordinate tasks and handle conflicting assessments inside an existing security operation. An automated workflow may join asset inventories, ticket systems and configuration interfaces that were previously governed by separate approval steps. Assessing the chain of authority behind each change is part of the problem, particularly when one agent acts on another’s uncertain conclusions.
In attack surface management, the NCSC asks whether agents could correlate inventories, prioritise vulnerabilities and propose configuration changes without overlooking operational dependencies. Disabling an exposed service might reduce one risk while interrupting a customer process or breaking a recovery pathway. The proposed work therefore covers context, explanation and approval as well as technical remediation capability.
Threat detection and incident response research would join related evidence from logs, alerts and threat intelligence. The chapter considers using AI to identify suspicious activity in software supply chains and assemble incident timelines, while requiring a clear account of uncertainty. Telemetry can be incomplete or manipulated; processing it faster does not establish that a resulting conclusion is sound.
Testing autonomous defences creates a further assurance problem because an agent may choose different sequences of tool calls as a network or attack changes. The NCSC proposes realistic failure testing, bounded authority, intervention mechanisms, explanation and monitoring after deployment. The ability to reconstruct why a system took an action must survive the operational pressure under which it made that decision.
AI technology itself is also a potential target. The fifth question covers the lifecycle of a model, from training data and software dependencies through deployment and maintenance. Poisoned data, tampered weights and unsafe fine-tuning may need controls alongside conventional software assurance, with trade-offs among resilience, performance, explainability and cost.
The sixth question deals with connected AI ecosystems. Deployed services can combine foundation models, tool gateways, memory, datasets, cloud infrastructure and agents acting on behalf of different users. A compromised component or malicious instruction could cross a trust boundary into a service with greater permissions. The NCSC asks for research into provenance, containment, incident investigation and the security of these dependencies.
At national scale, the NCSC asks how graph analysis, structured knowledge and machine learning might reveal shared dependencies or concentrated exposure. Security information is spread across sectors and organisations, and it may be incomplete or inconsistent. Proposed methods therefore have to remain explainable, reusable and sensitive to context so that aggregated data does not produce misleading assessments.
Human oversight appears throughout the seven questions, from proposed approval of live system changes to the evaluation of models operating within connected AI services. The chapter follows the NCSC’s earlier assessments of AI-enabled threats but does not assert that autonomous defenders are ready to operate critical systems independently. Contributions to the problem book will inform research into where automation is effective and what evidence is needed to trust it.
