Large language models are no longer confined to simple text generation. In operational roles, they query telemetry, propose configuration changes, and in some deployments, execute those changes directly against live infrastructure. What began as ticket drafting and alert summarization has evolved into what vendors call autonomous remediation or self-healing infrastructure. But a recent survey on agentic AI in network and IT operations gives it a more sobering name: a confused-deputy problem waiting to happen.
The Confused-Deputy Problem in Agentic AI Security
The classic confused-deputy attack occurs when an authorized program is tricked into misusing its privileges. In the context of agentic operations, the conditions are ideal for this kind of abuse. The AI agent holds legitimate access to change-management APIs, deployment pipelines, and network controllers. Its decisions are shaped by tickets, runbooks, chat transcripts, and log entries—the very artifacts an attacker can influence. Compromising the tool itself becomes unnecessary when an attacker can compromise the text the agent reads before it uses the tool. This shifts the attack surface from the model to the data it consumes, a subtle but critical distinction.
Agentic AI systems operate with a degree of autonomy that traditional automated tools lack. They interpret natural language instructions, reason about context, and take actions without human intervention at every step. This autonomy amplifies the potential for harm if an adversary can inject malicious commands into the operational pipeline. The problem is not unique to AI; similar issues exist with any software that acts on untrusted input. But the complexity and opacity of large language models make it harder to predict and prevent misuse.
Four Attack Categories Targeting LLM Operations
The survey catalogs several attack categories that deserve more attention from security teams. The most familiar is prompt injection through operational artifacts: malicious instructions embedded in a ticket or wiki page that steer the agent toward an unsafe action. For example, a support ticket describing a server issue might contain hidden text that tells the LLM to ignore its safety guidelines and execute a command that opens a backdoor. Subtler variants exist beyond direct injection.
Retrieval poisoning corrupts the runbooks and incident histories the agent consults during decision-making. If an attacker can insert false or misleading information into the knowledge base, the agent may be biased toward attacker-chosen conclusions. For instance, a corrupted runbook might instruct the agent that restarting a service is the correct response to any alert, even when the real cause is an ongoing intrusion. Retrieval jamming works in the opposite direction: it floods the knowledge base with blocker documents that trigger refusal loops, stalling incident response when it is most needed. An agent that gets stuck in a refusal loop might ignore a critical alert, allowing an attacker to maintain access.
Telemetry manipulation is another vector that targets LLM-driven operations agents. If an attacker can influence what metrics and logs say, they can steer mitigation decisions without touching the model. For instance, altering CPU utilization data could cause an agent to scale up resources unnecessarily, or hiding memory errors could prevent the agent from detecting a rootkit. These attacks are operationally dangerous because they do not look like attacks. They look like normal incident response that happens to go wrong, making them difficult to detect without careful forensic analysis.
The Propose-Commit Split as an Architectural Defense
The defense proposed by the survey is architectural rather than prompt-based. The authors argue for a strict propose-commit split: the language model can reason, retrieve evidence, and draft change proposals, but it cannot execute writes. Every action that touches production must pass through a non-bypassable gate over which the model has no authority. This gate enforces policy-as-code checks, invariant verification, human approval for high-blast-radius changes, and rollback-ready staged deployment. The model’s job is to draft a diff. The gate’s job is to decide whether that diff is allowed to apply.
This separation of concerns is not new; it mirrors principles from software engineering such as separation of duties and least privilege. In the context of AI, it creates a clear audit trail. Every proposed change is logged, and the gate records whether it was approved or rejected, along with the reasoning. Integrity-protected audit logs ensure that post-incident forensics can reconstruct what happened even if the agent is compromised. The propose-commit split also enables gradual adoption: organizations can start with read-only access, then move to bounded execution with gates, and only consider full autonomy when the controls are proven robust.
An example illustrates the concept. An agent monitoring a web server detects a spike in 500 errors. It retrieves a runbook that suggests increasing the connection pool size. Following the propose-commit split, the agent drafts a configuration change and submits it to the gate. The gate checks the change against policies (e.g., no more than 50% increase in connections), verifies that the new value does not violate any invariants (e.g., database connection limit), and if the risk is low, applies it automatically. For high-risk changes, it escalates to a human operator. The agent never directly modifies the configuration; it only proposes.
The Limits of Prompt-Based Agentic AI Security
This architecture matters because prompt-only defenses are brittle. Any system where the model’s text generation can directly cause production changes has built its security perimeter inside the most unpredictable component in the stack. Large language models are known to be susceptible to jailbreaks, adversarial inputs, and subtle prompt manipulations. Relying on system prompts to prevent misuse is like building a castle on quicksand. The OWASP excessive-agency pattern, the survey notes, is in practice a failure to implement the propose-commit split cleanly. Excessive agency occurs when an agent has more permissions than necessary for its task, and without a gate, those permissions can be exploited.
Even with careful prompt engineering, attackers have demonstrated techniques to bypass instructions. For example, they can encode malicious commands in Base64, use indirect injection through retrieved documents, or exploit multi-turn conversations to gradually erode safety measures. Prompt-based defenses are reactive and incomplete; they address symptoms rather than the root cause. The propose-commit split addresses the root cause by removing the model’s ability to execute actions directly, regardless of what the prompt says.
The Missing Evidence for Safe LLM Autonomy
A measurement problem sits alongside the architectural one. Many claims about safe agentic operations cannot be falsified because the supporting evidence is missing. The survey identifies what evaluations should report: tool-call traces, gate-violation rates, behavior under adversarial inputs, refusal-storm rates under jamming attacks, and rollback completeness. Most current benchmarks omit these metrics. A system that performs well on clean incidents may collapse the moment someone embeds a hostile instruction in a Jira ticket. Security teams evaluating agentic products should ask for adversarial evaluation data alongside success metrics on benign workloads.
The industry has a tendency to present early-stage technology as mature. Vendors often highlight success rates on standard tasks while downplaying failure modes under stress. This asymmetry creates a false sense of security. Without adversarial testing, organizations cannot know how their agent will behave when confronted with a targeted attack. The survey calls for standardized evaluation frameworks that include both functional and security dimensions, similar to how penetration testing is now standard for web applications.
Where Autonomy Earns Trust and Where It Does Not
The amount of autonomy an agent has is directly proportional to the damage it can do when things go sideways. Read-only assistance is useful and low-risk. Bounded execution with strong gates is defensible. Open-ended self-healing across large production environments, without the verification scaffolding the survey describes, is a harder problem than current deployments make it sound. Claims about fully autonomous remediation deserve skepticism, especially when the supporting evidence consists of demos on curated data sets.
Organizations should adopt a phased approach. Start with monitoring and alerting, then move to proposing changes for human approval. Only after extensive testing with adversarial scenarios should gates be tuned to allow automatic execution of low-risk actions. Even then, the gates themselves must be hardened against tampering. The propose-commit split is not a silver bullet—it is a minimum viable control that addresses the most obvious vulnerability. Additional measures such as anomaly detection on agent behavior, rate limiting, and regular audits of knowledge base content are necessary to create a defense-in-depth posture.
The future of agentic AI in operations is promising, but the path to safe autonomy runs through architectural controls, rigorous evaluation, and a healthy skepticism of vendor claims. Without these, the confused-deputy problem will continue to haunt production environments, turning AI assistants from assets into liabilities.
Source: Help Net Security News