GenAI Red Teaming Guide · 2025
Red
Teams
Attackers use the same AI you do - to write better scams, build malware faster, and find weaknesses before you patch them. The counter-move is simple: test your own systems.
That practice is called red teaming: playing the bad guy against your own AI, on purpose, to find and fix the holes before anyone else can.
Nine Ways Attackers Use AI
Drawn from the LLM tradecraft catalogued in the GenAI COMPASS and the GenAI Red Teaming Guide - the attacker's playbook, translated into plain English.
I
LLM-informed reconnaissance
The Scryer
“AI that quietly maps out everything about your system - before it strikes.”
+
The Scryer
“AI that quietly maps out everything about your system - before it strikes.”
What they do
LLMs gather actionable intelligence on your technology and vulnerabilities - accelerating OSINT, aggregating public and leaked data, and mapping your attack surface.
How to spot it
Suspicious OSINT scraping, abnormal LLM API usage, automated aggregation of social and leaked sources for targeted attacks.
How to stop it
- Monitor for anomalous scraping patterns
- Minimize exposed technical detail
- Treat public docs as hostile intel sources
II
LLM-enhanced scripting
The Scribe
“AI that writes attack scripts fast.”
+
The Scribe
“AI that writes attack scripts fast.”
What they do
LLMs generate and refine scripts for cyberattacks - from basic automation to fully weaponized tooling.
How to spot it
A high rate of script generation, AI-generated code artifacts.
How to stop it
- Detect high-volume script generation
- Flag AI-style code artifacts in logs
III
LLM-aided development
The Forger of Malware
“AI that builds malware faster than humans can.”
+
The Forger of Malware
“AI that builds malware faster than humans can.”
What they do
LLMs accelerate the development lifecycle of malicious tools - including malware - with fast iteration and near-perfect grammar.
How to spot it
AI-style malware source code, rapid tool iteration, playbooks with flawless prose.
How to stop it
- Hunt for AI-generated malware signatures
- Share indicators across communities
IV
LLM-supported social engineering
The Whisperer
“AI that writes flawless phishing emails in any language.”
+
The Whisperer
“AI that writes flawless phishing emails in any language.”
What they do
LLMs craft flawless, multilingual spear-phishing and business-email-compromise pretexts, removing the grammar errors that once betrayed attackers.
How to spot it
Sophisticated phishing, multilingual spear-phishing, executive impersonation.
How to stop it
- Verify unusual requests through a second channel
- Train staff on deepfake & AI phishing
V
LLM-assisted vulnerability research
The Diviner
“AI that finds weaknesses in your software.”
+
The Diviner
“AI that finds weaknesses in your software.”
What they do
LLMs understand and identify potential vulnerabilities in your software and systems, then target them for exploitation.
How to spot it
Abnormal vulnerability search patterns, AI-model queries against your stack.
How to stop it
- Patch aggressively; run your own AI-assisted audit first
VI
LLM-optimized payload crafting
The Alchemist
“AI that tweaks malware until it slips past detection.”
+
The Alchemist
“AI that tweaks malware until it slips past detection.”
What they do
LLMs craft and iteratively refine payloads, evolving obfuscation faster than signatures can keep up.
How to spot it
Fast-evolving, obfuscated payloads that evade static detection.
How to stop it
- Behavioral detection over signatures
- Sandbox dynamic analysis
VII
LLM-enhanced anomaly evasion
The Ghost
“AI that makes attacks blend in with normal activity.”
+
The Ghost
“AI that makes attacks blend in with normal activity.”
What they do
LLMs develop methods to blend malicious activity into normal behavior and traffic, evading detection systems.
How to spot it
Synthetic user behavior, adversarial noise injection.
How to stop it
- UEBA and aggregate-pattern detection
- Immutable logging of all AI events
VIII
LLM-directed security feature bypass
The Lockpick
“AI that finds ways around logins and CAPTCHAs.”
+
The Lockpick
“AI that finds ways around logins and CAPTCHAs.”
What they do
LLMs find ways around two-factor authentication, CAPTCHA, and other access controls.
How to spot it
MFA bypass attempts, CAPTCHA-solving patterns.
How to stop it
- Phishing-resistant MFA (FIDO2)
- Monitor for bypass attempts
IX
LLM-advised resource development
The Quartermaster
“AI that plans and preps an attack before it happens.”
+
The Quartermaster
“AI that plans and preps an attack before it happens.”
What they do
LLMs assist in tool development, modification, and strategic operational planning - the logistics of an attack, optimized.
How to spot it
Rapid tool iteration, playbooks with perfect grammar.
How to stop it
- Threat-intel sharing on emerging tooling
- Assume campaign-scale planning
GenAI Red Teaming
GenAI red teaming means simulating adversarial behavior against your own generative systems to find the flaws before real harm occurs. It goes beyond breaking into servers - it interrogates how the model can be made to produce harmful, deceptive, or dangerous output.
The risks it hunts: prompt injection, bias and toxicity, data leakage, data poisoning, hallucinations and confabulation, agentic vulnerabilities, and supply-chain risk.
The Four Things to Test
The Mind
Test the model's inherent weaknesses - toxicity, bias, hallucination, and alignment. Interrogate its soul before you interrogate its shell.
The Guardrails
Assess the prompts, filters, and guardrails wrapped around the model - and whether they can be bypassed.
The Fortress
Review the full application environment - APIs, storage, integration points, WAF bypass, gateway controls, monitoring blind spots.
The Living System
Probe over-reliance, automation bias, social engineering through the AI, agent boundary testing, and cascade failures across interacting components.
The 12 Tests for Autonomous AI
The red teaming guide's checklist of agentic attack tasks - twelve ways to try to break an autonomous agent, so you can fix it before anyone else tries.
Authorization Hijacking
Inject commands, spoof control signals, test permission revocation - can it be made to act beyond its grant?
Checker-Out-of-the-Loop
Simulate threshold breaches and alert suppression - does the human overseer stay informed?
Critical System Interaction
Test its reach into physical and critical digital systems.
Goal Manipulation
Redirect its objectives and instruction hierarchy.
Hallucination Exploitation
Turn its confabulations into harmful action.
Impact Chain & Blast Radius
Map how one fault propagates across agents and systems.
Knowledge Base Poisoning
Inject malicious training data and external inputs; test rollback.
Memory & Context Manipulation
Probe session isolation and cross-session data leaks.
Multi-Agent Exploitation
Intercept communication, abuse trust, trigger feedback loops.
Resource Exhaustion
Stress computation, memory, and API quotas to the breaking point.
Supply Chain & Dependencies
Introduce tampered dependencies and compromised services.
Agent Untraceability
Suppress logging, misuse role inheritance, obfuscate forensics.