Chapter 04 · GenAI Red Teaming Guide, 2025
How
attackers think
Attackers use the same AI you do - to write better scams, find weaknesses faster, and hide in normal-looking traffic. The counter-move is unglamorous: attack yourself first, on purpose, with a written scope.
Nine ways attackers use AI
Drawn from the tradecraft catalogued in the COMPASS playbook and the Red Teaming Guide. Read them as a buyer's guide: each one is a capability your own team could use to test itself, and someone else already does.
I
The scryer
LLM-informed reconnaissance
"AI that maps everything about your systems before it strikes."
The scryer
LLM-informed reconnaissance "AI that maps everything about your systems before it strikes."What they do
Use models to gather and correlate open-source intelligence: staff names, technology stacks, org charts, leaked credentials, and the exact version of the thing you are running.
How you notice
Scraping patterns across your public docs, careers page, and support forum; a spike in API calls that never convert to real usage.
What to do
- Assume public documentation is the attacker's briefing
- Watch for correlated scraping, not single requests
- Publish less version detail than you think you need
II
The scribe
LLM-enhanced scripting
"AI that writes attack scripts at typing speed."
The scribe
LLM-enhanced scripting "AI that writes attack scripts at typing speed."What they do
Generate and refine automation for the boring parts of an intrusion - enumeration, credential stuffing, data packaging - turning a week of work into an afternoon.
How you notice
Bursts of scripted requests with unnaturally consistent structure; tooling that adapts to errors faster than a human would.
What to do
- Detect on volume and consistency, not on individual requests
- Rate-limit the endpoints that would make a good target
III
The forger of malware
LLM-aided development
"AI that builds malicious tools faster than signatures can keep up."
The forger of malware
LLM-aided development "AI that builds malicious tools faster than signatures can keep up."What they do
Write and iterate tooling, including malware, with near-perfect grammar and fast variant generation. Each variant is slightly different, which is exactly what signature detection cannot handle.
How you notice
Malware source with unusually clean structure and comments; many near-identical samples appearing in a short window.
What to do
- Prefer behavioural detection over static signatures
- Share indicators; variant families are a shared problem
IV
The whisperer
LLM-supported social engineering
"AI that writes flawless phishing in any language, at any scale."
The whisperer
LLM-supported social engineering "AI that writes flawless phishing in any language, at any scale."What they do
Craft multilingual, context-aware pretexts with none of the grammar tells that used to give an attack away. Combine this with a cloned voice and the old advice - "look for bad spelling" - is worthless.
How you notice
Well-written phishing that quotes your real projects; a request that arrives from a correct-looking internal address but asks for something unusual.
What to do
- Verify unusual requests on a second channel, always
- Train on behaviour, not on spotting typos
- Make challenge normal, not heroic
V
The diviner
LLM-assisted vulnerability research
"AI that reads your code and finds the crack."
The diviner
LLM-assisted vulnerability research "AI that reads your code and finds the crack."What they do
Analyse source, configuration, and dependency graphs to locate exploitable weaknesses - the same tools your own engineers should be running first.
How you notice
You usually do not. The advantage goes to whoever audits their own stack first, which is the entire argument for doing it.
What to do
- Run AI-assisted review on your own code before someone else does
- Patch on a schedule you can prove
VI
The alchemist
LLM-optimized payload crafting
"AI that tweaks a payload until it slips past detection."
The alchemist
LLM-optimized payload crafting "AI that tweaks a payload until it slips past detection."What they do
Iterate obfuscation against known defences, changing the payload until it stops being recognised - a search problem, which is what models are good at.
How you notice
Payloads that mutate slightly between attempts, each testing one detection rule.
What to do
- Detonating in a sandbox beats matching a string
- Alert on the behaviour, not the encoding
VII
The ghost
LLM-enhanced anomaly evasion
"AI that makes an attack look like a Tuesday."
The ghost
LLM-enhanced anomaly evasion "AI that makes an attack look like a Tuesday."What they do
Shape malicious activity to resemble normal traffic and normal working hours, adding noise where detection expects a signal.
How you notice
Individually unremarkable events that only look wrong in aggregate: a pattern of small, well-timed actions from one identity.
What to do
- Detect on aggregates and sequences, not single events
- Baseline per user, then alert on drift
- Keep immutable logs - evasion works best against forgetful systems
VIII
The lockpick
LLM-directed security bypass
"AI that finds a way around the login page."
The lockpick
LLM-directed security bypass "AI that finds a way around the login page."What they do
Defeat CAPTCHAs, work around weak second factors, and talk helpdesks out of credentials. Voice cloning turns "call us to verify" into a formality.
How you notice
Password resets followed by immediate enrolment of a new device; support calls that pass verification but ask for nothing else.
What to do
- Phishing-resistant factors: passkeys, hardware keys
- Out-of-band confirmation for any access change
- Verify identity with knowledge only the real person has, checked in a second channel
IX
The quartermaster
LLM-advised resource development
"AI that plans the campaign before it begins."
The quartermaster
LLM-advised resource development "AI that plans the campaign before it begins."What they do
Plan tooling, infrastructure, and sequencing - the logistics that used to take a team and now take a prompt. This is where AI changes the economics of an attack more than anywhere else.
How you notice
Campaigns with unusually consistent tradecraft across unrelated targets: the same playbook, run at volume.
What to do
- Assume campaign-scale planning, not a lone actor
- Share threat intelligence; a playbook seen once will be seen again
What red teaming actually is
means simulating an adversary against your own system, with permission and a written scope, before someone does it without either. For generative AI it goes beyond breaking into servers: it asks what the model can be made to say, and what it can be made to do.
It is not a penetration test with a language model bolted on. A pentest asks "can I get in". A GenAI red team also asks "can I make it lie, leak, or obey", which are three different questions with three different fixes.
The mind
Toxicity, bias, hallucination, and refusal behaviour. Interrogate what the model is willing to say before you interrogate how it is deployed.
The guardrails
The prompts, filters, and policy layers wrapped around the model - and whether a determined stranger can walk around them.
The fortress
APIs, storage, integration points, gateway controls, and the monitoring blind spots that let an attack finish without anyone noticing.
The living system
Over-reliance, automation bias, social engineering through the assistant, and how failure propagates between agents.
The twelve tests
Pick a target and the plan fills itself in, or tick tests by hand. Copy the result into your issue tracker - it is written to be pasted, not admired.
Your plan
Copy and paste into a ticketYour first ninety minutes
A red team programme starts with one afternoon and one honest report. Here is the whole session, written so you can run it without a specialist.
Write the scope, out loud
One sentence: what system, which accounts, which data, and what is explicitly out of bounds. Name the person who can stop the session. If you cannot say all five, you are not ready to start.
Build a clean test account
A user with realistic but non-production access, a scratch mailbox, and one test document. Use it for everything, so every finding is reproducible and nothing real is touched.
Run five tests, not twelve
Goal manipulation, hallucination exploitation, authorization hijacking, memory isolation, and knowledge-base poisoning. Those five find most of what matters in an afternoon.
Write findings as behaviour, not payload
"When asked to summarise a document containing a hidden line, the agent sent mail to an external address" - not the exact prompt. Behaviour survives the fix; a payload becomes obsolete and hides whether the class of bug is gone.
Decide, assign, and retest
Each finding gets an owner, a date, and a retest. A finding without an owner is a paragraph, not a control.
The rule that keeps it legal and useful
Only test systems you own or have written permission to test, only within the scope you wrote down, and stop the moment the session owner says stop. Everything here is defensive: the goal is a fix, not a demonstration.
Sources
- OWASP GenAI Red Teaming Guide (2025) - genai.owasp.org
- OWASP GenAI COMPASS Playbook (2025) - genai.owasp.org
- Attacker tactics follow the LLM TTP catalogue published in the COMPASS playbook.