AI security testing means checking that an AI-powered application cannot be tricked, misused or made to leak data, and that the checks keep working when the model, prompts or tools change. For QA engineers it breaks into four jobs: test for prompt injection, test for jailbreaks, test the guardrails you put around the model, and test the application around the model like any other web app. Then automate the attacks so they run on every change. This page is the map. Each step links to a detailed guide; where we ran the tests ourselves, the guide says so and states the limits.
How to read this page
- Start here if: you test or build an app that calls an LLM and nobody has checked how it fails under attack.
- What is on it: a reading path, what each type of test finds, and a summary of the tests we actually ran (all on practice apps and mock chatbots on our own PC).
- What is not on it: legal or compliance advice, guarantees that any tool makes an app safe, or tests of commercial AI security products. Only attack systems you own or have written permission to test.
Table of Contents
What is AI security testing?
An LLM application takes text from users, documents and tools, and a model decides what to do with it. That creates failures a normal test suite does not look for: a hidden instruction in a web page that the model obeys, a role-play prompt that talks the model out of its rules, a reply that reveals a system prompt, or an agent that uses a tool it should not. AI security testing is the practice of attacking your own application in these ways, on purpose and under control, and recording what happens.
It overlaps with classic security testing but is not the same. A scanner finds known web flaws such as missing headers or injection in a form field. It does not know that a chatbot can be instructed to ignore its rules. The OWASP list of risks for LLM applications is the usual checklist; our tester’s version of the OWASP Top 10 for LLM applications turns it into tests.
The four jobs of an AI security tester
| Job | What you are checking | Start with |
|---|---|---|
| 1. Prompt injection | Can text in a user message, document or web page change what the model does? | Prompt injection testing and 9 attack examples |
| 2. Jailbreaks | Can role-play or encoded prompts push the model past its rules? | LLM jailbreak testing |
| 3. Guardrails | Do the filters you added actually block attacks, and do they block normal users? | AI guardrails testing |
| 4. The app around the model | Does the web app, API or login behind the chatbot have ordinary security flaws? | AI penetration testing |
| Automation | Can you run the attacks again on every change? | AI red teaming tools |
A reading path
- OWASP Top 10 for LLM applications: a tester’s checklist: the map of risks.
- Prompt injection testing then Prompt injection examples: the most common attack and how to test each variant.
- LLM jailbreak testing: nine patterns and a test suite.
- AI red teaming tools: Promptfoo vs PyRIT vs garak: automate the attacks, with a garak run on two mock chatbots.
- AI guardrails testing: measure what two jailbreak filters block and miss.
- AI penetration testing: a classic scanner against an AI pentest agent on a practice app.
- How to test LLM applications and how to test an AI chatbot: the wider quality picture around security.
What we ran ourselves, and what it showed
Every result below comes from a real run on our own PC, against practice apps or toy chatbots we wrote. Each linked page lists exactly what was and was not tested. None of these is a verdict on a product.
| Test | What we did | What came out |
|---|---|---|
| garak on two mock chatbots | Ran garak probes (DAN, prompt injection, base64) against a weak and a hardened mock bot | The weak bot failed the prompt-injection probe on 145 of 256 attempts by garak’s own detector; the hardened bot on none. garak also scored one refusal as a failure because of a substring match, so detectors need checking too. |
| ZAP and Strix on OWASP Juice Shop | A default ZAP full scan, then one narrow run of the Strix AI agent pointed at two endpoints | ZAP raised no injection alerts. Strix reported SQL injection in the login and in the product search. We told it which two endpoints to test, so this does not show it would find them unaided. |
| Guardrails AI and NeMo Guardrails | Put two jailbreak detectors in front of the two mock bots with 15 attacks and 20 harmless prompts | At documented settings neither blocked anything. At Guardrails AI’s package default, 5 of 15 attacks were blocked with no false alarms; at a lower threshold more attacks were caught and harmless prompts started to be blocked. |
Common mistakes
- Treating a guardrail as the fix. Our guardrail test showed a detector can miss whole attack families; the app should be fixed first.
- Only counting blocked attacks. Also count harmless messages wrongly blocked.
- Testing once. Models, prompts, tools and filter versions change; the attacks need to run again, ideally in CI.
- Trusting one tool’s detector. Check a sample of its verdicts by hand, in both directions.
- Testing systems you do not own. Keep practice apps local and get written permission for anything else.
Key terms
Prompt injection · LLM guardrails · AI red teaming · OWASP Top 10 for LLM applications · Agentic testing
Agents and tool access
Applications where a model can call tools or browse raise the stakes, because a successful injection can act, not just talk. We have not yet run hands-on tests on agent or MCP security, so this site has no tested guide on it yet. For background, see our notes on OWASP’s agentic security guide, NIST AI agent standards for QA teams and agentic testing. We will add tested material when we have run it.
FAQ
What is AI security testing?
Attacking your own AI application on purpose, in a controlled way, to see how it fails: prompt injection, jailbreaks, data leaks, misuse of tools, and flaws in the app around the model.
Is AI security testing the same as penetration testing?
No. Penetration testing targets a system’s technical weaknesses. AI security testing adds attacks that work on the model itself through language. Real projects need both; see our AI penetration testing guide.
Which tools should a QA engineer start with?
For automated attacks on an LLM app, our red teaming tools comparison covers Promptfoo, PyRIT and garak. For a classic scan of the app around the model, OWASP ZAP is a free option (see the penetration testing guide).
Do guardrails make an AI app secure?
No. They reduce risk for the attacks they recognise. In our test one detector blocked 5 of 15 attacks at its default setting and missed every plain injection prompt. Measure any guardrail on your own attacks and normal traffic. Details in AI guardrails testing.
Can I test any website or AI service?
Only systems you own or have written permission to test. Everything we describe ran on practice apps or toy chatbots on our own computer.
Sources
OWASP Top 10 for LLM Applications (OWASP GenAI Security Project). All test results on this page are from our own runs, linked above, dated 17 and 18 November 2026.
One useful email a week
New guides, tool price changes and AI testing news. No spam.





