AI Security Testing for QA Engineers: a tester's map of prompt injection, jailbreaks, guardrails and AI pentesting

AI Security Testing for QA Engineers (2026): Start Here

AI security
Quick answer

AI security testing means checking that an AI-powered application cannot be tricked, misused or made to leak data, and that the checks keep working when the model, prompts or tools change. For QA engineers it breaks into four jobs: test for prompt injection, test for jailbreaks, test the guardrails you put around the model, and test the application around the model like any other web app. Then automate the attacks so they run on every change. This page is the map. Each step links to a detailed guide; where we ran the tests ourselves, the guide says so and states the limits.

How to read this page

  • Start here if: you test or build an app that calls an LLM and nobody has checked how it fails under attack.
  • What is on it: a reading path, what each type of test finds, and a summary of the tests we actually ran (all on practice apps and mock chatbots on our own PC).
  • What is not on it: legal or compliance advice, guarantees that any tool makes an app safe, or tests of commercial AI security products. Only attack systems you own or have written permission to test.

What is AI security testing?

An LLM application takes text from users, documents and tools, and a model decides what to do with it. That creates failures a normal test suite does not look for: a hidden instruction in a web page that the model obeys, a role-play prompt that talks the model out of its rules, a reply that reveals a system prompt, or an agent that uses a tool it should not. AI security testing is the practice of attacking your own application in these ways, on purpose and under control, and recording what happens.

It overlaps with classic security testing but is not the same. A scanner finds known web flaws such as missing headers or injection in a form field. It does not know that a chatbot can be instructed to ignore its rules. The OWASP list of risks for LLM applications is the usual checklist; our tester’s version of the OWASP Top 10 for LLM applications turns it into tests.

The four jobs of an AI security tester

JobWhat you are checkingStart with
1. Prompt injectionCan text in a user message, document or web page change what the model does?Prompt injection testing and 9 attack examples
2. JailbreaksCan role-play or encoded prompts push the model past its rules?LLM jailbreak testing
3. GuardrailsDo the filters you added actually block attacks, and do they block normal users?AI guardrails testing
4. The app around the modelDoes the web app, API or login behind the chatbot have ordinary security flaws?AI penetration testing
AutomationCan you run the attacks again on every change?AI red teaming tools

A reading path

  1. OWASP Top 10 for LLM applications: a tester’s checklist: the map of risks.
  2. Prompt injection testing then Prompt injection examples: the most common attack and how to test each variant.
  3. LLM jailbreak testing: nine patterns and a test suite.
  4. AI red teaming tools: Promptfoo vs PyRIT vs garak: automate the attacks, with a garak run on two mock chatbots.
  5. AI guardrails testing: measure what two jailbreak filters block and miss.
  6. AI penetration testing: a classic scanner against an AI pentest agent on a practice app.
  7. How to test LLM applications and how to test an AI chatbot: the wider quality picture around security.

What we ran ourselves, and what it showed

Every result below comes from a real run on our own PC, against practice apps or toy chatbots we wrote. Each linked page lists exactly what was and was not tested. None of these is a verdict on a product.

TestWhat we didWhat came out
garak on two mock chatbotsRan garak probes (DAN, prompt injection, base64) against a weak and a hardened mock botThe weak bot failed the prompt-injection probe on 145 of 256 attempts by garak’s own detector; the hardened bot on none. garak also scored one refusal as a failure because of a substring match, so detectors need checking too.
ZAP and Strix on OWASP Juice ShopA default ZAP full scan, then one narrow run of the Strix AI agent pointed at two endpointsZAP raised no injection alerts. Strix reported SQL injection in the login and in the product search. We told it which two endpoints to test, so this does not show it would find them unaided.
Guardrails AI and NeMo GuardrailsPut two jailbreak detectors in front of the two mock bots with 15 attacks and 20 harmless promptsAt documented settings neither blocked anything. At Guardrails AI’s package default, 5 of 15 attacks were blocked with no false alarms; at a lower threshold more attacks were caught and harmless prompts started to be blocked.

Common mistakes

  • Treating a guardrail as the fix. Our guardrail test showed a detector can miss whole attack families; the app should be fixed first.
  • Only counting blocked attacks. Also count harmless messages wrongly blocked.
  • Testing once. Models, prompts, tools and filter versions change; the attacks need to run again, ideally in CI.
  • Trusting one tool’s detector. Check a sample of its verdicts by hand, in both directions.
  • Testing systems you do not own. Keep practice apps local and get written permission for anything else.

Key terms

Prompt injection · LLM guardrails · AI red teaming · OWASP Top 10 for LLM applications · Agentic testing

Agents and tool access

Applications where a model can call tools or browse raise the stakes, because a successful injection can act, not just talk. We have not yet run hands-on tests on agent or MCP security, so this site has no tested guide on it yet. For background, see our notes on OWASP’s agentic security guide, NIST AI agent standards for QA teams and agentic testing. We will add tested material when we have run it.

FAQ

What is AI security testing?

Attacking your own AI application on purpose, in a controlled way, to see how it fails: prompt injection, jailbreaks, data leaks, misuse of tools, and flaws in the app around the model.

Is AI security testing the same as penetration testing?

No. Penetration testing targets a system’s technical weaknesses. AI security testing adds attacks that work on the model itself through language. Real projects need both; see our AI penetration testing guide.

Which tools should a QA engineer start with?

For automated attacks on an LLM app, our red teaming tools comparison covers Promptfoo, PyRIT and garak. For a classic scan of the app around the model, OWASP ZAP is a free option (see the penetration testing guide).

Do guardrails make an AI app secure?

No. They reduce risk for the attacks they recognise. In our test one detector blocked 5 of 15 attacks at its default setting and missed every plain injection prompt. Measure any guardrail on your own attacks and normal traffic. Details in AI guardrails testing.

Can I test any website or AI service?

Only systems you own or have written permission to test. Everything we describe ran on practice apps or toy chatbots on our own computer.

Sources

OWASP Top 10 for LLM Applications (OWASP GenAI Security Project). All test results on this page are from our own runs, linked above, dated 17 and 18 November 2026.

Was this helpful?Suggest an edit
Zain Ul Rehman
Zain Ul Rehman

QA automation engineer. Each guide says what I ran myself and what I only researched, and I update it when things change. About me

One useful email a week

New guides, tool price changes and AI testing news. No spam.

Free. Confirm by email, unsubscribe in one click. Privacy

Scroll to Top