AI penetration testing (often shortened to AI pentesting) means using AI agents to plan and run attacks against an application, then confirm what actually works. It sits between a classic automated scanner and a human pentester. We ran both ends ourselves on the same practice app: an OWASP ZAP scan (the classic baseline) and one narrow run of the open-source AI agent Strix. Both are described below, with their limits.
Verdict box
- What we tested: an OWASP ZAP full scan, and one narrow Strix 1.7.0 run (free-tier model), both against OWASP Juice Shop on our own PC, 17 and 18 November 2026.
- What we did not test: any commercial AI pentest vendor; Strix without telling it where to look; other models.
- Use this page for: seeing what a default scanner and a pointed AI agent each reported on the same practice app, plus the tool landscape.
- Do not use it for: choosing an AI pentest vendor, or judging how often agents are right. One run proves very little.
Table of Contents
Table of Contents
AI pentesting vs scanners vs human pentesters
| Automated scanner (e.g. ZAP) | AI pentest agent | Human pentester | |
|---|---|---|---|
| How it works | Fixed rules, crawls and probes | An LLM plans steps, runs tools, reads results, tries again | Experience and manual testing |
| Strength | Fast, repeatable, cheap, good at configuration problems | Can chain steps and try to prove a finding works | Judgement, business logic, context |
| Weak spot | Misses flaws that need several steps | Cost, wrong findings, needs supervision | Time and price |
| Needs a human to check | Yes | Yes | Is the human |
This is our summary of how the categories are usually described, not a measured comparison.
How does an AI pentest agent work?
Most agents follow the same loop: map the target, decide what to try, run tools or write small exploit scripts, check whether the attack worked, and write a report. The important difference from a scanner is the last two steps. A scanner reports “this might be a problem”. An agent tries to show that it is, for example with a working proof of concept. Strix’s own page describes it this way: agents run the target, find vulnerabilities, and confirm them with proof-of-concept exploits (Strix on GitHub). That is the project’s description, and below is what it did on our machine.
What did ZAP find in OWASP Juice Shop?
Safety first. We scanned only OWASP Juice Shop, an app built to be vulnerable, running in Docker on our own PC and reachable only from that PC (127.0.0.1). Nothing on the internet, nothing belonging to an employer or client. Only test systems you own or have written permission to test.
Setup
- Juice Shop:
docker runof the officialbkimminich/juice-shopimage (image dated 8 Nov 2026). - Scanner: ZAP 2.17.0 (
zaproxy/zap-stableimage dated 3 Nov 2026),zap-full-scan.pywith a 3-minute spider, no login. - Machine: a Windows PC with Docker Desktop 4.99.0.
Run it yourself (Windows Command Prompt; on macOS or Linux use $(pwd) for the folder):
docker network create pentestlab
docker run -d --name juice-shop --network pentestlab -p 127.0.0.1:3000:3000 bkimminich/juice-shop
docker run --rm --network pentestlab -v "%cd%\zap-out:/zap/wrk:rw" zaproxy/zap-stable zap-full-scan.py -t http://juice-shop:3000 -m 3 -r zap-report.html -J zap-report.json -I
docker rm -f juice-shopThe 127.0.0.1: part keeps Juice Shop reachable only from your own machine. ZAP reaches it over the private Docker network.
Result
- Run time: about 14 minutes (21:41 to 21:55).
- 13 alert types, none rated High.
- Medium (5): backup file disclosure (31 instances), a “bypassing 403” check, CORS misconfiguration, Content-Security-Policy header not set, cross-domain misconfiguration.
- Low (4): missing cross-origin isolation headers (2), dangerous JavaScript functions (in main.js and one other bundle), deprecated Feature-Policy header.
- Informational (4).


ZAP’s report also noted that 50% of responses were slow, so a longer or slower-paced scan might give different results.
What it found that matters. The backup-file and “bypassing 403” alerts point at Juice Shop’s /ftp folder. ZAP listed files such as package.json.bak and coupons_2013.md.bak there. Juice Shop has challenges built around exactly this kind of exposure, including “Confidential Document”, “Forgotten Developer Backup” and “Forgotten Sales Backup” (OWASP Juice Shop companion guide). So the scanner did surface a real, intended weakness. We did not try to download or use those files beyond what ZAP’s scan requested.
What it did not report. ZAP raised no injection or cross-site scripting alerts. We have not matched the scan against the full challenge list, so we are not claiming a coverage percentage and not naming other missed flaws. The point is narrower: an unauthenticated baseline scan found exposed files and header problems and stopped there. Hold on to this: the AI agent below, pointed at the same app, found injection flaws that the scanner’s default run did not raise.
Limits of our run. One scan, one default configuration, no login, short spider. A tuned or authenticated ZAP run would likely find more. Treat these numbers as one baseline, not a verdict on ZAP.
What happened when we ran the Strix AI agent?
What we ran. Strix 1.7.0, an open-source agent (Apache-2.0) that runs locally in Docker with your own LLM key (source). The Windows build was blocked on our PC by application control (the message said strix.exe “was blocked by your organization’s Device Guard policy”), so we ran it inside WSL Ubuntu instead. We used a free OpenRouter model (nvidia/nemotron-3-ultra-550b-a55b:free), so the model cost was $0. Same target as ZAP: Juice Shop, reachable only from our PC.
What went wrong first. Our first run stopped when the free tier’s 50-requests-per-day limit was reached: no findings and an incomplete scan. Strix still exited with code 0, so the exit code alone would have fooled us. We read the run’s coverage.json, which said the scan was rate_limited and incomplete. A later attempt failed in seconds because Docker was not running. Two lessons if you try this: check completeness in coverage.json, not the exit code; and make sure Docker Desktop is fully started first.
The run that finished. To fit the free daily limit we narrowed the job. We told the agent to test only two things with curl and nothing else: SQL injection on the login endpoint and on the product search endpoint, with no browser, no crawling and no state-changing actions. It ran for about two and a half minutes (18 November 2026) and coverage.json reported the scan as completed with no caveats. It filed two findings, and we then reproduced both by hand:
- Login (critical). The agent sent
' OR 1=1--as the email with any password. The app returned HTTP 200 and a valid admin session token. We re-ran the same request ourselves: it logged us in as the admin account (admin@juice-sh.op), while a normal wrong email and password were correctly rejected with “Invalid email or password”. Strix rated this critical. - Product search (medium). The agent sent
' OR 1=1in the search box. We re-ran it: the search returned all 46 products in the database instead of the 3 a normal search (“apple”) returns, and malformed input produced a raw SQLite error. Strix rated this medium.
Each of the agent’s findings came with the request, the response, a check that a normal login was rejected, and a fix suggestion (use parameterised queries). We confirmed both on our own machine before writing this.
How to read this. This is not a fair race against ZAP, and we are not presenting it as one. ZAP ran unauthenticated with default settings and had to discover everything; the agent was told which two places to look. So the result shows that an AI agent can confirm known weakness types quickly and with working proof when it is pointed at them. It does not show that the agent would have found them unaided, how often it reports things that are not real, or how it compares with a tuned scanner or a human tester. Juice Shop’s login injection is a well-known intended challenge, so a capable model may simply recognise it. One model, one target, one run.
Which AI penetration testing tools exist (not tested by us)?
Facts below come from each vendor’s or project’s own pages, checked 17 Nov 2026. They are claims, not our results.
| Tool | Type | What its page says |
|---|---|---|
| Strix | Open source (Apache-2.0) | AI agents that test your app and confirm findings with proof-of-concept exploits; runs locally with Docker and your LLM key; authorised use only. (The tool we ran above.) |
| Shannon Lite | Open-source edition | White-box (needs source code). Its project reports 100 of 104 challenges (96.15%) on a cleaned XBOW benchmark, a result published by the project itself |
| XBOW | Commercial | “Autonomous offensive security platform”; says no human drives the attack and that findings are reviewed before you see them |
| Horizon3.ai NodeZero | Commercial | Autonomous pentesting platform; added web application testing in July 2026 that chains flaws and shows which are exploitable |
What are the limits and risks?
- Authorisation. An agent actively attacks what you point it at. Written permission and a clear scope are not optional.
- Wrong findings. Anything an AI reports should be reproduced by a person before it goes into a ticket. We did exactly that above, and it is the only reason we call our two findings confirmed.
- Cost and time. Agents call an LLM many times. A free model worked for us but its daily limit stopped our first run; a paid key removes that limit at a cost that depends on the provider.
- Benchmarks. Several published comparisons are run by the vendors being compared. Read them as claims.
What if the target is an AI app?
If your target is a chatbot, an LLM app, or an agent rather than a normal web app, you are testing for prompt injection, data leaks and tool misuse. That is a separate topic, covered in our AI red teaming tools page and prompt injection testing guide.
Frequently Asked Questions
What is AI penetration testing?
Using AI agents to plan, run and verify attacks against an application or network, usually with a human reviewing the result.
Is AI pentesting better than a scanner like ZAP?
Our one run does not settle it. ZAP, unauthenticated and on defaults, raised no injection alerts. A free-tier AI agent, told to look at two endpoints, confirmed SQL injection on both with proof that we then reproduced. That is not a fair comparison, because the agent was pointed at the targets. The usual claim is that agents can chain steps and prove exploits; that needs more testing than one run.
Can I try AI penetration testing for free?
ZAP is free. Strix is free software; we ran it with a free OpenRouter model at no cost, but the free tier allows only 50 requests a day, which stopped our first run. Test only systems you own.
Is AI penetration testing legal?
Only with permission. Unauthorised testing is illegal in most places.
Will AI replace pentesters?
Not on current evidence. Even the vendors above describe review or validation steps, and a scanner or agent cannot decide what is in scope, judge business risk, or sign off a report. Expect AI to take over more of the repetitive testing, with people checking and prioritising the results.
This guide is part of our AI security testing hub, which maps prompt injection, jailbreak, guardrail and penetration tests for QA engineers.
Sources
Strix · Shannon benchmark page · XBOW · Help Net Security on NodeZero · OWASP Juice Shop companion guide. Tests run on our own PC on 17 and 18 November 2026: ZAP 2.17.0, Strix 1.7.0, OWASP Juice Shop (image dated 8 Nov 2026).
One useful email a week
New guides, tool price changes and AI testing news. No spam.





