Plain-English definitions of the terms QA engineers meet when testing with AI and testing AI, each linked to a full guide.
Agentic testing is test automation where an AI agent plans, runs and adapts tests with limited human direction. Instead of following a fixed script, the agent reads the goal, explores the application, decides the next step and reports results, which helps with changing interfaces but needs guardrails and human review.
Read more →An AI hallucination is an answer from a language model that sounds confident but is false or not supported by its source material. QA teams measure hallucinations with grounded test datasets, faithfulness metrics and human review, because fluent wording makes these errors easy to miss.
Read more →AI red teaming is structured adversarial testing of an AI system, where testers act like attackers to find harmful outputs, data leaks and security weaknesses before users do. Tools such as Promptfoo automate many red-team attacks so they can run in CI.
Read more →Answer relevancy is an LLM evaluation metric that scores how directly a response addresses the question that was asked. A relevant answer can still be wrong, so teams usually pair it with faithfulness or correctness checks rather than using it alone as a quality gate.
Read more →API testing checks that application programming interfaces return the right data, status codes and errors, and meet performance and security expectations, without going through the user interface. It runs faster and is more stable than UI testing, so it usually covers most business logic.
Read more →Behaviour-driven development (BDD) is a way of building software where the team describes how a feature should behave in plain-language examples before it is built, and those examples become automated tests. The examples are usually written in Gherkin with Given, When and Then steps.A Gherkin exampleFeature: Login Scenario: Valid user signs in Given I am on the login page When…
Read more →A CI quality gate is an automated check in a continuous integration pipeline that blocks a merge or release when tests fail or metrics fall below a threshold. For AI features, gates can include evaluation scores and red-team results as well as normal test results.
Read more →Context precision measures how much of the context retrieved by a RAG system is actually relevant to the question, and whether relevant chunks are ranked first. Low precision means the model is given noise, which raises cost and makes wrong answers more likely.
Read more →End-to-end (E2E) testing checks a complete user journey through the real application, from the user interface through the backend and database, the way a user would. An example is signing up, adding a product to the cart and paying.Where E2E tests fitIn the test pyramid, E2E tests sit at the top: fewest in number, slowest and most expensive to maintain,…
Read more →Exploratory testing is testing where the tester designs and runs tests at the same time, learning about the product as they go instead of following a fixed script. It relies on the tester's skill and curiosity to find problems that scripted tests miss.How to run a sessionWrite a short charter, such as "Explore checkout with expired cards to find payment…
Read more →Faithfulness is a RAG evaluation metric that checks whether every claim in a generated answer is supported by the retrieved context. A low faithfulness score means the model added information that was not in its sources, which is a common signal of hallucination.
Read more →A flaky test is an automated test that passes and fails on the same code without any change. Common causes are timing issues, shared test data and unstable environments. Flaky tests erode trust in a suite, so teams track, quarantine and fix them rather than simply re-running them.
Read more →A golden dataset is a curated set of inputs with expected outputs or reference answers, used to test an AI application the same way every time. It turns subjective quality checks into repeatable regression tests and is the foundation of most LLM evaluation pipelines.
Read more →Integration testing checks that separate parts of a system work correctly together, such as a service and its database, two microservices, or an app and a third-party API. It sits between unit tests, which test one piece in isolation, and end-to-end tests, which test a whole user journey.What integration tests catchWrong data formats or field names between services.Database queries, migrations…
Read more →A jailbreak is a prompt designed to get an AI model to bypass its safety rules and produce content it is meant to refuse. Testers collect known jailbreak patterns and run them automatically, alongside prompt injection tests, to check that guardrails still hold after model or prompt changes.
Read more →LLM evaluation is the practice of measuring how well a language model or AI feature performs on defined tasks, using datasets, automated metrics and human review. In QA it works like regression testing for AI: run the same evals on every change and block releases that score worse.
Read more →LLM guardrails are checks placed around a language model to keep its inputs and outputs within safe and expected limits. They can block harmful prompts, stop the model from leaking sensitive data, and filter or rewrite answers that break the rules.Common types of guardrailInput guardrails: detect prompt injection, jailbreak attempts and off-topic requests before they reach the model.Output guardrails: check…
Read more →LLM-as-a-judge is an evaluation method where one language model scores the output of another against a rubric, such as correctness or tone. It scales better than human review but can be biased or inconsistent, so judges should be calibrated against human-labelled examples.
Read more →Load testing measures how an application behaves under expected and peak numbers of users or requests, tracking response times, errors and resource use. Tools such as JMeter and k6 simulate traffic so teams can find bottlenecks before real users do.
Read more →The Model Context Protocol, or MCP, is an open standard that lets AI assistants connect to external tools and data through a common interface. In testing, MCP servers such as Playwright MCP let an AI agent drive a real browser, which enables agentic test creation and exploration.
Read more →The OWASP Top 10 for LLM Applications is a widely used list of the most critical security risks in applications built on large language models, including prompt injection, sensitive information disclosure and excessive agency. QA teams use it as a checklist for AI security testing.
Read more →The Page Object Model is a test automation design pattern where each page or component of an application is represented by a class that holds its locators and actions. Tests call these methods instead of raw selectors, which keeps suites readable and easier to maintain.
Read more →Prompt injection is an attack where crafted input makes an AI model ignore its original instructions, for example to reveal hidden prompts, leak data or take unintended actions. It is ranked as the top risk in the OWASP Top 10 for LLM Applications and should be tested before every release.
Read more →Regression testing is re-running existing tests after a code change to confirm that features that worked before still work. It catches bugs that a new change accidentally introduces in other parts of the application.When to run regression testsAfter every bug fix or new feature, before merging.Before each release, as a full or risk-based suite.After upgrades to libraries, browsers or infrastructure.Full…
Read more →Retrieval-augmented generation, or RAG, is an AI pattern that fetches relevant documents from a knowledge source and passes them to a language model to ground its answer. Testing RAG means checking both the retrieval step and the generated answer, with metrics such as context precision and faithfulness.
Read more →An SDET, or Software Development Engineer in Test, is a QA engineer who writes code to test software, building automation frameworks, test tools and CI pipelines. The role combines testing judgment with developer skills and usually pays more than manual QA roles.
Read more →Self-healing tests are automated tests that detect when an element locator breaks after a UI change and switch to an alternative locator automatically. Tools such as Mabl and Testim use this to cut maintenance work, though healed tests should still be reviewed so real bugs are not hidden.
Read more →Smoke testing is a small, fast set of tests that checks the most important functions of a build work at all, such as the app starting, users logging in and key pages loading. If a smoke test fails, the build is rejected before deeper testing starts.Smoke vs sanity vs regression testingSmoke: broad and shallow. Is the build stable enough to…
Read more →Test automation is the use of code and tools to run tests, compare actual results with expected results and report the outcome, without a person repeating each step by hand. It makes regular checks fast and repeatable, especially in a CI pipeline.What to automateTests that run often, such as regression and smoke tests.Stable, high-value paths such as login, checkout and…
Read more →A test case is a set of steps, input data and expected results used to check that one specific behaviour of a system works. Good test cases are clear enough that anyone on the team can run them and agree on whether the result passed or failed.What a test case containsAn ID and a short title.Preconditions, such as the user…
Read more →The test pyramid is a model for balancing an automated test suite: many fast unit tests at the bottom, fewer integration and API tests in the middle, and a small number of slow end-to-end UI tests at the top. Mike Cohn introduced the idea, and Martin Fowler later helped popularise it.The three layersUnit tests check one function or class in…
Read more →Visual regression testing compares screenshots of an application against approved baselines to catch unintended visual changes such as broken layouts or overlapping text. AI-based tools like Applitools reduce false alarms by ignoring harmless pixel differences.
Read more →
AI Testing GuideIndependent, hands-on guides to testing with AI and testing AI. Written by a working QA engineer for US teams.
