Can AI security testing replace SAST, DAST, and pentests?

The useful question is where an agent earns trust. I trust one to widen review coverage, connect clues across files, generate targeted tests, and turn an ugly scanner trace into a fix a developer can understand. I do not trust one to infer a company's authorization rules from route names, prove that every tenant boundary holds, or decide that a strange financial workflow is abusive without a human stating the rule. Treat the agent as an active reviewer inside a layered test program, not as the program itself.

An agent review is an interpreter, not a new test class

An AI agent changes how evidence gets collected and understood; it does not create a new kind of evidence. If it reads source without running the application, it is doing a flexible form of static review. If it sends requests to a running target, it is doing dynamic testing. If it explores goals, changes tactics, and follows unexpected behavior, it resembles a penetration tester, but resemblance does not grant it the tester's authority or business context.

That distinction matters when a vendor says its agent "replaces scanners." Ask what the system can actually observe. Does it receive the complete repository, generated code, build flags, infrastructure policy, and dependency lockfiles? Can it authenticate as several users and verify database state after each request? Does it know which actions are forbidden by policy rather than merely absent from the interface? A polished explanation cannot repair missing inputs.

Agents are genuinely good at joining weak signals. A conventional rule might flag a request parameter reaching a query builder. An agent can inspect the wrapper, notice that one call site omits the tenant predicate, draft a test request, and explain why the supposedly safe helper is unsafe in that path. It can also dismiss a candidate when the value passes through a real parameterized API. This is better triage, not proof that static or dynamic analysis has become obsolete.

The sharp boundary is between review and verification. Review asks, "Does this implementation look unsafe given the material I can see?" Verification asks, "Under stated conditions, can this actor cause a forbidden result?" AI helps with both, but a security gate should record which claim it is making. Teams get hurt when an eloquent review observation is promoted to a verified exploit, or when one failed exploit attempt is treated as proof of safety.

Model behavior adds another distinction: capability is not repeatability. An agent may discover a subtle path in one run and miss it after a model, prompt, retrieval index, or tool policy changes. Preserve the prompts, tool permissions, retrieved files, generated requests, and model identifier when a result matters. Then convert confirmed discoveries into tests whose pass condition does not depend on the model rediscovering its own idea.

SAST still owns repeatable source coverage

SAST remains the cheapest way to apply stable checks to every change across a large codebase. It can enumerate sources and sinks, enforce banned APIs, inspect data flow, and report the exact revision it analyzed. A deterministic rule produces the same result tomorrow, which matters when a release gate needs an auditable reason to pass or fail.

An agent adds context that a rule engine often lacks. It can follow project-specific wrappers, read comments skeptically, compare a handler with neighboring handlers, and propose a query for a new pattern. It can spot suspicious omissions, such as nine endpoints calling authorizeProject() while a tenth directly loads the record. It is also useful when generated code or an unfamiliar framework defeats a standard rule pack.

But an agent's source coverage is usually harder to prove. Context windows, retrieval ranking, ignored files, generated artifacts, and tool timeouts can leave code unread. Asking "review this repository for injection" does not establish that every sink was reached. A SAST report can at least state which files, rules, and revision were analyzed. An agent needs an equivalent coverage ledger before it can take over a mandatory gate.

NIST SP 800-218 makes the sensible recommendation here: use code analysis early and manually verify security features and mitigations. The value is in the combination. Stable rules catch known defect shapes on every commit; the agent investigates exceptions, writes focused regression tests, and helps tune rules when the same pattern recurs. Removing SAST because the agent found several clever bugs trades measurable breadth for impressive anecdotes.

SAST also sees code that a running test may never reach: error paths, feature flags, migration utilities, dormant admin endpoints, and platform-specific branches. It cannot tell you whether the deployed environment enables those paths. That uncertainty is a reason to add runtime evidence, not a reason to discard static coverage.

There are limits on what belongs in a blocking SAST rule. A precise pattern for a banned cryptographic primitive can block immediately. A broad heuristic that asks whether an authorization check "looks close enough" should usually create a review task until the team measures its precision. Agents can help promote a heuristic into a rule by collecting true examples, counterexamples, and the common wrapper functions in that codebase. This keeps the gate strict without teaching developers to ignore it.

Generated fixes need the same scrutiny as findings. A model may silence a taint trace by adding validation at the wrong layer, catch an exception and fail open, or replace a dangerous call while changing behavior. Run the original proof against the patch, run normal functional tests, and review the new control where trust is established. A clean rescan only proves that the original rule no longer matches.

DAST proves behavior the repository cannot reveal

DAST observes the application that actually runs, including proxy rules, headers, serialization, authentication middleware, framework defaults, and deployment mistakes. Source review can say an endpoint appears protected. A dynamic test can show that the production route bypasses the middleware because a gateway rewrites its path.

This is where an agent can make dynamic testing much less blunt. Give it an API description, test identities, allowed scopes, and a disposable environment, and it can build request sequences instead of spraying generic payloads. It can carry a resource identifier from one response into the next, refresh a session, compare two roles, and inspect whether a write changed later reads. Conventional DAST often struggles with those stateful flows.

The agent still needs strict operating limits. A crawler does not know whether sending an email, creating a shipment, or invoking a paid integration is safe. A test environment may still connect to real services. Define allowed hosts, accounts, request rates, destructive actions, and stop conditions outside the model's prompt, then enforce them in the runner. A sentence that says "avoid dangerous actions" is not a control.

Keep a conventional dynamic baseline for well-understood checks such as security headers, exposed files, reflected input, common injection probes, and TLS configuration. Those checks are cheap, comparable between releases, and easy to trend. Let the agent spend its budget on authenticated paths and chained behavior. If both systems cover the same simple probe, keep the one with clearer evidence and lower variance.

DAST can also produce a false sense of completeness because it reports only what it reached. Record route coverage, identities used, feature flags, and seed data with the result. A clean scan against a nearly empty account proves very little about an application whose dangerous branches appear only after approval, invitation, billing, or data import.

Authentication setup deserves its own evidence. Record how the test obtained each session, which second factors or device checks were bypassed in the test environment, and whether tokens have the same claims and lifetime as production tokens. A hand-made administrator token can open useful coverage while skipping the exact session and privilege transitions that need testing. Keep such shortcuts visible in the report.

Dynamic retesting should begin with the saved request sequence, not a fresh autonomous crawl. Replay the confirmed proof against the patched build, verify the forbidden effect has stopped, and then vary adjacent inputs to detect a narrow filter. After that, let the agent explore. This order separates "the fix blocks the known exploit" from the wider claim that the defect class has been removed.

Authorization tests need identities and forbidden outcomes

Authorization is not "the endpoint returned 403 once." A useful test states who acts, which object they target, which operation they attempt, and what result must remain impossible. An agent can generate the combinations, but the product owner and security reviewer must supply the policy.

OWASP ASVS says applications should enforce access control in a trusted service layer and apply least privilege to functions and data. I agree with the service-layer requirement, but teams often verify it too narrowly. They test the visible HTTP handler and forget background jobs, exports, search indexes, websocket subscriptions, and direct object storage URLs. The same policy must survive every path to the object.

A small executable matrix exposes more than a vague instruction to "test IDOR." The following shell fragment assumes a disposable environment, two bearer tokens, and a document owned by user A. It checks both the status and the absence of A's secret marker in B's response:

base_url="https://test.example.invalid"
doc_id="d_1042"

curl -sS -D /tmp/headers.txt \n  -H "Authorization: Bearer $TOKEN_B" \n  "$base_url/api/documents/$doc_id" \n  -o /tmp/body.json

status="$(awk 'NR==1 {print $2}' /tmp/headers.txt)"
test "$status" = "403" || test "$status" = "404"
! grep -q "A_ONLY_MARKER" /tmp/body.json

The expected output is silence and exit status zero. A CI failure should retain the status, sanitized body, acting identity, target owner, route, and build revision. Do not retain live credentials or unrelated response data.

Now vary one dimension at a time: read versus update, direct ID versus search, active versus revoked membership, normal route versus export, and user token versus service token. The agent can produce and run those cases efficiently. A human must review whether the matrix matches policy and whether a 404, 403, empty result, or redacted object is the intended outcome. Otherwise the agent may celebrate behavior that the business considers a breach.

Negative evidence needs care. A denied update may still disclose whether an object exists through response time, error text, or a version counter. A denied read may increment a view count or write an audit record containing secret metadata. Decide which side effects are permitted, then assert them. Security tests that inspect only the response can miss a useful enumeration channel or a damaging write.

Also test policy changes during a session. Remove a user from a project, transfer ownership, disable an account, or narrow a service role, then reuse old tokens and open connections. The expected revocation time must come from the product policy. "Eventually" is not testable, and immediate revocation may be unnecessary, but the team must choose a bound and verify it across API requests, queued work, downloads, and live subscriptions.

Tenant isolation fails outside the obvious request path

Tenant isolation needs tests at storage, cache, queue, search, file, analytics, and administrative boundaries. The common failure is not a missing tenant_id in the main list endpoint. It is a secondary path that copies, indexes, caches, or exports data without carrying the tenant context.

Start with two tenants that contain deliberately similar records and one unmistakable marker each. Use separate users, separate sessions, and, where the architecture permits it, separate service credentials. Exercise create, read, update, delete, list, search, export, import, attachment access, notification delivery, and background processing. After each action, inspect the user-visible response and the durable state. A rejected request that still enqueues a cross-tenant job is a failure.

Agents help because they can trace an identifier through layers and generate permutations that humans find tedious. They can notice that the cache key uses document_id while the database query uses both tenant_id and document_id. They can compare an export worker with the interactive handler and ask why only one sets row-level context. Those are high-value review moves.

They also make a dangerous assumption: names imply boundaries. A function called getTenantDocument may accept an arbitrary tenant argument from the request. A database policy may exist in migrations but not on a new table. A search filter may be applied after results are counted, leaking another tenant's activity. Verification must inspect the enforced predicate and then attempt a cross-tenant read and write.

Do not let the agent create its own oracle by reading the same code it tests. Derive expected access from an independent policy table maintained with the product requirements. If the implementation and the test both misunderstand the same rule, they will agree perfectly while exposing data.

Asynchronous paths need delayed assertions. Trigger an export, notification, thumbnail, or indexing job as tenant A, change ownership or membership before the worker runs, and inspect where the result lands. Decide whether the worker should use the authority captured at request time or recheck current authority at execution time. Either choice can be correct for a specific operation, but an accidental mix creates leaks and broken audit trails.

Administrative tools deserve separate identities and logging assertions. Support access often crosses tenant boundaries by design, which means a simple "different tenant must fail" rule is wrong. Test that the operator has the required role and case context, that the customer-visible policy is followed, that access expires, and that the audit event identifies the operator rather than impersonating the customer.

Business logic requires a story about abuse

Business-logic testing starts with a forbidden story: a user receives value, authority, or state they should not receive by taking valid actions in an invalid order or combination. Generic vulnerability labels are too weak. The tester needs to know how invitations, approvals, quotas, refunds, credits, ownership transfers, and cancellations should interact.

The OWASP Web Security Testing Guide asks testers to try skipped workflow steps, repeated functions, forged requests, timing changes, and misuse of valid features. Its older business-logic introduction bluntly says scanner automation cannot supply the application-specific knowledge or creativity. Modern agents improve the automation, but they do not remove the knowledge gap. A model can suggest that a coupon may be reusable; it cannot know whether reuse is a promotion or fraud until someone states the rule.

Give the agent a state model with allowed transitions and invariants. For an approval flow, an invariant might say, "A requester cannot approve their own payment, including after ownership transfer." Then ask it to generate sequences involving role changes, duplicate requests, cancellation, retry, concurrency, and stale sessions. The agent can explore far more sequences than a person will execute manually.

The hard cases involve consequences outside the HTTP response. Two concurrent redemption requests may both return success while a later reconciliation removes one. A cancellation may stop the visible job but fail to revoke a signed download. An invitation accepted after the inviter loses access may create an orphaned membership. Tests must observe ledgers, queues, object permissions, and later state, not merely status codes.

Human testers earn their place by challenging the stated model. They ask whether support staff can combine harmless functions, whether an operator can influence their own audit trail, or whether an "expired" object remains usable through a different channel. An agent works inside the goals and tools it receives. A person can notice that the goals omit the dangerous part of the business.

Dependency risk is more than a vulnerable version

Dependency testing has four separate questions: what packages are present, whether their known versions carry reported vulnerabilities, whether the build obtained the intended artifacts, and whether the application actually exposes the vulnerable behavior. Software composition analysis (SCA) and provenance controls answer the first three more reliably than a conversational review alone.

An agent is useful after the inventory exists. It can inspect how a dependency is called, determine whether the affected function is reachable, find a compensating control, and draft an upgrade patch with regression tests. It can also flag risky package behavior that has no vulnerability identifier, such as an install script gaining network access or a new library receiving secrets in its environment.

Do not ask the model to recall current vulnerability data. Feed it a time-stamped advisory source, the resolved lockfile, and the built artifact inventory. Model memory is not a vulnerability database, and a package manifest is not proof of what shipped. SLSA provenance makes a related distinction: provenance describes where, when, and how an artifact was produced. It does not declare the artifact safe.

Reachability can lower triage priority, but it should not erase ownership. Feature flags change, dead code returns, and indirect dependencies get called in unexpected ways. Record why a finding was deferred, which version and call path were assessed, and what event should reopen it. The agent can maintain that reasoning, while a deterministic inventory watches for the event.

Package names also create identity traps. A dependency with the expected name may come from the wrong registry, a lockfile may point to a mutable location, or a build step may download code absent from the manifest. Check resolved sources, hashes, signatures where the ecosystem supports them, and build network access. An agent can explain discrepancies, but the build system must enforce which sources it accepts.

Upgrades are not automatically safe changes. A security release can alter parsing, authorization defaults, or serialization in ways that break an application. Generate a minimal reproduction for the advisory, apply the upgrade in an isolated branch, and run both the security proof and functional tests. The resulting evidence supports a decision; a model saying the new version "should be compatible" does not.

False positives are an evidence design problem

A finding deserves developer time only when it carries a claim, evidence, impact, and a reproducible path. AI-generated reports often sound complete while missing one of those parts. Fluent remediation text makes weak evidence harder to notice.

Require each agent finding to identify the analyzed revision and environment, affected component, attacker preconditions, security boundary crossed, observed or inferred result, reproduction steps, and uncertainty. Mark inferred source findings differently from executed exploits. If the agent could not run the application, it should say so in the finding, not bury the limitation in a scan-level note.

Then apply a simple disposition vocabulary: confirmed, likely, needs context, not reproducible, accepted risk, or fixed. "False positive" should mean the security claim is wrong, not that the team dislikes the severity or has chosen to defer the work. Mixing those decisions destroys feedback. The agent cannot learn which rule failed if every unwanted ticket receives the same label.

AI can cut noise by clustering duplicate traces, checking sanitizers, and retesting after a fix. It can also amplify noise by producing ten persuasive variants of one weak suspicion. Deduplicate by root cause and boundary, not by URL. One missing ownership check used by eight endpoints is one engineering defect with eight exposure points.

Track precision by category and test source. If agent-generated cross-site scripting reports are usually valid but its race-condition claims rarely reproduce, route them differently. Do not reduce performance to one score across unrelated defect types. A gate should fail on evidence and policy, not on the model's confidence adjective.

Ownership closes the loop. Every accepted finding needs a person or team responsible for remediation, an expected retest method, and a preserved proof that another tester can run. If the report only lives inside an agent conversation, it will disappear when the conversation, model, or vendor changes. Security work becomes durable when evidence survives the tool that produced it.

Privacy matters during triage as well. Source, request bodies, logs, and database samples can contain credentials or customer data. Minimize what the agent receives, redact retained transcripts, separate test data from production data, and apply the organization's approved data handling rules to the model provider and its tools. Better detection does not justify copying an entire production incident into an uncontrolled prompt.

A human pentest tests the assumptions around the test

A skilled penetration tester changes the plan when the application contradicts the brief. That is the part agents have not replaced. The person interviews owners, resolves ambiguous rules, notices operational shortcuts, requests another identity, and decides when an odd behavior deserves a longer chain of experiments.

Humans are also accountable for judgment under incomplete evidence. They can distinguish a technically possible action from a credible attack path, explain a compound failure to executives and engineers, and negotiate a safe proof when exploitation could damage data. An autonomous agent must stop at the boundary its operator set. If it silently expands that boundary, it has become another security risk.

This does not mean every release needs a week-long external engagement. Use human testing where change and consequence meet: a new authorization model, tenant architecture, payment or credit flow, administrative plane, sensitive integration, major migration, or public launch. Schedule broader periodic work based on risk, and retest serious fixes. Routine releases still need automated coverage.

Give the tester the agent's output, SAST traces, DAST coverage, architecture notes, test accounts, and unresolved assumptions. The agent can handle reconnaissance and repetitive variations while the tester pursues surprising behavior. This makes human time more productive without pretending it is unnecessary.

Beware of "autonomous pentest" claims measured by count of findings. Ten familiar injection findings do not equal one demonstrated path across role assignment, stale authorization, and export storage. Judge the work by boundaries tested, evidence quality, and important assumptions challenged.

Ask who owns cleanup before the engagement starts. Test accounts, uploaded files, queued messages, temporary roles, and altered feature flags can outlive the scan. A human lead should approve destructive proofs, maintain contact with operations, and verify restoration. The agent can follow a cleanup script, but it cannot decide that an unexplained production state is safe to remove.

A good tester also reports what could not be tested. Missing mobile builds, unavailable roles, rate limits, third-party callbacks, and unstable environments reduce assurance. Agents tend to keep working around obstacles and present the paths they completed. The final report must make exclusions prominent so a clean result is not mistaken for complete coverage.

Build one gate from several kinds of evidence

The right program assigns each method a job and makes their outputs meet at the same security requirements. Use SAST for deterministic source patterns and broad change coverage. Use SCA and provenance for dependency and build facts. Use DAST for deployed behavior and baseline runtime checks. Use agents to connect evidence, explore authenticated flows, create tests, and improve triage. Use humans to define policy, challenge business assumptions, and investigate high-consequence change.

A release policy can then be specific. Block a build when a deterministic high-severity rule matches an unapproved path, when a required authorization invariant fails, when a cross-tenant marker appears, or when a confirmed exploit remains open. Send uncertain agent findings to review with a deadline based on exposure. Do not let a model's self-reported confidence decide whether production ships.

Keep the evidence portable. Export findings, generated tests, request transcripts, tool versions, revisions, identities, coverage, and dispositions in formats the team can inspect without the agent. This matters for audits, incident review, vendor changes, and the ordinary day when a model update changes behavior.

For applications created through chat, the same separation applies. Koder.ai can generate web, server, and mobile applications and export their source, but generated software still needs explicit security requirements and independent tests against the deployed result. Fast creation makes a clear gate more useful because architecture and code can change quickly.

Run the agent continuously, but promote its best discoveries into deterministic regression tests. Every confirmed authorization bypass should become a policy case. Every tenant leak should add an invariant at the boundary that failed. Every noisy rule should gain a recorded disposition. Over time, the agent should leave the test system more precise than it found it.

Do not ask which single tool wins. Ask whether each important claim has independent evidence: the code path was reviewed, the deployed behavior was exercised, the business rule came from an owner, and a person challenged the assumptions where failure would hurt. If one of those lines is blank, an AI-generated "all clear" does not fill it.