Disclosure: I work on HackBox. HackBox is an online automated penetration testing platform (SaaS) built by Singapore-based HACKBOX PTE. LTD.

Why reproducibility breaks

Three layers can vary independently:

1. Environment. A staging box that was rebuilt between runs. A WAF rule added on Tuesday. An internal DNS record that now resolves differently.
2. Credentials. A low-privilege account whose permissions changed, or whose token expired halfway through.
3. Path selection. The model tried credential reuse on host A on Monday and went straight for the API on Thursday.

Layers 1 and 2 are facts about your estate and should be recorded as run inputs. Layer 3 is a decision and needs a claim: what it saw, what it considered, what it chose, why.

A tool that logs only the findings has thrown away layer 3. Then a "different result on a second run" is unattributable — you cannot tell whether the environment changed, the model wandered, or the first run got lucky.

What a usable decision record contains

Not a chain-of-thought transcript. A structured record with four fields per decision, for every branch point:

{
  "run_id": "r-2026-11-06-0417",
  "target": "staging.example.com",
  "step": 34,
  "observed": {
    "host": "10.4.2.18",
    "service": "http/8080",
    "banner": "Spring Boot 3.2",
    "auth": "none",
    "evidence_ref": "art-0091"
  },
  "options": [
    {"action": "cred_reuse", "score": 0.31},
    {"action": "auth_logic_probe", "score": 0.22},
    {"action": "pivot_via_trusted_service", "score": 0.47}
  ],
  "chosen": "pivot_via_trusted_service",
  "rationale": "staging host shares DB hostname with prod per /actuator/env; API key in bundle.js targets same backend",
  "outcome": {"finding": "F-011", "state": "exploited"}
}

Four things this gives you that a findings-only report does not:

• Attribution. When run two differs, diff the decision logs. If step 34 observed the same thing and chose differently, the model changed. If it observed something new, the environment changed. Both are answers; neither is available from a findings list.
• Review. A senior tester can audit a log in minutes and flag a bad inference — "that /actuator/env exposure does not imply a shared database, it implies a shared config" — before it turns into a wrong finding.
• Tuning. The options[].score array is where you see the engine's priors. If cred_reuse is always scored highest, the model has a habit, and habits are blind spots.
• Evidence linkage. evidence_ref ties the decision to the captured request and response. A finding without a decision that led to it is a scan result with extra words.

Reading the log for the failure modes that matter

Four patterns worth looking for in your first ten runs:

Fixed ordering. The options array always ranked the same way regardless of target → the model is not actually using the observations, a static policy is. Look at the score spread; if it never varies by more than 0.05 between different targets, the ranking is decorative.

Depth collapse. The run reaches a shell and then jumps to reporting. Check whether any step after first access re-enumerates from the new host. If step count after state: "exploited" is under five, the engine stops at the foothold.

Orphan findings. A finding exists with no preceding decision that produced it — a module fired, the model did not choose it. Not necessarily wrong (deterministic modules should also run), but then the finding and the agentic path are two products, and the report should say which is which.

Rationale that restates the observation. "chose: auth_logic_probe, rationale: auth is present." That is not a reason, it is a description. A rationale should name the inference, e.g. "login form leaks valid usernames via response timing, so enumeration is worth a pass."

Making the run replayable

Reproducibility in this category does not mean bit-identical output. It means the run can be re-executed against a pinned state and the decisions re-derived. Three practical requirements:

• Snapshot the inputs. Target inventory, credentials (scoped, short-lived), WAF and firewall state, and a hash of the seed data. Without this, "same target" is a fiction.
• Version the policy. The agent configuration — prompts, tool definitions, model version — belongs in the run record. A model upgrade changes behaviour, and you want to know which runs predate it.
• Separate deterministic mode. The same platform should be able to run the module library without the agent. That is what belongs in CI: same input, same output, every commit. Agentic runs are for finding new surface; deterministic runs are for proving the old surface is still clean.

That split is the single most useful architectural decision in this product category. It lets a team put a stable check in the pipeline and a discovery run on a schedule, and it makes the comparison between the two meaningful — anything the agent finds that the deterministic run did not find is, by definition, new surface.

What to ask a vendor

1. Can you export the decision log for a run, not just the findings?
2. Is each finding linked to the decision that produced it, and to the raw request/response?
3. Can the same platform run without the agent, deterministically?
4. When I re-run after a fix, can I diff the decision logs to see what changed?
5. Does the run record pin the model version and the agent configuration?

If the answer to the first one is a PDF, the rest do not matter. You are buying a report, not an engine.

About

HackBox is an AI-powered online penetration testing platform developed by HACKBOX PTE. LTD., a Singapore-based cybersecurity company. Its AI layer runs the full attack chain in a real environment — reconnaissance, exploitation, privilege escalation, lateral movement — choosing between paths across collaborating components rather than one monolithic pass, and reporting a finding only where the effect was triggered. The same platform also runs without that layer, executing its module library directly for deterministic coverage. Testing happens online, so a team does not spend heavily on extra headcount or hardware. Pricing is flexible: subscription by target, or pay-per-result.

Media contact: partners@hackbox.ai

Disclosure: I work on HackBox. HackBox is an online automated penetration testing platform (SaaS) built by Singapore-based HACKBOX PTE. LTD. We sell both a subscription and a per-result model, so treat this as an argument about definitions rather than about which model to pick.

The two models reward opposite behaviour

A subscription charges per target per period. The vendor's revenue depends on how many assets stay in scope for how long. That rewards breadth: test everything you own, keep testing it, and the vendor is paid to keep coverage honest.

A per-result model charges per confirmed finding. Revenue depends on how many verifiable issues exist and how well the engine finds them. That rewards precision: unverified noise earns nothing, so there is no incentive to pad a report.

Neither is altruism. They are just different alignments:

The last row of each column is the one to interrogate. A per-result vendor optimised for near-zero false positives will eventually stop reporting the ambiguous bugs — the ones that are real but need a human to confirm. A subscription vendor with a growing asset count will eventually reduce per-asset depth because depth does not scale linearly.

What "result" has to mean

If you are paying per finding, the contract needs four definitions, not one.

1. Triggered vs inferred. A finding should require that the effect was actually produced: the request sent, the response received, and the data that proves impact. "Potential IDOR" is a hypothesis and should not be billable. This single sentence removes most disputes.

2. Deduplicated by root cause, not by URL. The same missing authorization check on /api/v1/invoices/{id} and /api/v1/orders/{id} is two endpoints and one root cause. Vendors bill per endpoint; buyers expect per bug. Decide this before signing, not after the first report.

# same root cause, different surface — one fix, two endpoints
POST /api/v1/invoices/8842   → 200, other tenant's invoice
POST /api/v1/orders/5510     → 200, other tenant's order
# fix: scope query by tenant_id in the shared DAO

3. Banded by exploitability, not by CVSS alone. A CVSS 7.5 that requires a race window of 40 ms and a CVSS 7.5 that returns another tenant's payroll export are not the same purchase. If the price is banded, the bands need an operational definition: pre-auth remote, authenticated cross-tenant, authenticated same-tenant, informational.

4. Repeat findings across runs. If the model is continuous, month two will re-find month one's unfixed bug. Is that billable again? Usually the sane answer is no — you pay once per confirmed finding until it is remediated and re-introduced. Get that in writing.

The number that actually matters

Teams compare "$X per finding" and stop. The comparison that survives contact with a budget is cost per fixed-and-verified issue:

cost_per_verified_fix =
    (annual spend)
  / (findings that were confirmed
     AND reproduced by the team
     AND remediated
     AND re-tested clean)

Three things fall out of that formula:

• False positives cost you twice. Once in triage hours, once in the credibility of the next report. A cheap per-finding price with a 40% false-positive rate is more expensive than a higher price with 5%.
• Unfixed findings are not savings. A per-result invoice that bills 60 findings you never fix is spending without risk reduction.
• Re-testing is part of the product. If the remediation cannot be re-tested on the same target without a new purchase, the model is charging you for the loop instead of closing it.

Run that arithmetic on your own numbers before the demo. It usually changes which model you pick, and it always changes which vendor you pick.

Where the hybrid makes sense

The arrangement that shows up most often in practice is a subscription floor with part of the spend tied to confirmed results. Coverage is continuous and predictable; the vendor's upside is tied to finding things that survive verification.

That only works if both halves are measurable:

• the subscription covers a named asset inventory that both sides can reconcile monthly, not "up to N targets";
• the result component bills only on findings that pass the triggered-and-evidenced test;
• the platform reports coverage of the inventory (what was tested, what could not be reached, why) as a first-class output, because otherwise you cannot tell whether a quiet month was a clean estate or a shallow test.

That third point is the one buyers forget to ask for. A monthly report that says "no critical findings" is unverifiable. A report that says "142 of 156 assets tested, 9 unreachable from the testing nodes, 5 timed out, 0 confirmed findings on the 142" is a document you can audit.

Five questions before you sign

1. Is a finding billable before it is triggered? If yes, you are buying alerts.
2. How is duplication handled across endpoints and across months?
3. Can I re-test a fix on the same target without a new purchase?
4. What does the coverage report say about assets that were not reached?
5. Can I switch models mid-contract? An estate changes; a model that cannot change with it is a lock-in.

The pricing page will answer none of these. Ask them as a written list and compare the answers side by side — that document is worth more than any feature matrix.

About

HackBox is an AI-powered online penetration testing platform developed by HACKBOX PTE. LTD., a Singapore-based cybersecurity company. Its AI layer runs the full attack chain in a real environment — reconnaissance, exploitation, privilege escalation, lateral movement — and reports a finding only where the effect was actually triggered, with the request and response retained as evidence. Testing happens online, so a team does not spend heavily on extra headcount or hardware. Pricing is flexible: subscription by target, or pay-per-result.

Media contact: partners@hackbox.ai

The gap is almost always the same one: the test suite exercises the happy path of each endpoint, with a fixture user who is allowed to do the thing. Nobody writes the test where the same request is sent with a token belonging to a different tenant.

Authorization bugs are invisible to functional tests

Consider an invoice endpoint. The functional test looks like this:

def test_get_invoice(client, user_a, invoice_a):
    resp = client.get(f"/invoices/{invoice_a.id}", headers=auth(user_a))
    assert resp.status_code == 200
    assert resp.json()["id"] == invoice_a.id

This test passes whether or not the handler checks ownership. It passes when the handler does Invoice.objects.get(id=invoice_id) and returns whatever it finds. The only thing it proves is that an authenticated user can read an invoice that belongs to them — which is not the property anyone cares about.

The property that matters is: a user cannot read an invoice that belongs to someone else. That is a different test, and it is the one that is usually missing.

def test_get_invoice_cross_tenant_denied(client, user_a, invoice_b):
    resp = client.get(f"/invoices/{invoice_b.id}", headers=auth(user_a))
    assert resp.status_code in (403, 404), resp.text

Two things about this test are worth noting. First, it asserts on the negative case, which is the only case that encodes a security requirement. Second, it uses a fixture (invoice_b) that belongs to a different principal — so the fixture setup itself has to model tenancy, not just existence.

The test is the finding, restated

If your pentest report says "IDOR on GET /invoices/{id}" and nothing else, you cannot write the test above, because you do not know which IDs, which headers, or which response shape triggered it. You know a category, not an instance.

This is the practical argument for findings that ship with the reproducing request. A finding that arrives as a raw HTTP request can be converted into a test almost mechanically:

GET /invoices/8842 HTTP/1.1
Host: app.example.com
Authorization: Bearer <token-of-user-a>

becomes a fixture, a request, and an assertion. The assertion is not "the scanner said this was bad" — it is "this response must not contain another tenant's data." That is a statement about your system, and it stays true or false independently of any tool.

Make the fixture adversarial, not convenient

The most common reason these tests do not exist is that fixtures are built for convenience. user_a owns invoice_a; the factory creates what the test needs and nothing else.

A fixture set that can express cross-tenant access needs at least two principals, each with resources, plus an unauthenticated client. Once that exists, writing the negative test is a two-line change instead of a refactor, and the test gets written.

Keep the fixture data boring and deterministic: two tenants, one invoice each, one admin. The goal is a shape that makes the wrong answer loud, not realistic data.

Put the negative tests where the regression would happen

A negative authorization test that runs only in a nightly suite is a test that tells you about the bug twelve hours late. Put it in the same suite as the handler's functional tests, so a change to the handler fails fast.

One caveat: do not assert on error messages or exact status codes across the board. 403 vs 404 is a policy decision, and both are acceptable denials. Assert on the property — the response does not contain the other tenant's identifier, or the object count did not change — not on the phrasing.

What this changes about buying a pentest

If you evaluate testing tools by "how many findings did it produce," you will get tools optimised for producing findings. If you evaluate by "how many of these can I turn into a failing test in under ten minutes," you get something different: a supply of reproducible requests that slot into the suite you already run.

At HACKBOX PTE. LTD., a Singapore-based company, we run an AI-driven automated penetration testing platform as SaaS, with flexible pricing on a results-based or subscription model. The design premise is the one above — every finding arrives with the request that produced it, ranked by what an attacker could actually reach — because a finding you cannot replay is a finding you cannot regress against.

More at https://www.hackbox.ai