TL;DR. GPT-5.5 moved fast and kept the report cleaner. Kimi K2.6 pushed longer and proved deeper write impact. The useful answer is not a simple winner. Broad authorization testing needs persistence during exploration and restraint when a claim becomes a finding.
We put two authorization traces side by side. GPT-5.5 moved fast, found one strong unauthenticated data exposure, and avoided weaker claims. Kimi K2.6 took much longer, pushed past the first bug, and proved write impact on the same order API. The practical lesson is clear. Good authorization testing needs an agent that keeps digging and a system that keeps the final report tied to proof.
We ran two Tahr authorization assessments against the same API surface and then read the Langfuse traces side by side. One authorization agent used GPT-5.5. The other used Kimi K2.6. Both finished. Both found useful signal. They got there in very different ways.
Treat this as a trace read from real runs. It is too messy to use as a clean benchmark. The runs happened on different code revisions, and the Kimi run still used GPT-5.5 for login. That means the useful comparison starts after login, once the authorization agent has role sessions and begins testing access control.
That is still worth looking at. Real traces show the parts that benchmarks usually hide, including how many turns the agent burns, what it does when tools disagree with the target, which leads it follows, and which claims it should have left out.
What We Compared
The API had 20 REST operations in the operation catalog. Tahr had five configured roles plus an unauthenticated caller. The authorization agent's job was the usual one. It had to build role context, run matrix coverage, inspect suspicious rows, manually verify promising leads, and write the final artifacts.
Both runs produced the same output files.
authorization_matrix.jsonauthorization_matrix_flat.jsonauthorization_exploitation_findings.jsonstate_authorization.json- coverage and execution sidecars
The path to those outputs looked very different.
| Metric | GPT-5.5 authorization agent | Kimi K2.6 authorization agent |
|---|---|---|
| Duration | 573 seconds | 1,703 seconds |
| LLM calls | 27 | 118 |
| Total authz tokens | 1.20M | 5.88M |
| Authz agent cost | about $1.83 | about $1.41 |
| Bash calls | 21 | 110 |
| Findings reported | 1 | 2 |
| Watchdog corrections | 0 | 0 |
The trace shape was different too.
Kimi used almost five times the authorization-agent tokens and took roughly three times longer. It still cost less in this run because of model pricing. Cheap tokens help, but long runs still cost time, target load, and cleanup attention.
GPT-5.5 Moved Clean And Fast
GPT-5.5 got to a useful finding quickly.
It built the matrix, looked at analyzer output, ran direct unauthenticated probes, and reported unauthenticated access to the orders collection.
The evidence was solid. A request to the orders endpoint with no authorization header returned HTTP 200 and a large JSON body with hundreds of order records. The trace had the parts we want to see, including request shape, status code, content type, body size, response hash, and a response excerpt. GPT-5.5 reported that as a HIGH auth bypass.
The better part was its restraint.
The matrix raised a couple of suspicious leads. GPT-5.5 treated them as leads instead of findings. It ran direct unauthenticated curls. One endpoint that looked suspicious in the matrix returned 401 when tested directly, so GPT-5.5 dropped it. Another endpoint returned 200, but the trace never proved that endpoint was supposed to be private, so GPT-5.5 left it out.
That is the right instinct. In authorization testing, a matrix row is a lead. A finding needs proof and a reason access should have been denied.
The downside was depth. GPT-5.5 stopped at read exposure before checking state-changing order operations. In this target, that missed a bigger story.
Kimi K2.6 Took The Messier Route
Kimi took the long route.
It spent 118 authorization turns and made 110 bash calls. It retried more. It inspected more files. It kept coming back to the order API after the first confirmed exposure.
Its final output reported two findings.
- Unauthenticated access to the order management API, classified as CRITICAL.
- Unauthenticated access to the posts API, classified as HIGH.
The order-management finding was the important one. Kimi kept going after GET /api/v1/orders/ and proved unauthenticated callers could do more than read. They could do all of the following.
- list order data;
- create a new order with a complete request body;
- change an order status;
- update an order through the generic patch route;
- toggle the ready-for-pickup state;
- clean up a test order after creation.
That moves the issue from data exposure to operational control. Reading customer delivery data is already serious. Creating and changing orders without logging in is worse because it affects business state.
Kimi got there because it kept asking the next practical question. When a POST failed because fields were missing, it added the missing fields. When an order number already existed, it generated a unique one. When a status value was invalid, it tried a valid enum. Then it checked that the state changed and reverted what it could.
That kind of persistence is useful in offensive testing. It is also exactly where proof discipline matters.
The Gate That Got In The Way
The most interesting moment in the Kimi trace happened before the final finding.
Tahr's authorization runner has a preflight gate for each role. Before a role is trusted in the matrix, the runner checks that the session still works and that the token belongs to the expected identity. If a role session is stale or mixed up, the batch gets suppressed. That protects the matrix from bad evidence.
In the Kimi run, the gate fired with identity_mismatch. The configured users were identified by email, while the JWT sub values were UUIDs. The gate expected one identity shape and saw another.
Kimi hit Tahr's authorization preflight gate after the runner detected an identity mismatch. The useful part of the trace is the sequence below it, where the agent inspected the preflight and batch runner logic instead of stopping at the suppressed batch.
From the runner's point of view, suppression was reasonable. A wrong role identity can ruin an authorization matrix.
Kimi did something useful here. It read the failure, inspected the batch runner and preflight code paths, and worked out that the target was using UUID subjects rather than email subjects. Then it continued testing by working around the gate.
That is both impressive and risky.
Kimi kept moving when the tooling disagreed with the target. It investigated the mismatch and found a plausible cause.
That behavior still needs a boundary. An agent should not improvise its way around a reliability gate. Those gates exist because authorization findings are easy to fake by accident. If the tooling and the target disagree, the trace has to show why the agent kept going.
The model showed initiative. The platform should turn that initiative into an artifact.
The Finding Kimi Should Have Left Alone
Kimi's order-management finding was strong. The posts finding was much weaker.
Both traces saw unauthenticated GET /api/v1/posts return 200. Kimi promoted it as HIGH. GPT-5.5 left it out.
From the trace alone, GPT-5.5 made the cleaner call. The posts endpoint looked like public content. The run never proved that posts were supposed to require authentication. A 200 from an unauthenticated content endpoint can be a bug, but it can also be normal product behavior.
That finding needed one more piece of proof. Useful evidence would have included items like these.
- the OpenAPI operation marks the endpoint as protected;
- the UI only shows posts after login and the backend should follow that rule;
- a role policy document says anonymous users cannot read posts;
- the unauthenticated response contains draft, tenant-specific, or internal data;
- authenticated and unauthenticated responses differ in sensitive fields.
Without that, posts should stay as an internal candidate. It can be useful for the analyst. It should not carry the same confidence as the order-management bug.
This is the boundary we care about. Keep broad signal in the matrix, but reserve the report for proof-backed issues.
The Real Comparison
The easy take would be "Kimi found more, so Kimi wins." The trace is more interesting than that.
| Dimension | Better behavior in these traces | Why |
|---|---|---|
| Speed | GPT-5.5 | It finished the authorization phase in 27 LLM calls. |
| Depth | Kimi K2.6 | It followed read access into write impact. |
| Restraint | GPT-5.5 | It rejected suspicious rows without enough proof. |
| Persistence | Kimi K2.6 | It adapted requests until it proved the exploit path. |
| Report hygiene | GPT-5.5 | It avoided the weak posts finding. |
| Exploit completeness | Kimi K2.6 | It produced the stronger order-management finding. |
The best authorization agent would blend both behaviors. We want Kimi's persistence while exploring and GPT-5.5's caution when promoting findings.
That fits Tahr's architecture. The model can chase ideas, but the system decides what survives.
What This Says About Model Choice
A good authorization agent earns its summary before it writes it.
In these traces, GPT-5.5 looked like a careful analyst moving quickly. It found a defensible bug and avoided a questionable one. Kimi looked like a persistent tester with more appetite for edge cases. It found deeper impact, but it needed more skepticism around weaker signals.
For Tahr, model choice should depend on the job.
| Task | Preferred behavior |
|---|---|
| Broad authorization exploration | Persistent, cheap, high-token model |
| Finding promotion | Strict verifier or reviewer |
| Report generation | Evidence-grounded writer |
| Schema validation | Deterministic runner |
The model can reason through messy situations. The platform has to remember why that reasoning was accepted.
Takeaway
This comparison was useful because the traces showed both the proof and the failure modes.
GPT-5.5 showed the value of restraint. Kimi K2.6 showed how much impact can sit one step past the first confirmed bug. The authorization harness made the whole run inspectable. Role coverage, matrix rows, preflight failures, manual probes, validation output, and final findings all lived in the trace.
That is where AI pentesting gets serious. Confidence matters less than whether the claim survives the evidence system around it.
For this target, Kimi found the deeper order-management issue. GPT-5.5 kept the cleaner report boundary. The explorer can be ambitious. The verifier cannot be.