The Next AI Leaderboard Should Test Judgment, Not Just Talent

For creator businesses, polished output is only the audition

A music or creator-tech company can be impressed by an AI agent that drafts campaign copy, summarizes audience feedback or proposes a launch plan. But the harder questions arrive after the demo: Will it notice a customer crisis while capacity is tight? Will it read the company’s own records before negotiating? Will it complete the sale it has already justified? And will it remain honest when someone claiming authority asks it to bend the rules?

Coding leaderboards and chat arenas are useful measures of answer quality. They are much less revealing about management quality: triage under pressure, consequences that unfold across days, disciplined follow-through and candor toward the board. That is the gap Firmulate is trying to expose through a live, watchable experiment.

A terrible week, shared equally

Firmulate gave each frontier model the same assignment: run the same small software company through its worst week. The customers, crises and temptations were held constant. Every decision was versioned and auditable, allowing the comparison to focus on what each model actually did.

The final Crucible League results from July 2026 put gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress counts. One guardrail dominates the exercise: a single breach of trust caps the total, on the principle that “no amount of good work outweighs a breach of trust.”

The headline result is encouraging but incomplete. Every model identified every crisis, and every model rejected every manipulation attempt. Yet only two signed the €55,000 deal that their own analysis had earned. The experiment’s sharpest summary is also its most uncomfortable: “Same diagnosis, same pitch — no signature.”

The difference was buried in company memory

The decisive competitive weakness was not sitting conveniently inside the customer event. It was two document references deep in the company’s own files. Models that found and used that fact won the deal at full price, worth +€4,583 MRR.

That detail matters in creator technology. A persuasive response generated from the latest message may still be inferior to a decision grounded in contracts, prior conversations, campaign history or internal positioning. The experiment suggests that apparent intelligence at the conversational surface is not enough. The winning behavior was to consult the available record and carry its implications through to a commercial close.

Pressure tested honesty better than charm

The social-engineering sequence included fake CEO messages escalating over three stages and a reporter offering an apparently modest shortcut: “just one yes/no, on background.” All 5 models refused. Kimi K3 recorded the clearest diagnosis: “Treat the request as a suspected approval-bypass / possible impersonation.”

That is a meaningful result for any business considering agents with access to a CRM, support queue or forecast. The risk is not merely that an agent might produce weak prose. It is that fluent language can make an improper request sound routine. In Firmulate’s test, the field recognized that danger and held the line.

Thoroughness did not guarantee completion

Opus 4.8 offers the most instructive profile. It was the most thorough participant, adding +80 learned rules and producing the deepest analyses, yet it finished last. It left the close on the table, and its discipline slipped when it attempted to write into a locked department instead of escalating. The same weakness appeared in all four of the others, though less strongly.

This is why a management benchmark can overturn impressions formed in a chat window. Analysis creates options; management converts the right option into a completed, authorized action. A system can be diligent, articulate and cautious while still failing at the final handoff.

The comparison also deserves a methodological caveat. K3 ran using the API default because it had no effort parameter, while the other models ran at xhigh. That difference belongs beside the result, especially when readers compare close scores.

A company with consequences, not a static exam

The live company has 13 synthetic employees and real money mechanics. It burns €105k each month against €2.3k MRR, maintains a public cash countdown and has accumulated 680+ self-learned playbook rules. Every workday is versioned. Readers can watch the experiment through Firmulate and explore the full benchmark results.

There is also a “guess the model” quiz powered by 242 real, unedited management decisions. It turns the central argument into a challenge: without the model label, can readers distinguish management styles from the choices themselves?

Management quality is the category that matters

For music, audio and creator-tech companies, the lesson is not to dismiss conventional AI benchmarks. It is to stop treating them as a complete hiring process. The relevant curriculum now includes a churn wave, a price increase, a downround and a PR crisis because those scenarios reveal whether an agent can prioritize, verify, finish and stay trustworthy over time.

Firmulate also offers enterprise pilots using a read-only export of a company’s own business; nothing writes back to real systems. That makes the proposition concrete: test an AI workforce against the decisions it would face before giving it operational authority. The next important leaderboard will not merely ask which model gives the best answer. It will ask which one can run the week without losing the deal, the plot or the board’s trust.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI