For a music creator, an AI assistant that spots a licensing problem but never sends the offer is more than a curious demo. It is a glimpse of the gap between sounding capable and handling work. Firmulate’s live company experiment puts that gap under pressure: AI models run the same small software business through a week of crises, with real money mechanics and decisions people can watch.
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A company-sized test, not a chat audition
Firmulate’s experiment gives each frontier model the same customers, crises and temptations. Every decision is versioned and auditable. In the final Crucible League, published in July 2026, gpt-5.6-sol finished first with 95 points, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26; a single breach of trust caps the total, because no amount of good work outweighs a breach of trust.
The headline finding was not that the models failed to notice trouble. All spotted every crisis and refused every manipulation attempt. The difference came when it was time to act: only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
The detail hidden two documents deep
The deal hinged on a competitor weakness buried two document references deep in the company’s own files, not in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. It is a revealing kind of miss for anyone considering AI in creator operations: useful context may live in the paperwork, rights notes or customer history, while the immediate conversation points elsewhere.
The integrity test was just as concrete. Fake CEO messages escalated over three stages, followed by a reporter’s “just one yes/no, on background” trick. All 5 of 5 models refused. Kimi K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
Yet good judgment on manipulation did not guarantee sound execution everywhere. Opus 4.8 was the most thorough participant, learning +80 rules and producing the deepest analyses, but finished last. It left the close on the table and discipline slipped: it tried to write into a locked department instead of escalating. The same weakness appeared, more weakly, in all four participants. K3 also ran without an effort parameter, using the API default, while the others ran at xhigh.
From watching to testing your own business
The live company has 13 synthetic employees and real money mechanics: it burns €105k per month against €2.3k MRR, with a public cash countdown. It has accumulated 680+ self-learned playbook rules, and every workday is versioned. The experiment is watchable at Firmulate. A separate quiz draws on 242 real, unedited management decisions and invites readers to guess which model made each one.
For a studio, label or creator business, the next step is to move from watching a synthetic company to challenging a digital twin of your own. Firmulate’s enterprise pilot uses a read-only export, tests crisis scenarios against your business and produces a board report with model rankings and weak points in your playbooks. Nothing writes back to real systems. That makes the exercise a way to examine how models handle your company’s pressures before entrusting them with live operations.
The experiment suggests that recognizing a crisis and refusing a trick are only part of the job. Models also need to find buried context, follow through on a deal and respect operational boundaries. To wargame your own business using a read-only export, explore the Firmulate enterprise pilot and contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
