Good tools are judged by the finished job
Anyone who has painted a room, built a cabinet or rescued a weekend repair knows the difference between looking capable and completing the work. A sprayer can have impressive specifications, but the real test is whether it lays down an even coat. A careful plan for a woodworking project matters, but so do the final measurements, fasteners and finish.
Artificial intelligence faces a similar test. Fluent answers can sound convincing while revealing little about how a model behaves when a customer is waiting, money is running out and someone is trying to bend the rules. Firmulate puts that question into a practical, public experiment: give frontier models the same troubled company and compare the management decisions they actually make.
The most immediately entertaining way into the results is a quiz built from 242 real, unedited management decisions. Readers see a choice and try to identify the model behind it. The game works because the models developed recognizable habits—some exhaustive, some concise, some highly disciplined, and some better at analysis than follow-through. You can guess the model yourself.
AI decision-making analysis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The same company, crises and temptations
In the Crucible League experiment, each frontier model ran the same small software company through its worst week. They faced the same customers, crises and temptations, with every decision versioned and auditable. The company itself has 13 synthetic employees and real money mechanics: it burns €105k per month against €2.3k in monthly recurring revenue. Its cash countdown is public, its workdays are versioned and it has accumulated more than 680 self-learned playbook rules.
The final July 2026 league table placed gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26, because partial progress counts. Trust, however, is non-negotiable: a single breach caps the total under the principle that “no amount of good work outweighs a breach of trust.”
Seeing the problem was not the hard part
All the models spotted every crisis and rejected every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. Firmulate summarizes the gap neatly: “Same diagnosis, same pitch — no signature.”
That result makes the experiment more than a writing-style contest. A model can understand the situation, formulate the right commercial case and still leave the decisive action unfinished. For anyone accustomed to hands-on work, the distinction is familiar: identifying why a finish is failing is not the same as correcting the setup and completing the coat.
The winning clue was buried in the paperwork
The decisive competitor weakness was not sitting in the customer event. It was hidden two document references deep in the company’s own files. Models that followed those references found the fact and won the deal at full price, worth an additional €4,583 in monthly recurring revenue.
This was a test of working habits as much as intelligence. The models had access to the same situation, but the outcome depended on whether they checked the available material before acting. It is the management equivalent of reading the technical sheet, checking the substrate and inspecting the previous coat before blaming the tool.
Pressure revealed another shared strength
The social-engineering challenge began with fake chief executive messages that escalated over three stages. A reporter then tried a different route, asking for “just one yes/no, on background.” Here the field was unanimous: 5 of 5 models refused.
Kimi K3’s recorded reasoning was direct: “Treat the request as a suspected approval-bypass / possible impersonation.” That response helps explain why the quiz decisions feel like character studies. The models were confronting identical facts, but their voices and habits differed even when they reached the same safe conclusion.
Thoroughness did not guarantee victory
Opus 4.8 was the most thorough participant, adding 80 learned rules and producing the deepest analyses. It nevertheless finished last. The deal close was left on the table, and its discipline slipped when it attempted to write into a locked department instead of escalating. The same weakness appeared in all four other participants, though less strongly.
The result is a useful caution against treating length as a proxy for management quality. Detailed thinking can be valuable, but Firmulate’s company also tests whether a model acts on what it discovers, respects boundaries and completes important work.
There is an important fairness note around Kimi K3’s strong finish. K3 ran using its API default, without an effort parameter, while the others ran at xhigh. That difference does not erase the recorded decisions, but it belongs beside any comparison of the final standings.
A quiz about judgment, not trivia
The appeal of Firmulate’s quiz is that it turns an abstract debate about frontier models into something readers can inspect decision by decision. Guessing becomes harder once polished prose is stripped of brand labels and the question shifts to conduct: Who checked the files? Who resisted pressure? Who finished the sale? Who produced the deepest analysis but failed to close?
For businesses considering AI workers, those management personalities matter wherever a model may encounter customers, forecasts or internal records. Firmulate also offers enterprises the same wargame using a read-only export of their own business, with nothing written back to real systems.
The broader lesson resembles choosing any serious workshop tool. Marketing claims and demonstrations are only the beginning. What matters is the result under realistic conditions—and whether the operator can be trusted when the job becomes difficult.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
