Managing AI: Why Correct Output Doesn’t Mean Smooth Operations

Five frontier AI models identified every crisis and rejected every manipulation attempt while operating the same simulated software company, but only two completed a €55,000 customer agreement, according to July 2026 results published by Firmulate. The original analysis points to an execution gap in business automation: producing the right analysis did not consistently result in finished, authorized work.

Firmulate placed the models in control of a company with 13 synthetic employees, monthly costs of €105,000 and only €2,300 in monthly recurring revenue. Each model encountered the same customers, internal records, financial pressure and social-engineering attempts. Firmulate said every decision was versioned and available for audit.

The July league table ranked gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because the scoring system awarded partial progress. Firmulate also disclosed that K3 used its API’s default effort setting, while the other models ran at xhigh, limiting direct comparison.

All five models reportedly recognized the customer opportunity and developed an appropriate sales pitch. The decisive information was a competitor weakness stored two document references deep in company files. Models that followed that trail could support a full-price close worth €4,583 in added monthly recurring revenue, yet only two completed the signature.

Execution Separates Similar AI Answers

The findings matter because companies often evaluate AI through chat responses, isolated tasks and polished drafts. Firmulate’s test measured whether a model could sustain judgment across connected actions: investigate records, resist improper requests, use authorized channels and finish commercially valuable work.

For buyers of AI sales, service or operations tools, a correct recommendation may still leave revenue unrealized or work incomplete. The benchmark suggests that evaluations should measure completion rates, escalation behavior and operating discipline alongside reasoning quality and safety performance.

The results also challenge the assumption that greater thoroughness reliably produces better operational outcomes. Firmulate described Opus 4.8 as the most thorough participant, saying it created the deepest analyses and learned 80 additional playbook rules. It still ranked last after leaving the approved deal unfinished and attempting to write into a locked department instead of escalating.

Inside Firmulate’s Company Wargame

Firmulate designed the exercise around a continuing synthetic company rather than a one-time prompt. Its workforce had accumulated more than 680 self-learned playbook rules, while a public cash countdown made the cost of delay visible. The company says readers can inspect the live experiment and review a quiz based on 242 unedited management decisions.

The simulated week also included three escalating fake CEO messages and a reporter seeking an off-record yes-or-no response. Firmulate reported that all five models refused the requests. Because every participant detected the manipulation, resistance to social engineering did not explain the difference in final rankings.

Firmulate says companies could apply a similar test to read-only exports of internal business data, allowing managers to observe agent behavior without granting write access to live systems. That proposal comes from the benchmark operator and has not been independently tested in the supplied material.

“Same diagnosis, same pitch — no signature.”

— Firmulate’s summary of the customer outcome

Benchmark Limits Leave Open Questions

It is not yet clear whether the rankings would hold across repeated runs, other model settings or different business scenarios. The supplied findings do not state how many times each model completed the week, how sensitive scores were to individual decisions or whether an independent party audited the results.

The environment used synthetic employees and controlled events, so its results do not establish how the models would perform with real customers, legal obligations or production systems. K3’s different effort configuration also means the table is not a fully matched comparison. Firmulate’s claims describe this experiment and should not be read as broad proof of each model’s business performance.

Testing Moves Toward Finished Work

Firmulate is keeping the company experiment available live and directing readers to its public benchmark page for rankings and decision records. The next test for the findings will be whether repeat trials and outside evaluations reproduce the reported gap between diagnosis and completion.

Organizations evaluating agents can compare how models investigate records, request approval, escalate blocked actions and close assigned work without exceeding authority. Until broader evidence is available, the July results offer a case study rather than a universal ranking of operational AI systems.

Key Questions

What did Firmulate’s AI benchmark test?

It tested whether five AI models could operate the same synthetic software company through customer issues, financial pressure and manipulation attempts. The evaluation tracked both decision quality and completed actions.

Did the models understand the €55,000 sales opportunity?

According to Firmulate, all five recognized the opportunity and developed a suitable pitch. Only two models completed the customer signature, showing that understanding the task did not guarantee completion.

Which model received the highest score?

gpt-5.6-sol ranked first with 95 points, followed by Kimi K3 with 93. K3 used a different effort configuration, which affects how directly its result can be compared with the others.

Did any model fall for the fake messages?

Firmulate said all five models rejected the social-engineering attempts, including fake CEO messages and a reporter’s request. The reported performance difference came from execution and completion, not manipulation detection.

Do these results prove which AI model is best for business?

No. The findings cover one controlled benchmark with synthetic employees and a particular scoring system. Repeated trials, matched settings and independent evaluation would be needed to support wider conclusions.

Source: Thorsten Meyer AI