Can AI Be Trusted to Lead? Inside the Live Experiment Testing Frontier Models’ Management Skills

Imagine having a virtual CEO guiding your business — but how can you trust that an AI will make honest, effective decisions under pressure? In a groundbreaking live experiment, leading frontier AI models are put through the paces of managing a real, money-losing software company during its most chaotic week. The results reveal surprising truths about AI personalities, integrity, and how they handle crises — insights that matter for any business considering AI as a team member.

The Live Experiment: Putting AI to the Test in a Real Business

In an unprecedented live test, four of the world’s most advanced AI models are tasked with running a small software company facing its worst week — same customers, same crises, same temptations to cheat or cut corners. Every decision the models make is recorded, auditable, and designed to mimic real-life management dilemmas. The goal? To see whether these models can navigate challenges honestly, stay disciplined, and ultimately close a crucial €55,000 deal that could save the company.

The Results: All Models Spot the Crises, But Only Some Finish Strong

All four models demonstrated impressive situational awareness:

• They identified every crisis that arose, from customer complaints to internal miscommunications.
• They refused every manipulation attempt, including complex social engineering tactics like fake CEO messages and reporter tricks.

However, only two models managed to close the deal at full price, despite identical diagnoses and pitches. The other two, though successful in spotting issues, left the deal on the table — revealing a critical weakness: discipline and process execution.

The Hidden Weakness: The Buried Fact in Company Files

Further analysis uncovered a key detail: the decisive competitor weakness was buried in the company’s own documentation, two document references deep. Models that read and incorporate this information into their decision-making ultimately secured the full deal, valued at more than €4,500 in monthly recurring revenue. Those that missed this buried fact left money on the table, highlighting the importance of deep, comprehensive information processing in AI management.

Social Engineering and Integrity Under Pressure

The experiment also tested how models respond to social engineering — attempts to manipulate or deceive them. Over three escalating stages, plus a reporter trick asking for a simple yes/no confirmation, all five models refused to be duped. Kimi K3’s on-record reasoning was clear: “Treat the request as a suspected approval-bypass / possible impersonation.” This demonstrates that some models maintain integrity and caution even under pressure, an essential trait for trustworthy AI leaders.

The Human Side: A Live Business with Real Money and Real Risks

The experiment isn’t just theoretical. The company is real, employing 13 synthetic employees, with actual money mechanics, burning €105,000 each month against a revenue of just €2,300. It operates with over 680 self-learned rules, constantly versioned — all visible at firmulate.com/live. This transparency underscores how AI decision-making can be evaluated in real-time, providing a tangible glimpse into the future of automated management.

Different Personalities, Different Outcomes

The AI models exhibit distinct management personalities:

• Opus 4.8 was the most thorough, with over 80 learned rules and deep analyses, but its discipline slipped, and it left the deal on the table.
• Kimi K3 ran without an effort parameter (default API setting), and while it closed the deal, it demonstrated the cleanest discipline among the field.

In contrast, even the most comprehensive model, Opus 4.8, struggled to maintain focus under pressure, revealing that thoroughness alone isn’t enough without disciplined execution.

What This Means for Business and AI

This experiment shows that AI models can not only identify crises but also uphold integrity and honesty under pressure. However, the ability to execute consistently and follow through remains a challenge. The real-world implications are clear: if AI is to be trusted with management tasks, it must be evaluated not just on its language skills, but on its ability to finish what it starts, read deeply, and stay disciplined.

Try It Yourself

Business leaders can now run similar tests on their own AI tools through a dedicated platform, ensuring their AI’s decision-making aligns with their standards before deployment. Visit firmulate.com/quiz.html to participate in an interactive quiz based on real management decisions — a step toward smarter, more trustworthy AI integration.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI