Five things worth knowing before your first AI experiment goes live, from a founder who learned them the expensive way.
The first version of an AI tool almost always gets things wrong. The real decision is whether you keep improving it or walk away.
We asked Jem co-founder and CEO Simon Ellis, whose WhatsApp-based HR platform serves more than 250,000 frontline workers, what he learned from killing an AI experiment too early…
The move: plan for failure instead of reacting to it
Jem sells mobile phones and data contracts through a ten-step WhatsApp flow. When a customer stopped at step seven, a person called them the next morning. In late 2025, Jem put an AI voice agent on those calls. It made errors, so the team cut it. Simon now calls that a massive mistake. The agent is back, but it took a couple of months to restart.
“In order to push AI, you’re going to deal with a fair bit of failure.”
What to know about AI experiments from day one
1. The first version will make mistakes
Know this before you launch: the first version will get things wrong. Part of Jem’s problem was timing: the voice technology they used wasn’t where it is today. Tools improve quickly, so an experiment that struggles this quarter may work well next quarter, but only if it’s still running.
2. The bar is the people doing it now, not perfection
An AI agent will never be flawless, and neither are the people doing the job now. Before launch, measure how the current process performs: how many calls convert, how many go wrong, how long each takes. Then judge the AI against that baseline. An agent that makes some errors but reaches far more customers may already be winning.
3. You need to define success and failure before you start
Decide in advance what result would make you expand the experiment, and what would make you stop. One pilot framework calls these kill criteria. Without them, the decision gets made on how the first few mistakes feel, which is usually too early.
4. Narrowing an experiment beats switching it off
When errors show up, you have more options than on or off. Shrink the experiment rather than ending it. Limit it to the simplest cases, add a person to review its work, or change how it handles the moments where it goes wrong. You keep learning while the risk stays small.
5. Stopping has a cost too
Switching off an experiment feels free. It isn’t. Jem lost a couple of months restarting its voice agent, and Simon believes they would be significantly further ahead if they had kept going. The time it takes to rebuild, retrain and regain the team’s confidence is part of the real cost of quitting.
The big payoff
Teams that expect early failure keep their experiments running long enough to work. Every month an AI tool stays live, it improves and you learn, while a team that quits starts again from zero.
You might also be interested in
Want the full story?
Simon’s full panel from The Open Letter Cape Town 2026 is available to members inside the Founder Collab, where he goes further than we could cover here:
How to get Claude to spot your repetitive work every week
How to share context across offices in three cities
How Jem helps customers like McDonald’s adopt AI without fear
Why he wants a thousand AI-native people, not ten
You’ll also get access to 40+ masterclasses from SA founders and operators on sales, fundraising, UX, paid media and more inside The Founder Collab.
Get more SA tech and business news, tips and business-building workflows and subscribe to The Open Letter.
