- The one-minute version
- Why most AI programmes stall
- What a Monday looks like
- 1. You can change your mind about the model
- 2. Your data stays yours
- 3. You can prove it worked
- 4. Something tests the AI before your client does
- 5. You know exactly what the AI is allowed to do
- Three routes, five walls
- What we got wrong first
- Why the model race makes this urgent
- Five questions to ask any vendor — us included
- When we’re not the answer
- Sources
The one-minute version
We build AI for Australian asset managers — firms from $3 billion to $116 billion in funds under management. We do one industry, and we do it in production. This is what we have learned about which parts of an AI strategy matter.
The model you choose is the least durable decision you will make. It will be retired in twelve to eighteen months. What survives that is the layer underneath it — and there are five things that layer has to do.
We built all five into the platform before we sold it, because we had to run it ourselves. That is the argument of this article, and the rest of it is the detail.
Why most AI programmes stall
The industry numbers are blunt, and they do not describe a technology problem. Roughly three in ten organisations are piloting agents. One in ten is running one.
Nobody is failing to get a demo working. Demos are easy now. What stops programmes is everything between a working demo and a system a compliance officer will sign off, a distribution team will actually open on a Monday, and a CFO can still justify in year two.
What a Monday looks like
Five walls is an architecture argument, and architecture is abstract until you watch someone use the thing it holds up. So here is the thing, first.
Before. Someone on the distribution team logs into three platform portals, downloads yesterday’s flow files and scans them for anything notable. At 8:15 they open the CRM to prepare for a 9:00 meeting, find notes from last quarter, and go back to a portal for recent flows. In the meeting, the adviser mentions they have moved money to a competitor. The file showing that switch arrived ten days ago. Nobody had time to open it.
After. They ask one question in plain English and get a ranked list back — who moved, how much, when we last spoke and what was said. The follow-up emails come back drafted, with the right numbers already in them. They still decide who gets one.
That is the product. The five walls are what it takes to run that every Monday, in a business where being wrong in front of a client is expensive. They are also the part you only notice when one of them is missing.
1. You can change your mind about the model
The risk. Most people frame this as pricing leverage — one supplier, one price list, no negotiating room. That is real, but it is the smaller half.
The bigger half is that you will be forced to move whether you like it or not. Frontier models have a working life of roughly twelve to eighteen months before they are retired. OpenAI pulled GPT-4o from its API in 2026. Anthropic retired Claude 3.5 Sonnet in 2025. This is not a threat from a competitor. It is the normal metabolism of the market.
Why it hurts. Swapping models is not a settings change. Real prompts accumulate months of model-specific tuning — the phrasing that stopped one model waffling, the example that fixed another one’s formatting. Pulling the durable logic out of that is a rewrite. One documented enterprise migration ran five months of two systems in parallel and roughly $331,000 of doubled-up spend before a single dollar of saving arrived.
Every request in our platform goes through us, not straight to a model vendor. The model is a setting, not an architecture — in Sherpa it is literally a dropdown, and our clients can see which model answered them. Everything that took work to get right stays on our side of that line, so changing model is a decision we make on evidence rather than a project someone has to fund.
One caveat we will not pretend away: done badly, this makes every model mediocre, because you end up using only the features they all have in common. We choose per job — this task, this quality bar, this budget — and let each model do what it is actually good at.
2. Your data stays yours
The risk. This is the one with a regulator attached. Through 2026 the rules tightened here and in Europe — on who you can hand critical work to, and on telling people when a machine had a hand in a decision about them. Your compliance team will track the detail.
The part that belongs to you is simpler, and most firms cannot answer it: where does a question containing client data actually go, and who can reach it once it gets there?
The half nobody puts in the board pack. Roughly 47% of generative AI use inside enterprises runs through employees' personal accounts. The average large enterprise logs around 223 AI-related data policy violations a month. Nobody approved any of that. It happened because the approved tool was worse than the unapproved one.
Two distinctions vendors blur, and you should not. Residency is where the bytes sit. Sovereignty is whose courts can compel access to them — a hyperscaler region in Sydney gives you the first, not automatically the second. And training is not your only exposure: retention windows, abuse-monitoring copies, sub-processors and vendor support access are all routes your client data takes after the answer comes back.
Every model call runs through our own infrastructure in Sydney and Melbourne. Your data does not leave the country. The assistant can only show someone what that person could already open for themselves — which sounds obvious, and is the single most common way these systems leak internally. Audit logs and access controls are part of the platform, not a later project, and OpsFlow turns the compliance evidence itself into a workflow rather than a spreadsheet someone rebuilds each quarter.
And we treat shadow AI as our problem, not yours. If the sanctioned tool is good enough, nobody pastes a client list into a personal account. That is a product standard, not a policy.
3. You can prove it worked
The risk. Most AI budgets are approved on a business case and renewed on a feeling. The second year is where that catches up with you.
The correction worth making. Cost per token is the wrong number to manage. Driving it down rewards doing less work — a cheaper model, given less to go on. Cost per successful task rewards doing more, efficiently — which is what you wanted. A cheap model that fails a third of the time and falls back to a human is not cheap. It has moved the cost somewhere your finance system cannot see.
The rest is straightforward. Give every workflow an owner, put a ceiling on what it can spend, and prove the outcome against a number you agreed before you built it.
That last step is the one everyone skips, and it is the single largest documented cause of failure — 41% of stalled programmes, ahead of every technical gap.
We measure two things and we report both: adoption and impact. Who is using it, on the workflow it was bought for. Then what moved — which conversations happened, which flows were influenced, which assumption turned out to be wrong.
Adoption is the leading indicator, and it is the one almost nobody measures honestly. Licences issued is a procurement number. A platform with every seat licensed and a fifth of them opened weekly has not failed on a technical review — it has failed completely, and it will pass that review anyway.
So here is ours, defined before you ask for it. We count a person as adopted if they used Sherpa in the past week, on the workflow their firm bought it for. Not a licence. Not a login. On that definition we are at 100% across our subscription clients, we report it to you monthly, by name, and if it slips it is our problem to fix rather than a line item you keep paying while you wonder.
Our pricing is deliberately a fixed platform fee. Per-conversation pricing charges you more precisely as adoption improves, which means it penalises the exact outcome you were trying to buy.
4. Something tests the AI before your client does
The risk. This one is rarely on the list, and it should be near the top.
Per Forrester’s 2026 panel, only 38% of production AI agents have automated tests that run on every prompt change. The ones without that coverage were rolled back 47% of the time. The ones with it: 9%.
Notice how this wall holds up the first one. A test suite is what makes model portability real. Without it, “we can switch providers” is a sentence on a slide. With it, switching is an ordinary decision: run the suite against the new model, see the six things that regressed, fix four, accept two, ship it. The tests are what turn optionality from a claim into a capability.
Our test suites are built from the work itself — real questions about fund flows, adviser activity, platform data and compliance, with known-correct answers, contributed by the people who do the job. They run when a prompt changes, when a model changes, and when we add a feed. That is also why we can be specific with clients about what a new model changed rather than reassuring them in general terms.
We keep the scoring rules portable on purpose. When OpenAI’s evaluation product went read-only in 2026, the firms whose testing history lived inside it lost it. Ours is ours.
5. You know exactly what the AI is allowed to do
The risk. The shift from chat to agents changed the risk class completely. A chatbot that is wrong produces a bad paragraph. An agent that is wrong sends the email, updates the record, files the report. The failure is no longer a wrong answer — it is an action you did not authorise.
And capability is outrunning the guardrails around it. The labs set the thresholds that trigger stricter safety controls, and have raised them at least four times in two years — each time after a model in development had already crossed the previous line. One lab confirmed in 2026 that a model of its own broke into a third party’s systems during testing.
You cannot slow that race down. You can decide how much of your business sits inside the blast radius.
Four controls, none of them clever, all of them present from the start:
- Identity. Every agent has its own login, with the narrowest access that lets it do the job — not one shared account inherited from whoever built the first workflow.
- Tiered authority. Reading is free. Writing is logged. Anything irreversible, outward-facing or financial needs a person. Sherpa drafts the fifteen emails; it does not send them.
- Containment. Hard limits on what it can spend, how long it can run and which systems it can touch — and a stop button that works. We assume someone will eventually trick it, and design for what it can reach when they do.
- Reversibility. Every action logged and attributable. If you cannot reconstruct why the system did something eight weeks ago, you cannot answer a regulator and you cannot debug it either.
Three routes, five walls
There are three ways to get an AI capability into an asset management business. They differ less on the AI than on how many of these five walls you end up building yourself.
| Wall | Generic AI seats | Build it yourself | Datafabric |
|---|---|---|---|
| Change the model | Locked to your vendor | Yours to build and maintain | A setting you can change |
| Data stays yours | Offshore, vendor terms | Yours to build and certify | Australian-hosted, your access rules applied |
| Prove it worked | Seat counts only | One more thing to build | Adoption and impact, reported |
| Test before release | Not offered | Yours to build and staff | Industry test suites, run on every change |
| Bound what it can do | Generic guardrails | Yours to design | Per-agent identity, tiered authority |
| Who does the work | You | You, twice | Us |
We set out the dollar version of this comparison — line by line, first year, 20 seats — in what an AI platform actually costs. The short version is that the AI is the cheap part. The people who build and run these five walls are not.
What we got wrong first
We are not a large firm, and we do not sell into eleven industries. We build for Australian asset managers, and everything above came out of doing it rather than researching it. Some of it came out of being wrong.
We over-trusted demos. Workflows that were compelling in a walkthrough and useless in the field got deleted. Someone pays for that lesson once; we have already paid it, which is why our build row is shorter than yours would be.
We under-estimated the data, not the AI. The assistant was never the hard part. Australian platform files and unit registry extracts were — matching adviser and dealer-group codes, handling restated data, surviving a format change nobody announced. No major vendor ships a connector for any of it, because the market is too small to warrant one. That work is most of what we actually sell.
We learned that adoption is a product problem. Not a training problem, not a change-management problem. If people go around the tool, the tool is worse than the thing they went to. We stopped treating that as a rollout issue and started treating it as a defect.
Why the model race makes this urgent
A handful of extremely well-funded labs are competing for what may be the largest platform position in computing history. Enterprise spend on generative AI tripled year over year to roughly $37 billion in 2025. Three consequences land on your desk regardless of what you think about any of it:
- The thing you bought keeps changing. Models improve, prices fall, APIs shift, products retire. Keeping your options open is not scepticism about a particular lab. It is the only rational posture toward a market moving this quickly.
- Safety is a competitive variable, not a constant. Where thresholds move to accommodate capability, you cannot outsource your risk appetite to somebody else’s safety policy. The bounds have to be yours.
- Concentration is now an operational resilience question. 81% of enterprise executives report concern about depending on a single AI vendor, and 47% say losing their primary one would disrupt a key business function. That is no longer a procurement matter. It is a resilience question, and it gets asked at board level.
The strategic move is not to pick the winner. It is to build the layer that makes the winner interchangeable, and to keep the parts that are genuinely yours — your data, your workflows, your test history, your hard-won knowledge of what works — on your side of the line.
Five questions to ask any vendor — us included
Take these into your next three vendor conversations. The answers sort the field quickly.
| Ask | A real answer sounds like | A weak answer sounds like |
|---|---|---|
| When did you last move a live workflow to a different model? | A date, the workflow, and what broke | “We’re model-agnostic” |
| Where does a prompt with client data go, and whose courts can reach it? | A diagram, a jurisdiction, a retention window, a sub-processor list | “It’s encrypted and we don’t train on it” |
| What did your last client’s adoption look like at month six? | Weekly active users on a named workflow | A licence count |
| What runs automatically when someone changes a prompt? | A test suite, a pass threshold, a blocked release they can describe | “Our team reviews changes” |
| What is the worst thing your most privileged agent could do in an hour? | A bounded answer, because the permissions are bounded | “That couldn’t happen” |
When we’re not the answer
Every vendor article is written by the vendor, so here is where ours stops working.
Five walls is too much for a first experiment. If one team wants to automate one task, building all five walls first is theatre. Ship the thing. Keep your prompts somewhere you control and your data in your own systems — two habits that cost nothing and preserve most of your options — and come back to the rest when it is real.
If your data is simple, we are probably overkill. Our value concentrates in the messy parts: platform files, registry extracts, adviser mapping, compliance evidence. If everything you need is already clean and inside your CRM, a good generic assistant and a careful policy will serve you, and we would rather say so now than in month six.
And measurement can be theatre too. A dashboard with forty AI metrics and no decision attached to any of them is worse than three metrics with a name against them, because it looks like governance. If no number on it has ever caused someone to stop doing something, it is not measurement. It is decoration.
What does not change: the model is rented, the five walls are owned, and the only way to know whether any of it is working is to have measured it from the start.
Sources
- Single-vendor dependency concern among enterprise executives, and the share reporting a key business function at risk. AvePoint, and Why Enterprises Need a Multi-Model AI Strategy
- Model deprecation cycles, prompt portability, and the cost of a first migration. The Hidden Switching Costs of LLM Vendor Lock-In
- Why a thin, lowest-common-denominator layer degrades every model. The Abstraction Layer That Made Every Model Mediocre
- Evaluation portability and the 2026 OpenAI Evals deprecation. Keep the Decision Contract Portable
- Shadow AI prevalence, personal-account usage and monthly policy violations. Kiteworks, 2026 AI Data Security Crisis
- Data residency versus data sovereignty at the inference layer. NeuralTrust
- Australian privacy reform, automated decision-making transparency and EU AI Act obligations. OAIC, and the EU AI Act
- Cost per successful task as the governing metric for agent economics. Microsoft Azure
- Share of organisations piloting agentic AI versus running it in production. Deloitte, Tech Trends 2026
- Root causes of agent failure, automated test coverage and rollback rates. Forrester, The State of Agentic AI in 2026, as reported in Why Most Enterprise Agent Pilots Never Reach Deployment
- Frontier capability thresholds and safety policy revisions. METR
- Frontier models taking unauthorised action during evaluation. Frontier AI models breaching systems during testing
Survey figures are drawn from published 2026 industry research and vendor panels and are cited as reported; methodologies and sample sizes vary between sources.