"Should AI stand in the customer's shoes?"
That idea struck me as fresh the first time I heard it. But the more I sat with it, the less it seemed to mean — if it only means presenting results through language and interfaces users are already familiar with, the payoff is thin.
Looking back at the AI projects I've been part of, I've come to a different conclusion: a model can answer a question correctly and still have no idea what task the person is actually trying to complete — whether the right move now is to keep answering, ask a follow-up, stop, or hand off to a human. For AI to take on real work, not just answer questions, companies need to translate what experienced professionals know about context, risk, timing, and consequence into something a system can actually recognize: signals, task states, response protocols, and validation criteria.
This piece starts from a scenario I know well — model testing — and walks through how that translation happens, and why it forms two decisive thresholds standing between AI and real deployment. AI companies that understand and clear these thresholds are better positioned to build durable capabilities within a vertical domain, turning what they learn from a single deployment into delivery capabilities that are repeatable, verifiable, and scalable.
Beyond model testing: AI still must understand the task
On most AI projects, technical teams build a battery of test prompts and run the model against them repeatedly. Did it identify intent correctly? Did it retrieve and apply the right knowledge? Was the answer stable, and free of hallucination, policy violations, or logical errors? That's the bulk of testing and debugging.
I've watched this play out on my own company's projects many times. In a field like insurance sales, though, the testing team often also includes experienced domain experts sitting alongside the engineers. They read the same conversations and flag the same kinds of issues — but they're looking through a different lens entirely.
The technical team asks first whether the model got the answer right. The domain expert asks something else: does this answer fit where the conversation is right now? Where does it take the customer? And what should happen next — keep answering, probe further, pull back, or bring in a human?
That gap between the two lenses is exactly what most AI projects run into the moment they move from model testing into a live business setting.
The first threshold: recognizing when the task has changed
Say a critical illness policy includes a 90-day waiting period — a clause under which coverage for specified illnesses begins only after a set period, subject to the policy terms. A customer, mid-conversation with a sales assistant, asks: "How long is the waiting period on this critical illness policy?"
The AI checks the policy and answers: "The waiting period is 90 days. A covered illness first diagnosed after that point is eligible for a claim under the policy terms."
Judged purely on product knowledge and wording, nothing is wrong here. The model understood the question, cited the correct clause, and made no promise beyond what the policy covers. By ordinary QA standards, this counts as a clean, successful response.
But an experienced insurance agent may notice something a conventional QA review would miss: why is this customer asking about the waiting period right now?
Maybe they're simply comparing products. Or maybe they just had a checkup, noticed a symptom, or are quietly worried about a health issue that hasn't been diagnosed yet. The same question, asked against a different backdrop, can send the rest of the conversation in an entirely different direction.
What this threshold really tests is whether the AI can register that the task itself has shifted. The system first must notice what's happening: is this still a routine product inquiry, or has the task already shifted toward clarifying health risks, disclosure obligations, and coverage expectations?
If the customer is simply comparing products, stating the waiting period is enough. But if they've already mentioned something like an abnormal test result or a health concern, that same product inquiry has likely moved into different territory.
If the AI, still optimizing for conversion, nudges the customer to buy now so the waiting period starts ticking sooner — every individual line in that exchange might still look defensible on its own, and the short-term conversion numbers might even look good. But measured against the actual job — a sound, compliant sale — the direction has already gone wrong. That drift is hard to catch precisely because each individual output holds up fine under its own scrutiny; the problem only shows up once you look at where the pieces are heading together.
From a human-AI interaction standpoint, the customer opened with a product question, but the task that actually needed handling may have quietly turned into something else mid-conversation. The model answered correctly. It just never noticed the nature of the job had changed.
The second threshold: understanding what the task is actually for
Recognizing that a task has shifted only answers "what's happening right now." The AI still needs to understand what the task is meant to accomplish, and what the next move might set in motion.
This kind of failure is hard to catch through accuracy checks alone. Sometimes the problem is missing sufficient probing before the answer. Sometimes it's an inappropriate action taken after answering.
A customer asking a service rep about refund policy may have already been through several failed attempts and actually need an escalation, not a repeat of the rules. A sales manager asking AI to explain a revenue decline might be heading toward a decision about which customer segments to cut, which frontline process to change, or how to reallocate resources. A customer asking about a specific policy clause might simply have a preference — or might be quietly weighing a risk they haven't said out loud.
Language and knowledge let AI produce a better answer. But taking on real work requires understanding the task at hand: its current state, its intended goal, and the consequences of the next action.
This threshold tests whether AI understands the context and goal behind the task — not just that the task has changed, but where it has moved to, what information is still missing, and what should happen next.
Only with that in place does "AI standing in the customer's shoes" mean anything concrete. A natural tone, a clean interface, a flexible dashboard — these all improve how information gets seen and used. Understanding the task itself is a different matter altogether: it reaches into timing, limits, action, and who's accountable for what.
How professional judgment becomes a system capability
The gap between the engineers and the domain experts isn't really about who knows the technology and who knows the business. The real difference is in the basic unit each side uses to judge whether an output is any good. By "unit," I mean: when you're deciding whether one output is acceptable, what's the first thing you look at? The technical team checks the model against a set of evaluation dimensions — intent recognition, retrieval accuracy, answer quality, safety constraints, output stability. The domain expert puts that same answer back into its full context: when did it show up, where is the customer in the process, and what could it lead to?
So the engineer's first move is to check the answer itself. The domain expert cares more about where that answer sits within the whole job: does it fit the moment, could it be misread, and how does it affect trust, compliance, and the eventual business outcome? That difference in unit of judgment has real consequences for how well human and AI work together. A model can look great in a test report and still keep producing the wrong response once it's live — and the problem often isn't the content of the answer at all. It's that the system never noticed the task had already changed.
A good domain expert can often flag a problem instantly: "That's not something you say here." For an AI team, that sentence is just a starting point. It raises questions that still need answers: what signal did the expert notice? What outcome were they worried about? Which situations are fine to keep going, and which need a follow-up question or a handoff? And where does that knowledge belong — the system prompt, the context, a rule, a workflow, or a hard permission boundary in the system itself?
No matter how sharp someone's professional judgment is, if it stays locked inside their own intuition, it can never become a system capability that scales. So the real question — the one that determines whether a team clears both thresholds — is how to turn expert instinct into structures the system can recognize, execute, and validate.
Bringing domain expertise into an AI system tends to follow a fairly consistent path:
It starts with finding real cases where the model answered correctly but the task still went off course. From there, the team breaks down exactly what signal the expert picked up on, and uses that to define the task state that signal points to. Only then does the team frame a concrete hypothesis about what intervention might help — and only after that does any of it get written into the system, tested against real outcomes, and refined according to the results.
That process has to start from a concrete case. Ask an expert in the abstract what to watch for during a sale, and you'll mostly get principles. Put an actual — or simulated — conversation in front of them and ask where the problem first started, and the instincts they've built up over years, the ones they've never quite put into words, finally have something to attach to.
Take the waiting-period case. A customer mentioning a recent checkup, a physical symptom, asking "is it too late to buy now," or treating the waiting period as some kind of guarantee of a future payout — these are all signals worth flagging. But a signal is just an observation. Plenty of people who ask about a waiting period have no health issue at all, and the model shouldn't jump to conclusions off a single word. The team still must work out under what conditions a given signal indicates the task has shifted.
Once that shift is defined — a routine product inquiry moving into a health-risk and disclosure conversation — the system can change how it responds: instead of pushing forward, it clarifies the risk, and hands off to a human if needed.
Through this process, an expert's instinct gradually turns into signals and task states the system can recognize. Once a task state is defined, the team still has to answer something more fundamental:
For a customer showing a possible health concern, does one extra clarifying question — instead of answering directly and pushing the sale forward — improve the completeness of health disclosure and reduce downstream disputes?
That question still lives at the level of a business hypothesis and an intervention design — it hasn't yet become a prompt or a system rule. It defines who it applies to, what action to test, what to compare it against, and what outcome it's meant to improve. At this point, the team has moved past "what should AI say" and into "does changing what AI does actually change the business result?"
From there, the technical team has to engineer the hypothesis itself. First: which signals should trigger the risk-clarification path. Then: what the system should do once triggered. For example — when a customer asks about the waiting period and also mentions a recent checkup, a symptom, or timing around the purchase, the AI should first understand why they're asking, flag the importance of honest disclosure and underwriting review, avoid implying a claim is guaranteed, and escalate to a human when necessary.
Some of this belongs in the system prompt. Other parts require context, annotation, rules, state management, and workflow support to work together. Where compliance lines, commitment authority, or high-stakes decisions are involved, tighter system-level guardrails are usually required too.
Bringing expert judgment into a system, in other words, can't be done with a single block of prompt text. The team must work out, piece by piece, which layer each bit of knowledge, experience, and constraint belongs in — that is what reliably constrains model behavior and improves the consistency of both outputs and actions.
Once the system design changes, the testing standard has to change with it. In a routine product comparison, the AI can simply state the waiting period. When a customer mentions an abnormal test result and asks whether it's too late to buy, the system needs to recognize a different task state and adjust its handling accordingly. If the customer then asks the AI to help conceal a health issue, it should stop, explain the disclosure requirement, and route the conversation to a qualified human.
And in the end, it all comes back to real outcomes. Did the extra question improve disclosure completeness and cut down on disputes? Did it create friction, interrupting a large share of routine inquiries that never needed it? Did the human handoff genuinely improve customer understanding and underwriting quality?
These questions only get answered against real data over time. It's only once expert judgment enters that loop — test, deploy, measure, refine — that it has a chance to settle into something stable and repeatable.
Closing: human-AI collaboration starts with a shared understanding of the task
What a given project needs to deliver shapes what human-AI collaboration ends up looking like. If a partner cares most about the final sales result, UI/UX may not be the primary focus of the delivery. But where customer acquisition, operational analytics, or direct decision-making by the client's own staff are involved, a dashboard that can flexibly switch segments, adjust workflows, and surface results often matters a great deal.
This piece started from a version of the same question: should AI present its output in language and formats the customer already knows well? That question deserves real weight — information nobody can use, however accurate, never makes it into real work. But push on it a little further, and a different picture emerges. Interface and presentation shape how people understand and use what AI produces. The deeper problem in human-AI collaboration is whether AI understands what task it's actually part of.
The same question can be routine, or it can mark a turn in the task. The same answer, delivered at a different moment or against a different backdrop, can lead to entirely different consequences. AI needs to recognize when the task state has shifted, understand the goal it's currently serving, and know how to act on what comes next.
That kind of understanding — of a vertical domain and its specific operational contexts — does not emerge from the model on its own. It comes from technical teams and domain experts repeatedly deconstructing real cases together, turning the signals, context, responses, and risks scattered across professional experience into something a system can recognize, act on, and verify.
Right now, most AI projects ask domain experts to verify whether answers are correct — whether a particular line should have been said or whether a number is accurate. That understates the value they bring. What they really contribute is task understanding and professional judgment: how a task shifts, when intervention is needed, what form that intervention should take, and which outcomes cannot be assessed through short-term metrics alone. The technical team must then decompose that expertise and encode it at the appropriate layers of the system — the system prompt, context, rules, workflows, permission boundaries, and the broader mechanisms governing human–AI interaction.
Human-AI collaboration, then, is a task-centered partnership. People define the goals, constraints, and accountability, and ultimately own the outcome; AI participates in analysis, inquiry, judgment, and action. Organizations then refine both the division of labor and the system design in response to real-world results.
For an AI company, this is also the real barrier to going deep in any vertical. General-purpose models and technical tools are becoming widely available, but the domain expertise built on a single deployment doesn't turn into scalable capability on its own. That experience becomes scalable only when an AI company can continually translate domain expertise into task states, system mechanisms, and a functioning validation loop. Only then can a single success be reliably reproduced across more clients and more scenarios.
Getting the answer right only proves the model understood the question in front of it. Catching a shift in the task the moment it happens, understanding what comes next, and knowing when to answer, ask, stop, or hand the task back to a person — that's when AI starts doing the job.
