The fine-tune that answered with last week’s policy
Ashvale Benefits (a fictional employee-benefits administration company) helps mid-sized employers manage health-plan enrollment, eligibility rules, and employee policy questions. It had eleven days before its busiest open-enrollment season.
The company had built an AI support assistant inside its member portal. The assistant was supposed to answer routine questions from each employer’s approved policy handbook, show the supporting section, and hand uncertain cases to the support team.
During open enrollment, that team expected eighty thousand questions. Most would be some version of the same thing: Does my plan cover this? When does coverage begin? Which form do I need?
The business goal was simple. Answer the routine questions immediately. Send the hard ones to a human. Never invent a benefit that did not exist.
There was one more rule from legal: every answer had to point to the policy section that supported it.
Concepts in this story5 concepts
Parametric knowledge is encoded implicitly in a model’s learned weights. Non-parametric knowledge lives outside the model—such as in documents or an index—and is fetched when needed. External knowledge can be revised without retraining the model.
A generation pattern that retrieves relevant external information for the current query and includes it in the model’s input before the answer is generated. Retrieval changes the context, not the model’s weights.
RAG supplies external knowledge at inference time; fine-tuning changes model weights through training. Use retrieval as the default for changing, sourceable facts and fine-tuning for learned behavior, format, or task adaptation. They can be combined.
The indexing pipeline prepares source content before questions arrive: load, split, enrich, and store it. The query pipeline runs per request: interpret the question, retrieve evidence, assemble context, generate, and validate the answer.
Connecting generated claims to supplied evidence or an authoritative external source. Grounding makes answers inspectable and can reduce unsupported claims, but retrieval and generation can still fail.
The obvious shortcut
Rhea (the AI engineer leading the assistant project) was responsible for taking that system from an internal demo to production. Her service sat behind the member portal, received an employee’s question and plan details, called a language model, and returned either an answer or an escalation to a human agent.
The base model wrote clear answers, but it did not know Ashvale’s plans. The company had 4,800 approved question-and-answer pairs from past enrollment seasons. They were clean, reviewed, and already written in the support team’s voice.
Training on those examples felt like the shortest path. If the model did not know the handbook, teach it the handbook.
Rhea removed personal data, balanced the common and rare question types, and held back 600 examples for evaluation. Two days later, the new model looked much better:
- answer accuracy rose from 71% to 93% on the held-back set;
- the required JSON shape passed 99.6% of the time;
- answers were shorter and matched the support team’s tone;
- the model stopped recommending forms from other plan families.
The team called that a launch candidate.
The phrase they kept using was: “Now the model knows our policies.”
That sentence survived every review.
The answer from last week
On Friday afternoon, legal approved a policy amendment. New spouses would become eligible after 30 days, not 60. The document team published version 7.4 of the handbook and archived version 7.3.
On Monday morning, a support analyst ran the launch checklist.
Question: I got married last month. Can I add my spouse now?
Assistant: Spouses become eligible after a 60-day waiting period. See section 8.2 of the Ashvale Standard Plan handbook.
Section 8.2 did not say that anymore.
Worse, the answer looked excellent. It was direct. It used the approved tone. It included a citation-shaped sentence. Nothing in the wording signalled that the fact was stale.
Rhea tried the question twelve ways. Nine answers used 60 days. Two avoided a number. One used 30 days but cited the wrong section.
The launch dashboard still showed 93% accuracy because its questions came from version 7.3—the same policy era as the training data.
The model had passed the test it was given. The product had failed the test the business cared about.
What had actually changed
Rhea’s manager asked a blunt question.
“We replaced the handbook on Friday. Why is the assistant still reading the old one?”
It was not reading either handbook.
The training run had changed the model’s weights. The old examples had influenced statistical patterns inside those weights. There was no row Rhea could update from 60 to 30, no source record she could inspect, and no reliable pointer from an answer back to the training example that shaped it.
That was the distinction Rhea had missed: are stored differently.
The fine-tuned model held knowledge implicitly in its parameters. The handbook was explicit knowledge outside the model. Rhea had copied some of the handbook’s patterns into the parameters, then mistaken that process for connecting the model to the handbook.
Fine-tuning can teach facts. It is not physically incapable of doing so. But learning a fact from examples is not the same as maintaining a current, inspectable source of truth. Coverage can be uneven. Updates require more training. Provenance is weak. Old and new facts can compete.
Ashvale’s problem was not “make the model speak differently.” It was “answer from whichever policy is active for this employee today.”
That reframed .
Fine-tuning had been useful for behaviour: tone, response shape, and when to escalate. The changing policy facts needed a different path.
Two pipelines, not one training run
The team stopped asking the model to remember the handbook.
They built an .
Ashvale already ran its customer-facing services on Amazon EKS. The member portal was React. A NestJS service called assistant-api handled questions. Versioned policy PDFs lived in Amazon S3, PostgreSQL held plan and policy-version metadata, and a dedicated OpenSearch domain held searchable policy sections. A Python worker called policy-indexer prepared each new handbook.
The indexing pipeline ran when the document team published a policy. It extracted each approved section, attached the tenant, plan, version, and effective dates, then stored a searchable representation. Publishing version 7.4 created new records and retired the old records. It did not retrain the model.
The query pipeline ran for every question. It identified the employee’s plan and date, searched only the active policy records, and placed the best matching passages beside the question.
The model still generated the prose. But it now generated with current evidence in the request. That pattern was .
Rhea’s first implementation was deliberately boring:
async function answerPolicyQuestion(input: PolicyQuestion) {
const activePolicy = await policyVersions.findActive({
tenantId: input.tenantId,
planId: input.planId,
asOf: input.asOfDate,
});
const evidence = await openSearch.search({
index: 'policy-sections-v1',
query: input.question,
filters: {
tenantId: input.tenantId,
planId: input.planId,
policyVersion: activePolicy.version,
},
limit: 5,
});
if (!hasEnoughEvidence(evidence)) {
return escalate('No active policy passage supports an answer');
}
const answer = await model.generate({
instructions: [
'Answer only from the supplied policy passages.',
'Cite the source ID after every policy claim.',
'If the passages do not answer the question, say so.',
],
question: input.question,
evidence,
});
return citationsResolveToEvidence(answer, evidence)
? answer
: escalate('The answer contains an unsupported citation');
}
The important line was not model.generate. It was the refusal before it and the check after it.
Ashvale did not accept “the model probably knows” as evidence. Each policy claim had to connect to a retrieved, active passage. That connection was .
Grounding did not make the model truthful by definition. The search could retrieve the wrong section. The model could misread the right section. A citation could point to a passage that did not actually support the sentence.
So the team tested the pieces separately. Could the system retrieve the correct active section? Given that section, did the answer stay within it? Did every citation resolve? Did no-evidence questions escalate?
The second launch review
By Thursday, the updated-policy test set had 240 questions. Half depended on a recent policy change. The other half were designed to catch accidental regressions on stable rules.
The numbers were less pretty than the original 93%, but more useful:
- 91% of questions produced a correct answer supported by the active policy;
- 7% escalated because the retrieved evidence was weak or conflicting;
- 2% failed and stayed blocked from launch for investigation;
- every displayed citation resolved to a versioned policy section.
The spouse question now returned 30 days and linked to version 7.4, section 8.2. When the team marked version 7.4 inactive in a staging test, the assistant did not quietly fall back to 7.3. It escalated.
That behaviour cost them something.
Median response time rose from 720 milliseconds to 980. The 99th percentile rose by almost a second. Each answer consumed more input tokens. The team now operated a document parser, an index, freshness alarms, access filters, and two evaluation suites. A bad effective date could hide the right policy even when every other component worked.
The fallback also sent more conversations to humans. Legal considered that a feature. Finance saw a higher support cost.
Both were right.
What Rhea kept from the fine-tune
The team did not throw the fine-tuned model away.
It still followed Ashvale’s response format better than the base model. It used the right tone. It made cleaner escalation decisions. Those were changes in behaviour, and the training examples were good evidence for that behaviour.
The policy text remained outside the model, where it could be versioned, filtered, inspected, and replaced.
That was the answer Rhea carried into the next design review:
If the knowledge changes, retrieve it. If the behaviour changes, fine-tune it. If both change, combine them—but keep the source of truth outside the weights.
The rule was not absolute. It was a better starting question than “Which AI technique should we use?”
The architecture they took to production
The lifecycle view earlier in the story showed when the indexing and query pipelines ran. For the final design review, Rhea also needed a deployment view that answered a different question: where does each responsibility live?
The member and legal portals remained user-facing systems. The Python policy-indexer and NestJS assistant-api ran inside the Amazon EKS cluster. Versioned documents, publish events, active-version records, and searchable vectors lived in managed data services. Embeddings and generation crossed a separate AI API boundary.
Concepts in this story5 concepts
Parametric knowledge is encoded implicitly in a model’s learned weights. Non-parametric knowledge lives outside the model—such as in documents or an index—and is fetched when needed. External knowledge can be revised without retraining the model.
A generation pattern that retrieves relevant external information for the current query and includes it in the model’s input before the answer is generated. Retrieval changes the context, not the model’s weights.
RAG supplies external knowledge at inference time; fine-tuning changes model weights through training. Use retrieval as the default for changing, sourceable facts and fine-tuning for learned behavior, format, or task adaptation. They can be combined.
The indexing pipeline prepares source content before questions arrive: load, split, enrich, and store it. The query pipeline runs per request: interpret the question, retrieve evidence, assemble context, generate, and validate the answer.
Connecting generated claims to supplied evidence or an authoritative external source. Grounding makes answers inspectable and can reduce unsupported claims, but retrieval and generation can still fail.
Ashvale Benefits, its people, and its metrics are fictional. Technical grounding: the original RAG paper, research on fine-tuning with new factual knowledge, and an empirical study of groundedness in retrieval-augmented generation.