Artificial intelligence
AI with company documents: preparing a useful assistant
An internal assistant needs more than document access. It must retrieve the right information, respect user permissions and recognise when no supported answer exists.
1 — Define who it helps and why
Pick a concrete need: finding a procedure, answering catalogue questions or locating a condition. Covering the entire company makes evaluation harder.
Collect actual questions, expected answers and supporting sources. Include ambiguous questions, missing answers and conflicting documents.
2 — Organise sources and permissions
Check ownership, validity and access groups. The assistant must not expose information the user cannot see in the original system.
Define updates, removals and conflicting-version handling. A polished answer based on an outdated procedure is still wrong.
- Assign ownership and review dates.
- Check permissions before retrieval.
- Plan updates and removals.
- Avoid unnecessary sensitive test data.
3 — Understand document retrieval
Retrieval-augmented generation, or RAG, finds relevant passages and supplies them to a model as context. This supports proprietary knowledge and references. Microsoft’s guide describes the pattern and evaluation.
Retrieval does not guarantee correctness. Check selected evidence and whether the answer follows from it. A citation alone is not proof.
4 — Evaluate usefulness and cost
Review correct, incomplete and unsupported answers, including abstention. Measure human review effort and costs of querying, retrieval and maintenance.
Separate recommendations from execution before enabling record changes or messages. A read-only pilot can still be useful.
5 — Make the handover operational
Expect connected sources, access rules, evaluation, monitoring and an incident owner. Document updates and how to disable unacceptable behaviour.
At YUMI, conventional search and simple rules remain options when they solve the problem better.