Start with a baseline
Record the current workflow, volume, handling time, and common exceptions. A task with clear inputs and observable outcomes is easier to evaluate than an open-ended promise to improve productivity. Check whether ordinary rules or workflow software would solve it first.
Example: draft a support reply
An assistant could retrieve approved product information and draft a response for a support agent. Keep sending under human control during the pilot. Compare drafts against representative tickets, including ambiguous requests, outdated documentation, and cases requiring escalation.
Set limits before connecting tools
Define approved data sources, permitted actions, retention requirements, and escalation paths. Treat content retrieved from documents or messages as untrusted input. A model-generated instruction should not expand the system’s permissions.
Measure the complete workflow
Track usable output, incorrect answers, review effort, turnaround time, and operating cost. A draft that appears fast can still increase total effort if a reviewer must verify every detail. Compare results with the original baseline and capture exceptions, not just successful demonstrations.
Expand through evidence
Keep an evaluation set and rerun it when prompts, models, or sources change. Extend the scope only when observed results meet the agreed criteria. The pilot should make it possible to stop or roll back as well as to scale.