An AI email triage can gather messages, flag what is pending and draft replies. How good it is depends as much on what it leaves out as on what shows up in the summary. A tidy list can miss exactly the request that needed attention.
That is why the first test should use a period you know well, where someone can check which messages really needed a reply.
Explain what "needs a reply" means
Urgency can be spelled out in a subject line or hidden in an earlier commitment. A short message can change an important deliverable; a long notification may need no action at all. When a client writes only "are we still on?", understanding what is pending means checking the history. A system that only looks for the word "urgent" misses that.
Set the period, the sources it may use and the priority criteria. Ask for links back to the original threads and for a clear split between open requests, information received and commitments waiting for confirmation.
Review the drafts and the gaps
Check a sample of the messages it included, then look for requests it left out. Triage has to be judged on what it excluded, too.
In the drafts, check that they respect the history and do not add promises. A drafted reply is not permission to send it; the final log should make clear what is still waiting for a decision.
Measure the total time spent on setup, review and fixes. The test may show time saved on reading and more effort spent on checking. That is useful: it tells you whether to tighten the criteria, narrow the scope or drop the idea.
In AI Thinking by Dani Hayoshi, a routine is only delegated once the team knows where to check the doubtful cases. See also Delegating to AI agents.
Dani Hayoshi is the author of AI Thinking. Her work connects artificial intelligence, education and entrepreneurship. About Dani.