Evaluation and Benchmarks for Agents

A global enterprise operating across healthcare, finance, telecom, and education engaged Tbrain to stand up domain-specific Q&A agents and a practical evaluation framework they could operate in-house. The assignment prioritized realism, safety, and speed to value.

Deliver 6 production-grade agents grounded in authentic, approved knowledge and a turnkey evaluation package that the client could run immediately - achieved in 1 month from kickoff to handoff.

Each agent would answer only when evidence exists in its corpus and refuse clearly when evidence is absent, with every expected answer traceable to source material.

Timeline and Scale

• 4 weeks to deliver.

• At least 45 files curated per agent.

• 120 prompts authored per agent.

Evaluation Requirements

• 100 answerable prompts strictly from the corpus.

• 20 unanswerable prompts to validate safe refusal behavior.

Each corpus mixed formats such as PDF, DOCX, PPTX, XLSX/CSV, HTML, and SharePoint pages, with layout variety like nested headings, footnotes, long tables, charts, and images. Files spanned small, medium, and large sizes.

The query set had to feel human - covering fact-seeking, procedural, comparison, multi-part, hypothetical queries with realistic patterns like misspellings and domain-term paraphrases. Many prompts required combining evidence across 2, 5, and 10+ documents.

Tbrain executed a pod-based operating model so multiple teams could work in parallel while maintaining one central quality standard.

1. Corpus Curation - authentic documents sourced, normalized, and deduplicated.

2. Query Generation - roughly 120 realistic prompts created per agent.

3. Ground-Truth Mapping - span-level evidence attached to every answerable query.

4. Quality Review - rubric alignment, inter-rater checks, and policy verification.

5. Final Packaging - test-ready bundles approved by team leads for immediate handoff.

Every response is compared to the approved corpus with one outcome: Correct, Needs Correction, or Refusal Required.

Client Benefits

• Turnkey evaluation framework ready to run internally for benchmarking and fine-tuning

• Every answer mapped to precise supporting passages for streamlined review and audits

• Reproducible & scalable - includes templates and checklists to extend the program at the same pace

• Reduced time-to-value while raising confidence in both grounded accuracy and refusal behavior