An AI evaluation is only as useful as the record it leaves behind. This guide shows how to keep notes that make trials comparable, preserve failures as well as successes, and explain a decision months after the original demonstration.
How Should You Document AI Evaluations?
Use the same task cases and scoring rules for each trial, and save the inputs, configuration, outputs, and reviewer judgments together. Separate observed quality from effort, privacy findings, and cost rather than hiding everything inside one score. Record unknowns explicitly and attach evidence to conclusions so another person can check them.
Create a small evaluation packet
An evaluation packet is a collection of records for a specific decision. It should state the question being investigated, the cases used, the trial conditions, and the conclusion. It need not be a complex database; a structured document and a controlled folder may be enough for a small pilot.
Start with the decision question: “Can this tool prepare a draft summary of approved meeting notes that a coordinator can check efficiently?” That is narrower than “Is this AI good?” The question determines what belongs in the notes and what evidence would count against adoption.
Create stable identifiers for cases and runs. A case is the task and source material. A run is one attempt using a particular configuration. Keeping those concepts separate lets you compare several attempts without accidentally treating them as independent tasks.
- Freeze the initial case set. Why: Quietly changing examples makes comparisons unreliable. How: Save a dated version of each approved input and record later additions as a new case-set version.
- Write the expected characteristics. Why: Reviewers need a common target. How: List required facts, allowed labels, forbidden claims, and what should happen when information is absent.
- Record limitations upfront. Why: Small trials can be overinterpreted. How: State which languages, formats, departments, and risk levels the evaluation does not cover.
OpenAI's evaluation guide describes assessing outputs against specified criteria. The underlying habit is useful even for manual notes: define what you are checking before inspecting the answer.
Record the conditions needed to interpret each run
A product name alone is not enough context. Save the date, product or model identifier shown to you, plan, settings, prompt version, and relevant enabled features. If the provider does not expose an exact model version, write “not disclosed” rather than guessing.
| Field | Example entry | Why it matters |
|---|---|---|
| Case ID | SUMMARY-04 | Links repeated runs |
| Run date | Recorded date and time zone | Places results in context |
| Configuration | Model shown; search disabled | Explains available capabilities |
| Prompt version | Summary template v3 | Distinguishes instruction changes |
| Source version | Approved notes revision B | Prevents input confusion |
| Outcome | Missing one required action | Preserves a concrete finding |
| Reviewer effort | Timed checking and editing | Shows hidden work |
| Evidence | Restricted output record | Allows verification |
- Capture the real execution context. Why: A fresh chat and a long existing chat can behave differently. How: Note whether history, memory, retrieval, connectors, or tools were enabled and which sources were available.
- Keep raw outputs separate from edits. Why: A polished final draft can erase evidence of the original failure. How: Save the initial output, then store the corrected version with a short change note.
For agent evaluations, also record the permitted tools and action boundaries. A system that only proposes an update is not being tested under the same conditions as one authorized to write into a live database. Capture observable actions and tool results, not claims that you can inspect every internal reasoning step.
Do not store secrets or unrestricted personal information merely for completeness. Reference an approved restricted location when necessary, and use redacted evidence in general review documents. The evaluation record itself needs an owner, access rules, and an appropriate retention period.
Separate quality, effort, safety, and cost
Use dimensions that answer different questions. Quality asks whether the result was correct and complete. Effort asks how much human work was necessary. Safety and privacy ask whether boundaries were respected. Cost asks what resources the process consumed.
A simple quality rubric might distinguish “usable unchanged,” “usable after correction,” and “unusable.” Add a short reason to every judgment. “Usable after correction: omitted the owner of action three” is more informative than “3 out of 5.” Make critical failures explicit rather than averaging them away.
- Define scoring anchors. Why: One reviewer's minor issue may be another's failure. How: Write examples for each category and jointly review a few outputs before scoring independently.
- Separate observation from inference. Why: A result can suggest a cause without proving it. How: Write “the output used the old date” as the observation and “possibly retrieved the old policy” as a hypothesis.
- Track unknowns. Why: An unanswered privacy question is not a passing result. How: Mark it pending, assign an owner, and state what documentation would resolve it.
For security-related trials, record the attempted boundary crossing and the actual effect. OWASP's prompt-injection guidance explains why malicious instructions in external content deserve attention. Test only in authorized, contained environments, and avoid putting real credentials into adversarial examples.
Cost notes should distinguish measured usage from estimates. If the interface does not expose tokens or tool charges, say so. Do not infer precise API cost from a subscription chat session or treat a promotional allowance as the expected production price.
A worked example of a useful trial note
Suppose a hypothetical team compares two tools on a meeting-summary case. The notes contain four actions, three named owners, and one action with no deadline. The acceptance rule requires all actions and prohibits invented deadlines.
A useful record might read: “Case SUMMARY-04, prompt v2, fresh session. Tool A captured all four actions but assigned Friday to the action without a deadline. Quality: correction required. Checking took three minutes; editing took one minute. Source evidence: action four has no date. Reviewer removed Friday and wrote ‘not stated.’”
For Tool B, the note might read: “Captured three actions and omitted the unassigned task. Quality: correction required. Checking took four minutes; editing took two minutes.” These are hypothetical observations, not product test results. They show why an identical overall rating can conceal different defects and effort.
The decision note could then say: “Neither output met the no-correction criterion for this case. Repeat with the missing-data rule made explicit; preserve these original runs.” That conclusion is narrower and more defensible than “Tool A is better at summarization.”
Keep a counterexample alongside a successful result. If one tool performs well on routine cases but fails when the notes contain conflicting dates, that exception should remain visible in the recommendation rather than disappearing into an average.
Turn notes into a decision and a retest plan
Summarize findings by task and failure pattern, not just by product. A short decision memo should state what was tested, what was not, which requirements passed, and which questions remain open. Link each major conclusion to one or more case identifiers.
- State the scope of approval. Why: A narrow pilot can become a broad endorsement through repetition. How: Write the permitted task, data category, required review, and excluded uses beside the decision.
- Define retest triggers. Why: Product behavior and organizational needs change. How: Revisit important cases after model changes, prompt revisions, new connectors, or a significant incident.
Preserve the previous result when rerunning a case. Add a new run rather than overwriting history. That makes it possible to distinguish improvement from a change in test conditions and to understand why an earlier decision was reasonable at the time.
A concise, well-linked record is usually better than a large unstructured archive. Keep the evidence needed to reproduce the comparison where practical, while recognizing that hosted models may change and exact reproduction may not be possible.
Questions About Documenting AI Evaluations
How many examples should an evaluation include?
There is no universal minimum for every task. Cover the important task types, expected variations, and costly failure modes, then state the limits of the sample. A small pilot can identify problems, but it should not be presented as proof of performance across all future work.
Should reviewers know which tool produced the answer?
Where practical, hide product identity during quality review to reduce brand expectations. Keep configuration information available to the person managing the experiment. Blind review does not remove every bias, but it can separate judgment about the output from enthusiasm for a vendor.
Can another AI score the results?
It can help with clearly defined checks or triage, but its judgments also need validation. Compare automated scores with human-reviewed examples, especially for nuanced or consequential tasks. Keep the scoring prompt and model version as part of the record.
What if a provider changes its model without notice?
Record the identifier visible at the time and acknowledge what you cannot know. Retain representative cases and rerun them periodically. Your notes should support comparison under changing conditions rather than promise perfect reproducibility from a service that does not expose every detail.
Make Trial Notes Useful for Decisions
Good evaluation notes preserve the reasoning behind a decision without pretending a small trial proves everything. Keep concrete evidence, clear unknowns, and a defined date or trigger for review.
Next steps: Use the record when choosing an agent platform, learn about autonomous AI agents, and browse AI Agents.
Sources
- OpenAI — Working with evals
- OWASP — LLM01: Prompt Injection
This guide is general education, not professional advice.