Start with a task you can inspect, write what makes its result acceptable, and keep the evidence needed to explain a failure. Then compare the old and new versions on the same cases. An evaluation, often shortened to eval, is that repeatable check. Its value comes from helping you decide what to fix.

For an AI application, the result depends on more than the language model. The application may search documents, query a database, call tools and format an answer. Keep a trace: a record of the request, the information supplied to the model, the actions it requested, their results and the final response. That lets you connect a disappointing answer to a particular step.

Make success observable

“Give a good answer” leaves too much open. For a document-answering task, an acceptable result might need to answer the question, support its factual claims from the supplied source and acknowledge when the source lacks the answer. For a tool task, success may also require the correct action and arguments.

Choose the criteria before comparing versions. A parseable response can still contain the wrong answer. A successful database query can still leave the user without useful help. Keep those requirements separate so a failure points somewhere.

Hamel Husain’s Lucy case study offers a narrow check inside a real-estate application: requests for listings are tested against scenarios with one, several or no matching records. The returned array’s length is checked. That is useful evidence about a specific property, but it does not establish that every listed property satisfies every requested constraint or that the final explanation is useful. His broader method adds trace review and human judgment around these fast assertions. Read the original case study.

Read a failure from input to answer

Husain’s judge guide includes a worked order-tracking trace. A user supplies ABC123 as an order identifier. The database returns no match and gives the expected format, XXX-XXX-XXX. Retrieved material also tells a new user to check the confirmation email. The final response merely asks the user to check the number and retry.

The necessary help was available. The reply omitted it. Husain classifies this as missing user education. For this case, the next experiment should address how the response uses supplied guidance, rather than immediately replacing the search system. See the original trace and error analysis.

A useful acceptance rule for the next version is: when this lookup fails, explain the expected format and where the user can find the correct identifier, without inventing an order status. Compare the revised response with the old one on the same case. Also check successful lookups and other no-match cases, so a repair does not introduce an inappropriate instruction elsewhere.

That is a complete evaluation decision: the evidence locates the omission; the criterion names the missing behavior; the next change has a testable target. Whether the change works remains a question for the comparison.

Turn the verdict into the next experiment

From an observed failure to a tested fix

An evaluation loop for an application’s own tasks.

  1. Step 1

    Choose a task

    Write the user’s goal and the conditions that make its output acceptable.

  2. Step 2

    Keep the evidence

    Record the request, retrieved material, tool calls and results, final response, and configuration.

  3. Step 3

    Review the failure

    Give the output a pass/fail judgment and a specific reason. Find the first step that stopped the task from succeeding.

  4. Step 4

    Change one cause

    Choose a change that addresses the evidence: retrieval, instructions, tools or application code.

  5. Step 5

    Run the comparison

    Evaluate the old and new configurations on the same cases, and inspect regressions as well as improvements.

    Condition Keep examples for final checking separate from the examples used to tune an automated evaluator.

A result becomes actionable when it connects a particular failure to a change that can be checked.

The sequence is a reading and testing method. It does not imply that the application has already improved.

The same final symptom can have different causes. An unsupported answer may follow a missing source, a misleading retrieved passage or a model ignoring a correct passage. Reading only the last message hides that distinction.

Jason Liu’s retrieval guide makes this separation practical. Retrieval-augmented generation, or RAG, gives a model information found by a search step. Liu proposes testing whether questions retrieve their expected source material before optimizing the answer-generation step. He also explains why dates and other document metadata can matter: a request for the latest information cannot be satisfied reliably by text similarity alone. Read his original method.

For your next experiment, keep the diagnosis specific:

  • The required source never reaches the model: inspect indexing, filters and the search query.
  • The source arrives but the response contradicts or omits it: inspect the instructions and how context is presented.
  • The model requests the wrong tool or arguments: inspect the tool description, available information and action constraints.
  • The action succeeds but the user receives a poor explanation: evaluate the response separately from the tool result.

A trace supports these distinctions. It does not by itself prove the root cause; use it to choose a change, then check that change.

Choose a check you can trust

Use a code assertion for a property that code can decide: required fields exist, a value has the expected type, a result belongs to an allowed set, or a known identifier is absent. Human review handles judgments that need context, such as whether an answer resolves the user’s actual problem.

Eugene Yan’s product-evaluation method starts with labeled examples, aligns automated evaluators and runs them with configuration changes. He recommends clear pass/fail decisions for objective criteria and allows ties when comparing subjective qualities. Distinct criteria can have separate checks. That makes it easier to identify the requirement causing a regression. Read his original three-step guide.

An LLM judge is another model asked to assess an output. Treat its verdict as a prediction that needs checking. Husain’s updated validation guidance separates examples placed in the judge’s prompt, examples used to revise that prompt and a final test set. He also explains why overall agreement can hide an evaluator that misses rare failures. Read the validation section.

For the order task, inspect two kinds of judge error: accepting replies that omit the required help, and rejecting replies that genuinely provide it. An impressive aggregate agreement score cannot tell you which mistake the judge makes.

Keep the samples used to calibrate that judge distinct from a sample used to estimate everyday defect rates. A collection deliberately enriched with failures is useful for finding judge weaknesses, but its pass rate does not directly describe ordinary traffic.

Compare configurations, including what got worse

Save the inputs, relevant source or database state, acceptance rules and outputs alongside the configuration. Record the model version, prompt, retrieval settings and tool definitions. If those inputs change between runs, a score change becomes harder to interpret.

Chip Huyen’s production guide connects separate evaluation examples with prompt versioning and practical cost and latency. Its 2023 API prices are historical; the useful engineering principle is to track the configuration that produced each result and evaluate changes beyond examples embedded in the prompt. Read the original guide.

Change one intended factor first. Run both configurations against the same cases, retain the failed outputs and inspect the requirements behind their verdicts. Record how long the task takes and what it costs alongside acceptance. A better-looking answer that still omits the needed information has not fixed the task.

Choose the tolerated failure rate and important failure categories for your application. A small set of examples can expose a defect and guide a repair; it gives limited evidence about how often that defect will occur. Repeated runs can also reveal whether a result depends on a fortunate generation.

Follow the question that remains

The AI engineering collection connects these methods to their authors and original works. Open Liu when the missing evidence is in search, Husain when you need to inspect a failure, Yan when you need repeatable comparisons, and Huyen when you need to track the surrounding application.

If your next decision is whether a model can run usefully on your hardware, continue with what to check before downloading a local LLM. If tools and multi-step execution complicate the result, read what an agent demonstration establishes. When the question is whether untrusted content can redirect actions, the prompt-injection field note examines the boundary between an acceptable answer and an authorized action.