An AI agent demonstration gives you evidence about one task in one setup. To decide what that evidence is worth, identify the task, the tools it could use, the record of its actions, the result you can check and the access it had. Those five fields turn an impressive sequence into something you can assess.
A tool-using agent can request an action, receive its result and choose another action. The surrounding application executes those requests. A tool might search a document, query a database or send a message; the available actions help define what the agent can accomplish. Chip Huyen's explanation connects tools, planning and the environment in which an agent operates.
Read the demonstration in five fields
Start with this compact record. A trace is the sequence of tool requests and returned results. State is the information or external changes the task needs to keep track of as it progresses.
| Field | Evidence to look for |
|---|---|
| Task | A specific requested outcome, including constraints and a stopping point. |
| Tools | Names, arguments and capabilities: what can be read, changed or sent? |
| Trace | Actual calls and results, including errors and retries. |
| Outcome | An answer checked against its source, or a change verified in the target system. |
| Scope | Model and application setup, starting data, permissions and any saved progress. |
For a research assistant, the outcome might be an answer supported by retrieved passages. For an assistant that changes a booking, it includes the reservation after the operation. Its final message is a claim to compare with that evidence.
This also helps locate missing information. If a video shows only the finished reply, you can assess the reply's content, while the tool path and recovery behavior remain open questions. If it shows calls but hides the starting data, you have a trace whose outcome is harder to check.
A complete example: recovering from a database error
In a 27 May 2025 tool walkthrough, Simon Willison asks his application which repository has the most stars in a Datasette database. Datasette makes database tables available for queries; SQL is the language used to request rows from them.
Three calls in the published log
Simon Willison's LLM 0.26 post, 27 May 2025
-
Step 1
Query fails
A query using stars returns a missing-column error.
-
Step 2
Read the schema
Datasette_schema supplies the database structure.
-
Step 3
Query again
The corrected query uses stargazers_count and returns datasette with 10,020 stars.
The log shows error recovery using tool feedback. It contains one run, not a repeatability test.
The result belongs to the author's dated database example.
The useful detail is the middle call. An error becomes input to another action, which obtains information needed for a better request. The returned row then provides a basis for the final answer. Our assessment: this is a clear example of recovery through tool feedback, with an inspectable answer. It earns that bounded conclusion. Reliability across different questions, interrupted runs and permission settings would each need further evidence.
For your next test of a similar application, define success before running it: the reported repository and count must agree with the query result for the chosen database snapshot. Keep the snapshot fixed when comparing runs. A changing data source otherwise makes a changed answer ambiguous: the application may have failed, or the underlying data may have changed.
Turn one successful run into a useful test
The following test variations are our proposal for extending that small example. They preserve its task while making different failure paths visible.
| Variation | What to inspect |
|---|---|
| Supply an unfamiliar schema | Does the application retrieve the column names it needs? |
| Return a temporary query error | Does it stop, retry within a limit or explain the failure accurately? |
| Ask about a repository absent from the data | Does the final answer preserve the absence rather than fill it in? |
| Put two repositories at the highest count | Does the answer satisfy your stated rule for ties? |
| Repeat the same task from the same snapshot | How consistently does it reach the required outcome? |
Huyen's failure analysis gives a useful diagnostic vocabulary: choosing an unavailable tool, supplying bad arguments, receiving a wrong tool result and failing the overall goal are different problems. Her efficiency checks also include steps, cost and elapsed time. Record those alongside success, so a retry that eventually works remains visible in the comparison.
A research benchmark makes the distinction between a convincing conversation and a completed action particularly concrete. In the coauthored τ-bench paper, Shunyu Yao and colleagues evaluate agents in simulated retail and airline settings. They compare the final database with an annotated expected state and measure consistency across repeated trials. Their pass^k asks whether all of a set of trials succeed; pass@k asks whether at least one succeeds.
The authors also identify a scoring limit: a correct final state can still miss a policy violation, such as making a change without required user confirmation. That matters when designing your own check. Verify both the requested outcome and the constraints on how it may be achieved. The benchmark supplies a method to learn from, rather than a current success rate for your application.
Check what survives an interruption
An agent can use the previous tool result during a run without being able to resume that run after a restart. Ask separately what it remembers in the current conversation, what it has changed in external systems and what progress the application saves.
Harrison Chase's framework, runtime and harness essay is helpful here. In his proposed terminology, frameworks supply abstractions; runtimes address execution concerns such as persistence and human intervention; harnesses package choices such as prompts and tools. He explicitly acknowledges overlap. The terms help formulate a question, while the implementation has to supply the answer. His profile work is a starting point for that distinction.
A useful interruption test has a visible checkpoint: stop after a tool completes, restart the application and inspect what happens next. For an application that writes, check whether it repeats an already completed change. For a read-only query, check whether it can recover enough context to finish accurately. Put the observed restart behavior in the demonstration record rather than assuming it from the word “memory.”
Use the result to choose your next reading
You should now be able to write a short assessment: “The task was __. The available actions were __. The trace showed __. The checked outcome was __. The next untested condition is __.” For the Datasette example, that last field naturally leads to repeated trials and explicit failure cases.
To build those checks, continue with evaluating an AI application and the evaluation reading task. To assess private data and outward actions, follow the prompt-injection tool chain and permission boundaries. To implement and inspect the software itself, explore coding-agent review workflows.
The AI agents selection connects these questions to authors' implementations, working methods and control designs.