Getting Started With (Offline) Agent Evals

Context. Marky is a social media marketing platform. A few months ago we shipped Max, an in-app assistant that takes actions for users, not just answers questions. Max is built as a LangChain deepagents agent, and reaches Marky through an MCP server that exposes about 70 tools, mostly CRUD operations over Marky's resources: posts, businesses, topics, media, and so on.

Problem. With 70 tools and a deliberately non-frontier model (Gemini 3.1 Flash, chosen for cost), Max hallucinated tools often: calling one that did not exist, or the wrong one for the job. We needed to iterate on tool-calling to drive hallucinations down, and ideally cut token usage while we were at it. But agent changes are not like normal code: a prompt tweak has no obvious right answer you can read off the diff.

Goal. Similar to unit tests, we needed a way to validate that our changes were having the intended effect. Concretely: run experiments on Max's tool-calling offline, change a prompt, swap the tool-selection strategy, or even switch models, and get a trustworthy before-and-after on hallucination rate and token cost before anything reaches users. In short, a test suite for the agent. This post is how we built one: the three levels of eval you can choose from, and the LangSmith code to reproduce each. It took a day and about sixty dollars in model calls, and it let us cut Max's token usage in half at equal accuracy.

Three levels of environment

An agent eval is really a question about environments: how much of the real world do you let the agent touch before you check its work? There are three levels, and you should start at the top.

Level 1: stop at the first tool call

You give the agent a prompt. The model responds with the tool calls it wants to make. You stop right there, before anything runs, and check one thing: did it reach for the right tool with the right arguments? This is the cheapest eval you can build. There is no state, nothing executes, and one prompt costs one model call. Start here.

First, a dataset. Each example is a prompt plus the tools the agent should and should not reach for.

from langsmith import Client

client = Client()

# Each example is a user prompt plus assertions about the tools the agent
# SHOULD and SHOULD NOT reach for. must_call is an "acceptable set": any one
# of them satisfies it, so two tools that both legitimately answer the request
# each count as correct.
examples = [
    {
        "inputs": {"query": "how did my last post do?"},
        "outputs": {"assertions": {
            "must_call": ["get_post_analytics"],
            "must_not_call": ["get_post"],   # fetches content, not performance
        }},
    },
    {
        "inputs": {"query": "make that post punchier"},
        "outputs": {"assertions": {"must_call": ["update_post"]}},
    },
]

client.create_dataset(dataset_name="agent-tool-selection")
client.create_examples(
    dataset_name="agent-tool-selection",
    inputs=[e["inputs"] for e in examples],
    outputs=[e["outputs"] for e in examples],
)

How big should the dataset be? A useful floor is one example per tool the agent can call. Our agent had about 70 tools, so we built about 100 examples: enough to make each tool the expected answer at least once, plus some prompts ambiguous enough that two tools could each be right. Coverage beats raw count. A tool with no example is a tool whose selection you are never testing, and the tools with no coverage are exactly the ones a model quietly stops reaching for.

Then run one prompt through your agent and capture the first tools it reaches for. This works with any LangGraph or LangChain agent that exposes astream.

from langchain_core.messages import AIMessage, HumanMessage

async def first_tools(agent, query: str) -> list[str]:
    """Run the agent, stop at the first turn that emits tool calls, return names."""
    async for state in agent.astream(
        {"messages": [HumanMessage(content=query)]},
        stream_mode="values",
        config={"recursion_limit": 25},
    ):
        for msg in state["messages"]:
            if isinstance(msg, AIMessage) and msg.tool_calls:
                return [tc["name"] for tc in msg.tool_calls]
    return []

Wire that into a LangSmith experiment so every change gets a tracked score.

from langsmith import aevaluate
from langsmith.evaluation import run_evaluator

async def target(inputs: dict) -> dict:
    return {"called": await first_tools(agent, inputs["query"])}

@run_evaluator
def passed(run, example) -> dict:
    called = set(run.outputs["called"])
    a = example.outputs["assertions"]
    must_call = set(a.get("must_call") or [])          # any ONE of these is acceptable
    must_not_call = set(a.get("must_not_call") or [])  # none of these may be called

    # Fail if a tool was required but none of the acceptable ones was called.
    if must_call and not (called & must_call):
        return {"key": "passed", "score": 0}
    # Fail if any forbidden tool was called.
    if called & must_not_call:
        return {"key": "passed", "score": 0}
    return {"key": "passed", "score": 1}

results = await aevaluate(
    target,
    data="agent-tool-selection",
    evaluators=[passed],
    num_repetitions=1,          # one rep while iterating (see "watch the cost")
)

The limit of Level 1 shows up fast. Many real requests are not one tool call, they are a sequence. "How did my last post do" means find the most recent post, then pull its analytics. "Make that post punchier" means find the post, then edit it. At Level 1 you watch the agent reach for "find the post" and you mark it wrong, because you expected "edit." But the agent was right. It cannot edit a post it has not found yet. Level 1 cannot measure a request that needs more than one step, and most interesting requests need more than one step.

This is the trap we fell into. Our first pass scored 66 percent. When we looked at the misses, most of them were the agent correctly taking the first step of a two step task. The number was not measuring quality. It was measuring how many of our test prompts happened to be single step.

Level 2: let one turn run to completion

Same single prompt, but now you let the agent's own loop run. It calls a tool, gets a result, reasons about it, calls the next one, and stops when it is done. Now "find the post then edit it" can actually happen, and you can score the whole trace: did it get to the right final action, using an id it actually discovered rather than one it invented?

The catch at Level 2 is that the tools have to return something. If your "search posts" tool returns nothing, the agent has no post to act on and the trace dead ends. So you need mock tools. Search has to hand back a realistic post with a real id. Update has to look like it succeeded. You are building a small fake world for the agent to act in. That is more work than Level 1, but it is the first level that measures whether the agent finished the job, not just whether it picked a plausible first move.

Where you put that mock matters. We mocked at the client layer, not the MCP server. The harness still fetches the real tool catalog, every tool's name, description, and argument schema, from the live MCP server, so the model sees the exact surface it sees in production. Then it swaps each tool's execution for a local stub. The MCP server itself stays untouched. No mock flag to accidentally ship to production, no backend writes, and per-scenario control over exactly what each tool hands back.

At Level 1, that swap is a no-op stub. The model sees the real name, description, and schema; the body does nothing.

from langchain_core.tools import BaseTool

class RecordingStub(BaseTool):
    """Mirrors a real tool's name/description/schema, but runs nothing.
    The model sees the exact catalog, so its CHOICE is realistic; we read the
    tool calls off the transcript, so the stub only has to be a valid no-op."""

    async def _arun(self, **_):
        return "Success."

    def _run(self, **_):
        return "Success."

def stub_of(tool: BaseTool) -> BaseTool:
    return RecordingStub(name=tool.name, description=tool.description,
                         args_schema=tool.args_schema)

# Fetch the REAL catalog from the live MCP server, then swap execution for stubs.
mcp_tools = await load_mcp_tools(mcp_url)   # real names + descriptions + arg schemas
catalog = [stub_of(t) for t in mcp_tools]
agent = create_agent(model, tools=catalog, middleware=[...])

At Level 2, the same wrapper returns canned data instead: a small world of rows with stable ids, read tools that hand them back, and write tools that echo the args they were called with.

# A small canned world with stable ids the agent can discover and then act on.
WORLD = {
    "posts": [
        {"id": "post-1", "caption": "Summer sale is live!", "status": "scheduled"},
        {"id": "post-2", "caption": "Meet the team.", "status": "new"},
    ],
    "templates": [{"id": "tmpl-1", "name": "Bold Quote", "fields": ["headline", "subtext"]}],
    # ...comments, designs, connected accounts, etc.
}

# READ tools return rows WITH ids (so the model can discover one). Every other
# tool echoes the args it was called with (so the scorer can read back the id
# the model passed to the terminal action).
READ_FIXTURES = {
    "list_posts":     lambda a: {"posts": WORLD["posts"]},
    "get_post":       lambda a: next(p for p in WORLD["posts"] if p["id"] == a["post_id"]),
    "list_templates": lambda a: {"templates": WORLD["templates"]},
    "get_template":   lambda a: next(t for t in WORLD["templates"] if t["id"] == a["template_id"]),
}

def fixture_result(name, args):
    reader = READ_FIXTURES.get(name)
    if reader:
        return reader(args)
    return {"ok": True, "tool": name, "args_received": args}   # write tools echo args

class FixtureTool(BaseTool):
    def _run(self, **kwargs):
        return json.dumps(fixture_result(self.name, kwargs), default=str)

    async def _arun(self, **kwargs):
        return json.dumps(fixture_result(self.name, kwargs), default=str)

That echo is what lets the trace scorer confirm the agent acted on an id it actually discovered, rather than one it invented.

WORLD_IDS = {row["id"] for rows in WORLD.values() for row in rows if "id" in row}

def passed_trace(trace, example) -> int:
    called = {call["name"] for call in trace}
    hit = bool(called & set(example.must_call))
    # the agent passed an id that exists in the world => it discovered it
    grounded = any(v in WORLD_IDS
                   for call in trace for v in call["args"].values()
                   if isinstance(v, str))
    return int(hit and grounded)

When we moved to Level 2, the picture flipped. About 83 percent of what Level 1 had flagged as failures turned out to be correct discover-then-act traces. The real bug count was close to zero.

Level 3: multi-turn

A real conversation, back and forth, over several user turns. To run this offline you need a second model playing the user on the other side of the chat. This is the most faithful and the most expensive, and keeping the simulated user honest and deterministic is its own problem. Most teams do not need to start here. You get most of the value from Levels 1 and 2.

Watch what it costs

Evals are model calls, and model calls are money. Two habits saved us a lot of it, after we learned them the expensive way.

Run one repetition, not three, while you iterate. It is tempting to run every example three times and average, so the number is stable. But while you are still changing the dataset and the harness, you do not need a publication grade number. You need to know if you moved. One repetition per example is a third of the cost. Save the multi-run averaging for the final confirmation. We did not, at first, and spent about sixty dollars in a day on runs that mostly told us the same thing three times.

Only re-run the failures. If you are passing 60 percent, do not re-run the 60 percent that already passes every time you tweak something. Tag the failures into a split and point the next iteration at that subset. Every loop now costs 40 percent of a full run, and you are looking at exactly the examples you are trying to fix.

# After a run, tag the examples that failed into a named split...
for example_id in failing_example_ids:
    client.update_example(example_id=example_id, split=["failures"])

# ...then point the next iteration at just that split.
data = client.list_examples(dataset_name="agent-tool-selection", splits=["failures"])
results = await aevaluate(target, data=data, evaluators=[passed], num_repetitions=1)

Beware invoke-time state

This is the subtle one, and it is where a lot of evals quietly lie to you.

Agents rarely see just the system prompt. At the moment you invoke them, dynamic context gets injected: the user's memory, their saved skills, the current date, which post is on their screen. Every one of those makes a run harder to reproduce, and worse, it makes your eval's copy of that context drift away from what production actually sends.

We hit exactly this. An example failed in the eval because the eval never told the agent which post was on screen, so "make that post punchier" had no referent and the agent, correctly, asked which post. In production that context is always there. The failure was ours, not the agent's.

The fix that made this manageable was structural. We moved the dynamic parts of the prompt out of one big string and into composable middleware. Skills, for example, became a single component that owns the skill tools and the skill instructions and injects them, with an on/off switch. Memory, the current-post context, and the rest followed the same shape. Now the eval can turn a piece off, or feed it a controlled version, with one flag.

We stopped hand-copying production context into the eval, which is the thing that drifts. If any part of your prompt is built at invoke time, make it a component you can toggle before you try to eval around it.

The payoff: real experiments

Here is why all of that is worth it. Once the eval is trustworthy, it stops being a report card and becomes an experiment engine. The same target, run against a few variants of the agent, with the same scorers.

for variant in ["all_tools", "top_k", "search_on_demand"]:
    agent = build_agent(variant)          # same prompt + tools, different selector
    await aevaluate(
        make_target(agent),
        data="agent-tool-selection",
        evaluators=[passed, cost_usd, args_valid],
        experiment_prefix=variant,
        num_repetitions=1,
    )

The question we actually wanted to answer: how should the agent pick which tools to show the model, out of a catalog of 83? We compared three methods on the same cleaned dataset.

Embedding based top-k selection ties the naive "show everything" approach on accuracy, 70.6 percent either way, at half the input tokens. The search-on-demand method, where the model asks for tools as it needs them, lost 11 points of accuracy and used more tokens than top-k, because the extra lookup round trip is not free. At 83 tools, narrowing beats searching. We kept top-k in production, now with a number we trust behind the decision.

None of that comparison means anything if the underlying eval is counting correct behavior as failure. Fix the measurement first. Then the experiments write themselves.

Where to start

If you are building your first agent eval: start at Level 1, and know its ceiling. Move to Level 2 with mock tools the moment "search then act" shows up in your prompts, which is sooner than you think. Keep the cost down with single runs and failure splits while you iterate. Make your invoke-time context composable so you can turn it off. And do not trust a single delta until you have chased a few "failures" down to the trace and confirmed they are real.

The eval you can trust is worth more than the agent you are testing with it. One is a number. The other is every decision you make after.