Run It Exactly Once: Idempotent Tool Calls for AI Agents

An agent issues a $25 refund. The request leaves, and the connection times out before the answer comes back. Did the customer get their money?

Nobody knows. Retry, and they might be refunded twice. Don't, and they might not be refunded at all. Most agent frameworks pick one of those answers without telling you which, and the usual pick is to retry, because retrying is what you do with timeouts.

For a read, that's fine. Searching an order twice costs a few milliseconds. For a refund, an email or a deleted record, a second run is a second refund, a second email or an error on a record that's already gone. Agents make this worse than ordinary services do: their runs pause for hours, resume on other processes, and are driven by a model that can decide a call "didn't seem to work" and make it again.

This post explains how Algen Agent Runtime makes a tool call take effect once, and what it does when it honestly can't tell whether it did.

The short version

1. Give every side effect a key. Each tool call gets an idempotency key from its run and call, <run_id>:<tool_call_id>, and the tool passes it to the system it changes.
2. Write it down before you run it. The call is reserved in a ledger under that key first. A key that already completed returns its recorded result instead of running again.
3. Retry reads, not writes. A tool that changes something gets one attempt. A timeout on a write isn't a failure; it's an unknown.
4. Unknown is a status of its own. A call that may or may not have taken effect is recorded indeterminate and never run again by itself. The tool can look it up, or an operator decides.
5. Undo only on purpose. Compensation exists, but the runtime never runs it automatically.

"Exactly once" doesn't exist. Here's what does

Distributed systems people will tell you, correctly, that exactly-once delivery is impossible. A message is either sent until it's acknowledged, which can mean twice, or sent once, which can mean never.

What you can have is an effect that happens once: send the request as many times as you need to, and make the receiver recognise a repeat. That's what an idempotency key is for. Stripe has taken an Idempotency-Key header on payment requests for years for exactly this reason. Send the same key twice and the second request gets the first one's answer instead of a second charge.

So "run it exactly once" really asks three things of an agent runtime:

• Don't send a request again when you already know the answer.
• When you do have to send it again, send the same key.
• When you can't know, say so, and don't guess.

Four ways an agent runs a tool twice

The framework retries a timeout. A generic retry policy can't tell a refund from a search. Three attempts with backoff is a good default for reads and a bad one for writes.

The process restarts mid-step. The run is recovered from its last checkpoint, which was saved before the tool ran. Replaying the step replays the call.

Two processes pick up the same run. A deploy overlaps old and new workers, or recovery starts on a second machine while the first is still alive. Both drive the run forward.

The model calls it again. The tool returned an error, or a result the model didn't expect, so it tries once more, perhaps with slightly different arguments. To the model, that's diligence.

The first three are the runtime's job, and the rest of this post is about them. The fourth is partly a matter of what the model is told, and we come back to it at the end.

A key per call, passed downstream

Every call a tool receives carries an idempotency key in its context: the run's id and the tool call's id, so it's unique to that call in that run and stable across every replay of it. A tool that changes something passes it on:

@billing.tool(external(approval=True))
async def issue_credit(account_id: str, amount_cents: int, context: ToolContext) -> Credit:
    """Credit a customer's account."""
    client = await billing.client()
    return await client.credit(account_id, amount_cents, idempotency_key=context.idempotency_key)

The key isn't derived from the arguments. Two different calls with identical arguments, say two separate $25 credits the customer really is owed, are two effects, and a hash of the arguments would merge them. And a key that changed whenever the model reworded an argument wouldn't catch a replay at all.

Passing the key is half the job; the other system has to honour it. Many payment, messaging and ticketing APIs do. If yours doesn't, store the key on the record you create, in a reference, metadata or description field, so you can at least look it up later. That lookup comes back below.

The ledger: reserve first, then run

Before a tool runs, the runtime reserves its key in a tool-execution ledger. In production that ledger is a PostgreSQL table, next to the run's checkpoints.

• No record: the call is new. It's recorded started and runs.
• completed: the call already ran. Its recorded result is returned, and the tool isn't called. This is what makes a recovered run safe to replay: every call that finished before the crash comes back from the ledger.
• failed: the call definitely didn't take effect, so it may run again.
• started or indeterminate: the outcome is unknown. The call doesn't run.

The reservation is atomic, so when two processes drive the same run, only one of them gets to make the call. A version check on the run's checkpoint makes the other one stand down at its next save.

Retry reads, not writes

Each tool declares its side effect, and the side effect sets its retry rule:

The runtime's test helper assert_tool_contract fails any tool that writes, sends or deletes and is allowed more than one attempt or has no idempotency key, so the rule is checked in CI, not just documented.

One distinction saves a lot of reconciliation. A tool that refuses a call before sending anything, because the user's token expired or the account is locked, raises ToolNotAttemptedError. The call is recorded failed, not indeterminate: nothing left the building, so there's nothing to look up, and it can run again once the problem is fixed. Only a call that may have reached the other system becomes unknown.

When the answer is "we don't know"

A call ends up indeterminate when it timed out or lost its connection after the request left, and stays started when the process died while it ran. In both cases, someone has to find out what happened. The runtime gives that job three ways to get done.

Ask the tool. A tool can have a reconcile hook that looks the key up in the external system:

async def find_credit(record: ToolExecutionRecord, context: ToolContext) -> ReconcileOutcome:
    """Look the credit up by its idempotency key."""
    client = await billing.client()
    credit = await client.find_credit(idempotency_key=record.idempotency_key)
    if credit is None:
        return ReconcileOutcome(status="unknown")
    return ReconcileOutcome(status="completed", value=credit.receipt(), result_reference=credit.id)


@billing.tool(external(approval=True), reconcile=find_credit, compensation=void_credit)
async def issue_credit(account_id: str, amount_cents: int, context: ToolContext) -> Credit:
    ...

The hook only reads, so it can run more than once. When a recovered run replays a call that's still unknown, the runtime asks the hook first; if the external system says the credit exists, the run carries on with the real result as if the timeout never happened.

Ask a person. When the hook can't tell, an operator can. GET /v1/tool-executions lists a tenant's unknown calls, the longest-waiting first. POST /v1/tool-executions/{key}/resolve records what the operator found, completed or failed, with a required reason. Every decision is compare-and-set on the ledger status, so of two operators acting at once exactly one wins, and it's appended to the record with the operator's id. A call that finally returns after someone settled it can't overwrite their decision; the late outcome is logged and ignored.

Undo it. If an effect shouldn't stand, POST .../compensate runs the tool's compensation, such as voiding the credit. The runtime never compensates by itself, not when a run fails, is cancelled or is recovered. It can't tell whether a failure later in the run makes an earlier effect wrong, and compensating a call that never happened, like voiding a credit that was never issued, can be a side effect of its own. Someone who can check the other system decides.

Meanwhile, the run can wait. With tool_policy.on_indeterminate: pause, a run whose call is unknown pauses in awaiting_reconciliation instead of failing, and settling the call resumes it. The model then hears the truth: the recorded result, or that the call didn't take effect, or that it was compensated.

agents:
  - name: billing-assistant
    tool_policy:
      on_indeterminate: pause

We kill it on purpose

Claims like "takes effect once" are easy to write and hard to keep. The runtime's persistence drills run real runtime processes against live PostgreSQL and Redis and SIGKILL them in the middle of model calls and in the middle of tool calls, then let a fresh process recover the runs. The check is simple: one ledger record and one external effect per run, however many times the process died.

See it in a minute

No API key, network or database needed:

python -m pip install --pre algen-agent-runtime
python -m examples.pattern_side_effect_reconciliation.app

It runs four scenarios against a fake billing system: a credit applied before the timeout and found by the reconcile hook; one that never landed, settled failed by an operator over the HTTP API; a compensation an operator asked for; and a call refused before anything was sent, which leaves nothing to reconcile.

Or, if you'd rather break something, play Heist the Agent. Its fourth trick is a double payout: Penny the refund bot pays you, crashes before recording it, and pays you again on restart.

What this doesn't solve

The other system has to cooperate. The key makes a repeat recognisable; only the receiver can refuse it. Against an API with no idempotency support and nowhere to store the key, the ledger still stops the runtime from repeating itself, but an unknown call can only be settled by a person checking by hand.

A new call from the model is a new key. If the model hears that a call failed and decides to make it again, that's a different tool call and the ledger treats it as one. on_indeterminate: pause exists for this reason: the model doesn't hear "failed" while the honest answer is "unknown". By default an unknown call is handled like any failed call: the run fails, or the model is told the call failed. Settling it afterwards is record-keeping only.

Workflow retries are at-least-once. A workflow node set to recovery_policy: retry starts a fresh agent run, which makes its own calls with its own keys. Use it only for nodes whose tools are read-only or have a reconcile hook.

It's early. At the time of writing the runtime is 0.1.0a23, a pre-release hardened for a single node. Coordinated recovery across several processes is still open work.

Five things worth copying

• Key every side effect by call, not by arguments, and pass the key to the system you're changing.
• Record the call before you make it. A replay should read the ledger, not rerun the tool.
• Give writes one attempt. Retries are for reads.
• Treat "unknown" as an answer. Record it, pause on it and look it up. Never resolve it by trying again.
• Undo by decision, not by reflex. Compensation is an operator's call.

An agent that does something twice is worse than one that does nothing, because nobody notices until the customer does.