Automated rollback for agent side effects

An agent rollback can restore code, configuration, or selected application data. It cannot make another organization forget a request, pull a delivered email from an inbox, or pretend a card charge never reached a payment processor. Teams that label all of these operations "rollback" build a comforting control that fails precisely when the consequences matter.

The safe design separates four mechanisms: application snapshots, Git reverts, database recovery, and compensating actions. Each owns a different boundary. Any action that crosses the application boundary needs its own approval, evidence, retry rule, and recovery path before the agent executes it. If nobody can state that path in a sentence, the action is not ready for unattended execution.

Rollback has four separate meanings

Rollback is only useful when the team names the state it will restore and the state it cannot touch. The word usually hides four mechanisms with very different guarantees.

A snapshot restores a captured version of an application or workspace. Depending on the product, that snapshot may include generated code, configuration, and selected managed state. It says nothing about outside services unless the snapshot contract explicitly includes them.

A Git revert records a new commit whose changes reverse an earlier commit. It repairs source history without deleting that history. It does not contact the services that the old code called while it was running.

Database recovery changes records held by the database. A transaction rollback discards uncommitted writes in one transaction. Restoring a backup or using point-in-time recovery is a much larger operation that moves a database cluster toward an earlier state. Neither operation automatically reconciles systems outside that database.

A compensating action creates a new effect intended to offset an old one. A refund compensates for a captured payment. A cancellation request compensates for an order. A correction email may mitigate a mistaken email, though it cannot remove the first message. Compensation preserves the uncomfortable truth that the original action happened.

I use an effect ownership table during design reviews because it forces precise answers:

The last column matters most. An inverse operation is not evidence that the original disappeared. Audit records, recipient copies, settlement entries, webhooks, and physical work may persist.

A code revert changes the program, not the past

A code revert prevents or changes future behavior; it does not reverse behavior the old version already caused. This remains true whether the team uses Git, a platform snapshot, or a deployment rollback.

The Git documentation describes git revert as recording commits that reverse changes introduced by earlier commits. That wording is exact: Git applies an inverse patch to repository content. Git has no knowledge of emails, payments, cloud resources, support tickets, or partner APIs created when the reverted commit ran.

Suppose an agent modifies a billing function, deploys it, and invokes it twice before monitoring catches the defect. Reverting the commit can stop the bad function from running again. It leaves two payment attempts at the processor. If the local database recorded only one attempt, the revert may even make investigation harder by removing the code path that understood the second response.

Deployment rollback has the same boundary. Moving traffic back to a previous build restores executable behavior. Requests already accepted by the replaced build keep their consequences. Queued jobs may also survive the deployment and execute old assumptions against the restored version.

Before reverting, preserve the operational evidence that the old build produced:

• Deployment identifier and source commit
• Agent run and intent identifiers
• Queue message identifiers and lease status
• External request identifiers
• Provider responses and timestamps

Then stop new execution, reconcile incomplete operations, and restore code. Reverting first and asking questions later often destroys the easiest route to understanding which effects escaped.

A snapshot can be broader than a Git commit, but the same rule applies. The snapshot contract must say exactly which resources it contains. If it contains code and configuration, call it a code and configuration recovery mechanism. Do not promote it to universal undo through hopeful wording in the interface.

Database recovery has a narrower job than people assume

Database recovery restores database state, not business reality across every participant in a transaction. A transaction can be atomic inside one database while the surrounding operation remains split across several systems.

Consider this sequence:

1. The agent inserts an invoice row.
2. It calls a payment API.
3. The processor accepts the charge.
4. The database commit fails.

The local rollback removes the invoice row. The charge still exists. Retrying the whole operation without reconciliation can charge the customer again. This is the classic dual write failure: the application tried to make one business action atomic across systems that do not share a transaction coordinator.

Reversing the order does not solve it. If the application commits the invoice first and the payment call then fails, the database contains an unpaid invoice. That state is easier to inspect, but the application still needs a state machine that distinguishes payment_pending, payment_confirmed, payment_failed, and payment_unknown.

PostgreSQL documentation explains point-in-time recovery as restoring a base backup and replaying write ahead log records to a chosen recovery target. This is an operator procedure for recovering a database cluster. It is not a selective undo for one agent run, and it cannot ask a payment processor or mail service to return to the same timestamp.

Moving the database backward can create a second mismatch. Imagine restoring to 10:00 after an outage. A provider accepted requests until 10:07, but the restored database no longer contains their records. Agents that see "missing" rows may recreate all seven minutes of work. Recovery therefore needs an external reconciliation phase before workers resume.

An outbox pattern reduces one dangerous gap. The application commits its business change and an effect intent in the same local transaction:

BEGIN;

INSERT INTO invoices (invoice_id, customer_id, status)
VALUES ('inv_2048', 'cust_91', 'payment_pending');

INSERT INTO effect_intents
  (intent_id, operation, subject_id, status)
VALUES
  ('eff_7f31', 'capture_payment', 'inv_2048', 'pending');

COMMIT;

A worker later claims eff_7f31, calls the provider with a stable idempotency value, and records the result. The outbox does not make the external call atomic. It gives the system durable evidence that work was intended, which makes retries and reconciliation possible.

External actions need compensations, and some have none

An external side effect needs a provider-specific compensation or an explicit statement that no meaningful compensation exists. Treating every action as reversible is worse than admitting that some actions require approval.

Email is the simplest example. Before the provider accepts the message, the application may cancel a queued job. After acceptance, the provider might allow cancellation for a brief internal stage, but a general recall guarantee does not exist across recipients and mail systems. Once delivered, a second message can correct the record, not erase it. Sensitive data, legal notices, and reputational harm remain exposed.

Payments have several states that teams often collapse into "charged." An authorization reserves spending capacity. A capture requests movement of funds against that authorization. A void may release an authorization that has not settled. A refund creates a later financial entry returning money after capture. These operations have different timing, fees, permissions, and customer impact. A generic undo_payment tool hides information the agent needs to act safely.

Other APIs are even less cooperative. A request may order inventory, provision infrastructure, publish content, grant access, send a package, or trigger a person to begin work. A DELETE endpoint does not prove reversibility. Deleting a resource may leave audit logs, copied data, notifications, dependent resources, or physical consequences.

Classify compensation by what it can honestly achieve:

• Exact local inverse returns controlled state to its prior value.
• Provider cancellation stops work that has not completed.
• Financial compensation creates a refund or credit.
• Corrective communication acknowledges that the first message remains visible.
• Manual remediation handles effects whose context cannot fit a safe automated rule.

Compensation can also fail. The refund endpoint may time out. The cancellation window may close. The recipient address may reject the correction. The account used by the agent may lack permission. For that reason, the system must track compensation as another operation with its own intent identifier, status, attempts, evidence, and approval policy.

Do not build a recursive "rollback the rollback" feature. Model the history as a ledger of actions. If a compensation causes a new error, issue another explicit action after reviewing the current state. That history is longer, but it remains understandable during an incident.

Idempotency stops repeats but does not reverse success

Idempotency protects retries from producing duplicate intended effects; it does not undo the first successful effect. Teams routinely blur those ideas and then discover the difference after a timeout.

RFC 9110 defines an idempotent request method by the intended effect of several identical requests being the same as one such request. It identifies PUT, DELETE, and safe methods as idempotent at the protocol semantics level. POST is not generally idempotent, although an API can add idempotency behavior through its own contract.

The qualification matters. An idempotent DELETE can still produce a new log entry, metric, or response each time. A provider's idempotency implementation may also expire records, scope identifiers to one account, reject changed parameters, or cache only selected outcomes. Read the provider contract instead of inferring guarantees from an HTTP verb.

Every effect intent should receive one stable idempotency value before its first attempt. Retries for that same intent reuse it. A new business intent receives a new value. Never derive it only from mutable parameters such as customer, amount, and date, because two legitimate purchases can share those values.

A timeout creates an unknown outcome, not a failure. Use this sequence:

1. Mark the attempt outcome_unknown; do not create a replacement intent.
2. Query the provider using the idempotency value or operation reference.
3. If the provider confirms success, record that success locally.
4. If it confirms no operation, retry with the same value.
5. If it cannot answer, hold the operation for reconciliation or human review.

This response shape gives the agent enough information to distinguish acceptance from transport uncertainty:

{
  "intent_id": "eff_7f31",
  "attempt": 2,
  "idempotency_key": "eff_7f31",
  "transport_status": "timeout",
  "provider_status": "unknown",
  "provider_reference": null,
  "next_action": "reconcile"
}

An idempotency value should travel through logs, queue messages, API headers, and provider metadata where supported. If operators cannot search by it on both sides of the boundary, they will guess during recovery.

Approval belongs at the effect boundary

Approval must occur after the agent has formed the exact external action and before the first irreversible request leaves the system. Approval at the start of a broad task gives the agent too much room to change recipients, amounts, scopes, or tools later.

"Handle this customer's account" is not adequate approval to charge a card or email every user. A valid approval describes the concrete operation: recipient, amount and currency, message or payload digest, target account, tool, expiration, and permitted attempt count. If any approved field changes, the approval no longer matches.

A useful policy sorts effects by consequence rather than by which agent or model requested them. Read access can still expose private data, but it does not create the same recovery problem as an outbound action. Drafting an email is local and reversible. Sending it crosses the boundary. Creating a payment proposal is local. Capturing funds crosses the boundary.

Require explicit approval for actions that:

• Move money or create a financial obligation
• Send information to a person or outside organization
• Publish, delete, or disclose data outside controlled storage
• Change identity, access, ownership, or security settings
• Start physical work or another process that cannot be recalled reliably

Low consequence, repetitive actions may use bounded standing approval. The bound should name a maximum amount, recipient set, allowed tool, expiration time, rate, and total operation count. "Approved for billing" has no enforceable boundary.

Approval also needs protection against replay. Bind it to an immutable digest of the operation and mark it consumed when policy allows only one execution. If execution returns an unknown outcome, do not ask for a fresh approval and create a second intent. Reconcile the approved intent first.

The approval screen should state the recovery truth in ordinary language. "This message cannot be recalled after delivery" is useful. "This operation is reversible" is misleading when the actual recovery is a refund that may take time and remain on financial statements.

Tool contracts should expose the full effect lifecycle

An agent tool contract should describe intent, execution, observation, and compensation as separate operations. A single function that performs an effect and returns success: true leaves too little evidence for retries, approvals, or incident response.

The following policy fragment is small enough to enforce and specific enough to review:

tools:
  send_email:
    effect: irreversible
    approval: required
    idempotency_field: intent_id
    evidence_field: provider_message_id
    compensation: null

  capture_payment:
    effect: compensatable
    approval: required
    idempotency_field: intent_id
    evidence_field: provider_payment_id
    compensation: refund_payment

  update_draft:
    effect: local_reversible
    approval: none
    compensation: restore_version

effect tells the planner which recovery class applies. approval blocks execution until the exact arguments are authorized. The idempotency field makes retries reuse one identity. The evidence field tells operators what must survive. The compensation field points to a separate tool rather than pretending the original call can run backward.

Execution should accept an immutable envelope:

{
  "intent_id": "eff_7f31",
  "operation": "capture_payment",
  "arguments": {
    "invoice_id": "inv_2048",
    "amount_minor": 12900,
    "currency": "USD"
  },
  "approval": {
    "approval_id": "apr_662",
    "scope_hash": "sha256:8b4f...",
    "expires_at": "2026-07-27T18:00:00Z"
  }
}

The executor calculates the operation digest, compares it with scope_hash, checks expiration, reserves the intent, and only then contacts the provider. It stores the request metadata before the call and the response afterward. If it crashes between those writes, the durable intent remains available for reconciliation.

The executor should reject four conditions without improvising: changed arguments, expired approval, reuse of an intent for a different operation, and an attempt to compensate an effect whose successful completion has not been confirmed. Agents are good at finding plausible continuations. Financial and communication controls must prefer a visible stop to a plausible guess.

Give observation its own tool, such as get_payment_status(intent_id). Observation should not create an effect. Keeping it separate lets the agent resolve ambiguous outcomes without receiving permission to retry the original action.

One failed run can cross every recovery boundary

A single agent run can leave code, database records, and outside systems at different points in time. Walking through that failure before launch exposes gaps that a generic rollback control hides.

Assume an agent builds a membership application, deploys a change, imports a customer list, charges annual fees, and sends welcome emails. The task sounds unified, but it crosses at least four recovery boundaries.

At 14:00, the agent deploys code that calculates the annual fee from the wrong column. At 14:02, it writes 40 payment intents and email drafts to the database. At 14:03, a worker captures several payments. At 14:04, the mail provider accepts welcome messages containing the incorrect fee. At 14:05, monitoring stops the worker. Some payment calls timed out after reaching the processor, so local status does not reveal whether they succeeded.

Restoring the 13:59 code snapshot stops the faulty calculation in future runs. It does not change the fee copied into existing intents. Reverting the Git commit documents the source correction but has the same limit.

Restoring the database to 13:59 would remove the local intent records that contain provider references and idempotency values. That makes the external mismatch worse. The better database action is a logical correction: preserve the intents, mark uncertain operations for reconciliation, and correct rows only after matching provider state.

The response team should then proceed by effect class. It queries every uncertain payment using its stable operation identity. Confirmed captures move to a refund review, failed attempts close without retry, and unresolved attempts remain blocked. Delivered emails receive a carefully approved correction. Queued messages that have not reached the provider are cancelled. Each compensation gets a new intent linked to the original.

This example also shows why automatic compensation can be dangerous. If the system immediately refunds every local payment_unknown record, it may issue refunds for charges that never existed or call a refund endpoint with the wrong reference. If it resends every missing email after database recovery, recipients may receive duplicates. Reconciliation must precede compensation whenever local and provider state disagree.

The run is complete only when every intent reaches a terminal state such as succeeded, confirmed_failed, compensated, or manual_exception. "The application was rolled back" describes only the first part of the incident.

Recovery works only when the evidence survives

Recovery controls fail when rollback deletes the records needed to decide what happened. Store an append only effect ledger outside the application state that routine snapshots or restores can replace, and retain enough provider evidence to reconcile each operation.

The ledger should record intent creation, argument digest, approval, execution lease, attempt, transport result, provider reference, observed provider state, and compensation links. Restrict changes to state transitions instead of allowing agents to overwrite old entries. Corrections should add events rather than edit history.

Monitor unresolved states, not just explicit errors. outcome_unknown that sits for ten minutes can be more dangerous than a clean rejection because an operator may retry it manually. Also alert when an approval expires during execution, an idempotency value appears with a different argument digest, or a compensation fails.

Run recovery drills with deliberately awkward failure points. Terminate a worker after the provider accepts a request but before the local success write. Restore application data from an earlier snapshot while preserving the effect ledger. Expire approval between planning and execution. Make the observation API unavailable. The drill passes when the system stops, reconciles, and exposes the unresolved decision without duplicating the effect.

When using Koder.ai, treat its snapshots and rollback as the application recovery layer, then design separate controls for database history and every outside action the application can trigger. Source export, deployment controls, and rollback help restore software, while the application's tool contracts still own approvals and compensation.

A rollback button should name its boundary beside the button: code, configuration, managed data, or external effects. If the interface cannot make that sentence precise, it should not promise rollback. The honest control may look less magical, but it gives the incident team the one thing they need at 2 a.m.: a trustworthy account of what changed, what escaped, and which action is safe to take next.